Problem
Currently, optimize.py runs each Optuna trial via run_ABM(), which calls bdm run as a subprocess for the full simulated duration (total_minutes_to_simulate).
If a trial is interrupted partway through (e.g., killed by a cluster job scheduler or timeout), the entire simulation must restart from scratch. This wastes compute time and makes long calibration runs fragile.
Existing functionality
BioDynaMo already provides a native checkpoint/restore mechanism based on ROOT object serialization (WriteObject<bdm::Simulation>), exposed through:
--backup <FILE> — periodically writes a full simulation snapshot
--restore <FILE> — restores the simulation from a snapshot and continues execution
Proposed solution
Modify run_ABM() in optimize.py to:
- Pass a
backup_file (and optionally backup_interval) as part of the generated params.json/configuration for each trial.
- Before launching
bdm run, check whether a backup already exists for that trial instead of starting a new simulation.
- Remove the backup file after the trial finishes successfully to avoid accumulating stale checkpoints.
Benefits
- Allows long-running calibration trials to survive interruptions without losing progress.
Problem
Currently,
optimize.pyruns each Optuna trial viarun_ABM(), which callsbdm runas a subprocess for the full simulated duration (total_minutes_to_simulate).If a trial is interrupted partway through (e.g., killed by a cluster job scheduler or timeout), the entire simulation must restart from scratch. This wastes compute time and makes long calibration runs fragile.
Existing functionality
BioDynaMo already provides a native checkpoint/restore mechanism based on ROOT object serialization (
WriteObject<bdm::Simulation>), exposed through:--backup <FILE>— periodically writes a full simulation snapshot--restore <FILE>— restores the simulation from a snapshot and continues executionProposed solution
Modify
run_ABM()inoptimize.pyto:backup_file(and optionallybackup_interval) as part of the generatedparams.json/configuration for each trial.bdm run, check whether a backup already exists for that trial instead of starting a new simulation.Benefits