Skip to content

Add support for resuming interrupted simulations in optimize.py using BioDynaMo's backup/restore #23

Description

@salva24

Problem

Currently, optimize.py runs each Optuna trial via run_ABM(), which calls bdm run as a subprocess for the full simulated duration (total_minutes_to_simulate).

If a trial is interrupted partway through (e.g., killed by a cluster job scheduler or timeout), the entire simulation must restart from scratch. This wastes compute time and makes long calibration runs fragile.

Existing functionality

BioDynaMo already provides a native checkpoint/restore mechanism based on ROOT object serialization (WriteObject<bdm::Simulation>), exposed through:

  • --backup <FILE> — periodically writes a full simulation snapshot
  • --restore <FILE> — restores the simulation from a snapshot and continues execution

Proposed solution

Modify run_ABM() in optimize.py to:

  • Pass a backup_file (and optionally backup_interval) as part of the generated params.json/configuration for each trial.
  • Before launching bdm run, check whether a backup already exists for that trial instead of starting a new simulation.
  • Remove the backup file after the trial finishes successfully to avoid accumulating stale checkpoints.

Benefits

  • Allows long-running calibration trials to survive interruptions without losing progress.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions