feat(vmm): add experimental systemd process manager

[kvinwang]

Summary

• add an opt-in systemd VMM process-manager backend that bypasses the standalone supervisor
• launch each VM as a transient systemd service with a dedicated cgroup
• persist the original process metadata so VMM restart/reconciliation keeps working after transient units are collected
• preserve the existing supervisor backend as the default
• add an auto migration mode: new launches use systemd while VMs already running in Supervisor remain pinned there until their next restart
• document configuration, runtime behavior, inspection commands, and current experimental limitations

Lifecycle

The transient service uses:

• Type=exec
• ExitType=cgroup
• KillMode=mixed
• KillSignal=SIGTERM
• SendSIGKILL=yes
• TimeoutStopSec=<systemd.stop_timeout>

For swtpm VMs, the existing vm-launcher still handles swtpm readiness and graceful QEMU/swtpm shutdown. systemd owns the enclosing cgroup and keeps the unit in deactivating while processes remain during long kernel-side teardown.

Enable with:

[cvm]
pm = "auto"

[systemd]
unit_prefix = "dstack-vm"
state_dir = "/var/lib/dstack-vmm/systemd-processes"

Experiment

On systemd 255, started a transient service with the same lifecycle properties and submitted an asynchronous stop. Immediately after stop, systemd reported:

ActiveState=deactivating
SubState=stop-sigterm
ControlGroup=/system.slice/dstack-vm-experiment-....service

After the child shutdown completed, it reported:

ActiveState=inactive
SubState=dead
Result=success

Review hardening

• close auto-mode ghost-pin races and self-heal vanished Supervisor records
• preserve detailed Supervisor probe errors and ignore only definitive Unix ECONNREFUSED stale sockets
• use CLOCK_MONOTONIC for systemd monotonic timestamps
• tolerate the stop-vs-unit-collection race and preserve non-zero crash status after stop
• reset failed transient units before relaunch and retain metadata on launch failure
• fail closed when a configured Supervisor socket cannot be probed; direct systemd mode rejects running legacy VMs
• pin stopped and running Supervisor records in auto mode and use one filtered list call
• retain cached pinned state if the legacy Supervisor becomes unavailable
• make stop escalation configurable (systemd.stop_timeout), defaulting to infinity for long encrypted-memory teardown
• normalize systemd exit status to the Supervisor raw-wait representation
• propagate systemd bus/query failures instead of mapping running VMs to exited, and expose systemd monotonic start/stop timestamps
• skip isolated corrupt/unavailable metadata records instead of failing the entire VMM
• rename the App process-manager field and add systemd state/config parsing tests

Tests

• cargo test --manifest-path dstack/Cargo.toml -p dstack-vmm (108 passed)
• cargo clippy --manifest-path dstack/Cargo.toml -p dstack-vmm --all-targets -- -D warnings
• git diff --check

[Copilot]

Pull request overview

Adds an experimental, opt-in systemd-based process manager for dstack-vmm so VMs can be launched as transient systemd services (dedicated cgroups) while preserving existing supervisor-backed behavior as the default.

Changes:

• Introduces a ProcessManager abstraction with a new SystemdProcessManager backend driven by systemd-run/systemctl, persisting process metadata for restart reconciliation.
• Extends VMM configuration to select the backend and configure systemd unit naming/state storage.
• Adds documentation describing the runtime model, inspection commands, and current limitations.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
dstack/vmm/vmm.toml	Adds config knobs to select the process-manager backend and configure systemd unit/state settings.
dstack/vmm/src/process_manager.rs	Implements the new systemd process-manager backend and shared abstraction over supervisor vs systemd.
dstack/vmm/src/main.rs	Wires backend selection to instantiate either the existing supervisor client or the new systemd manager.
dstack/vmm/src/config.rs	Adds ProcessManagerBackend and systemd-specific configuration defaults.
dstack/vmm/src/app.rs	Switches the app’s process control dependency from SupervisorClient to the new ProcessManager.
docs/experimental-systemd-vm-processes.md	Documents configuration and operational behavior for the experimental systemd backend.

Suppressed comments (1)

dstack/vmm/src/process_manager.rs:235

• Error message capitalization is inconsistent with the repo convention (errors should start lowercase).

            bail!("Process is running");
        }
        let record = self.read_record(id)?;
        if record.started {
            bail!("Process is started");

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.