bench-campaigns: config-driven campaign runner for ingest benchmarks

[marwen-abid]

Why

We run ingest benchmark campaigns on-demand. One campaign is dozens of bench-ingest invocations — every (dataset, chunk) cell, cold and hot, repeated N times — plus dataset preparation and result collection. Done by hand this is slow and error-prone, and the results weren't self-describing enough to compare across machines and dates.

This PR turns a campaign into a single config file and one command.

How it works

A campaign is a small bash config:

NAME=pubnet-oct                 # results key: pubnet-oct-<sha>-<stamp>
REF=main                        # git ref to benchmark (default: current checkout)
INGEST=both                     # cold, hot, both, or none
QUERY=no
CLOSE_INTERVAL=2s               # pace hot ingest at a 2s close cadence
RUNS=5                          # repetitions per (dataset, chunk) cell
PUBLISH_URI=gs://my-bucket/benchmarks
DATASETS=(
  "pubnet|packs-gs|gs://my-bucket/ledgers/pubnet/packs/cold|1 2"
)

Then, on the benchmark machine:

BENCH_ROOT=/mnt/nvme/bench ./scripts/bench-campaigns/campaign.sh pubnet-oct.cfg

campaign.sh walks the config top to bottom:

1. REF (default: your checkout's HEAD) is built in a persistent clone at $BENCH_ROOT/src into a versioned binary ($BENCH_ROOT/bin/stellar-rpc-<sha>)
2. DATASETS are materialized into $BENCH_ROOT/golden/<name>/: packs-local is used in place, packs-gs is fetched once from GCS (staged atomically, so a failed fetch never looks like a complete dataset) and reused across runs.
3. INGEST/RUNS/CLOSE_INTERVAL drive the benchmark loop: for each (dataset, chunk) cell, RUNS fresh bench-ingest cold / hot processes, each with its own --out directory. Each run writes an invocation.json (binary version, commit, flags, timings, error if it failed) next to its CSVs — this is the Go-side change that makes every result directory self-describing.
4. Results land under $BENCH_ROOT/results/<NAME>-<sha>-<stamp>/ with a metadata.json manifest (config, datasets, hardware, hostname, timings) and are bundled (tar); with PUBLISH_URI set, publish.sh uploads the bundle to object storage (immutable — it refuses to overwrite an existing run).

After a run, $BENCH_ROOT looks like:

$BENCH_ROOT/
├── bin/stellar-rpc-<sha>                                    # binary built from REF, kept per sha
├── src/                                                                  # persistent build clone (build caches survive across campaigns)
├── golden/<dataset>/                                         # materialized datasets, reused across campaigns
├── scratch/<dataset>/<chunk>/                        # cold ingest output, wiped before every run
├── hot/<dataset>/<chunk>/                               # hot DBs (query-hot reads the one hot ingest leaves)
└── results/<NAME>-<sha>-<stamp>/             # one directory per campaign = the published bundle
    ├── <campaign>.cfg                                              # the exact config that ran
    ├── binary.txt                                                          # identity of the benchmarked binary
    ├── machine-metadata.txt                                    # hardware/OS/disk probe
    ├── metadata.json                                                 # manifest: config, datasets, hardware, timings
    └── ingest-cold-<ds>-c<chunk>-run<N>/        # one dir per invocation: CSVs + invocation.json

The results directory is also tarred to /tmp/bench-results-<NAME>-<sha>-<stamp>.tgz (the EBS root, so the bundle survives an instance stop).

--dry-run prints every command the campaign would execute without building, downloading, or benchmarking anything.

What changed

• scripts/bench-campaigns/ — new: campaign.sh (the runner, which also owns building the binary under test), bootstrap.sh (idempotent devbox prep after a stop/start: mounts the NVMe, installs the toolchain and native libs — no builds), publish.sh (bundle upload), and a commented example-campaign.cfg.
• bench-ingest — each cold/hot run writes invocation.json alongside its CSVs, on failure too.
• pace_lag fix — zero-lag (on-time) ledgers are now included in the pace_lag percentiles, so they describe the whole paced run rather than only the late ledgers.

Notes for reviewers

• campaign.sh also knows how to drive query benchmarks (QUERY=yes) and fixture datasets; those subcommands land in follow-up PRs. Until then, configs using them fail at runtime with "unknown command" — pack-based ingest campaigns don't use either.
• BENCH_ROOT (the working root) stays an environment variable; everything else is validated config keys.

Closes #896

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds reproducible, config-driven ingest benchmark campaigns for dedicated devboxes.

Changes:

• Adds campaign orchestration, bootstrap, publishing, and example configuration.
• Records per-invocation provenance for cold/hot ingest benchmarks.
• Includes zero-lag samples in paced-run statistics.

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
scripts/bench-devbox/campaign.sh	Orchestrates campaign builds, datasets, runs, and metadata.
scripts/bench-devbox/bootstrap.sh	Prepares benchmark devboxes.
scripts/bench-devbox/publish.sh	Publishes campaign results.
scripts/bench-devbox/example-campaign.cfg	Documents campaign configuration.
bench/invocation.go	Generates invocation manifests.
bench/invocation_test.go	Tests manifest generation.
bench/cold.go	Records cold-run provenance.
bench/hot.go	Records hot-run provenance.
bench/command.go	Passes Cobra commands into runners.
bench/bench_test.go	Updates runner tests.
bench/csvsink.go	Includes zero pace lag values.
bench/csvsink_test.go	Tests zero/mixed lag aggregation.

Comments suppressed due to low confidence (1)

scripts/bench-devbox/campaign.sh:247

• The REF build also bypasses the Makefile's injected version/commit/build metadata, leaving every generated invocation.json with default binary identity fields. Use the existing identity-aware build target here as well.

  run make build-libs
  run go build -o "$BIN" ./cmd/stellar-rpc

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (7)

scripts/bench-devbox/campaign.sh:232

• When no REF is supplied, this provenance can identify an altered binary as a clean commit: git describe --dirty ignores untracked files, even though untracked Go files are compiled (as the later comment notes). Refuse a dirty/untracked checkout here, or bundle/hash all working-tree changes so SHA, built_commit, and the retained metadata can identify the actual source.

  BUILT_COMMIT=$(git rev-parse HEAD)
  SHA=$(git describe --always --dirty --abbrev=8)

scripts/bench-devbox/campaign.sh:603

• NAME may start with -, so a valid value such as --help produces a member name that tar parses as an option and the campaign fails only after all benchmark work finishes. Terminate option parsing before the generated member name.

tar -C "$BENCH/results" -czf "$TARBALL" "$NAME-$SHA-$STAMP"

scripts/bench-devbox/campaign.sh:611

• The advertised retry command is not executable from the campaign's current directory: the script has cd'd to the repository root, while publish.sh is under scripts/bench-devbox and is not necessarily on PATH. Printing the absolute script path and shell-escaping each argument makes this the exact copyable retry command promised by the message.

    note "publish failed — data is safe in $RES and $TARBALL; retry with: publish.sh $RES $PUBLISH_URI"

cmd/stellar-rpc/internal/rpcv2/bench/cold.go:158

• Failed backfills can write partial CSVs at lines 146-148, but they return before this success-only invocation write. Those partial result directories therefore remain non-self-describing. Record invocation metadata from a finalization path for both success and failure, ideally including the completion status/error.

	if err := writeInvocationJSON(opts.OutDir, cmd, captureFlags(cmd), startedAt, time.Now().UTC()); err != nil {

cmd/stellar-rpc/internal/rpcv2/bench/hot.go:166

• Failed or incomplete hot runs write partial CSVs at lines 156-158 and return before this success-only invocation write, so those output directories still lack the provenance this change is intended to guarantee. Finalize invocation metadata on both paths and distinguish failed/incomplete runs in the record.

	if err := writeInvocationJSON(opts.OutDir, cmd, captureFlags(cmd), startedAt, time.Now().UTC()); err != nil {

scripts/bench-devbox/bootstrap.sh:39

• If /mnt/nvme is already a mountpoint, the script never verifies that it is backed by NVME_DEV. It can therefore silently run benchmarks on an EBS or other filesystem, invalidating the storage/fsync measurements despite having validated a different instance-store device. Check the existing mount source (for example with findmnt) and fail unless it resolves to NVME_DEV.

if ! mountpoint -q "$MOUNT"; then
  sudo mkdir -p "$MOUNT"
  sudo mount -o noatime "$NVME_DEV" "$MOUNT"
  sudo chown "$USER" "$MOUNT"
fi

scripts/bench-devbox/campaign.sh:132

• This prefix check does not make the sourced config assignment-only. For example, NAME=ok; REPO=/tmp passes and then overwrites an internal variable, while NAME=$(some-command) executes a command during source. As a result, unknown assignments and commands are not actually rejected. Parse the config as data, or validate the complete Bash syntax and reject separators/substitutions before sourcing it.

This issue also appears on line 231 of the same file.

  [[ $_stripped =~ $_re_cfg_key ]] ||