From 0380b31e990c6b9045b401ed7061ad607b3cbb8a Mon Sep 17 00:00:00 2001 From: Baris Erdem Date: Tue, 18 Aug 2026 17:20:12 +0200 Subject: [PATCH] Add mix belay.replay: a production failure is a test case MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Belay already journals the result of every step, so a failed job carries everything needed to run its code again on the exact values it saw. That was reachable only as Belay.Replay.dry_run/2, against a configured instance. mix belay.replay 4821 --url $DATABASE_URL Answers one of four ways: completed, diverged at , blocked, or raised. Divergence is the useful one — it names the point where today's code departs from what production ran, which is also how you find out whether in-flight jobs survive a worker change before deploying it. mix belay.replay 4821 --url $DATABASE_URL \ --out test/regressions/job_4821_test.exs writes a self-contained ExUnit file: worker, input, journal. No database, no instance. Fix the bug, keep the file, and the production data that broke you is in CI forever. Belay.Replay.from_journal/4 is the storage-free entry point behind it; it stands up an ephemeral in-memory instance for the call and tears it down after, so fixtures run anywhere mix test runs. Fixtures embed values copied from the database. The task docs, the guide, and the generated file header all say so. New guides/debugging.md, linked from the README and registered in docs. 201 tests on Memory, 219 on PostgreSQL. --- CHANGELOG.md | 24 +++- README.md | 1 + guides/debugging.md | 114 ++++++++++++++++++ lib/belay/replay.ex | 58 +++++++++ lib/mix/tasks/belay.replay.ex | 190 ++++++++++++++++++++++++++++++ mix.exs | 1 + test/belay/events_replay_test.exs | 27 +++++ test/belay/replay_task_test.exs | 75 ++++++++++++ 8 files changed, 489 insertions(+), 1 deletion(-) create mode 100644 guides/debugging.md create mode 100644 lib/mix/tasks/belay.replay.ex create mode 100644 test/belay/replay_task_test.exs diff --git a/CHANGELOG.md b/CHANGELOG.md index 1df752b..5c44c23 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -70,7 +70,29 @@ affect you (see the end of this entry). measured in minutes or days, `Belay.sleep/3` parks the job and holds nothing. -4. **`BELAY_PG_URL`** points the Postgres half of the test suite at any +4. **`mix belay.replay JOB_ID --url URL`** — re-run a job's code against the + journal it actually committed. Nothing executes: steps, sleeps and spawns + replay from the recording, `emit/2` and `debit/3` are inert, and the + database is only read, so replaying a *live* job is safe. + + It answers one of four ways — completed, **diverged at ``**, blocked, + or raised. Divergence names the exact point where today's code departs from + what production ran, which is also how you find out whether in-flight jobs + will survive a worker change *before* deploying it. + + `--out FILE` writes a self-contained ExUnit fixture — worker, input, and + journal, with no database and no instance required — so an incident becomes + permanent CI coverage. See the new [debugging guide](guides/debugging.md). + + Fixtures embed values copied from your database; the guide and the task docs + both say so, but it bears repeating here. + +5. **`Belay.Replay.from_journal/4`** — the storage-free replay entry point the + fixtures call. Stands up an ephemeral in-memory instance for the call and + tears it down after, so it runs from a test, a script, or an IEx session + with no database configured. + +6. **`BELAY_PG_URL`** points the Postgres half of the test suite at any instance, so contributors no longer have to reproduce the CI setup exactly. ### Deprecations diff --git a/README.md b/README.md index afff341..fbec23e 100644 --- a/README.md +++ b/README.md @@ -266,6 +266,7 @@ run. [Building agents](guides/agents.md) · [Operations](guides/operations.md) · [Testing](guides/testing.md) · +[Debugging](guides/debugging.md) · [Honest comparison](guides/comparison.md) · [Formal verification](verify/README.md) · [Architecture](DESIGN.md) · diff --git a/guides/debugging.md b/guides/debugging.md new file mode 100644 index 0000000..68c4e04 --- /dev/null +++ b/guides/debugging.md @@ -0,0 +1,114 @@ +# Debugging: a production failure is a test case + +A job failed in production three hours ago. The process is gone, the machine +may be gone, and the logs tell you *that* it failed, not what it saw. + +But the journal is still there. Every step the job committed, with the exact +value it returned. That is enough to run the job's code again, locally, and +watch it make the same decisions on the same data. + +```console +$ mix belay.replay 4821 --url $DATABASE_URL + +replaying job 4821 (MyApp.Reconcile) over 6 steps + +diverged at step "settle_v2" + +The journal has: fetch, normalize, match, price, settle +Today's code asked for "settle_v2", which production never ran. +``` + +Nothing executed. Steps, sleeps, and spawns replayed from the recording, +`emit/2` and `debit/3` were inert, and the database was only read. Replaying a +job that is *still running* is safe — no ack, no attempt consumed, no lease +touched. + +## The four answers + +| | what it means | +|---|---| +| **completed** | today's code walks the path the journal recorded, and returns this | +| **diverged at ``** | the code asked for a step the journal doesn't have | +| **blocked on ``** | the recording stops at a signal or sleep that never resolved | +| **raised** | the code itself raised during the replay | + +**Divergence is the interesting one.** It names the exact point where today's +code departs from what production ran — which is also how you answer *"will +the jobs currently in flight survive this deploy?"* before you deploy. Replay a +sample of live jobs against the new code; every divergence is a job that will +break on resume. + +## Freezing it as a test + +```console +$ mix belay.replay 4821 --url $DATABASE_URL \ + --out test/regressions/job_4821_test.exs + +wrote test/regressions/job_4821_test.exs (6 steps) +``` + +The file is self-contained — the worker, the input, and the journal: + +```elixir +defmodule Belay.Regression.Job4821Test do + use ExUnit.Case, async: true + + @input %{"account" => "ACC-9931"} + + @steps [ + %{name: "fetch", value: <<131, ...>>, usd_micros: 0, tokens: 0}, + ... + ] + + test "job 4821 replays against its recorded journal" do + assert {:ok, _result, _trace} = + Belay.Replay.from_journal(MyApp.Reconcile, @input, @steps) + end +end +``` + +No database. No Belay instance. It runs anywhere `mix test` runs, which is what +turns an incident into permanent coverage: fix the bug, keep the file, and the +exact production data that broke you is now in CI forever. + +> #### Fixtures carry production values {: .warning} +> +> The step values are copied out of your database and may contain personal data +> or secrets. Review a fixture before committing it, and treat it like any +> other file holding real values. If the instance encrypts at rest, values are +> embedded as stored and replaying them needs the same key. + +## From the API + +`mix belay.replay` is a wrapper over two functions you can call directly. + +Against a running instance: + +```elixir +Belay.Replay.dry_run(MyApp.Belay, 4821) +#=> {:blocked, {:missing_step, "settle_v2"}, trace} +``` + +Against a journal you already hold, with no instance at all: + +```elixir +Belay.Replay.from_journal(MyApp.Reconcile, input, steps) +``` + +`from_journal/4` is what the generated fixture calls. It stands up an ephemeral +in-memory instance for the duration and tears it down afterwards, so it is safe +to call from a test, a script, or an IEx session on a laptop with no database +configured. + +## Why this works + +Belay journals the *result* of every step, not just that it ran. Replay is +therefore not a simulation — it is the same code, taking the same branches, on +the same values, because every value it reads is the value production wrote. + +The one thing replay cannot reproduce is work that was never journaled: code +*between* steps runs again for real during a replay. If that code is +non-deterministic — a branch on `:rand`, on the wall clock, on ambient state — +a replay may take a path the original never did. Keeping decisions inside steps +is what makes replay trustworthy, and is a good idea for +[durability](durable-steps.md) anyway. diff --git a/lib/belay/replay.ex b/lib/belay/replay.ex index 7c2c7bc..6b5b115 100644 --- a/lib/belay/replay.ex +++ b/lib/belay/replay.ex @@ -50,6 +50,64 @@ defmodule Belay.Replay do end end + @doc """ + Replay a recorded journal with no database and no running instance. + + This is what `mix belay.replay` writes into a fixture: the worker, the input + it ran on, and the steps it committed, which together are everything a replay + needs. A production failure becomes a test case you can commit. + + Belay.Replay.from_journal(MyApp.Worker, %{"id" => 7}, [ + %{name: "fetch", value: <<131, ...>>}, + %{name: "score", value: <<131, ...>>} + ]) + + Each step needs `:name` and `:value` (the stored binary, in either encoding + `Belay.Codec` accepts); `:usd_micros` and `:tokens` are optional and default + to zero. Returns the same shapes as `dry_run/2`. + + An ephemeral in-memory instance is started for the call and stopped + afterwards, so this is safe to run anywhere — including against a codebase + whose real instance is not configured. + """ + @spec from_journal(module(), map(), [map()], keyword()) :: + {:ok, term(), trace} + | {:blocked, term(), trace} + | {:raised, {atom(), term()}, trace} + def from_journal(worker, input, steps, opts \\ []) do + name = Module.concat(Belay.ReplayFixture, "R#{System.unique_integer([:positive])}") + + {:ok, pid} = + Belay.start_link( + name: name, + storage: [adapter: :memory], + queues: [], + crons: [], + notifiers: [], + clock: Keyword.get(opts, :clock, Belay.Clock.System) + ) + + try do + {:ok, job} = Belay.insert(name, {Belay.Worker, worker, input, Keyword.get(opts, :job, [])}) + + {storage, ref} = Config.fetch!(name).storage_ref + now = DateTime.utc_now() + + for step <- steps do + cost = %{ + usd_micros: Map.get(step, :usd_micros, 0), + tokens: Map.get(step, :tokens, 0) + } + + {:ok, _} = storage.put_step(ref, job.id, to_string(step.name), step.value, cost, now) + end + + dry_run(name, job.id) + after + Supervisor.stop(pid) + end + end + defp fetch(storage, ref, job_id) do case storage.get_job(ref, job_id) do {:ok, job} -> {:ok, job} diff --git a/lib/mix/tasks/belay.replay.ex b/lib/mix/tasks/belay.replay.ex new file mode 100644 index 0000000..2b6f4a8 --- /dev/null +++ b/lib/mix/tasks/belay.replay.ex @@ -0,0 +1,190 @@ +defmodule Mix.Tasks.Belay.Replay do + @shortdoc "Replay a job against its recorded journal, or write it out as a test" + + @moduledoc """ + Re-run a job's *code* against the *journal it actually committed*, so a + production failure becomes something you can debug locally — and commit. + + # What did job 4821 do? + mix belay.replay 4821 --url postgres://localhost/my_app + + # Freeze it as a regression test. + mix belay.replay 4821 --url postgres://localhost/my_app \\ + --out test/regressions/job_4821_test.exs + + Nothing is executed. Every memoized read — steps, sleeps, spawns — returns + its recorded value, `emit/2` and `debit/3` are inert, and the database is + opened read-only. Replaying a live job is safe: no ack, no attempt consumed, + no lease touched. + + ## What it tells you + + * **completed** — the current code walks the same path the journal + recorded, and returns this value. + * **diverged at ``** — the code asked for a step the journal does not + have. This is the useful one: it names the exact point where today's code + departs from what production ran. Run it before deploying a worker change + to find out whether in-flight jobs will survive it. + * **blocked on ``** — the recording stopped at a signal or sleep + that never resolved. + * **raised** — the code itself raised during replay. + + ## Fixtures + + `--out` writes a self-contained ExUnit file: the worker, the input, and the + steps, with no database dependency and no Belay instance required. It runs + anywhere `mix test` runs, which is what makes a production incident into a + permanent regression test. + + > #### Fixtures contain journal values {: .warning} + > + > A fixture embeds the step values the job committed, which may include + > personal data or secrets. It is a copy of production data — review it + > before committing, and treat it like any other fixture carrying real + > values. If the instance encrypts at rest, values are embedded as stored + > (encrypted), and replaying them needs the same key. + + ## Options + + * `--url` — Postgres URL of the database holding the job (required) + * `--out` — write a fixture here instead of replaying now + * `--module` — module name for the generated test (inferred from `--out`) + + Run this from your application, so worker modules are loadable. + """ + + use Mix.Task + + alias Belay.Storage.Postgres + + @switches [url: :string, out: :string, module: :string] + + @impl Mix.Task + def run(argv) do + {opts, args, _} = OptionParser.parse(argv, strict: @switches) + + job_id = parse_job_id(args) + url = opts[:url] || Mix.raise("--url is required (the database holding job #{job_id})") + + Mix.Task.run("app.start") + + {job, steps} = fetch(url, job_id) + + case opts[:out] do + nil -> replay_now(job, steps) + path -> write_fixture(path, opts[:module], job, steps) + end + end + + defp parse_job_id([id]) do + case Integer.parse(id) do + {int, ""} -> int + _ -> Mix.raise("expected a job id, got #{inspect(id)}") + end + end + + defp parse_job_id(_), do: Mix.raise("usage: mix belay.replay JOB_ID --url URL [--out FILE]") + + # A throwaway connection, closed before anything is replayed: the task reads + # the journal and then has no further business with the database. + defp fetch(url, job_id) do + conn_opts = + url + |> Postgres.parse_url() + |> Keyword.put(:types, Belay.Storage.PostgresTypes) + + {:ok, conn} = Postgrex.start_link(conn_opts) + + try do + case Postgres.get_job(conn, job_id) do + {:ok, job} -> + {:ok, steps} = Postgres.list_steps(conn, job_id) + {job, steps} + + :error -> + Mix.raise("job #{job_id} not found") + end + after + GenServer.stop(conn) + end + end + + defp replay_now(job, steps) do + worker = Belay.Job.worker_module!(job) + + Mix.shell().info("replaying job #{job.id} (#{job.kind}) over #{length(steps)} steps\n") + + case Belay.Replay.from_journal(worker, job.input, steps) do + {:ok, result, _trace} -> + Mix.shell().info("completed: #{inspect(result, pretty: true)}") + + {:blocked, {:missing_step, name}, trace} -> + Mix.shell().error("diverged at step #{inspect(name)}") + + Mix.shell().info( + "\nThe journal has: #{Enum.map_join(trace, ", ", & &1.name)}\n" <> + "Today's code asked for #{inspect(name)}, which production never ran." + ) + + {:blocked, reason, _trace} -> + Mix.shell().error("blocked: #{inspect(reason)}") + + {:raised, {kind, reason}, _trace} -> + Mix.shell().error("raised: #{Exception.format_banner(kind, reason)}") + end + end + + defp write_fixture(path, module, job, steps) do + module = module || infer_module(path) + + File.mkdir_p!(Path.dirname(path)) + File.write!(path, fixture(module, job, steps)) + + Mix.shell().info( + "wrote #{path} (#{length(steps)} steps)\n\nRun it with:\n\n mix test #{path}" + ) + end + + defp infer_module(path) do + path + |> Path.basename(".exs") + |> Macro.camelize() + |> then(&"Belay.Regression.#{&1}") + end + + defp fixture(module, job, steps) do + # job.kind is the module name as a string ("MyApp.Worker"); the fixture + # needs the module itself, which inspect/1 renders back as valid source. + worker = Belay.Job.worker_module!(job) + + step_terms = + Enum.map_join(steps, ",\n ", fn step -> + "%{name: #{inspect(step.name)}, value: #{inspect(step.value)}, " <> + "usd_micros: #{step.usd_micros}, tokens: #{step.tokens}}" + end) + + """ + # Generated by `mix belay.replay #{job.id}`. + # + # Replays #{job.kind} against the journal job #{job.id} committed in + # production. No database and no Belay instance required. + # + # NOTE: the step values below are a copy of production data. Review before + # committing. + defmodule #{module} do + use ExUnit.Case, async: true + + @input #{inspect(job.input, limit: :infinity)} + + @steps [ + #{step_terms} + ] + + test "job #{job.id} replays against its recorded journal" do + assert {:ok, _result, _trace} = + Belay.Replay.from_journal(#{inspect(worker)}, @input, @steps) + end + end + """ + end +end diff --git a/mix.exs b/mix.exs index 68c1c01..ad3cec7 100644 --- a/mix.exs +++ b/mix.exs @@ -34,6 +34,7 @@ defmodule Belay.MixProject do "guides/agents.md", "guides/operations.md", "guides/testing.md", + "guides/debugging.md", "guides/comparison.md", {"verify/README.md", filename: "formal-verification", title: "Formal verification"}, "DESIGN.md", diff --git a/test/belay/events_replay_test.exs b/test/belay/events_replay_test.exs index aac34dd..bc909ab 100644 --- a/test/belay/events_replay_test.exs +++ b/test/belay/events_replay_test.exs @@ -68,4 +68,31 @@ defmodule Belay.EventsReplayTest do test "dry_run of a missing job", %{name: name} do assert {:error, :not_found} = Replay.dry_run(name, 424_242) end + + describe "from_journal/4" do + test "replays a recorded journal with no instance and no database" do + steps = [ + %{name: "a", value: Belay.Codec.encode(20)}, + %{name: "b", value: Belay.Codec.encode(22)} + ] + + assert {:ok, {:ok, 42}, trace} = + Belay.Replay.from_journal(Belay.Test.Divergent, %{}, steps) + + assert Enum.map(trace, & &1.name) == ["a", "b"] + end + + test "reports divergence when the code asks for a step the journal lacks" do + steps = [ + %{name: "a", value: Belay.Codec.encode(20)}, + %{name: "b", value: Belay.Codec.encode(22)} + ] + + :persistent_term.put({Belay.Test.Divergent, :path}, :changed) + on_exit(fn -> :persistent_term.put({Belay.Test.Divergent, :path}, :original) end) + + assert {:blocked, {:missing_step, "b_new"}, _trace} = + Belay.Replay.from_journal(Belay.Test.Divergent, %{}, steps) + end + end end diff --git a/test/belay/replay_task_test.exs b/test/belay/replay_task_test.exs new file mode 100644 index 0000000..b9e8610 --- /dev/null +++ b/test/belay/replay_task_test.exs @@ -0,0 +1,75 @@ +# The task reads a real journal out of Postgres, so it only runs on that half +# of the matrix. +if Application.get_env(:belay, :test_storage) == :postgres do + defmodule Mix.Tasks.Belay.ReplayTest do + use Belay.Test.Case, async: false + + alias Belay.Test.Budgeted + + setup do + Mix.shell(Mix.Shell.Process) + on_exit(fn -> Mix.shell(Mix.Shell.IO) end) + + {:ok, start_belay!()} + end + + defp url, do: Application.fetch_env!(:belay, :test_pg_url) + + test "replays a completed job and reports the result", %{name: name} do + {:ok, job} = Belay.insert(name, Budgeted.new(%{"steps" => 2, "usd" => 0.01})) + + assert %{succeeded: 1} = Testing.drain(name, :default) + + Mix.Tasks.Belay.Replay.run([to_string(job.id), "--url", url()]) + + assert_received {:mix_shell, :info, [banner]} + assert banner =~ "replaying job #{job.id}" + assert banner =~ "over 2 steps" + + assert_received {:mix_shell, :info, [outcome]} + assert outcome =~ "completed:" + end + + test "writes a self-contained fixture that is valid Elixir", %{name: name} do + {:ok, job} = Belay.insert(name, Budgeted.new(%{"steps" => 2, "usd" => 0.01})) + + assert %{succeeded: 1} = Testing.drain(name, :default) + + path = + Path.join(System.tmp_dir!(), "belay_replay_#{System.unique_integer([:positive])}.exs") + + on_exit(fn -> File.rm(path) end) + + Mix.Tasks.Belay.Replay.run([to_string(job.id), "--url", url(), "--out", path]) + + source = File.read!(path) + + # It must parse, name the worker as a module (not the "kind" string), + # and carry the journal it will replay against. + assert Code.string_to_quoted!(source) + assert source =~ "Belay.Replay.from_journal(Belay.Test.Budgeted" + assert source =~ ~s(name: "s1") + assert source =~ ~s(name: "s2") + assert source =~ "Review before" + + assert_received {:mix_shell, :info, [message]} + assert message =~ "wrote #{path}" + end + + test "a missing job fails with a clear message" do + assert_raise Mix.Error, ~r/job 99999999 not found/, fn -> + Mix.Tasks.Belay.Replay.run(["99999999", "--url", url()]) + end + end + + test "requires a job id and a url" do + assert_raise Mix.Error, ~r/usage: mix belay.replay/, fn -> + Mix.Tasks.Belay.Replay.run(["--url", url()]) + end + + assert_raise Mix.Error, ~r/--url is required/, fn -> + Mix.Tasks.Belay.Replay.run(["1"]) + end + end + end +end