Add mix belay.replay: a production failure is a test case

CHANGELOG.md

@@ -70,7 +70,29 @@ affect you (see the end of this entry).
    measured in minutes or days, `Belay.sleep/3` parks the job and holds
    nothing.
 
-4. **`BELAY_PG_URL`** points the Postgres half of the test suite at any
+4. **`mix belay.replay JOB_ID --url URL`** — re-run a job's code against the
+   journal it actually committed. Nothing executes: steps, sleeps and spawns
+   replay from the recording, `emit/2` and `debit/3` are inert, and the
+   database is only read, so replaying a *live* job is safe.
+
+   It answers one of four ways — completed, **diverged at `<step>`**, blocked,
+   or raised. Divergence names the exact point where today's code departs from
+   what production ran, which is also how you find out whether in-flight jobs
+   will survive a worker change *before* deploying it.
+
+   `--out FILE` writes a self-contained ExUnit fixture — worker, input, and
+   journal, with no database and no instance required — so an incident becomes
+   permanent CI coverage. See the new [debugging guide](guides/debugging.md).
+
+   Fixtures embed values copied from your database; the guide and the task docs
+   both say so, but it bears repeating here.
+
+5. **`Belay.Replay.from_journal/4`** — the storage-free replay entry point the
+   fixtures call. Stands up an ephemeral in-memory instance for the call and
+   tears it down after, so it runs from a test, a script, or an IEx session
+   with no database configured.
+
+6. **`BELAY_PG_URL`** points the Postgres half of the test suite at any
    instance, so contributors no longer have to reproduce the CI setup exactly.
 
 ### Deprecations

README.md

@@ -266,6 +266,7 @@ run.
 [Building agents](guides/agents.md) ·
 [Operations](guides/operations.md) ·
 [Testing](guides/testing.md) ·
+[Debugging](guides/debugging.md) ·
 [Honest comparison](guides/comparison.md) ·
 [Formal verification](verify/README.md) ·
 [Architecture](DESIGN.md) ·

guides/debugging.md

@@ -0,0 +1,114 @@
+# Debugging: a production failure is a test case
+
+A job failed in production three hours ago. The process is gone, the machine
+may be gone, and the logs tell you *that* it failed, not what it saw.
+
+But the journal is still there. Every step the job committed, with the exact
+value it returned. That is enough to run the job's code again, locally, and
+watch it make the same decisions on the same data.
+
+```console
+$ mix belay.replay 4821 --url $DATABASE_URL
+
+replaying job 4821 (MyApp.Reconcile) over 6 steps
+
+diverged at step "settle_v2"
+
+The journal has: fetch, normalize, match, price, settle
+Today's code asked for "settle_v2", which production never ran.
+```
+
+Nothing executed. Steps, sleeps, and spawns replayed from the recording,
+`emit/2` and `debit/3` were inert, and the database was only read. Replaying a
+job that is *still running* is safe — no ack, no attempt consumed, no lease
+touched.
+
+## The four answers
+
+| | what it means |
+|---|---|
+| **completed** | today's code walks the path the journal recorded, and returns this |
+| **diverged at `<step>`** | the code asked for a step the journal doesn't have |
+| **blocked on `<control>`** | the recording stops at a signal or sleep that never resolved |
+| **raised** | the code itself raised during the replay |
+
+**Divergence is the interesting one.** It names the exact point where today's
+code departs from what production ran — which is also how you answer *"will
+the jobs currently in flight survive this deploy?"* before you deploy. Replay a
+sample of live jobs against the new code; every divergence is a job that will
+break on resume.
+
+## Freezing it as a test
+
+```console
+$ mix belay.replay 4821 --url $DATABASE_URL \
+    --out test/regressions/job_4821_test.exs
+
+wrote test/regressions/job_4821_test.exs (6 steps)
+```
+
+The file is self-contained — the worker, the input, and the journal:
+
+```elixir
+defmodule Belay.Regression.Job4821Test do
+  use ExUnit.Case, async: true
+
+  @input %{"account" => "ACC-9931"}
+
+  @steps [
+    %{name: "fetch", value: <<131, ...>>, usd_micros: 0, tokens: 0},
+    ...
+  ]
+
+  test "job 4821 replays against its recorded journal" do
+    assert {:ok, _result, _trace} =
+             Belay.Replay.from_journal(MyApp.Reconcile, @input, @steps)
+  end
+end
+```
+
+No database. No Belay instance. It runs anywhere `mix test` runs, which is what
+turns an incident into permanent coverage: fix the bug, keep the file, and the
+exact production data that broke you is now in CI forever.
+
+> #### Fixtures carry production values {: .warning}
+>
+> The step values are copied out of your database and may contain personal data
+> or secrets. Review a fixture before committing it, and treat it like any
+> other file holding real values. If the instance encrypts at rest, values are
+> embedded as stored and replaying them needs the same key.
+
+## From the API
+
+`mix belay.replay` is a wrapper over two functions you can call directly.
+
+Against a running instance:
+
+```elixir
+Belay.Replay.dry_run(MyApp.Belay, 4821)
+#=> {:blocked, {:missing_step, "settle_v2"}, trace}
+```
+
+Against a journal you already hold, with no instance at all:
+
+```elixir
+Belay.Replay.from_journal(MyApp.Reconcile, input, steps)
+```
+
+`from_journal/4` is what the generated fixture calls. It stands up an ephemeral
+in-memory instance for the duration and tears it down afterwards, so it is safe
+to call from a test, a script, or an IEx session on a laptop with no database
+configured.
+
+## Why this works
+
+Belay journals the *result* of every step, not just that it ran. Replay is
+therefore not a simulation — it is the same code, taking the same branches, on
+the same values, because every value it reads is the value production wrote.
+
+The one thing replay cannot reproduce is work that was never journaled: code
+*between* steps runs again for real during a replay. If that code is
+non-deterministic — a branch on `:rand`, on the wall clock, on ambient state —
+a replay may take a path the original never did. Keeping decisions inside steps
+is what makes replay trustworthy, and is a good idea for
+[durability](durable-steps.md) anyway.

lib/belay/replay.ex

@@ -50,6 +50,64 @@ defmodule Belay.Replay do
     end
   end
 
+  @doc """
+  Replay a recorded journal with no database and no running instance.
+
+  This is what `mix belay.replay` writes into a fixture: the worker, the input
+  it ran on, and the steps it committed, which together are everything a replay
+  needs. A production failure becomes a test case you can commit.
+
+      Belay.Replay.from_journal(MyApp.Worker, %{"id" => 7}, [
+        %{name: "fetch", value: <<131, ...>>},
+        %{name: "score", value: <<131, ...>>}
+      ])
+
+  Each step needs `:name` and `:value` (the stored binary, in either encoding
+  `Belay.Codec` accepts); `:usd_micros` and `:tokens` are optional and default
+  to zero. Returns the same shapes as `dry_run/2`.
+
+  An ephemeral in-memory instance is started for the call and stopped
+  afterwards, so this is safe to run anywhere — including against a codebase
+  whose real instance is not configured.
+  """
+  @spec from_journal(module(), map(), [map()], keyword()) ::
+          {:ok, term(), trace}
+          | {:blocked, term(), trace}
+          | {:raised, {atom(), term()}, trace}
+  def from_journal(worker, input, steps, opts \\ []) do
+    name = Module.concat(Belay.ReplayFixture, "R#{System.unique_integer([:positive])}")
+
+    {:ok, pid} =
+      Belay.start_link(
+        name: name,
+        storage: [adapter: :memory],
+        queues: [],
+        crons: [],
+        notifiers: [],
+        clock: Keyword.get(opts, :clock, Belay.Clock.System)
+      )
+
+    try do
+      {:ok, job} = Belay.insert(name, {Belay.Worker, worker, input, Keyword.get(opts, :job, [])})
+
+      {storage, ref} = Config.fetch!(name).storage_ref
+      now = DateTime.utc_now()
+
+      for step <- steps do
+        cost = %{
+          usd_micros: Map.get(step, :usd_micros, 0),
+          tokens: Map.get(step, :tokens, 0)
+        }
+
+        {:ok, _} = storage.put_step(ref, job.id, to_string(step.name), step.value, cost, now)
+      end
+
+      dry_run(name, job.id)
+    after
+      Supervisor.stop(pid)
+    end
+  end
+
   defp fetch(storage, ref, job_id) do
     case storage.get_job(ref, job_id) do
       {:ok, job} -> {:ok, job}