⚓ Belay

Durable execution for Elixir — without a workflow server.

Resume at the first unfinished step. Add budgets, signals, DAGs, replay, and an embedded dashboard. Keep it all on Postgres.

Background jobs get awkward when one run contains twelve paid API calls, a human approval, and a fan-out across the cluster. Classic queues retry the whole job. Belay makes the step the unit of recovery: once a step's result commits to the journal, later attempts replay it in microseconds and continue from the first unfinished step. The same journal powers spend accounting and replay debugging.

Dependencies: Postgrex, Jason, telemetry — no Ecto requirement, no Redis, no separate service. PostgreSQL 14–18 are supported and exercised across the CI compatibility matrix. Apache-2.0.

Quick start

# mix.exs
{:belay, "~> 2.0"}

# config/config.exs — the Ecto/Phoenix convention, keyed by the instance name
config :my_app, MyApp.Belay,
  queues: [default: 10, mailers: [limit: 20]],
  crons: [[name: "digest", expr: "0 8 * * 1-5", worker: MyApp.Digest]]

# config/runtime.exs — runtime values (the database URL)
config :my_app, MyApp.Belay,
  storage: [adapter: :postgres, url: System.fetch_env!("DATABASE_URL")]

# application.ex — otp_app pulls the config above; inline opts still override
children = [
  {Belay, otp_app: :my_app, name: MyApp.Belay},
  {Belay.Dashboard, belay: MyApp.Belay, port: 4004}
]

# once, at deploy time (idempotent)
Belay.Storage.Postgres.migrate!(db_url)

Prefer everything inline in the supervision tree? That works too — pass the same keys to {Belay, name: ..., storage: ..., queues: ...} and skip otp_app.

defmodule MyApp.WelcomeEmail do
  use Belay.Worker, queue: :mailers, max_attempts: 5

  @impl Belay.Worker
  def run(ctx) do
    MyApp.Mailer.deliver_welcome(ctx.job.input["user_id"])
  end
end

Belay.insert(MyApp.Belay, MyApp.WelcomeEmail.new(%{"user_id" => 42}))

Testing is deterministic — drain synchronously, travel through time:

{:ok, job} = Belay.insert(name, MyApp.Pipeline.new(%{"id" => 1}))
assert %{succeeded: 1} = Belay.Testing.drain(name, :default)

Belay.Clock.Sim.advance(clock, 3_600)   # backoff, cron, rate windows, leases…

Steps, waits, fan-out — one worker

Where Belay pulls ahead of a classic queue is jobs with expensive parts, long waits, and moving pieces:

defmodule MyApp.ResearchPipeline do
  use Belay.Worker, queue: :pipelines, max_attempts: 10

  @impl Belay.Worker
  def run(ctx) do
    # Committed steps replay from the journal; unfinished steps run again.
    text    = Belay.step(ctx, :transcribe, fn -> whisper!(ctx.job.input["url"]) end)
    summary = Belay.step(ctx, :summarize, fn -> llm!(text) end, cost: [usd: 0.02, tokens: 1200])

    # Fan out real jobs across the cluster; park at zero cost until all land.
    checks = Belay.map_children(ctx, :verify, MyApp.FactCheck,
               Enum.map(summary.claims, &%{"claim" => &1}))

    # Human in the loop: durable wait, instant wake on signal.
    case Belay.await(ctx, :approval, timeout: 86_400) do
      %{"approved" => true} -> {:ok, %{summary: summary, checks: checks}}
      _ -> {:cancel, :rejected}
    end
  end
end

# Attach a job budget and enforce uniqueness at insert:
Belay.insert(MyApp.Belay,
  MyApp.ResearchPipeline.new(%{"url" => url}, budget: [usd: 1.00], unique: "research:#{url}"))

Step bodies have at-least-once execution until their result is committed. If a process dies after an external side effect but before the journal write, that body can run again. Use provider idempotency keys or make external writes idempotent. Once the journal row exists, normal retries do not re-execute it.

The same engine runs the unglamorous fleet — emails, webhooks, image processing, nightly crons — and the newer shapes those queues were never built for: declared workflow DAGs, month-long approval flows parked at zero cost, and LLM/agent pipelines where each API call is a journaled step with a price on it.

Budgets and spend control

LLM-heavy workloads add a failure mode classic queues don't model: cost. A retry that re-runs completed API calls buys them twice; a usage spike outruns any dashboard. Belay enforces spend in the engine, in three layers:

# 1. Per-job fail-after-crossing limit. The step that crosses the limit runs;
#    no later unfinished step is admitted. Overshoot is bounded by that
#    step's declared cost in the modeled single-attempt path.
MyApp.ResearchPipeline.new(input, budget: [usd: 1.00])

# 2. Fleet-wide admission budget per window — a resource bucket spans every
#    queue that names it. Claims debit estimates atomically across the fleet.
queues: [
  ai: [limit: 20,
       rate: [resource: "anthropic_tokens", allowed: 2_000_000, period: 60, estimate: 3_000]]
]
# When the estimated allowance is spent, claims stop and work queues.

# 3. True-up — the claim debits the estimate; inside the job you correct it
#    with actuals, so the window converges on what the invoice will say.
#    A job debits its own queue's resource:
Belay.step(ctx, :summarize, fn ->
  %{text: text, usage: usage} = llm!(prompt)
  Belay.debit(ctx, "anthropic_tokens", usage.total_tokens)
  text
end, cost: [usd: 0.02])

Committed steps replay from the journal at zero declared cost, and budget checks read durable spend, so a later attempt cannot forget recorded cost. As with any at-least-once system, an external charge made before a crash but not journaled may repeat and is not visible to Belay; use the provider's idempotency mechanism for that window.

Why it's built this way

• The journal is the core. Steps, costs, events, and results are first-class columns — so budgets, token-true-up rate limits, streaming, and replay debugging fall out of the schema instead of being bolted on.
• Leaderless everything. No peer election exists in the codebase: cron dedupes through a unique index, recovery is idempotent row-level work any node performs. The "leader stalled, nothing runs" failure class is structurally absent.
• Millisecond dispatch on a polling-floor guarantee. Adaptive burst polling plus an opt-in pg_notify accelerator — measured 11ms p50 insert→result across unconnected processes — while nothing load-bearing touches LISTEN/NOTIFY, so PgBouncer transaction pooling and serverless Postgres just work.
• Leases with fencing, not rescue heuristics. Crashed workers' jobs are reclaimed in seconds; zombie acks are rejected by attempt fencing.
• Deterministic by design. Storage behind a behaviour with an in-memory reference adapter; the clock injectable everywhere (SQL included). One suite runs against both adapters, and time-dependent behavior is tested by time travel, never sleeps.

The workflow DAG view

Workflows, fan-outs, and spawned children render as a live graph — deep-linkable (#workflow=<id>), with the full step journal one click away:

Feature tour

Durable steps + budgets	step/4 with cost:; budget: [usd:, tokens:] — committed steps replay, jobs fail after crossing a limit
Human-in-the-loop	await/3 parks at zero cost; signal_job/4 wakes instantly; deadlines
Steering & cancellation	steer_job/3 injects guidance mid-run; cooperative cancel at step boundaries
Dynamic children	spawn/3, await_children/1, map_children/5 — replay-safe runtime DAGs
Workflows & batches	declared DAGs, transactional release, cascade/ignore policies
Event streams	emit/2 + live subscriptions + offset replay — survives crashes
Replay debugging	Belay.Replay.dry_run/2 — re-run code against the recorded journal
Cluster limits	global_limit, sliding-window rate (request- or token-based with true-up), per-tenant partition fairness (exact, skew-proof)
Transactional enqueue	Belay.Txn.insert/3 in your Postgrex/Ecto transaction; wake-ups deliver exactly on commit
Unique jobs	constraint-backed: while-incomplete, windowed, or forever
Chunk workers	chunk: [size:, gather_ms:] — N jobs, one invocation (batch-priced APIs, bulk INSERTs), per-job partial failure
Adaptive concurrency	limit: [min:, max:] — per-node scaling under load, exactly bounded by cluster limits
Encrypted inputs	AES-256-GCM at rest; plaintext only in the executing process
Runtime CRUD	Belay.Queues / Belay.Crons — change queues and schedules with no deploy
Scheduling	schedule_in, durable sleep/3, leaderless cron with exactly-once slots
Embedded dashboard	zero dependencies, one child spec — everything in the GIFs above
MCP server	mix belay.mcp — AI assistants inspect the queue; writes are disabled by default and require an authorizer or explicit opt-in
Oban migration	mix belay.migrate_oban — pending jobs move in one command: dry-run analyzer, port verification, idempotent

Measured, not claimed

Numbers from this repo's reproducible harnesses on a laptop (Postgres 16):

What	Result	Harness
Dispatch, same node	8.6ms p50 insert→result	bench/run.sh
Dispatch, cross-process + pg_notify	11.0ms p50 / 24.5ms p99	bench/run.sh
Throughput (unbatched acks, 3 workers)	~416 jobs/s end-to-end	bench/throughput.exs
Endurance soak (7h)	99,004 jobs · 4,978 kill -9 · 13 DB restarts — surfaced one real bug (below); 13/13 invariants on the post-fix revalidation	soak/run.sh → reports in soak/reports/
Durable-core model	Recorded complete TLC run: 187,975,659 distinct states, zero violations, depth 49	verify/spec/RESULT.md
Trace conformance	7/7 real execution traces admitted (60/60 events), including crash/reclaim and cancel windows	verify/traces/ (via Attest.trace_check/2)
Attempt fencing	Complete 10-state model passes; removing the fence produces the expected zombie-ack counterexample	verify/attempt_fence/
Schedule exploration	READ COMMITTED wake race reproduced + fix proven across 400 schedules	verify/wake_protocol/
Suites	137 (Memory) + 149 (PostgreSQL), same behavioral suite plus PostgreSQL-only integration checks, --warnings-as-errors	mix test

Six real bugs were found by these harnesses before any user could: two by the chaos soak (a READ COMMITTED wake race among them), one by the 7-hour endurance run (the budget crash window), one by adapter-equivalence property testing (lost cancel requests), and two by extending the formal model (retry semantics). The model is validated by rediscovery: revert any fix in it and TLC reproduces the production failure, step for step (CHANGELOG, verify/). The race lessons are codified in the wire contract.

Proven on a real app

The first design-partner port replaced Oban in a production-shaped Phoenix ingestion service (4 workers, webhooks, GDPR deletion flows, hourly cron, uniqueness everywhere): 222/222 tests green and a live-producer run on the first attempt, with zero engine changes required. Dead-node recovery tightened from a conservative 45-minute rescue plugin to ~2× a 60-second lease — renewed leases replace guessing.

Coming from Oban?

Workers port mechanically (perform/1 → run/1), the plugins you delete are engine config (Pruner → retention:, Lifeline → renewed leases, Cron → crons:), and pending jobs move in one command:

mix belay.migrate_oban --url postgres://.../my_app            # dry-run report
mix belay.migrate_oban --url postgres://.../my_app --execute

The migration guide has the full porting map, the uniqueness translation, and the archive pattern for historical rows — with field notes from a real port that went 222/222 on the first run.

Docs

Getting started · Migrating from Oban · Durable steps · Building agents · Operations · Testing · Honest comparison · Formal verification · Architecture · Wire contract

Status

1.0.0. The public API and wire contract are stable. Belay is tested against PostgreSQL 14–18 and Elixir 1.15–1.19 / OTP 26–28. Production adoption remains early, so evaluate it against your failure modes before replacing a queue that already works. Post-1.0 candidates: SQLite storage, batched acking, and additional language clients.

License

Apache-2.0. Contributions welcome — CONTRIBUTING.md's five rules are the soul of the codebase.