Stepwise — durable workflow orchestrator

Stepwise runs DAG workflows of dependent tasks on a pool of workers. PostgreSQL is the single source of truth: tasks are claimed with row locks, held with heartbeated leases, and protected by fencing tokens, so workers can crash, be stopped or be partitioned without losing or double-applying work. Executions survive API and worker restarts and keep a complete attempt and event history.

Status: backend, infrastructure and CI are implemented and verified; the React dashboard, Prometheus/Grafana provisioning and the k6 load test are not built yet. See docs/IMPLEMENTATION_STATUS.md for exact test results and next steps.

What works today

• Workflows: drafts with server-side validation (cycles, missing/self dependencies, unknown task types, retry/timeout bounds, size limits, SSRF policy, template references), immutable published versions, activation, soft delete.
• Execution: dependency-ordered scheduling, bounded worker concurrency, exact per-tenant concurrency limits, leases + heartbeats + fencing, task timeouts, exponential backoff with jitter, retryable error categories, graceful worker drain, expired-lease recovery.
• Task types: NO_OP, DELAY, HTTP_REQUEST (allowlist, DNS-pinned private/metadata address blocking, re-validated redirects, size and time limits, stable Idempotency-Key), TRANSFORM (data-only templates), FAIL_FOR_TESTING. No shell or script execution.
• Operations: idempotent submission (Idempotency-Key), cancel, retry-in-place, rerun, dead-letter retry/dismiss, stuck-task listing, forced release, cron schedules with time zones and misfire policies, transactional outbox, SSE live events with Last-Event-ID replay, audit log, retention job.
• Security: registration/login, BCrypt, 15-minute JWT access tokens, rotating HttpOnly refresh cookies with reuse detection, ADMIN/USER roles, tenant scoping on every query, rate limits, problem+json errors.
• Observability: JSON logs with request/execution ids, Prometheus metrics, health/readiness probes, trace context propagated from the submitting request to worker spans (OTLP export opt-in).

Architecture

flowchart LR
    client[Client / dashboard] -->|REST + SSE| api
    subgraph api[workflow-api]
      rest[REST controllers] --> core1[(workflow-core engine)]
      jobs[Scheduler / outbox publisher / reaper / retention] --> core1
      hub[SSE hub]
    end
    subgraph workers[workflow-worker x N]
      loop[Claim loop + handlers + heartbeats] --> core2[(workflow-core engine)]
    end
    core1 <-->|transactions, SKIP LOCKED| pg[(PostgreSQL)]
    core2 <-->|transactions, SKIP LOCKED| pg
    jobs -->|publish outbox events| redis[(Redis pub/sub)]
    redis -->|fan-out| hub

Loading

Workers never call the API; all coordination goes through PostgreSQL. Design decisions are recorded in docs/adr.

Quick start (Docker)

Requirements: Docker with Compose v2, Python 3 (for the helper scripts).

python scripts/bootstrap_env.py          # writes .env with random secrets (never committed)
docker compose up -d --build --wait      # postgres, redis, api (:8080), 2 workers
python scripts/smoke_test.py             # end-to-end check (register -> run -> live events -> cancel)

If port 8080 is taken, set STEPWISE_API_PORT in .env and pass --base-url http://localhost:<port>. API docs: http://localhost:8080/swagger-ui.html (disable with STEPWISE_API_DOCS_ENABLED=false).

Failure-injection scenarios (the stack must be running):

python scripts/failure_injection.py crash        # SIGKILL the worker holding a task
python scripts/failure_injection.py graceful     # docker stop the worker holding a long task
python scripts/failure_injection.py api-restart  # restart the API mid-execution

Building and testing

Requirements: JDK 21 (the Maven wrapper downloads Maven), Docker for Testcontainers.

cd backend
./mvnw verify                                   # unit + integration tests (PostgreSQL via Testcontainers)
./mvnw test -Dtest='*Test' -Dsurefire.failIfNoSpecifiedTests=false   # unit tests only, no Docker

make env|build|test|test-unit|up|down|logs|smoke|clean wrap the same commands.

Configuration

All settings are environment variables; .env.example documents them. Required secrets (STEPWISE_DB_PASSWORD, STEPWISE_REDIS_PASSWORD, STEPWISE_JWT_SECRET ≥ 32 bytes) have no defaults, and the API refuses to start without a valid JWT secret. HTTP tasks are disabled until STEPWISE_HTTP_ALLOWED_HOSTS is set; the API and workers must use the same value.

API overview

Area	Endpoints
Auth	POST /api/auth/register, /login, /refresh, /logout, /switch-organization, GET /api/auth/me
Workflows	GET/POST /api/workflows, POST /api/workflows/validate, GET/PUT/DELETE /api/workflows/{id}, POST /{id}/versions, GET /{id}/versions[/{v}], POST /{id}/activate, /deactivate, POST /{id}/executions
Executions	GET /api/executions, GET /{id}, GET /{id}/events, GET /{id}/tasks/{key}/attempts, POST /{id}/cancel, /retry, /rerun
Live	GET /api/events/stream[?executionId=] (SSE, bearer auth, Last-Event-ID)
Schedules	GET/POST /api/schedules, GET/PUT/DELETE /{id}, POST /{id}/pause, /resume, GET /preview
Operations	GET /api/stats/overview, /workers, /analytics
Admin (ADMIN role)	GET /api/admin/dead-letters, POST /dead-letters/{task}/retry, /dismiss, GET /stuck-tasks, POST /tasks/{task}/release, POST /recovery/expired-leases, GET /audit-logs; PUT /api/org/settings, GET/POST /api/org/members

Guarantees and limitations

• Tasks execute at least once. A worker can perform an external side effect and crash before recording it; the task then runs again after its lease expires. HTTP tasks send Idempotency-Key: <executionId>:<taskKey> so receivers can de-duplicate. Exactly-once is not promised.
• Recovery after a worker crash takes up to one lease duration (30 s by default) plus the retry backoff.
• Live events are at-least-once and can arrive out of order after outbox retries; clients de-duplicate by id. Running more than one API instance requires Redis (STEPWISE_REDIS_ENABLED=true); without it, live events would be split between instances.
• Single-region, single-database design; see the ADRs and the status document for the full list.