PaperCrew is a local-first, evidence-grounded multi-agent system that turns a scientific paper into a tested Python repository. It preserves the Paper2Code/PaperCoder research workflow while making orchestration durable, typed, inspectable, and safe enough to present as an engineering portfolio project.
The Lite release contains the complete workflow in one process: document ingestion, evidence extraction, scoped planning, coding through validated patches, Docker-only verification, bounded repair, independent fidelity/code reviews, deterministic evaluation, an optional model judge, and immutable final reports.
Requirements: Python 3.12 or 3.13, uv, and Docker for generated-code execution.
git clone https://github.com/cuizhaonuo/PaperCrew.git
cd PaperCrew
uv sync --locked
uv run papercrew doctor
uv run pytest tests/e2e/test_complete_fake_paths.py::test_direct_fake_model_run_is_reproducible_and_reported -qThe smoke test uses deterministic fake model responses, calls no external API, and still traverses the complete Root Graph through final reporting. A minimal durable CLI run also needs no key:
uv run papercrew run --require-approval
uv run papercrew approve <run-id>
uv run papercrew artifacts <run-id>
uv run papercrew export <run-id> --destination exported-runTo run the fixed real-model evaluation, configure PAPERCREW_OPENAI_API_KEY and optionally
PAPERCREW_MODEL, then run:
uv run papercrew run tests/fixtures/papers/tiny_algorithm/paper.md --profile smokeThe profile fixes scope, budget, rubric, and required outputs in evals/smoke.json. An actual DeepSeek V4 Flash run is published as a secret-filtered artifact manifest and final Markdown report.
flowchart LR
CLI[Typer CLI] --> APP[Application services]
APP --> GRAPH[LangGraph Root StateGraph]
GRAPH --> I[Ingestion]
I --> P[Planning Crew]
P -->|interrupt / resume| H[Human approval]
P --> C[Coding Crew]
C --> V[Docker verification]
V -->|bounded failure| D[Debugger repair]
V --> R[Fidelity + Code review]
R --> E[Deterministic evaluation]
E --> J[Optional model judge]
J --> F[Immutable final report]
GRAPH --> SQ[(SQLite checkpoints + runs)]
APP --> AS[(Content-addressed artifacts)]
The graph checkpoint contains only compact typed state and Artifact references. Large paper, prompt, patch, log, repository index, review, evaluation, and report payloads live in the immutable Artifact store. Non-serializable dependencies enter nodes through LangGraph runtime context.
Key design choices:
- StateGraph makes routing, terminal paths, interrupts, and bounded loops explicit and testable.
- SQLite checkpoints allow approval to resume in a different process without replaying prior model calls.
- Every reproduction claim is tied to an Evidence ID and exact source location; downstream agents may not invent unknown evidence.
- Agents propose typed unified-diff Patches. A validator and atomic applicator own canonical workspace mutation.
- Generated code executes only in a non-root, network-disabled, resource-limited Docker container; there is no host fallback.
- Deterministic verification overrides optimistic model-judge scores. Final reports disclose cost, tokens, latency, repairs, limitations, and Artifact provenance.
run create and execute a durable run
status query persisted state
watch follow state until terminal or approval wait
resume resume with an explicit decision
approve approve and resume a pending checkpoint
revise request a revision with feedback
reject reject a pending decision
cancel idempotently cancel a run
artifacts list public immutable artifacts
export export a secret-filtered manifest and public artifacts
doctor inspect local SQLite, Docker, and model capability
All commands emit stable JSON. Operator commands call application services rather than repositories directly. Internal model-call, task-context, and execution-log artifacts are excluded from normal listing/export; exported JSON receives an additional sensitive-key redaction pass.
Paper text and generated repositories are untrusted input. PaperCrew normalizes relative paths, rejects symlinks and special files, caps patch/file/context sizes, applies patches transactionally, passes commands as argv rather than through a shell, exposes only allowlisted environment variables, disables container networking by default, drops capabilities, uses a read-only root/workspace where possible, and enforces CPU, memory, PID, and wall-time limits. Model API credentials never enter the container or public export.
Docker is mandatory for generated-code verification. If it is unavailable, the run fails closed.
The fake-model E2E produces a generated src/model.py, a repository index, verification state,
separate review artifacts, deterministic metrics, and JSON/Markdown reports. Inspect it with:
uv run pytest tests/e2e/test_complete_fake_paths.py -q -s
uv run pytest tests/e2e/test_docker_paths.py -q -m dockerCurrent release benchmark (local CI fixture, Python 3.12):
| Scenario | External model | Docker | Expected terminal result |
|---|---|---|---|
| Direct full graph | Fake | No | succeeded + report |
| Approval after process restart | None | No | succeeded |
| Seeded syntax repair | Fake debugger | Yes | repaired within budget |
| Unrepairable syntax | None | Yes | failed, workspace unchanged |
| Fixed smoke profile | DeepSeek V4 Flash | Yes | partial report; deterministic gate passed, review requested coding |
This table describes acceptance fixtures, not scientific model-quality claims. Real model results must come from the manual workflow and its uploaded Artifact report. The checked-in real run used 13 model calls, 31,761 tokens, 68.988 seconds of model latency, cost approximately $0.004185, and produced three Patch Artifacts. These values describe one run, not a quality or price guarantee.
uv sync --locked
uv run ruff format --check .
uv run ruff check .
uv run pyright
uv run pytest -q -m "not docker and not real_model"
uv run pytest tests/integration/sandbox tests/e2e/test_docker_paths.py -q -m docker
uv run papercrew doctorThe real-model evaluation is intentionally manual because it incurs cost. Use the GitHub Actions
workflow Real model smoke evaluation; it uploads the secret-filtered immutable report even when a
run fails.
Lite deliberately defers HTTP/UI surfaces, PostgreSQL, queues/workers, object storage, vector search, Kubernetes, and concurrent agents mutating one canonical workspace. Paper2Code is a design and fixture reference only; it is not a runtime dependency.
PaperCrew is licensed under the MIT License.