Each milestone is a set of small pull requests into main. A milestone is done when its
checklist is ticked, CI is green and status.md is updated. Target dates are plans,
not deadlines; GSoC and upstream pull requests come first.
Done for the whole project means: a live demo, a README with setup steps and screenshots, tests in CI, and seed data.
- Flask app factory, config per environment (
dev,test,prod), health endpoint - SQLAlchemy 2 + Alembic against MariaDB 11.4 in Docker Compose (MariaDB in tests too, never SQLite)
- Redis and one RQ worker in Compose; a test job that runs end to end
- pytest with coverage, Ruff, mypy (strict)
- GitHub Actions:
backendjob, CodeQL - Add
backend,frontendandcodeqlas required checks in the ruleset - Extra: accounts (Argon2id), CSRF, rate limits, security headers, list CRUD, OpenAPI contract
- Extra: prerendered landing site with SEO (meta, JSON-LD, sitemap, robots, llms.txt)
- Dependabot for Actions, Python, npm and Docker
- Wikimedia client: one
User-Agentwith contact details,maxlag, retries with backoff, continuation handled, responses recorded for tests - Pasted list: normalise titles, resolve redirects, report missing pages
- Category: members in namespace 0, subcategories up to a depth limit, loop-safe
- Wikidata: SPARQL query, items mapped to articles on the chosen wiki
- Top by pageviews: top N for a month from the Wikimedia REST API
-
POST /api/v1/lists,GET /api/v1/lists/<id>, validation and one error format - Build pipeline:
POST /lists/<id>/build, versions, articles; builders plug in viaapp/core/builders.py(core-contract.md) - Builder logic in plain Python with 100% branch coverage (CI gate)
- Every build is an RQ job; list status
queued,building,ready,failed - Retries with
Retryfor network errors only; a failed build keeps the last good version - Email when a build fails (one per list per day at most)
- Monthly rebuilds (hourly dispatcher,
schedulerservice) - Each version records the articles added and removed since the previous ready version
- Large lists (100k titles) written in batches without loading everything in memory (tested: build peaks near 2 MiB, TSV export near 3 MiB)
- Combinations of up to 10 lists: API, sources table, build job, form in
/app(core-contract.md) - Union, intersection and difference in
app/core/combine.py - A combination rebuilds when one of its source lists gets a new version
- Export TSV (title, page ID, revision ID), streamed, with attribution lines
- Export a static HTML bundle (Parsoid HTML, local links rewritten, attribution page)
- Downloads are streamed and expire after 7 days
- React + TypeScript + Tailwind app: list overview, builder forms, combination editor
- Build progress (polls while a build runs), version history, articles per version
- Diffs between versions, download buttons
- Generated API types from the OpenAPI schema, checked in CI
- Accessible forms (labels, keyboard, contrast), works on a phone
-
frontendCI job added as a required check
- Stretch: ZIM export with
python-libzimfor small lists (articles, main page, metadata, full-text index) - Seed data: three example lists (a category, a Wikidata query, a combination):
python -m app.seed <email> - README setup steps and screenshots
- Deploy at
offlineshelf.ahmershah.dev; tagv1.0.0
- End-to-end tests with Playwright: sign up, every builder, combinations, exports, delete
- Smoke tests against a fresh Docker Compose stack on every release tag
- Stress test: 15 to 20 simulated users in Docker building, combining and exporting at once, watching queue depth, build times, memory and Wikimedia request rates
- 100% line and branch coverage for the whole backend and frontend (today:
app/core100%, backend about 97%)