Skip to content

Build the fixture database in one transaction - #2971

Open
quotentiroler wants to merge 1 commit into
simonw:mainfrom
quotentiroler:fixture-transaction
Open

quotentiroler wants to merge 1 commit into
simonw:mainfrom
quotentiroler:fixture-transaction

Conversation

@quotentiroler

Copy link
Copy Markdown

Refs #2970

executescript() does not open a transaction, so the ~1,440 INSERTs in TABLES each committed separately.

Wrapping the script in BEGIN; / COMMIT; takes populate_fixture_database() from 11,453 ms to 19 ms here (single process, best of three, Windows 11 / Python 3.14), producing an identical database: a sha256 over every table's full contents matches.

This runs on every make_app_client(), so it is the bulk of the suite's setup cost. pytest tests/test_html.py tests/test_queries.py tests/test_stored_queries.py went from 470.6s to 28.7s with the same pass/fail counts.

@quotentiroler

Copy link
Copy Markdown
Author

@simonw CI on this PR never ran: every fork PR opened Sep 30 to Oct 1 failed instantly with zero jobs ("workflow file issue"), the same for #2973 and several others. I've rebased onto current main to trigger it again.

The speedup holds on Linux too. In python:3.13-slim, building the fixture database takes a median of 21.8 s on main and 65 ms with this change, with identical contents. make_app_client rebuilds that file on every call, so it adds up: pytest -n auto tests/test_fixtures.py tests/test_html.py takes 398 s on main and 12 s on this branch, both with 150 passed, 1 xpassed. (Docker on WSL2, whose disk syncs are slower than a GitHub runner's, so expect a smaller but still large gain in CI.)

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant