Skip to content

Drop the query planner statistics tables - #1027

Merged
epompeii merged 1 commit into
develfrom
drop-planner-statistics
Sep 14, 2026
Merged

epompeii merged 1 commit into
develfrom
drop-planner-statistics

Conversation

@epompeii

Copy link
Copy Markdown
Member

What changed

  • New migration 2026-09-14-120000_drop_planner_statistics drops sqlite_stat1 and sqlite_stat4 (DROP TABLE IF EXISTS, since a database that never ran ANALYZE has neither and a build without ENABLE_STAT4 never creates the second). Its down.sql is a comment: the statistics are not restored, because re-running a limited ANALYZE would reintroduce the problem. The 2026-08-17 migration is left as is; applied migrations are history.
  • New test migrations_leave_no_planner_statistics_behind in bencher_schema asserts that no sqlite_stat% table exists after run_migrations.
  • One rule in AGENTS.md, next to the other database rules: migrations never run ANALYZE or PRAGMA optimize, and nothing creates the sqlite_stat tables; the query planner runs without statistics.
  • Changelog line under a new Pending v0.6.13 section.

Why

The 2026-08-17 migration ended with a limited ANALYZE. The sampled statistics overstate how many rows share a key on indexes whose leading rows cluster on one value. Trusting them, the planner scans the whole alert table for a status filtered alerts list and walks every head version for a perf query. A full ANALYZE does not help: it keeps the alert scan and adds scans on other query shapes. Without statistics the planner falls back on its structural heuristics, which pick the indexes, as it did before the statistics were introduced.

How it was tested

  • Red then green: the new test was run on devel before the migration existed and failed with found ["sqlite_stat1", "sqlite_stat4"], since ANALYZE creates both tables even on an empty database. With the migration it passes.
  • Round trip: lib/api_projects/tests/metric_migration.rs reverts and reapplies the migrations, so the comment-only down.sql and the re-drop after the re-run ANALYZE are both exercised.
  • cargo nextest run --all-features --profile ci, cargo test --doc --all-features, cargo clippy --no-deps --all-targets --all-features -- -Dwarnings, cargo check --no-default-features, cargo fmt --all -- --check.

@github-actions

github-actions Bot commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

🐰 Bencher Report

ProjectBencher
Branchdrop-planner-statistics
Testbedintel-v1
Click to view all benchmark results
BenchmarkLatencyBenchmark Result
microseconds (µs)
(Result Δ%)
Upper Boundary
microseconds (µs)
(Limit %)
Adapter::Json📈 view plot
🚷 view threshold
5.23 µs
(+9.64%)Baseline: 4.77 µs
5.65 µs
(92.67%)
Adapter::Magic (JSON)📈 view plot
🚷 view threshold
5.03 µs
(+8.66%)Baseline: 4.63 µs
5.40 µs
(93.10%)
Adapter::Magic (Rust)📈 view plot
🚷 view threshold
28.20 µs
(+7.91%)Baseline: 26.13 µs
28.56 µs
(98.75%)
Adapter::Rust📈 view plot
🚷 view threshold
4.68 µs
(+25.70%)Baseline: 3.72 µs
5.47 µs
(85.52%)
Adapter::RustBench📈 view plot
🚷 view threshold
4.68 µs
(+25.82%)Baseline: 3.72 µs
5.46 µs
(85.71%)
🐰 View full continuous benchmarking report in Bencher

@epompeii
epompeii marked this pull request as ready for review September 14, 2026 04:10
@epompeii
epompeii marked this pull request as draft September 14, 2026 04:15
The 2026-08-17 migration ended with a limited ANALYZE. Its sampled
statistics overstate how many rows share a key on indexes whose leading
rows cluster on one value, and trusting them the planner scans the whole
alert table for a status filtered alerts list and walks every head
version for a perf query. A full ANALYZE keeps the alert scan and adds
scans elsewhere. Without statistics the planner falls back on its
structural heuristics, which pick the indexes.

A new migration drops sqlite_stat1 and sqlite_stat4, with IF EXISTS
because a database that never ran ANALYZE has neither and a build
without ENABLE_STAT4 never creates the second. Its down is a comment:
the statistics are not restored, since re-running a limited ANALYZE
would reintroduce the problem. The 2026-08-17 migration is left as is.

A bencher_schema test asserts that no sqlite_stat table exists after
run_migrations. It failed before the new migration, since ANALYZE
creates both tables even on an empty database, and passes with it.
@epompeii
epompeii force-pushed the drop-planner-statistics branch from 9b8072b to 4cacc3e Compare September 14, 2026 04:20
@epompeii
epompeii marked this pull request as ready for review September 14, 2026 04:37
@epompeii
epompeii merged commit 4329f73 into devel Sep 14, 2026
49 checks passed
@epompeii
epompeii deleted the drop-planner-statistics branch September 14, 2026 04:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant