Skip to content

perf: eliminate O(docs) disk reads in check_all_figures_have_license - #67

Merged
lkdmc merged 1 commit into
mainfrom
fix/performance
Mar 31, 2026
Merged

perf: eliminate O(docs) disk reads in check_all_figures_have_license#67
lkdmc merged 1 commit into
mainfrom
fix/performance

Conversation

@lkdmc

@lkdmc lkdmc commented Mar 31, 2026

Copy link
Copy Markdown
Collaborator

Summary

check_all_figures_have_license previously called env.get_doctree(docname) for every document in env.found_docs, which reads and unpickles each doctree file from disk on every build — including incremental builds where only a few documents changed. For large projects this produced significant unnecessary I/O proportional to the total number of documents.

Changes

MetadataFigure.run() — record each processed figure's image_uri and license value into env.metadata_figure_license_index[docname] as a plain dict. This is pure in-memory bookkeeping with no extra I/O.

check_all_figures_have_license() — iterate the index directly; fall back to env.get_doctree() only for documents not present in the index (unchanged docs from a previous incremental build, or figures emitted by third-party extensions).

clear_page_defaults() — also clear env.metadata_figure_license_index entries for documents that are about to be rebuilt, so stale entries from a previous run do not survive into the new build.

merge_figure_license_index() — new env-merge-info event handler that merges the index from each parallel read worker back into the main env, keeping parallel_read_safe = True fully functional.

Complexity

Scenario Before After
Full build (N docs) O(N) disk reads O(0) disk reads (index used)
Incremental build (k changed docs) O(N) disk reads O(N-k) disk reads (only unchanged docs fall back)

In the common incremental case where most docs are unchanged, the number of disk reads drops from O(N) to O(N-k), approaching zero for small changesets.

Test plan

  • Enable summaries: true in metadata_figure_settings.license and run a full build; confirm missing/unrecognized license warnings still appear correctly.
  • Run an incremental build after changing one document; confirm the summary still covers all documents, not just the changed one.
  • Run a parallel build (sphinx-build -j auto) and confirm the summary is complete (index merge working).
  • Confirm that figures produced by non-MetadataFigure directives (standard .. figure:: not overridden) still appear in the summary via the doctree fallback.

https://claude.ai/code/session_01ApsmWPWsKBesMV1rq2VenA

Previously the license summary check re-read every doctree pickle from
disk (env.get_doctree) for all found_docs, even when the build had only
modified a handful of documents.  For large projects this caused
significant I/O on every incremental build.

New approach:
- MetadataFigure.run() records each figure's image_uri and license value
  in env.metadata_figure_license_index[docname] as directives are
  processed.  This is pure in-memory bookkeeping with no extra I/O.
- check_all_figures_have_license iterates the index directly; it falls
  back to env.get_doctree only for documents not present in the index
  (e.g. figures emitted by third-party extensions or unchanged docs from
  a previous incremental build whose index entry was cleared).
- clear_page_defaults now also clears the index for documents that are
  about to be rebuilt, preventing stale entries from surviving across
  incremental builds.
- A new merge_figure_license_index handler wired to env-merge-info
  merges index data from parallel read workers back into the main env so
  that parallel_read_safe=True builds are handled correctly.

https://claude.ai/code/session_01ApsmWPWsKBesMV1rq2VenA
@lkdmc
lkdmc requested a review from douden as a code owner March 31, 2026 16:11
@lkdmc
lkdmc merged commit e72e957 into main Mar 31, 2026
3 checks passed
@lkdmc
lkdmc deleted the fix/performance branch March 31, 2026 18:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants