Repository navigation
Check validation plots against their committed data in every run - #25
Merged
Merged
Conversation
Tests for the data summary a validation plot carries (counts and ranges of lines, scatter points and images, the comparison with a relative tolerance on the scale of each range, and its round trip through SVG and AVIF files) and for the fixture without --validation-plots: it renders both themes and applies the size cap, compares the summary with the committed pair, fails on a missing or stale pair, warns on a pair without a summary, and fails a test that takes the fixture but makes no plot. The two existing tests that ran the fixture without the option now expect these checks.
Without --validation-plots the validation_plot fixture now calls the factory, renders both themes in memory and applies the size cap, so a plot that raises or grows above 60000 B fails wherever its test runs. It compares a data summary of the figure with the committed light file and fails when that file is missing or stale; it writes nothing. A test that takes the fixture and passes without calling it fails at teardown. The summary holds, per axes, the point count and the min, max and mean of each coordinate of every line, point set and image. save() stores it in the SVG description and the AVIF XMP packet, so it does not depend on the renderer. Counts must match and values must agree within SUMMARY_RTOL times the scale of the committed coordinate. A committed file without a summary passes with a warning.
The summary must cover categorical and datetime axes, meshes, fills, bars, error bars and filled contours, find a reversed series, compare an offset coordinate on the scale of its range, leave out masked scatter points, keep a large mean finite, name its format and the changed quantity, and read back a test id that holds the summary label. The fixture must check the dark file as well, refuse a name that differs from the committed file only in case, fail on an unreadable summary, warn on a summary of another format, find the pair under the rootdir from a subfolder, give the absolute path of a missing file, and let a test that expects an argument error or an xpass end without a teardown error.
The summary reads lines through their converted data, so categorical and date axes work, and covers the vertices of fills, error bars and contours, mesh coordinates, colour values and patches as well as scatter points and images. Each coordinate gets a ramp-weighted mean beside min, max and mean, which finds a reversed or reordered series; masked scatter points are left out and the means cannot overflow. The summary carries a format number. A value now passes within SUMMARY_RTOL of the range of the committed coordinate, or of a tenth of its largest magnitude when that is larger, so offset data are compared on their spread while round-off near 1e-10 of the values still passes. The fixture checks the light and the dark file, by their exact names, and gives the absolute path of a missing file. An unreadable summary fails with a hint to make the pair again, a summary of another format warns, and a stale message names the quantity that changed. read_summary takes the text after the last summary label. A test that expects an argument error from the fixture, or an xpass, no longer fails at teardown.
The summary must find a changed inset, hexbin, quiver, image extent and mesh coordinates, read the offsets of a collection whose paths are one shape, leave out arrow patches, compare a range too large for a float, and let an image without data, empty rows and a summary read back compare equal to themselves. Rows with a wrong number of statistics differ, an image shape change is named a shape, and render refuses a factory whose light and dark figures hold different data. The fixture must report a summary of the wrong shape and a file it cannot read as unreadable, and the stale message counts the other differences.
The summary walks inset axes, reads the offsets of every collection that has them (scatter points, hexbins, quivers, ellipses), the quiver U and V, and each image extent, and leaves out arrow patches. An image without data gives a two-entry shape, an image shape change is named a shape, and statistics are compared on halved values so a range near the largest float cannot make the tolerance infinite. render refuses a factory whose light and dark figures hold different data, after the size check. The fixture reports a committed file it cannot read as unreadable. README and CHANGELOG list what the summary leaves out, its size and its dependence on the matplotlib version.
The summary must find changed barbs, a moved single point, moved quiver positions, a masked quiver arrow, a flipped image origin with a fixed extent, a change in a nested inset and in parasite axes; it must take a single point at the origin as one point, pass a marker change at the origin and an added secondary axis, and leave out 3D axes. The mesh test asserts that a difference exists, the empty-artist cases share one test, and the unreadable-file test skips where the file stays readable.
A collection's offsets count as data unless they are one (0, 0) in an identity transform, so a single point at the origin is that point and not its marker. The summary keeps the quiver mask, adds the u and v of barbs and the image origin, walks parasite axes, and leaves out secondary axes, which hold no data, and 3D axes, whose collections depend on the view and on whether the figure was drawn. save documents the refusal of theme-dependent data. The README gives the measured summary size, names what is not compared, and advises a floor above the noise for round-off coordinates.
The summary reads contour sets and other collections in the structure matplotlib has had since 3.8, and the tests pass on 3.8.0, 3.8.4, 3.9.4, 3.10.7 and 3.11.2, so 3.8 is the lowest supported version; Python 3.9 follows from it and from the Pillow the extras need. CI runs a third job on matplotlib 3.8.0 with numpy below 2. Secondary axes are recognized by their type name instead of a private matplotlib import.
A change on a log axis must be found and a scale change reported, with log values compared in decades. Labelled contours must not depend on figure size, font size or dpi, while a change of the field or of the levels is found. Category names are compared and survive XML characters, a summary label inside the test id is not the summary, save renames a pair committed in other case on a case-insensitive file system and leaves a second file alone on a case-sensitive one, the check names a pair committed in other case, and an empty, cut, LFS-pointer or garbage file is refused.
On a log axis the statistics of a coordinate are taken of its log10, and each axes records its scales and category names, so a scale change or a reordered category list is a difference. A labelled line contour set is read from its contour generator, so label gaps, figure size, font size and dpi do not change it. read_summary refuses a file that is not a whole SVG or AVIF, finds the summary label at the start of the description or after "; ", and unescapes XML entities. save renames a committed file whose name differs only in case before the new file replaces it, on a case-insensitive file system, and the fixture names such a file and asks for a git mv.
The README lists what the fixture compares and what it does not: log axes in decades, scales and category names, labelled contours from their generator, the floor on a date axis, the order of scatter points and bars, broken and case-mismatched files, the warning under filterwarnings = error, and pairs made with the matplotlib the repository's CI installs; the measured summary size is 90 to 170 B per line. CI adds a Python 3.9 job on matplotlib 3.8.0, and the test extra needs pytest 7 or newer, which the plugin's stash keys need.
…t id A file cut 200 bytes before its end keeps a header Pillow can open, so only a full decode refuses it; a description whose only summary label sits inside the test id carries no summary.
Labelled tricontours and labelled contour sets without a contour generator must give the summary of their unlabelled form; colour values under a log norm must be compared in decades with the norm kind; a log coordinate near 1 must pass round-off and fail a change of 4e-6 decades.
…tricontours Labelled line contours are read with create_contour, which contour and tricontour generators both have; a labelled set without one is read from its paths. Colour values and images under a log norm are summarized in log10, and the summary keeps the norm scale of each value row and image, so a norm change is a difference. A coordinate in log10 has a floor of 0.1 decade in place of a tenth of its magnitude, so round-off near 1 passes and the tolerance is the same at any magnitude.
A 0.1 decade floor on the range gives a tolerance of 1e-6 x 0.1 = 1e-7 decade. A test pins the edge: 9e-8 decade passes and 1.1e-7 decade fails.
This was referenced Oct 24, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
validation_plotfixture now checks every validation plot in every test run, not only when the plots are written. Without--validation-plotsit renders both themes in memory, applies the 60 000 B cap and compares a data summary of the figure with the summary stored in the committed light and dark files, so a plot whose data drifts, grows past the cap or whose committed file is missing fails the test that makes it.validation.savestores that summary in each file it writes.Changes
validation.summarizebuilds a summary per axes: point counts and min, max, mean and a ramp-weighted mean of every line, collection, colour value, patch and image, in log10 on a log axis and under a log norm; labelled contours are read from their contour generator, so label gaps, figure size and dpi do not count.validation.compare_summariescompares two summaries withinSUMMARY_RTOL(1e-6) of the range of each coordinate, with a floor of a tenth of its magnitude, or 0.1 decade in log10.validation.renderrefuses a factory whose light and dark figures hold different data.Validation
python -m pytest testsinfigures/proteus-mplwithPYTHONPATH=src, matplotlib 3.11.0: 259 passed, 1 skipped.python -m coverage run --source=src/proteus_mpl -m pytest tests: 98 % in total,validation.py100 %; the changedsrclines of this PR are 218 of 218 covered.srcas the head): VULCAN on Linux, Python 3.11 to 3.13, https://github.com/FormingWorlds/VULCAN/actions/runs/40059633256, and aragog on macOS and Linux, Python 3.12 and 3.13, https://github.com/FormingWorlds/aragog/actions/runs/40059635451.Physics impact
None: the package draws and checks plots and changes no model output.
Breaking changes
proteus-mpl>=1.4.0gets the check on its next run after a release: each test that callsvalidation_plotnow calls its factory and needs both committed files; a map plot needs Pillow with AVIF support in that job. Pairs made with 1.4.0 carry no summary and pass with a warning until they are made again.Config changes
None.