Check validation plots against their committed data in every run

.github/workflows/proteus-mpl-tests.yml

@@ -20,13 +20,21 @@ jobs:
     strategy:
       matrix:
         python-version: ["3.11", "3.13"]
+        matplotlib: [""]
+        include:
+          - python-version: "3.11"
+            matplotlib: "==3.8.0"
+            numpy: "<2"
+          - python-version: "3.9"
+            matplotlib: "==3.8.0"
+            numpy: "<2"
     steps:
       - uses: actions/checkout@v4
       - uses: actions/setup-python@v5
         with:
           python-version: ${{ matrix.python-version }}
       - name: Install the package with its test extra
-        run: python -m pip install "./figures/proteus-mpl[test]" coverage
+        run: python -m pip install "./figures/proteus-mpl[test]" coverage "matplotlib${{ matrix.matplotlib }}" "numpy${{ matrix.numpy }}"
       - name: Check that Pillow can write AVIF
         run: python -c "from PIL import features; assert features.check('avif'), 'Pillow without AVIF'"
       - name: Run the tests with coverage

CHANGELOG.md

@@ -10,6 +10,12 @@ it.
 
 ## [Unreleased]
 
+### Changed
+- proteus-mpl needs matplotlib 3.8 or newer (was 3.5) and Python 3.9 or newer (was 3.8); the tests run in CI on the newest matplotlib and on matplotlib 3.8.0 with Python 3.9 and 3.11. The test extra needs pytest 7 or newer.
+- Without `--validation-plots` the `validation_plot` fixture now calls the factory, renders both themes in memory and applies the 60 000 B cap, so a plot that raises or grows above the cap fails in every run that runs its test. It compares a data summary of the rendered figure with both committed files in `docs/assets/validation` and fails when a file is missing, is committed under a name that differs only in case, is not a whole SVG or AVIF, or its summary cannot be read or differs; it writes nothing. With the option, on a case-insensitive file system, a committed file whose name differs only in case is renamed. A committed file without a summary of the current format passes with a warning. A test that takes the fixture and passes without calling it fails at teardown; a test that expects an argument error from the fixture, or an xpass, does not.
+- The summary (format 1) holds, per axes, inset axes and parasite axes, the point count and the min, max, mean and a ramp-weighted mean of each coordinate, in axis units and in log10 on a log axis, of every line, every collection and its colour values (in log10 under a log norm) and vectors, the patches except annotation arrows, and every image with its origin and extent. Counts, axis and norm scales and category names must match; labelled line contours are read from their contour generator; a value must agree within `validation.SUMMARY_RTOL` (1e-6) times the range of the committed coordinate, or a tenth of its largest magnitude when that is larger (0.1 decade in log10). 3D and secondary axes, axis labels, limits, text, colours, artist transforms, annotation arrows and the shapes of offset markers are not compared; the README lists the limits. `validation.render` refuses a factory whose light and dark figures hold different data.
+- `validation.save` stores the summary in the SVG description and the AVIF XMP packet, after the test id and the command. `validation.render`, `validation.summarize`, `validation.read_summary` and `validation.compare_summaries` are public.
+
 ## [1.4.0] - 2026-10-23
 
 ### Added

figures/proteus-mpl/README.md

@@ -12,6 +12,8 @@ pip install proteus-mpl
 From a checkout of the repository: `pip install figures/proteus-mpl` (or cd
 into the folder and `pip install .`).
 
+The package needs Python 3.9 and matplotlib 3.8 or newer; CI runs the tests on the newest matplotlib and on matplotlib 3.8.0 with Python 3.9 and 3.11. The pytest plugin needs pytest 7 or newer.
+
 ## Use
 
 ```python
@@ -87,7 +89,18 @@ def test_melt_fraction(validation_plot):
     validation_plot(lambda: plot_case(t, model), "melt_fraction")
 ```
 
-Without the option the fixture checks its arguments and writes nothing, so the test only asserts. `pytest -m reference_pinned --validation-plots` writes every plot to `docs/assets/validation` under the pytest rootdir; `--validation-plots=DIR` writes to another folder, relative to the rootdir. A bare `--validation-plots` followed by a word, which it would read as its folder, is refused, also from `addopts` or an `@argfile`, so write `--validation-plots=DIR` (always in `addopts` and `PYTEST_ADDOPTS`) or put the option after the test paths. A plot name used twice for one kind in a session is refused, with or without the option, so a parametrized test needs one name per parameter, such as `f"case_{param}"`; a rerun of the same test may use its names again. The `reference_pinned` marker belongs to the repository that runs the tests, which registers it in its pytest configuration. Under pytest-xdist each worker keeps its own list of names, so a name repeated in tests that run on different workers is not caught. The stamped command names a folder given with `--validation-plots=DIR`.
+Without the option the fixture checks the plot against the committed pair:
+
+- It calls the factory, renders both themes in memory, applies the size cap and writes nothing, so each run of the test pays for two renders (a smooth 200 x 200 map takes about 0.6 s on an arm64 Mac, most of it in rasterizing and encoding the two files).
+- It compares a data summary of the figure with the summary stored in both committed files in `docs/assets/validation`. The summary holds, per axes, inset axes and parasite axes, the axis scales, the category names of categorical axes, and the point count and the min, max, mean and a ramp-weighted mean of each coordinate of every line, every collection (offsets of scatter points, hexbins, quivers and barbs, mesh coordinates, vertices of fills, error bars and contours; labelled line contours from their contour generator, so label gaps, figure size, font size and dpi do not count; a labelled contour set without one, such as one built from segments, is read from its drawn paths, where inline label gaps count) and its colour values (in log10 under a log norm, and the norm scale of each is kept; and the vectors of quivers and barbs, masks kept), the corners of all patches except annotation arrows (bars, histograms, `ax.arrow`), and every image with its origin and extent.
+- Coordinates are in axis units; on a log axis the statistics are of the log10 of the values (values at or below zero are left out), and symlog and logit axes stay linear. The ramp weight runs from -1 at the first point to 1 at the last, so a reversed series is found, and the order of scatter points and bars counts; a change inside a series that leaves all four statistics within the tolerance is not found.
+- A value passes within `validation.SUMMARY_RTOL` (1e-6) of the range of the committed coordinate, or of a tenth of its largest magnitude when that is larger (on a log axis and under a log norm, of the range of the log10 values or 0.1 decade when that is larger, so the tolerance is at least 1e-7 decade at any magnitude: 1 and 1 + 1e-12 agree, 1 and 1 + 1e-5, 4.3e-6 decade apart, differ); counts, axis and norm scales and category names must match. Large offsets set the floor: on a date axis near 2024 it is about 170 s. A coordinate that is round-off noise, such as a residual near 1e-16, differs between machines on its own scale; plot it with a floor well above the noise, such as `np.maximum(residual, 1e-12)` for a residual near 1e-16 on a log axis, or leave it out.
+- Not compared: 3D and secondary axes, the base of a log scale, axis labels, limits, text, colours, artist transforms, annotation arrows, the shapes of offset markers and the interior cell edges of `pcolorfast` and `NonUniformImage` images; a colorbar is compared as an axes, so a colormap with another number of colours changes it.
+- Which artists a plot type makes depends on the matplotlib version, so make the pairs with the matplotlib the repository's CI installs, and make them again after an upgrade. The summary adds about 90 to 170 B per line and 90 B per axes to each file, which counts toward the size cap.
+- A factory whose light and dark figures hold different data is refused. A missing committed file, a file committed under a name that differs only in case (the message asks for a `git mv`), a file that is not a whole SVG or AVIF (empty, cut, an LFS pointer), an unreadable summary or a summary that differs fails the test with a hint to make the pair again. A committed file without a summary of the current format passes with a warning, which fails the test in a repository that sets `filterwarnings = error`.
+- A test that takes `validation_plot` and passes without calling it fails at teardown.
+
+With the option the fixture writes and does not compare; on a case-insensitive file system it renames a committed file whose name differs only in case. `pytest -m reference_pinned --validation-plots` writes every plot to `docs/assets/validation` under the pytest rootdir; `--validation-plots=DIR` writes to another folder, relative to the rootdir. A bare `--validation-plots` followed by a word, which it would read as its folder, is refused, also from `addopts` or an `@argfile`, so write `--validation-plots=DIR` (always in `addopts` and `PYTEST_ADDOPTS`) or put the option after the test paths. A plot name used twice for one kind in a session is refused, with or without the option, so a parametrized test needs one name per parameter, such as `f"case_{param}"`; a rerun of the same test may use its names again. The `reference_pinned` marker belongs to the repository that runs the tests, which registers it in its pytest configuration. Under pytest-xdist each worker keeps its own list of names, so a name repeated in tests that run on different workers is not caught. The stamped command names a folder given with `--validation-plots=DIR`.
 
 ## Conventions
 

figures/proteus-mpl/pyproject.toml

@@ -7,10 +7,10 @@ name = "proteus-mpl"
 version = "1.4.0"
 description = "PROTEUS Thermocline matplotlib theme: brand color cycles, phase colormaps, light + dark variants."
 readme = "README.md"
-requires-python = ">=3.8"
+requires-python = ">=3.9"
 license = { text = "Apache-2.0" }
 authors = [{ name = "PROTEUS collaboration", email = "dev@proteus-framework.org" }]
-dependencies = ["matplotlib>=3.5"]
+dependencies = ["matplotlib>=3.8"]
 keywords = ["matplotlib", "style", "theme", "PROTEUS", "planetary-science"]
 classifiers = [
     "Framework :: Matplotlib",
@@ -27,7 +27,7 @@ Changelog = "https://github.com/FormingWorlds/proteus-visual-language/blob/main/
 
 [project.optional-dependencies]
 validation = ["pillow>=11.3"]
-test = ["pytest", "pillow>=11.3", "hypothesis"]
+test = ["pytest>=7.0", "pillow>=11.3", "hypothesis"]
 
 [project.entry-points.pytest11]
 "proteus_mpl.pytest_plugin" = "proteus_mpl.pytest_plugin"

figures/proteus-mpl/src/proteus_mpl/pytest_plugin.py

@@ -1,21 +1,33 @@
 """pytest plugin: the `validation_plot` fixture and the ``--validation-plots`` option.
 
-Without the option the fixture checks its arguments and writes nothing, so a
-test asserts its tolerance without writing files. ``pytest -m reference_pinned
---validation-plots`` makes every plot again in ``docs/assets/validation`` under
-the pytest rootdir; ``--validation-plots=DIR`` writes to another folder.
+Without the option the fixture renders both themes in memory, applies the size
+cap and compares the data summary with both files of the committed pair in
+``docs/assets/validation`` under the pytest rootdir; it writes nothing, and a
+missing or stale file fails the test. ``pytest -m reference_pinned
+--validation-plots`` makes every plot again in that folder;
+``--validation-plots=DIR`` writes to another folder and compares nothing.
 """
 
 import os
 import re
 import shlex
 import sys
+import warnings
 from pathlib import Path
 
 import pytest
 
 DEFAULT_DIR = "docs/assets/validation"
 _SEEN = pytest.StashKey[dict]()
+_PASSED = pytest.StashKey[bool]()
+
+
+@pytest.hookimpl(hookwrapper=True)
+def pytest_runtest_makereport(item, call):
+    """Keep whether the call phase passed, not as an xpass, for the fixture's teardown."""
+    report = (yield).get_result()
+    if report.when == "call":
+        item.stash[_PASSED] = report.passed and not hasattr(report, "wasxfail")
 
 
 def pytest_addoption(parser):
@@ -112,14 +124,18 @@ def pytest_configure(config):
 
 @pytest.fixture
 def validation_plot(request):
-    """Return ``plot(fig_factory, name, kind="line")``, which saves a validation plot pair.
-
-    The call returns the written paths, or an empty list without
-    ``--validation-plots``; `fig_factory` is then not called. A name used twice
-    for one kind in a session raises ValueError, with or without the option; a
-    rerun of the same test may use its names again.
+    """Return ``plot(fig_factory, name, kind="line")``, which makes a validation plot pair.
+
+    With ``--validation-plots`` the call writes the pair and returns the written
+    paths. Without it the call renders both themes in memory (the size cap
+    applies), compares the data summary with both committed files and returns an
+    empty list; a missing or stale committed file fails the test, a committed file
+    without a summary of this version gives a warning. A name used twice for one kind in a
+    session raises ValueError, with or without the option; a rerun of the same
+    test may use its names again. A test that takes the fixture and passes
+    without calling it fails at teardown.
     """
-    from .validation import check_args, save
+    from .validation import check_args, render, save
 
     config = request.config
     outdir = config.getoption("validation_plots")
@@ -130,8 +146,10 @@ def validation_plot(request):
     seen = config.stash[_SEEN]
     for key in [key for key, owner in seen.items() if owner == nodeid]:
         del seen[key]
+    calls = []
 
     def plot(fig_factory, name, kind="line"):
+        calls.append(name)
         check_args(name, kind)
         key = (name.lower(), kind)
         if key in seen:
@@ -140,15 +158,62 @@ def plot(fig_factory, name, kind="line"):
                 "give each plot its own name"
             )
         seen[key] = nodeid
-        if outdir is None:
-            return []
-        return save(
-            fig_factory,
-            name,
-            Path(config.rootpath, outdir),
-            kind,
-            test_id=nodeid,
-            command=f"pytest {shlex.quote(nodeid)} {option}",
+        if outdir is not None:
+            return save(
+                fig_factory,
+                name,
+                Path(config.rootpath, outdir),
+                kind,
+                test_id=nodeid,
+                command=f"pytest {shlex.quote(nodeid)} {option}",
+            )
+        files, summary = render(fig_factory, name, kind)
+        for fname, _ in files:
+            _check_committed(name, Path(config.rootpath, DEFAULT_DIR, fname), summary)
+        return []
+
+    yield plot
+    if not calls and request.node.stash.get(_PASSED, False):
+        pytest.fail(f"{nodeid} took validation_plot but made no plot", pytrace=False)
+
+
+def _check_committed(name, path, summary):
+    """Fail when the committed file `path` is missing, unreadable or stale for `summary`."""
+    from .validation import compare_summaries, read_summary
+
+    again = "make it again with --validation-plots"
+    others = [n for n in os.listdir(path.parent) if n.lower() == path.name.lower()] if path.parent.is_dir() else []
+    if others and path.name not in others:
+        pytest.fail(
+            f"{name}: the plot is committed as {others[0]}, not {path.name}; git mv it to {path.name}",
+            pytrace=False,
+        )
+    if not path.is_file():
+        pytest.fail(
+            f"{name}: no committed plot at {path}; make it with --validation-plots and commit it",
+            pytrace=False,
+        )
+    try:
+        committed = read_summary(path)
+        same = committed is not None and committed.get("v") == summary["v"]
+        diffs = compare_summaries(summary, committed) if same else []
+    except (OSError, ValueError, TypeError, KeyError, AttributeError) as err:
+        pytest.fail(
+            f"{name}: {path.name} carries an unreadable data summary ({err}); {again} and "
+            "commit it",
+            pytrace=False,
+        )
+    if not same:
+        version = "" if committed is None else " of this version"
+        warnings.warn(
+            f"{name}: {path.name} carries no data summary{version}; {again} so that it can "
+            "be compared",
+            stacklevel=3,
+        )
+    elif diffs:
+        more = f" and {len(diffs) - 1} more" if len(diffs) > 1 else ""
+        pytest.fail(
+            f"{name}: the committed plot {path.name} is stale ({diffs[0]}{more}); {again} "
+            "and commit it",
+            pytrace=False,
         )
-
-    return plot