copier: modules variants are never tested, leaving import std UUID coverage unverified

[steve-downey]

Summary

copier/test_standard_project.sh never tests the C++ modules variants of a generated project. It defines the machinery to do so and then does not call it, so import std support in generated projects is unverified — including by the copier-cmake-matrix CI job, whose CMake version list exists specifically to cover the import std UUID branches.

This lands with #393; it is not a regression introduced by it, and deliberately not fixed there.

What is wrong

test_project_variant takes a use_modules parameter and translates it into a configure flag:

test_project_variant() {
    local variant_name="$1"
    local unit_test_library="$2"
    local use_modules="$3"
    ...
    if [[ "$use_modules" == "true" ]]; then
        cmakelists_args="-DBEMAN_TEST_PROJECT_USE_MODULES=ON"
    fi

But the only two call sites both pass false:

# 1. GTest + No Modules
test_project_variant "gtest-no-modules" "gtest" "false"

# 2. Catch2 + No Modules
test_project_variant "catch2-no-modules" "catch2" "false"

# Do not run modules locally if we cannot guarantee modern tooling, but CI will use clang/gcc containers
# We check if we are in github actions to enforce building modules, as locally it may fail CMake module requirements.

The trailing comment describes gating the modules variants on GITHUB_ACTIONS, and ci_tests.yml duly sets GITHUB_ACTIONS: true in the env for both the copier-test (line 36) and copier-cmake-matrix (line 74) jobs — but no script reads that variable. The -DBEMAN_TEST_PROJECT_USE_MODULES=ON path is dead code.

copier/MAINTAINERS.md line 64 documents the mode as if it works:

Locally, you can simulate specific container behaviors by targeting a custom CMake preset or enabling actions mode: GITHUB_ACTIONS=true ./copier/test_standard_project.sh llvm-release.

Setting GITHUB_ACTIONS=true currently changes nothing, so that sentence should be fixed or made true along with the script.

Why it matters

The copier-cmake-matrix job runs the template across a list of CMake versions chosen to cover the branches in infra/cmake/enable-experimental-import-std.cmake, which selects a different experimental import std UUID per CMake version range. Since no variant ever enables modules, the job exercises those branches only insofar as the file is included before project() — it never actually builds with CXX_MODULE_STD, which is the thing the UUID gates.

So a wrong or stale UUID would not fail CI. That is not hypothetical: CMake has changed this UUID within a patch series (4.0.2 uses a9e1cf81…, 4.0.3 uses d0edc3af…, which is why enable-experimental-import-std.cmake carries a VERSION_LESS "4.0.3" branch).

Suggested fix

Add the modules variants behind the intended guard, e.g.:

if [[ "${GITHUB_ACTIONS:-}" == "true" ]]; then
    test_project_variant "gtest-modules" "gtest" "true"
    test_project_variant "catch2-modules" "catch2" "true"
fi

Worth deciding as part of this:

• Whether modules variants should run across the whole copier-cmake-matrix version list or only on the newest CMake. The matrix is what makes them valuable for UUID coverage, but it multiplies build time.
• Whether to fail or skip when the toolchain cannot do import std, so the guard degrades usefully rather than being CI-only.

Related gap in the version matrix

While confirming the above: the matrix in ci_tests.yml is

cmake_version: ["3.30.9", "3.31.10", "4.0.3", "4.1.3", "4.2.3", "4.3.2", "4.4.2"]

enable-experimental-import-std.cmake has a branch for 4.0.0–4.0.2 (a9e1cf81…), but 4.0.3 is the first version of the next range (d0edc3af…), so that branch has no matrix entry. Adding "4.0.2" would close it.

Verified by reading the accepted UUIDs out of each CMake binary:

CMake	import-std UUID in binary	branch in enable-experimental-import-std.cmake
4.0.2	a9e1cf81…	VERSION_LESS "4.0.3" — no matrix entry
4.0.3	d0edc3af…	VERSION_LESS "4.3.0"
4.3.2	451f2fe2…	>= 4.3.0
4.3.4	451f2fe2…	>= 4.3.0
4.4.0	f35a9ac6…	>= 4.4.0
4.4.2	f35a9ac6…	>= 4.4.0

(PyPI publishes only 4.4.0 and 4.4.2 for the 4.4 series, so 4.4.2 covers that branch fully.)

[ClausKlein]

I would suggest to test all latest patch version of all CMake minor version starting with CMake 3.31.x, because import std; should be tested with latest bug fixed CMake version starting from v3.31:

ClausKlein@b573c43
and
ClausKlein@0cf80f6

NOTE: import std; needs at leased CMake v3.31! CK

[steve-downey]

I think we should test the full range in a distinct CI job. I want the interactive test cycle to stay short enough I stay engaged.

Perhaps part of the fuller testing of the real generated code? Should we do it also for the generated project?

[ClausKlein]

I think we should test the full range in a distinct CI job. I want the interactive test cycle to stay short enough I stay engaged.

My understanding is: at the moment, most of the CMake versions since v3.31.x are tested until today at CI.

What we must have attention on are new releases and bugfix releases.
That is always the highest patch version: i.e 4.3.x, 4.4.x, because here the UUID may change with future CMake releases!

If we see a problem at exemplar project, we have to adapt the infra project first.

Perhaps part of the fuller testing of the real generated code? Should we do it also for the generated project?

No, not necessary I think.