DR8 integration workflow by PiotrKorkus · Pull Request #398 · eclipse-score/reference_integration

PiotrKorkus · 2026-10-13T11:56:11Z

DEMO

Demo PR exists in private fork, as to demonstrate it we need to push workflow to main branch first.

Overview

What changed. ref_int no longer runs every module's unit tests inside its own workspace. The old way used quality_runners.py unit tests, the generated rust_coverage/BUILD, and resolved_dependencies.py / module_patches.py to inject modules into MODULE.bazel. Now each target_sw module runs its own tests in its own workspace, but against the dependency versions ref_int resolves.

Flow

test_and_docs.yml
 ├─ module_tests (module_tests.yml)
 │   ├─ List modules: jq .modules.target_sw keys from known_good.json
 │   └─ Test <module>: one matrix job per module (fail-fast: false)
 │        python3 scripts/module_tests.py run <module>
 │        → uploads artifact module-tests-<module>
 └─ test_and_docs (needs module_tests, runs unless cancelled)
      ├─ feature + platform integration tests (as before)
      ├─ download module-tests-* → _module_reports/
      ├─ module_tests.py report → unit_test_summary.md,
      ├─ collect test.xml into tests-report/ (module results go under external/<module>+/)
      └─ docs build (source code linker reads tests-report/), release ZIP on tags

What module_tests.py run does (scripts/module_tests.py:281)

Resolves ref_int's dependency set. It runs bazel mod graph in ref_int with --lockfile_mode=error, so every job gets the same set from the committed lockfile.
Fetches the module source through ref_int, at the known_good commit with ref_int's patches applied. It copies the source to _module_tests//.
Makes one edit to the module: it appends include("//:unified_toolchains.MODULE.bazel"). This is needed because toolchains can only be defined in MODULE files.
Pins the module's direct deps to ref_int's versions. For each direct dep that ref_int also resolves, it writes a --override_module==<ref_int's fetched copy> line into an overrides bazelrc. Bazel resolves the deeper dependencies from there.
Wraps bazel with its own --output_base, plus --bazelrc=unified.bazelrc and --bazelrc=overrides.bazelrc.
Runs the module's own test command: bazel coverage <test_args from known_good.json> --config=unified [--config=unified-rust] --combined_report=lcov.
Collects the results: test.xml files, lcov.dat, resuldes.bazelrc go into _module_tests/reports//.
Fails if no test ran. Bazel exits 0 when every test is skipped, so a run with no test results counts as a failure.

Shared config

bazel_common/unified.bazelrc: the platform, the GCC 12.2 toolchain, the Ferrocene Rust toolchain, the ferrocene-coverage config and the ANDROID_HOME workaround. ref_int's .bazelrc imports it, and so does every module run, so both are built the same way.
bazel_common/unified_toolchains.MODULE.bazel: defines unified_gcc_x86_64_toolchain. The unified_ prefix keeps it from clashing with a module's own toolchain names.
known_good.json → metadata.test_args: each module's own CI test arguments (configs, filters, targets).

Running it locally

python3 scripts/module_tests.py run score_baselibs
python3 scripts/module_tests.py report _module_tests/rep

Closes #264

OliverHeilwagen

Really nice solution. Using --override_module and bazel fetch from ref_int context reduced the complexity drastically, also patching a include() for the unified toolchain solves the former open point.

I only see pinning of the whole module graph as required change (pinning of transitive dependencies) see suggestion line 181, all other comments are optional suggestions.

OliverHeilwagen · 2026-10-15T20:28:25Z

+        "USE_BAZEL_VERSION": (REF_INT / ".bazelversion").read_text().strip(),
+    }
+
+    shared = direct_deps(mod_graph(workspace, env)) & resolved


Please change to:

Suggested change

shared = direct_deps(mod_graph(workspace, env)) & resolved

shared = (graph_modules(mod_graph(workspace, env)) & resolved) - {module.name}

Only pinning direct dependencies can lead to version drifts compared to the dependency resolution done by reference_integration.
All modules in scope of reference integration version resolution should be enforced on the modules as long as they are a dependency of the given module under test.
Please also update autodoc description in line 18.

Note: direct_deps can be removed after this change.

Follow up: A override from reference integration may lead to changes is the mod graph of the module, which may require additional --overide_module=, i assume we have to run this step in a loop till we are stable.

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T20:46:16Z

+def override_rc(paths: dict[str, Path]) -> str:
+    return "".join(f"common --override_module={name}={paths[name]}\n" for name in sorted(paths))


👍 usage of --override_module simplifies the approach drastically, thanks for finding this.

OliverHeilwagen · 2026-10-15T21:01:48Z

+    A repo that fails to fetch (e.g. a dead mirror of a module no build ever touches) is just
+    missing from the result instead of failing the run.
+    """
+    bazel(["fetch", "--keep_going", "--lockfile_mode=error", *(f"--repo=@@{n}+" for n in sorted(names))])


👍 Using bazel fetch from reference_integration context also has the benefit that git_override / singe_version_override patches from known_good.json are applied automatically, again nice simplification of the approach!

Please add this effect to the docstring of fetch().

Should we guard agains local_path overrides?

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T21:13:42Z

+    work = args.work_dir.resolve()
+    workspace, out = work / module.name, work / "reports" / module.name
+    shutil.rmtree(workspace, ignore_errors=True)
+    shutil.rmtree(out, ignore_errors=True)
+    out.mkdir(parents=True)
+    source = fetch({module.name}).get(module.name)
+    if source is None:
+        sys.exit(f"could not fetch {module.name} in ref_int")
+    shutil.copytree(source, workspace, symlinks=True)
+    shutil.copy(UNIFIED_TOOLCHAINS, workspace)
+    module_file = workspace / "MODULE.bazel"
+    module_file.chmod(0o644)  # read-only when copied from Bazel's repo cache
+    with module_file.open("a") as f:
+        f.write(f'\ninclude("//:{UNIFIED_TOOLCHAINS.name}")\n')


nitpick (optional)
tiny refactor would further increase already good readability of the script:
_setup_working_dir(...)
_copy_fetch_module(...)
_append_unified_toolchain(...)

similar changes could be done for:
_collect_coverage(...)
_write_summary(...)

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T21:19:36Z

+    shutil.copy(UNIFIED_TOOLCHAINS, workspace)
+    module_file = workspace / "MODULE.bazel"
+    module_file.chmod(0o644)  # read-only when copied from Bazel's repo cache
+    with module_file.open("a") as f:


👍 again nice solution for ensuring identical toolchain configurations.

OliverHeilwagen · 2026-10-15T21:24:26Z

+    env = {
+        **os.environ,
+        "PATH": f"{wrapper.parent}{os.pathsep}{os.environ['PATH']}",
+        "USE_BAZEL_VERSION": (REF_INT / ".bazelversion").read_text().strip(),


Please extend comment stating that USE_BAZEL_VERSION overrules the modules .bazelversion file.

optional, change not required.

OliverHeilwagen · 2026-10-15T22:13:53Z

+
+    shared = direct_deps(mod_graph(workspace, env)) & resolved
+    pinned = fetch(shared)
+    rc.write_text(override_rc(pinned))


Should we not also write the extra_test_config from known_good.json here?

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T22:22:39Z

+    startup = f"--output_base={work / 'output_base' / module.name} --bazelrc={UNIFIED_RC} --bazelrc={rc}"
+    wrapper.write_text(f'#!/bin/sh\nexec {shutil.which("bazel")} {startup} "$@"\n')


Suggested change

startup = f"--output_base={work / 'output_base' / module.name} --bazelrc={UNIFIED_RC} --bazelrc={rc}"

wrapper.write_text(f'#!/bin/sh\nexec {shutil.which("bazel")} {startup} "$@"\n')

startup = [f"--output_base={work / 'output_base' / module.name}", f"--bazelrc={UNIFIED_RC}", f"--bazelrc={rc}"]

wrapper.write_text(f'#!/bin/sh\nexec {shlex.quote(shutil.which("bazel"))} {shlex.join(startup)} "$@"\n')

Fixed. Comment can be resolved.

OliverHeilwagen · 2026-10-15T22:43:33Z

+    found = {}
+    for name in names:
+        # ponytail: "<name>+" is bzlmod's canonical name; a few built-ins (platforms) have none.
+        for repo in (external / f"{name}+", external / name):


Please document that multiple_version_override is excluded for now.
I also think this is fine, we can react once it is required.

Fixed. Comment can be resolved.

OliverHeilwagen

No findings remaining.
test_and_docs workflow issue should be resolved once PR is merged (pull_request_target).
build_and_test_qnx and _autosd issue was not investigated but appears to be unrelated to this PR.

Copilot

🟡 Changes recommended

Unresolved test-result, coverage-reporting, release-execution, and legacy CLI regressions undermine the new validation workflow.

6 open issues

What changed in this PR

This PR moves module unit testing out of the reference integration workspace. Each module runs its own tests against dependencies resolved by the integration, with results collected for documentation and release reports.

Changes:

Adds per-module test jobs, a Python runner, and helper tests.
Shares platform and toolchain settings across integration and module workspaces.
Replaces legacy dependency-injection and Rust coverage infrastructure with module report collection.

File	Description
scripts/tests/test_module_tests.py	Tests runner argument and report helpers.
scripts/module_tests.py	Runs module tests and aggregates reports.
scripts/known_good/update_module_from_known_good.py	Stops generating legacy Rust coverage targets.
scripts/known_good/tests/test_resolved_dependencies.py	Removes dependency-injection tests.
scripts/known_good/tests/test_module_patches.py	Removes standalone patch-application tests.
scripts/known_good/resolved_dependencies.py	Removes legacy dependency injection.
scripts/known_good/module_patches.py	Removes standalone patch application.
scripts/known_good/models/module.py	Adds module-specific test arguments.
scripts/known_good/BUILD	Removes the dependency-resolution binary.
scripts/BUILD	Adds runner library and test targets.
rust_coverage/BUILD	Removes legacy Rust coverage targets.
patches/tooling/BUILD	Removes the obsolete tooling patch package.
patches/tooling/001-expose-aspect-rules-lint-non-dev.patch	Removes the tooling dependency patch.
patches/time/005-use-sanitizer-bool-flags.patch	Updates Time sanitizer flags.
patches/time/004-rename-ubuntu-image-pull.patch	Avoids an image repository name collision.
MODULE.bazel	Includes shared toolchain definitions.
known_good.json	Adds test arguments and updates patches and tooling.
bazel_common/unified.bazelrc	Centralizes platform and coverage settings.
bazel_common/unified_toolchains.MODULE.bazel	Defines the shared GCC toolchain.
bazel_common/score_test_artifact_versions.MODULE.bazel	Updates the Rust toolchain override.
bazel_common/score_rust_toolchains.MODULE.bazel	Updates the Rust toolchain dependency.
bazel_common/score_modules_tooling.MODULE.bazel	Reflects the tooling pin and patch removal.
bazel_common/score_modules_target_sw.MODULE.bazel	Includes the new Time patches.
bazel_common/score_gcc_toolchains.MODULE.bazel	Removes the relocated GCC definition.
.gitignore	Ignores module-test work directories.
.github/workflows/test_and_docs.yml	Consumes module reports for documentation and releases.
.github/workflows/module_tests.yml	Adds the per-module test matrix.
.github/workflows/internal_tests.yml	Includes the new runner tests.
.bazelrc	Imports shared settings and toolchain names.
.bazelignore	Changes recursive package-discovery exclusions.

_{Review effort: Balanced}

@@ -1 +1 @@
-build
+build_module_tests


+          repository-cache: true
+          cache-save: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
+      - name: Run module tests
+        run: python3 scripts/module_tests.py run ${{ matrix.module }}


+      - name: Run module tests
+        run: python3 scripts/module_tests.py run ${{ matrix.module }}


+    command = coverage_args(module.metadata.test_args, module.metadata.langs, module.metadata.exclude_test_targets)
+    exit_code, log = streamed(["bazel", *command], workspace, env)
+
+    results = list((workspace / "bazel-testlogs").glob("**/test.xml"))


+    # Bazel exits 0 when every test is skipped (e.g. all incompatible with the platform).
+    no_tests = not results
+    exit_code = exit_code or int(no_tests)
+    _collect_coverage(workspace, out, env)


+        if "cpp" in module.metadata.langs:
+            numbers = lcov_percentages(lcov) if lcov.is_file() else {}
+            coverage[f"{module.name}_cpp"] = {**numbers, "exit_code": 0 if numbers.get("lines") else 1}


Squash of: - feat: two-stage DR-008 test workflow with per-module Stage 2 execution - fix: give central-mode test runs their own exclusion list - ci: split DR-008 test execution into its own workflow file, keep old job unchanged - fix: restore itf conftest.py - ci(stage2): isolate score_communication under a dedicated Stage-2 rc - fix(stage2): transport dependency patches into Stage 2 injection - style: apply ruff formatting - fix(stage2): add score_config_management to ref_int's GCC toolchain - chore: align DR-008 changes with upstream main Co-authored-by: Piotr Korkus <piotr.korkus.ext@qorix.ai>

- ci: add module tests workflow for PR testing - refactor: test target_sw modules via --override_module instead of MODULE.bazel injection - fix(module_tests): give module runs their own output base - fix(unified.bazelrc): clear ANDROID_HOME for module runs too - fix(module_tests): pin direct deps only and inject unified toolchains - fix(time): give time's ubuntu image pull its own name - fix(module_tests): fail runs without executed tests; patch time sanitizer flags - build: bump score_toolchains_rust to 0.10.0 - build: use score_tooling v2.2.3

- unified config: clang 22 (libstdc++) as the cc toolchain under `bazel coverage`, gcc stays for builds. Exports LLVM_PROFDATA, which score_communication's coverage merger needs for ferrocene profraw. - rules_rust: backport upstream -imacros/-resource-dir handling for cargo build scripts (toolchains_llvm passes both execroot-relative). - score_communication: list the exposed com-api-example in the visibility guard golden; exclude the tooling-only sync_skills check. - Exclude/relax what fails under clang in time, baselibs and lifecycle, each with a TODO in the new metadata.test_notes field. - module_tests.py: allow yanked versions when reading the module's own graph.

PiotrKorkus · 2026-10-26T09:58:15Z

test_and_docs failing as it uses code from main (pull_request_target). The workflow was executed successfully in a fork: https://github.com/PiotrKorkus/reference_integration/actions/runs/40293215478?pr=5

OliverHeilwagen · 2026-10-26T10:03:08Z

🥳 Nice!

PiotrKorkus requested review from AlexanderLanin, FScholPer, MaximilianSoerenPollak, antonkri, dcalavrezo-qorix, lurtz, nradakovic, opajonk and pawelrutkaq as code owners October 13, 2026 11:56

PiotrKorkus mentioned this pull request Oct 13, 2026

DR-008 Option 4: two-stage test-execution workflow (PR 2 of 2) #280

Closed

OliverHeilwagen suggested changes Oct 15, 2026

View reviewed changes

OliverHeilwagen approved these changes Oct 19, 2026

View reviewed changes

evinoth1206 requested a balanced review from Copilot October 22, 2026 18:02

Copilot started reviewing on behalf of evinoth1206 October 22, 2026 18:02

View session

Copilot AI reviewed Oct 22, 2026

View reviewed changes

Subramanian-K812 and others added 5 commits October 26, 2026 09:30

Ensure excluded test targets are still taken if present in known_good

2af40e1

review fixes

8de37bd

PiotrKorkus force-pushed the piotrkorkus_dr8 branch from 568f2f1 to 226b2b6 Compare October 26, 2026 08:35

pawelrutkaq approved these changes Oct 26, 2026

View reviewed changes

PiotrKorkus merged commit 6a7e4f8 into main Oct 26, 2026
17 of 18 checks passed

PiotrKorkus deleted the piotrkorkus_dr8 branch October 26, 2026 09:59

	shared = direct_deps(mod_graph(workspace, env)) & resolved
	shared = (graph_modules(mod_graph(workspace, env)) & resolved) - {module.name}

		def override_rc(paths: dict[str, Path]) -> str:
		return "".join(f"common --override_module={name}={paths[name]}\n" for name in sorted(paths))

		startup = f"--output_base={work / 'output_base' / module.name} --bazelrc={UNIFIED_RC} --bazelrc={rc}"
		wrapper.write_text(f'#!/bin/sh\nexec {shutil.which("bazel")} {startup} "$@"\n')

		@@ -1 +1 @@
		build No newline at end of file
		build_module_tests

		- name: Run module tests
		run: python3 scripts/module_tests.py run ${{ matrix.module }}

Conversation

PiotrKorkus commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DEMO

Overview

Flow

Shared config

Running it locally

Uh oh!

OliverHeilwagen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

OliverHeilwagen left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

PiotrKorkus commented Oct 26, 2026

Uh oh!

Uh oh!

OliverHeilwagen commented Oct 26, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

PiotrKorkus commented Oct 13, 2026 •

edited

Loading