[Feature request] Add test artifacts to XML.

[TDuffinNTU]

Hello! I'm not sure if this project is still developed actively, but I thought to suggest this for yourself or someone else if they were up for it.

Note I have only ever used Azure Devops Pipelines for test automation and CI/CD on flutter projects. The terminology may not match what others use!

Feature description

In CI/CD runners such as Azure Devops, there is ad hoc support for "attachments" which may be added to test results. While this isn't within the Junit spec, and may not be standard across automation platforms, it would be a useful feature to allow test results to publish supporting artifacts such as Golden "diff images" and log files that may not be surfaced by the test runner automatically.

In essence, they are simply relative paths to existing files at the time of the test execution, and it would be up to the automation platform to handle finding and publishing of the artifacts. The XML attachments simply couple this artifact to the test that produced it.

Benefits

Platforms such as ADO display test results within a "test run", which allow developers to see which tests are failing, how often, and assign bug tickets to them among other things. When an attachment is detected, it is uploaded during a "Publish test results" job on the pipeline, producing a result like so:

As mentioned before, you could use this to automate golden test reporting to levels not possible without the feature. Currently, unless I'm mistaken, devs usually upload the files within "test/**/goldens/failures/*.png" after a test run, usually requiring some detection to ensure the pipeline doesn't try publishing on a successful run (which throws an error on ADO, not sure for others) and have to manually thumb through potentially dozens of files to download/view and then reupload if they wanted to use the test results UI to, say, make a bug report.

Implementation

I'm not too sure how this could be achieved necessarily, it may be one for upstream dartlang devs to tackle as they could add it to the JSON reporter in a trivial manner (making your work much easier!)... but

In most cases a test runner will log the path of any artifacts it produces. Or you may be able to infer it from the logs of a particular test:

══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test:
Expected: one widget whose rasterized image matches golden image "goldens/ci/login_page_light.png"
  Actual: <Instance of 'Future<Image>'>
   Which: does not match

When the exception was thrown, this was the stack:
#0      fail (package:matcher/src/expect/expect.dart:187:31)
#1      _expect.<anonymous closure> (package:matcher/src/expect/expect.dart:155:13)
<asynchronous suspension>
<asynchronous suspension>
#8      expectLater.<anonymous closure> (package:flutter_test/src/widget_tester.dart:508:19)
<asynchronous suspension>
#9      FlutterGoldenTestAdapter.withForceUpdateGoldenFiles (package:alchemist/src/golden_test_adapter.dart:198:14)
<asynchronous suspension>
#10     FlutterGoldenTestRunner.run (package:alchemist/src/golden_test_runner.dart:137:9)
<asynchronous suspension>
#11     goldenTest.<anonymous closure> (package:alchemist/src/golden_test.dart:177:7)
<asynchronous suspension>
#12     testWidgets.<anonymous closure>.<anonymous closure> (package:flutter_test/src/widget_tester.dart:192:15)
<asynchronous suspension>
#13     TestWidgetsFlutterBinding._runTestBody (package:flutter_test/src/binding.dart:1952:5)
<asynchronous suspension>
<asynchronous suspension>
(elided 7 frames from dart:async and package:stack_trace)

The test description was:
  renders correctly on light theme (variant: CI)
════════════════════════════════════════════════════════════════════════════════════════════════════

In this case, the file goldens/ci/login_page_light.png suggests that goldens/ci/failures/* would contain the artifacts. Of course, relatively to the path of the test itself which the test reporter will have logged by this point.

This feels very specific to just one example of a test output, though, so perhaps there would be work needed within the dart team to surface artifacts within the JSON to be consumed later.

Alternatively, you could do the inverse and, given a file test/login_page/goldens/ci/failures/login_page_light.png, you could read the file hierarchy to infer that the nearest test file (or some other condition) is the "owner". It would be up to the user to supply the location of files they wish to attach. You could even provide subcommands to add arbitrary files to a particular test result, regardless of file hierarchy.

Conclusion

Let me know what you think, or if this is perhaps something out of scope for the project. I wonder what others would think of this as a standalone tool 🤔

[futabooo]

Thanks for the detailed write-up — and for thinking through the implementation options yourself rather than just dropping a request.

My take: attachments as a feature are out of scope for junitify, but there's a smaller change here that I think is worth doing and would get you most of the way.

Why attachments themselves are out of scope

junitify is a pure stdin→stdout filter: it converts dart test --reporter=json output to JUnit XML, and nothing more. It doesn't touch the filesystem and doesn't know anything about the project layout. That constraint is what keeps it simple and predictable.

The Dart JSON reporter doesn't emit artifact paths at all, so every option you listed requires stepping outside the input:

• Inferring paths from failure logs would hardcode a specific Flutter golden-failure message format — you flagged this yourself as "very specific", and I agree.
• Walking the file hierarchy to find the owning test would require filesystem access, which breaks the pipe-filter model.
• A subcommand for arbitrary attachments is really a different tool at that point.

Also, attachments aren't part of the JUnit format, and "support the JUnit format" is the whole remit of this project.

You're right that the cleanest fix is upstream: if package:test surfaced artifact paths in the JSON reporter, this would become a trivial passthrough here. That's worth filing with dart-lang.

The part I think is worth doing

Looking at how Jenkins' junit-attachments plugin solves this, the design is deliberately in-band: the test prints [[ATTACHMENT|path]] to stdout, the CI platform resolves and publishes the file, and every layer in between just has to pass the text through unchanged. No schema extension, no inference. The converter doesn't need to understand attachments at all.

The reason that doesn't work with junitify today is a gap on my side: dart test's print events carry a testID, but junitify currently throws that away and aggregates all output into a single suite-level <system-out>. Both Jenkins and Datadog treat suite-level output as unattributed and only bind <system-out> inside a <testcase> to the individual test, so the attribution is lost before it reaches your CI.

If junitify emitted per-testcase <system-out>, you could do this from your test today:

print('[[ATTACHMENT|test/login_page/goldens/ci/failures/login_page_light.png]]');

…and ADO / Jenkins would pick it up, with junitify doing nothing but faithfully passing your output along. That stays squarely inside "convert the input accurately", so it seems like a reasonable direction for this tool — probably behind an opt-in flag at first, to avoid duplicating log content for anyone already ingesting the suite-level output. It's better tracked as its own issue, separate from attachments.

Would that cover your use case? If ADO needs something more structured than the [[ATTACHMENT|...]] convention, let me know — I'd like to see a concrete example of the XML it expects before ruling anything else in or out.