Regression in Microsoft.Testing.Platform 2.3.0: dotnet test no longer shows the test host's command-line validation errors #11964

Describe the bug

With Microsoft.Testing.Platform (MTP) 2.5.1 and .NET SDK 10.0.402, a test application that fails command-line validation under dotnet test in MTP mode reports Zero tests ran and Exit code: 5, followed by the SDK's summary and a handshake-failure recap. The host's own error message, which names the problem, is absent from all of it.

The case I hit is two extensions declaring the same option. No command-line argument is involved, so the output gives the user nothing to go on. Running the test executable directly prints the reason:

Option '--my-option' is declared by multiple providers: 'ProviderA', 'ProviderB'

Other validation errors are lost the same way. An unknown option is one, including the "Option '--report-gh' is provided by the 'Microsoft.Testing.Extensions.GitHubActionsReport' extension. Add a package reference to use it." hint added in 2.4. So is dotnet test -m:4: the SDK forwards -m:4 to the host, which rejects --m as an unknown option, and dotnet test again shows exit code 5 without the host's message.

This regressed between MTP 2.2.3 and 2.3.0. With 2.2.3, dotnet test printed the host's message under Standard output:. The message is absent in every later version tested, up to 2.5.1, 2.5.2-preview.26512.15 and the 2.6.0-preview.26515.10 nightly tested on 2026-10-15.

Version used

• Microsoft.Testing.Platform 2.5.1 (via MSTest 4.5.1). The message is also missing with 2.3.0, 2.3.1, 2.3.3, 2.4.0, 2.4.1, 2.5.0, and with 2.5.2-preview.26512.15 and 2.6.0-preview.26515.10 from the test-tools feed (tested on 2026-10-15). Last good version tested: 2.2.3 (MSTest 4.2.3).
• .NET SDK 10.0.402. The host's message is also missing with 10.0.301, 10.0.401, 11.0.100-preview.7.26381.103 and 11.0.100-rc.2.26504.105. SDK 10.0.402 and SDK 11 print a handshake-failure recap, but the recap does not contain the message either. Runtime 10.0.13.
• Ubuntu 26.04, x64.

Steps To Reproduce

Create the four files below in an empty directory. The test project uses MSTest, with two option providers that both declare --my-option.

global.json:

{
  "sdk": { "version": "10.0.402", "rollForward": "disable" },
  "test": { "runner": "Microsoft.Testing.Platform" }
}

Repro.csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <OutputType>Exe</OutputType>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <EnableMSTestRunner>true</EnableMSTestRunner>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="MSTest" Version="4.5.1" />
  </ItemGroup>

  <ItemGroup>
    <TestingPlatformBuilderHook Include="6E0E8F6A-7C1B-4F5E-9C1D-3B2A1F0E9D8C" DisplayName="TwoProviders" TypeFullName="Repro.TwoProvidersHook" />
  </ItemGroup>

</Project>

SmokeTests.cs:

namespace Repro;

[TestClass]
public sealed class SmokeTests
{
    [TestMethod]
    public void Passes()
    {
    }
}

TwoProviders.cs:

using Microsoft.Testing.Platform.Builder;
using Microsoft.Testing.Platform.CommandLine;
using Microsoft.Testing.Platform.Extensions;
using Microsoft.Testing.Platform.Extensions.CommandLine;

namespace Repro;

// Two command-line option providers that both declare --my-option.
public static class TwoProvidersHook
{
    public static void AddExtensions(ITestApplicationBuilder builder, string[] _)
    {
        builder.CommandLine.AddProvider(() => new ProviderA());
        builder.CommandLine.AddProvider(() => new ProviderB());
    }
}

public abstract class ProviderBase : ICommandLineOptionsProvider
{
    public abstract string Uid { get; }
    public string Version => "1.0.0";
    public string DisplayName => Uid;
    public string Description => Uid;
    public Task<bool> IsEnabledAsync() => Task.FromResult(true);
    public IReadOnlyCollection<CommandLineOption> GetCommandLineOptions() =>
        [new CommandLineOption("my-option", "Declared by two providers.", ArgumentArity.Zero, false)];
    public Task<ValidationResult> ValidateOptionArgumentsAsync(CommandLineOption commandOption, string[] arguments) =>
        ValidationResult.ValidTask;
    public Task<ValidationResult> ValidateCommandLineOptionsAsync(ICommandLineOptions commandLineOptions) =>
        ValidationResult.ValidTask;
}

public sealed class ProviderA : ProviderBase { public override string Uid => "ProviderA"; }
public sealed class ProviderB : ProviderBase { public override string Uid => "ProviderB"; }

Then run dotnet test. To compare with the last good release, change only the MSTest version to 4.2.3 and run it again.

I originally hit this with xunit.v3.mtp-v2 3.2.2 plus Microsoft.Testing.Extensions.JUnitReport, which both declare --report-junit and --report-junit-filename. That pair behaves the same way with the three JUnitReport alphas I tried, which resolve MTP 2.3.0, 2.4.1 and 2.5.1. The MSTest-only repro above avoids the third-party package.

Expected behavior

dotnet test shows the host's error, as it did with 2.2.3 (MSTest 4.2.3, same files):

.../bin/Debug/net10.0/Repro.dll (net10.0) Zero tests ran
Exit code: 5
  Standard output: Option '--my-option' is declared by multiple extensions: 'ProviderA', 'ProviderB'
  Usage Repro [option providers] [extension option providers]
  ...

Actual behavior

In the measured 2.5.1 (MSTest 4.5.1) run, dotnet test exited with 5 and stderr was empty. Stdout was as follows (paths abbreviated; the duration varies):

.../bin/Debug/net10.0/Repro.dll (net10.0) Zero tests ran
Exit code: 5

Test run summary: Failed!
  error: 1

  total: 0
  failed: 0
  succeeded: 0
  skipped: 0
  duration: 82ms
Test run completed with non-success exit code: 5 (see: https://aka.ms/testingplatform/exitcodes)

Handshake failures:
  .../bin/Debug/net10.0/Repro.dll (net10.0)
Exit code: 5

With SDK 10.0.401 the output is the same except that the summary line reads Test run summary: Zero tests ran and there is no Handshake failures: recap.

Running the same build directly (./bin/Debug/net10.0/Repro) exits with 5 and prints the reason, followed by the help text:

Option '--my-option' is declared by multiple providers: 'ProviderA', 'ProviderB'
Usage Repro [option providers] [extension option providers]

Execute a .NET Test Application.
...

Additional context

Bisect

The exit code is 5 in every row, and the direct run prints the message in every row. To pin the platform version directly, I used the same two providers with xunit.v3.mtp-v2 3.2.2 as the test framework, pinning Microsoft.Testing.Platform and Microsoft.Testing.Platform.MSBuild to each version. A project without the two providers passes at both 2.2.3 and 2.5.1.

• SDK. 10.0.301, 10.0.401, 10.0.402, 11.0.100-preview.7 and 11.0.100-rc.2 give the same split: 2.2.3 relays the message and 2.5.1 does not.
• Microsoft.Testing.Platform.dll alone. I took a 2.2.3 build and replaced only that assembly with the one from the 2.3.0 package, and separately with the one from 2.5.1. In both cases dotnet test stopped showing the message, while the direct run still printed it.

What dotnet test receives

I set DOTNET_CLI_TEST_TRACEFILE. With 2.5.1, the SDK starts the app with --server dotnettestcli --dotnet-test-pipe <name> (10.0.402 adds --no-banner) and then logs Test Process exited with non-zero exit code: 5. There is no captured output and no handshake. With 2.2.3, the same trace has Output Data: Option '--my-option' is declared by multiple extensions: ....

The SDK prints whatever the process wrote before exiting without a handshake. In 10.0.401 this is TerminalTestReporter.HandshakeFailure, which prints Standard output: / Standard error: when they are non-empty. SDK 10.0.402 and SDK 11 also print a Handshake failures: recap. With 2.5.1 that recap contains only the module path and Exit code: 5. The host writes nothing, so the SDK has nothing to relay.

Suspected cause

Links are at the 2.5.1 commit 0bf77e3. The same lines are unchanged on main at df8183a.

1. Under --server dotnettestcli, the only output device is DotnetTestPassthroughOutputDevice (OutputDeviceManager.cs#L41-L44).
2. When command-line validation fails, the error is sent to that device as ErrorMessageOutputDeviceData, the help follows, and the builder returns an InformativeCommandLineHost with exit code 5 (TestHostBuilder.CommonServices.cs#L302-L311).
3. The passthrough device forwards errors only through a DotnetTestConnection. When none is registered it returns without writing anything (DotnetTestPassthroughOutputDevice.cs#L99-L108). The help text is informational, so it is discarded by design.
4. BuildHostAsync returns the early host before the DotnetTestConnection is created, connected and registered (TestHostBuilder.Modes.cs#L31-L50).

So on this path the error reaches neither the pipe nor the console, and the process exits 5 before any handshake.

This appears to come from #7564 ("Avoid plugging in TerminalOutputDevice when running with pipe protocol"), which first shipped in 2.3.0. #9521, also in 2.3.0, added error forwarding over the pipe, but that forwarding needs the connection, and on this path the connection does not exist yet. I found this by reading the 2.2.3..2.3.0 history and checking ancestry. I did not build testfx at each commit.

The FormatException branch just above (malformed option in testconfig.json, CommonServices.cs#L273-L285) has the same shape. I have not tested it.

Two possible directions. When no connection exists yet, the host could write error messages to stdout or stderr as it did before; the SDK already relays that output for a process that exits before the handshake. Alternatively, the host could connect before command-line validation. I'm happy to send a PR for whichever direction you prefer.

Related

#10868 and #11528 report the same missing-diagnostic symptom for extension options and are closed; #11528 was closed as a duplicate of #10868. The discussion on #10868 covers installing the missing extension, improving the documentation, the MTP 2.4 missing-extension hint, and the SDK's exit-code summary. None of these addresses the early command-line failure path described here:

• The 2.4 hint improves the host's message, but that message is discarded before the pipe connection exists.
• Backport descriptive MTP exit-code summaries to .NET 10 and .NET 11 dotnet/sdk#56104 improves the generic exit-code description; SDK 11.0.100-rc.2 prints The command-line arguments are invalid. for this case. Without the host's message it cannot name the offending option or providers.
• Surface and re-print handshake failures in dotnet test dotnet/sdk#54501 changes the failure summary and repeats captured output in a handshake-failure recap. SDK 10.0.401 already shows captured pre-handshake output, and SDK 10.0.402 and SDK 11 also show the recap. Neither can show host text that was never emitted.

I also checked an MSTest 4.5.1 project without the extension: passing --report-gh under dotnet test still produces no host diagnostic, while the direct run shows the missing-extension hint. This reproduces the missing-option condition reported in #11528, but it is not a rerun of that issue's full command. The two-provider repro above shows the same failure with no user option at all.