Skip to content

MSTest on MTP silently ignores custom .runsettings sections without a warning #11776

Closed

Description

Description

MSTest on Microsoft.Testing.Platform warns for some unsupported .runsettings entries, but silently ignores custom top-level sections that would be loaded by VSTest ISettingsProvider extensions. In particular, <Playwright> is ignored, which makes browser, launch, and assertion configuration appear to work while defaults are actually used.

This was reported in microsoft/playwright-dotnet#3124 and is relevant to the native testconfig.json support requested in microsoft/playwright-dotnet#3375. Playwright should not work around this by independently reading MSTest's runsettings file.

Reproduction

Use this project with the published MSTest 4.4.1 / MTP 2.4.1 packages:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net8.0</TargetFramework>
    <OutputType>Exe</OutputType>
    <EnableMSTestRunner>true</EnableMSTestRunner>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="MSTest" Version="4.4.1" />
  </ItemGroup>
</Project>

Add an ordinary passing MSTest test and this ignored.runsettings file:

<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>2</MaxCpuCount>
  </RunConfiguration>
  <Playwright>
    <BrowserName>firefox</BrowserName>
  </Playwright>
</RunSettings>

Run:

dotnet run --project WarningRepro.csproj -- --settings ignored.runsettings --output Detailed

Actual behavior

Verified with the published packages on Windows, targeting net8.0. The process exits 0 and runs one passing test. Output contains the control warning:

Runsettings attribute 'MaxCpuCount' is not supported by Microsoft.Testing.Platform and will be ignored

There is no warning for the ignored <Playwright> section. MSTestRunSettings.WarnOnUnsupportedEntries warns for loggers, data collectors, and a fixed list of RunConfiguration entries, but not unsupported custom top-level sections.

Expected behavior

Warn when a custom runsettings section will not be consumed on MTP, naming the ignored section and explaining that VSTest ISettingsProvider extensions are not invoked. Do not warn for sections MSTest actually consumes. Ideally the platform/framework can distinguish sections consumed by registered native extensions, so applications are not forced to guess or parse internal runsettings transport details.

Environment

  • MSTest 4.4.1
  • Microsoft.Testing.Platform 2.4.1
  • Windows, net8.0
  • .NET SDK 11.0.100-rc.1.26425.128
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/mstestMSTest framework (not analyzers or assertions).

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions