diff --git a/.github/docs/trigger-azdo-pipeline-setup.md b/.github/docs/trigger-azdo-pipeline-setup.md index 7a2c7252262f..823b30723cec 100644 --- a/.github/docs/trigger-azdo-pipeline-setup.md +++ b/.github/docs/trigger-azdo-pipeline-setup.md @@ -79,12 +79,19 @@ az identity federated-credential create \ --audiences "api://AzureADTokenExchange" ``` -> **Subject claim mapping:** The OIDC token's `sub` claim is what Azure AD matches -> against the `--subject` parameter. For `issue_comment` events (like the `/review` -> command), the workflow runs from the default branch, so the subject is -> `repo:dotnet/maui:ref:refs/heads/main`. For `pull_request` events, the subject -> would be `repo:dotnet/maui:pull_request`. This is why the case-sensitivity -> warning above is critical — the `sub` claim value must match exactly. +The trusted CI-fix validator deliberately runs from `main` and therefore reuses +the branch-scoped credential above: + +- `pull_request_target` provides the immediate path for eligible PRs targeting + `main`; +- `workflow_run` reconciles all eligible open CI-fix PR heads after either the + main or net11 fixer completes. This path covers PRs and pushes created with + `GITHUB_TOKEN`, whose normal PR events can be suppressed or approval-gated, + and covers net11 without granting OIDC trust to PR events or to `net11.0`. + +Do not add repository-wide `repo:dotnet/maui:pull_request` or net11 credentials +for this automation. The privileged queue step always uses the existing +`repo:dotnet/maui:ref:refs/heads/main` subject and trusted default-branch code. Add more federated credentials for other branches or trigger types as needed: @@ -152,7 +159,7 @@ The identity needs **"Queue builds"** permission on the target pipeline(s): | AzDO Organization | Project | Example Pipelines | |---|---|---| -| `dnceng-public` | `public` | 302 (maui-pr), 314 (maui-pr-devicetests) | +| `dnceng-public` | `public` | 302 (maui-pr), 313 (maui-pr-uitests), 314 (maui-pr-devicetests) | | `DevDiv` | `DevDiv` | 27723 | ## Step 4: Set GitHub Repository Secrets @@ -172,6 +179,54 @@ In **dotnet/maui** → **Settings** → **Secrets and variables** → **Actions* See [`.github/workflows/review-trigger.yml`](../workflows/review-trigger.yml) for a ready-to-use workflow. +Automated CI-fix PR validation is implemented separately by +[`ci-fix-azdo-validation.yml`](../workflows/ci-fix-azdo-validation.yml). It: + +- runs only from trusted default/base-branch code via `pull_request_target` and + `workflow_run`; +- never checks out or executes the PR head; +- requires the exact same-repo CI-fix bot fingerprint; +- reconciles every live eligible CI-fix head on each configured PR-target event, + including unrelated PR events, so GitHub's single-pending-run concurrency + behavior cannot strand an eligible head; +- treats the live open-PR scan as authoritative so delayed webhook snapshots + cannot restore revoked eligibility or queue obsolete head/merge commits; +- uses the list response only to nominate eligible PRs, then refreshes each + nominee individually within bounded retries and verifies the candidate + GitHub test-merge commit object has that refreshed source head as its second + parent before the head/merge pair can enter Azure queueing or deduplication. + The current base SHA is not required to equal the first merge parent because + the base can advance independently; +- reports a bounded missing/stale/conflicting test merge as three explicit + failed pipeline results for that PR without making Azure requests, while + continuing scan-all recovery for other verified eligible heads. Unexpected + GitHub failures and shared-budget exhaustion remain fatal discovery errors; +- starts an absolute seven-minute deadline before checkout and applies its + remaining allowance across PR discovery, authentication, queue-time + revalidation, dedupe reads, queue POSTs, retries, and reconciliation. The + ten-minute job timeout therefore preserves a real three-minute reserve for + summaries and an explicit failed outcome instead of a hard cancellation; +- caps every HTTP timeout and retry sleep to the remaining shared budget. If it + expires, no new requests start, completed results remain visible, all + unprocessed PR/pipeline work is marked failed, and an ambiguous one-time POST + remains explicitly uncertain rather than being retried; +- queues definitions 302, 313, and 314 against `refs/pull//merge`; +- supplies only a verified test-merge commit as `sourceVersion` and its paired + PR head as + `triggerInfo["pr.sourceSha"]`, matching normal Azure Pipelines PR build + metadata intended to attach checks to the PR head; +- supplies Azure Build Queue `parameters` as a serialized JSON string containing + the producer-equivalent `system.pullRequest.*` values. These serialized + system parameters provide the bare target branch used by pipeline conditions; + `triggerInfo` remains provider/build metadata. Exact PR-head check association + still requires live rollout canary verification for all three definitions; +- deduplicates each pipeline by PR head SHA and reports partial failures. + +The workflow and script need to exist only on `main`. Before declaring net11 +coverage enabled, verify that the net11 fixer's compiled workflow name remains +`CI Failure Fixer (net11.0)` so the main-branch `workflow_run` reconciliation +fires after its create/push run. + ## How It Works (Token Flow) ``` @@ -182,8 +237,9 @@ See [`.github/workflows/review-trigger.yml`](../workflows/review-trigger.yml) fo (grant_type=client_credentials) for the managed identity's client_id 4. Azure AD validates the JWT against the federated credential and returns a bearer token scoped to AzDO (resource: 499b84ac-1321-427f-aa17-267ca6975798) -5. Step 3 calls POST dev.azure.com/{org}/{project}/_apis/pipelines/{id}/runs - with the bearer token +5. Step 3 calls the appropriate Azure DevOps queue endpoint with the bearer + token: the Pipelines Runs API for `/review`, or the Build Queue API for + CI-fix PR validation 6. AzDO validates the token, checks the identity's permissions, and queues the build ``` diff --git a/.github/scripts/Queue-CiFixAzdoValidation.Tests.ps1 b/.github/scripts/Queue-CiFixAzdoValidation.Tests.ps1 new file mode 100644 index 000000000000..937b6c749b29 --- /dev/null +++ b/.github/scripts/Queue-CiFixAzdoValidation.Tests.ps1 @@ -0,0 +1,1835 @@ +#!/usr/bin/env pwsh +#Requires -Modules Pester + +BeforeAll { + $scriptPath = Join-Path $PSScriptRoot 'Queue-CiFixAzdoValidation.ps1' + $script:MaxHttpAttempts = 4 + $script:RetryBaseDelaySeconds = 2 + $script:DispatcherBudgetPrefix = '[dispatcher-budget-exhausted]' + $tokens = $null + $parseErrors = $null + $ast = [System.Management.Automation.Language.Parser]::ParseFile($scriptPath, [ref]$tokens, [ref]$parseErrors) + if ($parseErrors -and $parseErrors.Count -gt 0) { + throw ($parseErrors | ForEach-Object Message) -join [Environment]::NewLine + } + + foreach ($functionName in @( + 'Get-ObjectPropertyValue', + 'Get-DispatcherElapsedSeconds', + 'Get-DispatcherRemainingSeconds', + 'Test-IsDispatcherBudgetException', + 'Get-DispatcherHttpTimeoutSeconds', + 'Invoke-DispatcherSleep', + 'Test-CiFixPrFingerprint', + 'Get-CiFixPipelineDefinitions', + 'Get-CiFixContextFromPullRequest', + 'Get-GitHubApiHeaders', + 'Get-FixturePullRequestDetail', + 'Get-FixtureCommit', + 'Get-CiFixMergePairDiagnostic', + 'Resolve-VerifiedCiFixContext', + 'Get-CiFixEventContext', + 'Test-TrustedCiFixWorkflowRun', + 'Get-OpenCiFixContexts', + 'Test-IsTransientHttpException', + 'Get-HttpStatusCode', + 'Invoke-WithHttpRetry', + 'Get-AzdoToken', + 'Find-AzdoDuplicateBuild', + 'Get-AzdoDuplicateBuild', + 'New-AzdoQueueRequest', + 'Invoke-AzdoPipelineQueue', + 'Write-CiFixJobSummary', + 'New-CiFixFailureResult', + 'Invoke-CiFixQueueWork')) { + $function = $ast.Find({ + $args[0] -is [System.Management.Automation.Language.FunctionDefinitionAst] -and + $args[0].Name -eq $functionName + }, $true) + if (-not $function) { + throw "Function '$functionName' not found" + } + Invoke-Expression $function.Extent.Text + } + + function New-TestPullRequest { + param( + [int]$Number = 123, + [string]$Repository = 'dotnet/maui', + [string]$HeadRepository = 'dotnet/maui', + [string]$Author = 'github-actions[bot]', + [string]$HeadRef = 'ci-fix/issue-123', + [string]$BaseRef = 'main', + [string]$Title = '[ci-fix] Repair CI (refs #123)', + [string[]]$Labels = @('agentic-workflows'), + [string]$State = 'open', + [string]$HeadSha = '1111111111111111111111111111111111111111', + [string]$MergeSha = '2222222222222222222222222222222222222222' + ) + + return [pscustomobject]@{ + number = $Number + id = 456789 + $Number + state = $State + draft = $true + title = $Title + merge_commit_sha = $MergeSha + user = [pscustomobject]@{ login = $Author } + labels = @($Labels | ForEach-Object { [pscustomobject]@{ name = $_ } }) + base = [pscustomobject]@{ + ref = $BaseRef + repo = [pscustomobject]@{ full_name = $Repository } + } + head = [pscustomobject]@{ + ref = $HeadRef + sha = $HeadSha + repo = [pscustomobject]@{ full_name = $HeadRepository } + } + } + } + + function New-TestEvent { + param( + [string]$Action = 'synchronize', + [string]$Repository = 'dotnet/maui', + [string]$HeadRepository = 'dotnet/maui', + [string]$Author = 'github-actions[bot]', + [string]$HeadRef = 'ci-fix/issue-123', + [string]$BaseRef = 'main', + [string]$Title = '[ci-fix] Repair CI (refs #123)', + [string[]]$Labels = @('agentic-workflows'), + [string]$State = 'open', + [string]$EventLabel = 'agentic-workflows' + ) + + return [pscustomobject]@{ + action = $Action + label = [pscustomobject]@{ name = $EventLabel } + pull_request = New-TestPullRequest ` + -Repository $Repository ` + -HeadRepository $HeadRepository ` + -Author $Author ` + -HeadRef $HeadRef ` + -BaseRef $BaseRef ` + -Title $Title ` + -Labels $Labels ` + -State $State + } + } + + function New-TestMergeCommit { + param( + [string]$MergeSha = '2222222222222222222222222222222222222222', + [string]$HeadSha = '1111111111111111111111111111111111111111', + [string]$BaseParentSha = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + ) + + return [pscustomobject]@{ + sha = $MergeSha + parents = @( + [pscustomobject]@{ sha = $BaseParentSha }, + [pscustomobject]@{ sha = $HeadSha } + ) + } + } + + function New-TestPullRequestFixture { + param([Parameter(Mandatory = $true)][object[]]$PullRequests) + + $details = [ordered]@{} + $commits = [ordered]@{} + foreach ($pullRequest in $PullRequests) { + $number = [string]$pullRequest.number + $details[$number] = $pullRequest + if ([string]$pullRequest.merge_commit_sha -match '^[0-9a-fA-F]{40}$') { + $commits[[string]$pullRequest.merge_commit_sha] = New-TestMergeCommit ` + -MergeSha ([string]$pullRequest.merge_commit_sha) ` + -HeadSha ([string]$pullRequest.head.sha) + } + } + + return [pscustomobject]@{ + pullRequests = $PullRequests + pullRequestDetails = [pscustomobject]$details + commits = [pscustomobject]$commits + } + } +} + +Describe 'Test-CiFixPrFingerprint' { + It 'accepts the exact main automated PR fingerprint' { + Test-CiFixPrFingerprint ` + -Repository dotnet/maui ` + -HeadRepository dotnet/maui ` + -Title '[ci-fix] Repair CI' ` + -BaseRef main ` + -HeadRef ci-fix/issue-123 ` + -AuthorLogin 'github-actions[bot]' ` + -Labels @('agentic-workflows') | Should -BeTrue + } + + It 'accepts the exact net11 automated PR fingerprint' { + Test-CiFixPrFingerprint ` + -Repository dotnet/maui ` + -HeadRepository dotnet/maui ` + -Title '[ci-fix-net11] Repair CI' ` + -BaseRef net11.0 ` + -HeadRef ci-fix/issue-123 ` + -AuthorLogin 'github-actions[bot]' ` + -Labels @('agentic-workflows') | Should -BeTrue + } + + It 'fails closed for each mismatched trust attribute' -ForEach @( + @{ Repository = 'fork/maui'; HeadRepository = 'dotnet/maui'; Author = 'github-actions[bot]'; Head = 'ci-fix/issue-123'; Base = 'main'; Title = '[ci-fix] Repair'; Labels = @('agentic-workflows') } + @{ Repository = 'dotnet/maui'; HeadRepository = 'fork/maui'; Author = 'github-actions[bot]'; Head = 'ci-fix/issue-123'; Base = 'main'; Title = '[ci-fix] Repair'; Labels = @('agentic-workflows') } + @{ Repository = 'dotnet/maui'; HeadRepository = 'dotnet/maui'; Author = 'attacker'; Head = 'ci-fix/issue-123'; Base = 'main'; Title = '[ci-fix] Repair'; Labels = @('agentic-workflows') } + @{ Repository = 'dotnet/maui'; HeadRepository = 'dotnet/maui'; Author = 'github-actions[bot]'; Head = 'feature/issue-123'; Base = 'main'; Title = '[ci-fix] Repair'; Labels = @('agentic-workflows') } + @{ Repository = 'dotnet/maui'; HeadRepository = 'dotnet/maui'; Author = 'github-actions[bot]'; Head = 'ci-fix/issue-123'; Base = 'main'; Title = '[ci-fix-net11] Repair'; Labels = @('agentic-workflows') } + @{ Repository = 'dotnet/maui'; HeadRepository = 'dotnet/maui'; Author = 'github-actions[bot]'; Head = 'ci-fix/issue-123'; Base = 'net11.0'; Title = '[ci-fix] Repair'; Labels = @('agentic-workflows') } + @{ Repository = 'dotnet/maui'; HeadRepository = 'dotnet/maui'; Author = 'github-actions[bot]'; Head = 'ci-fix/issue-123'; Base = 'main'; Title = '[ci-fix] Repair'; Labels = @() } + ) { + Test-CiFixPrFingerprint ` + -Repository $Repository ` + -HeadRepository $HeadRepository ` + -Title $Title ` + -BaseRef $Base ` + -HeadRef $Head ` + -AuthorLogin $Author ` + -Labels $Labels | Should -BeFalse + } +} + +Describe 'trusted workflow configuration' { + It 'pins checkout to the reviewed v7.0.1 commit in the id-token job' { + $workflow = Get-Content -Raw -LiteralPath (Join-Path $PSScriptRoot '../workflows/ci-fix-azdo-validation.yml') + + $workflow | Should -Match 'actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7\.0\.1' + $workflow | Should -Match 'ref: \$\{\{ github\.sha \}\}' + $workflow | Should -Not -Match 'github\.event\.pull_request\.base\.sha' + $workflow | Should -Match 'persist-credentials: false' + $workflow | Should -Match 'id-token: write' + $workflow | Should -Match 'timeout-minutes: 10' + $workflow | Should -Match 'CI_FIX_DISPATCHER_DEADLINE_UNIX_SECONDS=.*date \+%s.*\+ 420' + $workflow.IndexOf('Start dispatcher deadline') | + Should -BeLessThan $workflow.IndexOf('Checkout trusted workflow revision') + $workflow | Should -Match 'Queue-CiFixAzdoValidation\.ps1 -DispatcherBudgetSeconds 420 -DispatcherDeadlineUnixSeconds \$env:CI_FIX_DISPATCHER_DEADLINE_UNIX_SECONDS' + } + + It 'requires exact-head originating build evidence before build-only readiness in both fixer prompts' { + foreach ($workflowName in @('ci-status-fix.md', 'ci-status-fix-net11.md')) { + $workflow = Get-Content -Raw -LiteralPath ( + Join-Path $PSScriptRoot "../workflows/$workflowName" + ) + + $workflow | Should -Match 'branchName == refs/pull/

/merge' + $workflow | Should -Match 'triggerInfo\["pr\.sourceSha"\] == C\.headSha' + $workflow | Should -Match 'Existence alone is NOT validation' + $workflow | Should -Match 'RELEVANT platform \*\*BUILD\*\* leg' + $workflow | Should -Match 'state == "completed".*result == "succeeded"' + $workflow | Should -Match 'relevant evidence is absent/unknown' + $workflow | Should -Match 'pending/inProgress' + $workflow | Should -Match 'failed/canceled/aborted' + $workflow | Should -Match 'Do NOT require unrelated test legs' + $workflow | Should -Match 'ORIGIN_PIPELINE.*ORIGIN_BUILD_ID.*ORIGIN_BUILD_LEGS' + $workflow | Should -Match 'originating failure was\s+def 302 itself' + $workflow | Should -Match 'This audit must name every pipeline' + } + } +} + +Describe 'Get-CiFixEventContext' { + It 'admits opened, reopened, synchronize, and matching labeled events' -ForEach @( + @{ Action = 'opened' } + @{ Action = 'reopened' } + @{ Action = 'synchronize' } + @{ Action = 'labeled' } + ) { + $context = Get-CiFixEventContext ` + -Event (New-TestEvent -Action $Action) ` + -Repository dotnet/maui ` + -EventName pull_request_target + + $context.PullRequestNumber | Should -Be 123 + } + + It 'ignores a labeled event for any other label' { + $context = Get-CiFixEventContext ` + -Event (New-TestEvent -Action labeled -EventLabel unrelated) ` + -Repository dotnet/maui ` + -EventName pull_request_target + + $context | Should -BeNullOrEmpty + } + + It 'ignores an otherwise valid event until the required label exists' { + $context = Get-CiFixEventContext ` + -Event (New-TestEvent -Labels @()) ` + -Repository dotnet/maui ` + -EventName pull_request_target + + $context | Should -BeNullOrEmpty + } + + It 'rejects any event type other than pull_request_target' { + { + Get-CiFixEventContext ` + -Event (New-TestEvent) ` + -Repository dotnet/maui ` + -EventName pull_request + } | Should -Throw "*Unexpected event 'pull_request'*" + } + + It 'ignores an unrelated PR before requiring a merge SHA' { + $event = New-TestEvent -Author attacker + $event.pull_request.merge_commit_sha = $null + + $context = Get-CiFixEventContext ` + -Event $event ` + -Repository dotnet/maui ` + -EventName pull_request_target + + $context | Should -BeNullOrEmpty + } + + It 'defers an eligible PR whose merge commit is not available yet' { + $event = New-TestEvent + $event.pull_request.merge_commit_sha = $null + + $context = Get-CiFixEventContext ` + -Event $event ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -WarningAction SilentlyContinue + + $context | Should -BeNullOrEmpty + } +} + +Describe 'Test-TrustedCiFixWorkflowRun' { + It 'accepts only the trusted default-branch fixer completions' -ForEach @( + @{ Name = 'CI Failure Fixer (main)'; Path = '.github/workflows/ci-status-fix.lock.yml' } + @{ Name = 'CI Failure Fixer (net11.0)'; Path = '.github/workflows/ci-status-fix-net11.lock.yml' } + ) { + $event = [pscustomobject]@{ + action = 'completed' + repository = [pscustomobject]@{ full_name = 'dotnet/maui' } + workflow_run = [pscustomobject]@{ + name = $Name + path = $Path + head_branch = 'main' + head_repository = [pscustomobject]@{ full_name = 'dotnet/maui' } + } + } + + Test-TrustedCiFixWorkflowRun -Event $event -Repository dotnet/maui | Should -BeTrue + } + + It 'rejects a matching workflow name from a non-main ref' { + $event = [pscustomobject]@{ + action = 'completed' + repository = [pscustomobject]@{ full_name = 'dotnet/maui' } + workflow_run = [pscustomobject]@{ + name = 'CI Failure Fixer (main)' + path = '.github/workflows/ci-status-fix.lock.yml' + head_branch = 'feature/untrusted' + head_repository = [pscustomobject]@{ full_name = 'dotnet/maui' } + } + } + + Test-TrustedCiFixWorkflowRun -Event $event -Repository dotnet/maui | Should -BeFalse + } +} + +Describe 'Get-OpenCiFixContexts' { + BeforeEach { + Mock Invoke-DispatcherSleep {} + } + + It 'enumerates a top-level REST Object[] and keeps both eligible PRs' { + $script:restPullRequests = @( + (New-TestPullRequest -Number 123), + (New-TestPullRequest ` + -Number 124 ` + -BaseRef net11.0 ` + -HeadRef ci-fix/issue-124 ` + -Title '[ci-fix-net11] Repair CI (refs #124)' ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444'), + (New-TestPullRequest -Number 125 -Author attacker -HeadRef feature/unrelated) + ) + $script:restDetails = @{ + 123 = $script:restPullRequests[0] + 124 = $script:restPullRequests[1] + } + Mock Invoke-WithHttpRetry { + if ($OperationName -like 'GitHub open pull request query page *') { + return ,$script:restPullRequests + } + if ($OperationName -match 'pull request #(?\d+) detail') { + return $script:restDetails[[int]$Matches.number] + } + if ($OperationName -match 'test merge commit (?[0-9a-f]{40})') { + $detail = $script:restDetails.Values | + Where-Object merge_commit_sha -EQ $Matches.sha | + Select-Object -First 1 + return New-TestMergeCommit -MergeSha $Matches.sha -HeadSha $detail.head.sha + } + throw "Unexpected operation '$OperationName'." + } + + $contexts = @( + Get-OpenCiFixContexts ` + -Repository dotnet/maui ` + -GitHubToken token ` + -FixturePath '' + ) + + $contexts.Count | Should -Be 2 + $contexts.PullRequestNumber | Should -Be @(123, 124) + Should -Invoke Invoke-WithHttpRetry -Times 5 -Exactly + Should -Invoke Invoke-WithHttpRetry -Times 0 -Exactly -ParameterFilter { + $OperationName -match 'pull request #125 detail' + } + } + + It 'handles an empty REST array' { + Mock Invoke-WithHttpRetry { return ,@() } + + $contexts = @( + Get-OpenCiFixContexts ` + -Repository dotnet/maui ` + -GitHubToken token ` + -FixturePath '' + ) + + $contexts.Count | Should -Be 0 + Should -Invoke Invoke-WithHttpRetry -Times 1 -Exactly + } + + It 'uses the enumerated count to fetch the next page at the 100 item boundary' { + $script:page = 0 + $script:fullPage = @( + 1..100 | ForEach-Object { + New-TestPullRequest -Number (1000 + $_) -Author attacker -HeadRef "feature/unrelated-$_" + } + ) + Mock Invoke-WithHttpRetry { + $script:page++ + if ($script:page -eq 1) { + return ,$script:fullPage + } + return ,@() + } + + $contexts = @( + Get-OpenCiFixContexts ` + -Repository dotnet/maui ` + -GitHubToken token ` + -FixturePath '' + ) + + $contexts.Count | Should -Be 0 + Should -Invoke Invoke-WithHttpRetry -Times 2 -Exactly + } + + It 'refreshes a stale list response and verifies the current head and test merge pair' { + $listPullRequest = New-TestPullRequest ` + -HeadSha '1111111111111111111111111111111111111111' ` + -MergeSha '2222222222222222222222222222222222222222' + $staleDetailPullRequest = New-TestPullRequest ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '2222222222222222222222222222222222222222' + $freshDetailPullRequest = New-TestPullRequest ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444' + $script:detailRefresh = 0 + Mock Invoke-WithHttpRetry { + if ($OperationName -like 'GitHub open pull request query page *') { + return ,@($listPullRequest) + } + if ($OperationName -like '*detail refresh') { + $script:detailRefresh++ + if ($script:detailRefresh -eq 1) { + return $staleDetailPullRequest + } + return $freshDetailPullRequest + } + if ($OperationName -like 'GitHub test merge commit 2222*') { + return New-TestMergeCommit ` + -MergeSha '2222222222222222222222222222222222222222' ` + -HeadSha '1111111111111111111111111111111111111111' + } + if ($OperationName -like 'GitHub test merge commit 4444*') { + return New-TestMergeCommit ` + -MergeSha '4444444444444444444444444444444444444444' ` + -HeadSha '3333333333333333333333333333333333333333' + } + throw "Unexpected operation '$OperationName'." + } + + $contexts = @(Get-OpenCiFixContexts -Repository dotnet/maui -GitHubToken token -FixturePath '') + + $contexts.Count | Should -Be 1 + $contexts[0].HeadSha | Should -Be '3333333333333333333333333333333333333333' + $contexts[0].MergeSha | Should -Be '4444444444444444444444444444444444444444' + Should -Invoke Invoke-DispatcherSleep -Times 1 -Exactly + } + + It 'fails closed after bounded refreshes when the test merge remains stale' { + $pullRequest = New-TestPullRequest + Mock Invoke-WithHttpRetry { + if ($OperationName -like 'GitHub open pull request query page *') { + return ,@($pullRequest) + } + if ($OperationName -like '*detail refresh') { + return $pullRequest + } + if ($OperationName -like 'GitHub test merge commit *') { + return New-TestMergeCommit ` + -MergeSha $pullRequest.merge_commit_sha ` + -HeadSha '3333333333333333333333333333333333333333' + } + throw "Unexpected operation '$OperationName'." + } + + $contexts = @(Get-OpenCiFixContexts -Repository dotnet/maui -GitHubToken token -FixturePath '') + + $contexts.Count | Should -Be 1 + $contexts[0].VerificationError | + Should -BeLike "*PR #123 head '1111111111111111111111111111111111111111' did not obtain a verified test merge after 4 attempts*source parent '3333333333333333333333333333333333333333' does not match head*No Azure DevOps validation was queued or deduplicated*" + + Should -Invoke Invoke-WithHttpRetry -Times 4 -Exactly -ParameterFilter { + $OperationName -like '*detail refresh' + } + Should -Invoke Invoke-WithHttpRetry -Times 4 -Exactly -ParameterFilter { + $OperationName -like 'GitHub test merge commit *' + } + Should -Invoke Invoke-DispatcherSleep -Times 3 -Exactly + } + + It 'uses an individually refreshed revocation as authoritative' { + $listPullRequest = New-TestPullRequest + $revokedPullRequest = New-TestPullRequest -Labels @() + Mock Invoke-WithHttpRetry { + if ($OperationName -like 'GitHub open pull request query page *') { + return ,@($listPullRequest) + } + if ($OperationName -like '*detail refresh') { + return $revokedPullRequest + } + throw "Unexpected operation '$OperationName'." + } + + $contexts = @(Get-OpenCiFixContexts -Repository dotnet/maui -GitHubToken token -FixturePath '') + + $contexts.Count | Should -Be 0 + Should -Invoke Invoke-WithHttpRetry -Times 0 -Exactly -ParameterFilter { + $OperationName -like 'GitHub test merge commit *' + } + } + + It 'stops stale merge refreshes when the shared dispatcher budget expires' { + $pullRequest = New-TestPullRequest + Mock Invoke-WithHttpRetry { + if ($OperationName -like 'GitHub open pull request query page *') { + return ,@($pullRequest) + } + if ($OperationName -like '*detail refresh') { + return $pullRequest + } + if ($OperationName -like 'GitHub test merge commit *') { + return New-TestMergeCommit ` + -MergeSha $pullRequest.merge_commit_sha ` + -HeadSha '3333333333333333333333333333333333333333' + } + throw "Unexpected operation '$OperationName'." + } + Mock Invoke-DispatcherSleep { + throw "$($script:DispatcherBudgetPrefix) No retry time remains." + } + + { + Get-OpenCiFixContexts -Repository dotnet/maui -GitHubToken token -FixturePath '' + } | Should -Throw '*dispatcher-budget-exhausted*' + + Should -Invoke Invoke-WithHttpRetry -Times 1 -Exactly -ParameterFilter { + $OperationName -like '*detail refresh' + } + Should -Invoke Invoke-WithHttpRetry -Times 1 -Exactly -ParameterFilter { + $OperationName -like 'GitHub test merge commit *' + } + } + + It 'rejects malformed test merge metadata before a context can be returned' -ForEach @( + @{ + Name = 'wrong commit identity' + Commit = [pscustomobject]@{ + sha = '5555555555555555555555555555555555555555' + parents = @( + [pscustomobject]@{ sha = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }, + [pscustomobject]@{ sha = '1111111111111111111111111111111111111111' } + ) + } + Expected = '*requested merge*returned commit*' + }, + @{ + Name = 'missing source parent' + Commit = [pscustomobject]@{ + sha = '2222222222222222222222222222222222222222' + parents = @([pscustomobject]@{ sha = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }) + } + Expected = '*has 1 parent(s), expected exactly 2*' + }, + @{ + Name = 'wrong source parent' + Commit = [pscustomobject]@{ + sha = '2222222222222222222222222222222222222222' + parents = @( + [pscustomobject]@{ sha = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' }, + [pscustomobject]@{ sha = '3333333333333333333333333333333333333333' } + ) + } + Expected = '*source parent*does not match head*' + }, + @{ + Name = 'malformed base parent' + Commit = [pscustomobject]@{ + sha = '2222222222222222222222222222222222222222' + parents = @( + [pscustomobject]@{ sha = '' }, + [pscustomobject]@{ sha = '1111111111111111111111111111111111111111' } + ) + } + Expected = '*has an invalid first parent*' + } + ) { + $pullRequest = New-TestPullRequest + Mock Invoke-WithHttpRetry { + if ($OperationName -like 'GitHub open pull request query page *') { + return ,@($pullRequest) + } + if ($OperationName -like '*detail refresh') { + return $pullRequest + } + if ($OperationName -like 'GitHub test merge commit *') { + return $Commit + } + throw "Unexpected operation '$OperationName'." + } + + $contexts = @(Get-OpenCiFixContexts -Repository dotnet/maui -GitHubToken token -FixturePath '') + + $contexts.Count | Should -Be 1 + $contexts[0].VerificationError | Should -BeLike $Expected + + Should -Invoke Invoke-DispatcherSleep -Times 3 -Exactly + } +} + +Describe 'Get-AzdoToken' { + BeforeEach { + $script:oldTenant = $env:AZDO_TRIGGER_TENANT_ID + $script:oldClient = $env:AZDO_TRIGGER_CLIENT_ID + $script:oldRequestToken = $env:ACTIONS_ID_TOKEN_REQUEST_TOKEN + $script:oldRequestUrl = $env:ACTIONS_ID_TOKEN_REQUEST_URL + $env:AZDO_TRIGGER_TENANT_ID = 'tenant' + $env:AZDO_TRIGGER_CLIENT_ID = 'client' + $env:ACTIONS_ID_TOKEN_REQUEST_TOKEN = 'request-token' + $env:ACTIONS_ID_TOKEN_REQUEST_URL = 'https://example.test/oidc?x=1' + + Mock Write-Host {} + Mock Invoke-WithHttpRetry { + if ($OperationName -eq 'GitHub OIDC token request') { + return [pscustomobject]@{ value = 'oidc-token' } + } + if ($OperationName -eq 'Azure AD token exchange') { + return [pscustomobject]@{ access_token = 'azdo-token' } + } + throw "Unexpected operation $OperationName" + } + } + + AfterEach { + $env:AZDO_TRIGGER_TENANT_ID = $script:oldTenant + $env:AZDO_TRIGGER_CLIENT_ID = $script:oldClient + $env:ACTIONS_ID_TOKEN_REQUEST_TOKEN = $script:oldRequestToken + $env:ACTIONS_ID_TOKEN_REQUEST_URL = $script:oldRequestUrl + } + + It 'returns exactly one success-stream value containing only the access token' { + $result = @(Get-AzdoToken) + + $result.Count | Should -Be 1 + $result[0] | Should -BeExactly 'azdo-token' + Should -Invoke Write-Host -Times 2 -Exactly + } +} + +Describe 'New-AzdoQueueRequest' { + It 'queues the PR merge ref and merge commit while preserving the source head identity' { + $context = Get-CiFixEventContext ` + -Event (New-TestEvent) ` + -Repository dotnet/maui ` + -EventName pull_request_target + + $request = New-AzdoQueueRequest -DefinitionId 302 -Context $context + + $request.definition.id | Should -Be 302 + $request.reason | Should -Be 'pullRequest' + $request.sourceBranch | Should -Be 'refs/pull/123/merge' + $request.sourceVersion | Should -Be '2222222222222222222222222222222222222222' + $request.triggerInfo.'pr.sourceSha' | Should -Be '1111111111111111111111111111111111111111' + $request.triggerInfo.'pr.targetBranch' | Should -Be 'main' + $request.triggerInfo.'pr.number' | Should -Be '123' + $request.triggerInfo.'pr.providerId' | Should -Be 'github' + } + + It 'sets the bare net11 target branch in trigger metadata' { + $context = Get-CiFixEventContext ` + -Event (New-TestEvent ` + -BaseRef net11.0 ` + -Title '[ci-fix-net11] Repair CI' ` + -HeadRef ci-fix/issue-124) ` + -Repository dotnet/maui ` + -EventName pull_request_target + + $request = New-AzdoQueueRequest -DefinitionId 302 -Context $context + + $request.triggerInfo.'pr.targetBranch' | Should -Be 'net11.0' + } + + It 'serializes producer-equivalent system pull request parameters for both target branches' { + foreach ($targetBranch in @('main', 'net11.0')) { + $title = if ($targetBranch -eq 'main') { '[ci-fix] Repair CI' } else { '[ci-fix-net11] Repair CI' } + $headRef = if ($targetBranch -eq 'main') { 'ci-fix/issue-123' } else { 'ci-fix/issue-124' } + $context = Get-CiFixEventContext ` + -Event (New-TestEvent -BaseRef $targetBranch -Title $title -HeadRef $headRef) ` + -Repository dotnet/maui ` + -EventName pull_request_target + $request = New-AzdoQueueRequest -DefinitionId 302 -Context $context + $outerJson = $request | ConvertTo-Json -Depth 10 -Compress + $outerRequest = $outerJson | ConvertFrom-Json -Depth 10 + + $outerRequest.parameters.GetType() | Should -Be ([string]) + $parameters = $outerRequest.parameters | ConvertFrom-Json + @($parameters.PSObject.Properties.Name | Sort-Object) | Should -Be @( + 'system.pullRequest.isFork', + 'system.pullRequest.mergedAt', + 'system.pullRequest.pullRequestId', + 'system.pullRequest.pullRequestNumber', + 'system.pullRequest.sourceBranch', + 'system.pullRequest.sourceCommitId', + 'system.pullRequest.sourceRepositoryUri', + 'system.pullRequest.targetBranch', + 'system.pullRequest.targetBranchName' + ) + + $parameters.'system.pullRequest.pullRequestId' | Should -Be '456912' + $parameters.'system.pullRequest.pullRequestNumber' | Should -Be '123' + $parameters.'system.pullRequest.mergedAt' | Should -Be '' + $parameters.'system.pullRequest.sourceBranch' | Should -Be $headRef + $parameters.'system.pullRequest.targetBranch' | Should -Be $targetBranch + $parameters.'system.pullRequest.targetBranchName' | Should -Be $targetBranch + $parameters.'system.pullRequest.sourceRepositoryUri' | Should -Be 'https://github.com/dotnet/maui' + $parameters.'system.pullRequest.sourceCommitId' | Should -Be '1111111111111111111111111111111111111111' + $parameters.'system.pullRequest.isFork' | Should -Be 'False' + foreach ($property in $parameters.PSObject.Properties) { + $property.Value.GetType() | Should -Be ([string]) + } + $outerRequest.sourceVersion | Should -Be '2222222222222222222222222222222222222222' + $parameters.'system.pullRequest.sourceCommitId' | Should -Not -Be $outerRequest.sourceVersion + } + } + + It 'defines all three MAUI validation pipelines' { + $pipelines = @(Get-CiFixPipelineDefinitions) + $pipelines.Name | Should -Be @('maui-pr', 'maui-pr-uitests', 'maui-pr-devicetests') + $pipelines.DefinitionId | Should -Be @(302, 313, 314) + } +} + +Describe 'Find-AzdoDuplicateBuild' { + It 'deduplicates any prior build for the same pipeline PR head regardless of completion state' { + $builds = @( + [pscustomobject]@{ + id = 9001 + sourceBranch = 'refs/pull/123/merge' + sourceVersion = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + status = 'completed' + result = 'failed' + triggerInfo = [pscustomobject]@{ + 'pr.number' = '123' + 'pr.sourceSha' = '1111111111111111111111111111111111111111' + } + } + ) + + $duplicate = Find-AzdoDuplicateBuild ` + -Builds $builds ` + -PullRequestNumber 123 ` + -HeadSha '1111111111111111111111111111111111111111' ` + -MergeSha '2222222222222222222222222222222222222222' + + $duplicate.id | Should -Be 9001 + } + + It 'does not deduplicate an older head on the same PR' { + $builds = @( + [pscustomobject]@{ + id = 9001 + sourceBranch = 'refs/pull/123/merge' + sourceVersion = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + triggerInfo = [pscustomobject]@{ + 'pr.number' = '123' + 'pr.sourceSha' = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' + } + } + + ) + + Find-AzdoDuplicateBuild ` + -Builds $builds ` + -PullRequestNumber 123 ` + -HeadSha '1111111111111111111111111111111111111111' ` + -MergeSha '2222222222222222222222222222222222222222' | + Should -BeNullOrEmpty + } + + It 'does not deduplicate sourceVersion-only legacy metadata' { + $builds = @( + [pscustomobject]@{ + id = 9002 + sourceBranch = 'refs/pull/123/merge' + sourceVersion = '2222222222222222222222222222222222222222' + triggerInfo = [pscustomobject]@{} + }, + [pscustomobject]@{ + id = 9003 + sourceBranch = 'refs/pull/123/merge' + sourceVersion = '1111111111111111111111111111111111111111' + triggerInfo = [pscustomobject]@{ + 'pr.number' = '123' + 'pr.sourceSha' = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' + } + } + ) + + Find-AzdoDuplicateBuild ` + -Builds $builds ` + -PullRequestNumber 123 ` + -HeadSha '1111111111111111111111111111111111111111' ` + -MergeSha '2222222222222222222222222222222222222222' | + Should -BeNullOrEmpty + } + + It 'does not let a previous-head build deduplicate a current verified head through an old merge SHA' { + $builds = @( + [pscustomobject]@{ + id = 9004 + sourceBranch = 'refs/pull/123/merge' + sourceVersion = '2222222222222222222222222222222222222222' + triggerInfo = [pscustomobject]@{ + 'pr.number' = '123' + 'pr.sourceSha' = '1111111111111111111111111111111111111111' + } + } + ) + + Find-AzdoDuplicateBuild ` + -Builds $builds ` + -PullRequestNumber 123 ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444' | + Should -BeNullOrEmpty + } +} + +Describe 'Invoke-AzdoPipelineQueue retry safety' { + BeforeAll { + $queueFunctionDefinitions = @( + 'Get-ObjectPropertyValue', + 'Get-DispatcherElapsedSeconds', + 'Get-DispatcherRemainingSeconds', + 'Test-IsDispatcherBudgetException', + 'Get-DispatcherHttpTimeoutSeconds', + 'Invoke-DispatcherSleep', + 'Get-CiFixPipelineDefinitions', + 'Test-IsTransientHttpException', + 'Get-HttpStatusCode', + 'Invoke-WithHttpRetry', + 'Resolve-VerifiedCiFixContext', + 'Find-AzdoDuplicateBuild', + 'Get-AzdoDuplicateBuild', + 'New-AzdoQueueRequest', + 'Invoke-AzdoPipelineQueue', + 'Write-CiFixJobSummary', + 'New-CiFixFailureResult', + 'Invoke-CiFixQueueWork' + ) | ForEach-Object { + $functionName = $_ + $ast.Find({ + $args[0] -is [System.Management.Automation.Language.FunctionDefinitionAst] -and + $args[0].Name -eq $functionName + }, $true).Extent.Text + } + $script:queueTestModule = New-Module -Name QueueCiFixAzdoValidationTest -ScriptBlock { + param([string[]]$FunctionDefinitions) + + $script:AzureDevOpsOrganization = 'dnceng-public' + $script:AzureDevOpsProject = 'public' + $script:TransientHttpStatusCodes = @(408, 429, 500, 502, 503, 504) + $script:MaxHttpAttempts = 4 + $script:RetryBaseDelaySeconds = 2 + $script:DispatcherBudgetSeconds = 480 + $script:DispatcherStopwatch = [System.Diagnostics.Stopwatch]::StartNew() + $script:DispatcherBudgetPrefix = '[dispatcher-budget-exhausted]' + foreach ($definition in $FunctionDefinitions) { + Invoke-Expression $definition + } + } -ArgumentList (, $queueFunctionDefinitions) + Import-Module $script:queueTestModule -Prefix QueueTest -Force + } + + AfterAll { + Remove-Module $script:queueTestModule -Force + } + + Context 'behavioral paths' { + BeforeEach { + $script:queueContext = [pscustomobject]@{ + PullRequestNumber = 123 + PullRequestId = 456789 + Draft = $true + Title = '[ci-fix] Repair CI (refs #123)' + BaseRef = 'main' + HeadRef = 'ci-fix/issue-123' + HeadSha = '1111111111111111111111111111111111111111' + MergeSha = '2222222222222222222222222222222222222222' + } + Mock Start-Sleep {} -ModuleName QueueCiFixAzdoValidationTest + Mock Get-DispatcherElapsedSeconds { return 0 } -ModuleName QueueCiFixAzdoValidationTest + Mock Resolve-VerifiedCiFixContext { + return $NominatedContext + } -ModuleName QueueCiFixAzdoValidationTest + } + + It 'returns a successful POST without duplicate reconciliation' { + $script:postedBuild = [pscustomobject]@{ id = 7001 } + Mock Invoke-RestMethod { return $script:postedBuild } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { + throw 'Duplicate lookup must not run after a successful POST.' + } -ModuleName QueueCiFixAzdoValidationTest + + $result = Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + + $result.Build.id | Should -Be 7001 + $result.Reconciled | Should -BeFalse + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Post' } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + Should -Invoke Start-Sleep -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + } + + It 'reconciles a timed-out POST to the exact build without retrying the POST' { + $script:duplicateCalls = 0 + $script:correlatedBuild = [pscustomobject]@{ id = 7002 } + Mock Invoke-RestMethod { + throw [System.TimeoutException]::new('queue response timed out') + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { + $script:duplicateCalls++ + if ($script:duplicateCalls -eq 2) { + return $script:correlatedBuild + } + return $null + } -ModuleName QueueCiFixAzdoValidationTest + + $result = Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + + $result.Build.id | Should -Be 7002 + $result.Reconciled | Should -BeTrue + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Post' } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + Should -Invoke Start-Sleep -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + } + + It 'recognizes the PowerShell timeout exception chain and reconciles exactly once' { + $socket = [System.Net.Sockets.SocketException]::new([System.Net.Sockets.SocketError]::TimedOut) + $io = [System.IO.IOException]::new('socket timed out', $socket) + $innerCanceled = [System.Threading.Tasks.TaskCanceledException]::new('socket read canceled', $io) + $timeout = [System.TimeoutException]::new('request timed out', $innerCanceled) + $canceled = [System.Threading.Tasks.TaskCanceledException]::new('request canceled', $timeout) + $script:exactBuild = [pscustomobject]@{ + id = 7004 + sourceBranch = 'refs/pull/123/merge' + sourceVersion = '2222222222222222222222222222222222222222' + triggerInfo = [pscustomobject]@{ + 'pr.number' = '123' + 'pr.sourceSha' = '1111111111111111111111111111111111111111' + } + } + $script:wrongBuild = [pscustomobject]@{ + id = 7005 + sourceBranch = 'refs/pull/123/merge' + sourceVersion = 'aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa' + triggerInfo = [pscustomobject]@{ + 'pr.number' = '123' + 'pr.sourceSha' = 'bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb' + } + } + Mock Invoke-RestMethod { + if ($Method -eq 'Post') { + throw $canceled + } + return [pscustomobject]@{ value = @($script:wrongBuild, $script:exactBuild) } + } -ModuleName QueueCiFixAzdoValidationTest + + (Test-QueueTestIsTransientHttpException -Exception $canceled) | Should -BeTrue + $result = Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + + $result.Build.id | Should -Be 7004 + $result.Reconciled | Should -BeTrue + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Post' } + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Get' } + } + + It 'keeps nested timeout POST acceptance uncertain when the exact reconciliation read fails' { + $socket = [System.Net.Sockets.SocketException]::new([System.Net.Sockets.SocketError]::TimedOut) + $io = [System.IO.IOException]::new('socket timed out', $socket) + $innerCanceled = [System.Threading.Tasks.TaskCanceledException]::new('socket read canceled', $io) + $timeout = [System.TimeoutException]::new('request timed out', $innerCanceled) + $canceled = [System.Threading.Tasks.TaskCanceledException]::new('request canceled', $timeout) + Mock Invoke-RestMethod { + if ($Method -eq 'Post') { + throw $canceled + } + throw [System.InvalidOperationException]::new('duplicate query unavailable') + } -ModuleName QueueCiFixAzdoValidationTest + + { + Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + } | Should -Throw '*may have been accepted*exact reconciliation failed: duplicate query unavailable*POST was issued exactly once*acceptance remains uncertain*' + + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Post' } + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Get' } + } + + It 'does not classify plain cancellation or authentication failures as transient' { + (Test-QueueTestIsTransientHttpException -Exception ([System.Threading.Tasks.TaskCanceledException]::new('caller canceled'))) | Should -BeFalse + $unauthorized = [System.Net.Http.HttpRequestException]::new( + 'unauthorized', + $null, + [System.Net.HttpStatusCode]::Unauthorized) + (Test-QueueTestIsTransientHttpException -Exception $unauthorized) | Should -BeFalse + $wrappedUnauthorized = [System.Net.Http.HttpRequestException]::new( + 'unauthorized', + [System.TimeoutException]::new('inner timeout'), + [System.Net.HttpStatusCode]::Unauthorized) + (Test-QueueTestIsTransientHttpException -Exception $wrappedUnauthorized) | Should -BeFalse + } + + It 'reconciles a 5xx POST to the exact build without retrying the POST' { + $script:correlatedBuild = [pscustomobject]@{ id = 7003 } + Mock Invoke-RestMethod { + $exception = [System.Exception]::new('service unavailable') + $exception | Add-Member -NotePropertyName StatusCode -NotePropertyValue 503 + throw $exception + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { + return $script:correlatedBuild + } -ModuleName QueueCiFixAzdoValidationTest + + $result = Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 313 ` + -Context $script:queueContext ` + -AuthToken test-token + + $result.Build.id | Should -Be 7003 + $result.Reconciled | Should -BeTrue + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Post' } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + Should -Invoke Start-Sleep -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + } + + It 'throws after bounded reconciliation when no exact build appears' { + Mock Invoke-RestMethod { + throw [System.TimeoutException]::new('queue response timed out') + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { return $null } -ModuleName QueueCiFixAzdoValidationTest + + { + Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 314 ` + -Context $script:queueContext ` + -AuthToken test-token + } | Should -Throw '*may have been accepted*no exact correlated build appeared after reconciliation*POST was issued exactly once*acceptance remains uncertain*' + + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Post' } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 4 -Exactly + Should -Invoke Start-Sleep -ModuleName QueueCiFixAzdoValidationTest -Times 4 -Exactly + } + + It 'preserves possible acceptance when reconciliation itself fails' { + Mock Invoke-RestMethod { + throw [System.TimeoutException]::new('queue response timed out') + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { + throw [System.InvalidOperationException]::new('duplicate lookup unavailable') + } -ModuleName QueueCiFixAzdoValidationTest + + { + Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + } | Should -Throw '*may have been accepted*exact reconciliation failed: duplicate lookup unavailable*POST was issued exactly once*acceptance remains uncertain*' + + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { + $Method -eq 'Post' + } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + } + + It 'throws a nontransient POST failure without reconciliation or retry' { + Mock Invoke-RestMethod { + throw [System.InvalidOperationException]::new('invalid request') + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { + throw 'Duplicate lookup must not run for a nontransient failure.' + } -ModuleName QueueCiFixAzdoValidationTest + + { + Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + } | Should -Throw '*invalid request*' + + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { $Method -eq 'Post' } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + Should -Invoke Start-Sleep -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + } + + It 'caps HTTP timeouts and retry sleeps to the shared remaining budget' { + Mock Get-DispatcherElapsedSeconds { return 455 } -ModuleName QueueCiFixAzdoValidationTest + + Get-QueueTestDispatcherHttpTimeoutSeconds -OperationName 'bounded request' | + Should -Be 25 + + Mock Get-DispatcherElapsedSeconds { return 475 } -ModuleName QueueCiFixAzdoValidationTest + Invoke-QueueTestDispatcherSleep -OperationName 'bounded sleep' -RequestedSeconds 10 + + Should -Invoke Start-Sleep -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { + $Seconds -eq 4 + } + } + + It 'prevents a queue POST after the shared budget expires' { + Mock Get-DispatcherElapsedSeconds { return 480 } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-RestMethod { throw 'HTTP must not run after budget expiry.' } -ModuleName QueueCiFixAzdoValidationTest + + { + Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + } | Should -Throw '*dispatcher-budget-exhausted*before*queue POST*' + + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + } + + It 'prevents nested HTTP retries from overrunning the shared budget' { + $script:elapsedCalls = 0 + Mock Get-DispatcherElapsedSeconds { + $script:elapsedCalls++ + if ($script:elapsedCalls -le 2) { return 470 } + return 479.5 + } -ModuleName QueueCiFixAzdoValidationTest + $script:operationCalls = 0 + + { + Invoke-QueueTestWithHttpRetry -OperationName 'nested retry' -Operation { + param($timeoutSeconds) + $script:operationCalls++ + $timeoutSeconds | Should -Be 10 + throw [System.TimeoutException]::new('transient') + } + } | Should -Throw '*dispatcher-budget-exhausted*' + + $script:operationCalls | Should -Be 1 + Should -Invoke Start-Sleep -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { + $Seconds -eq 2 + } + } + + It 'keeps an ambiguous POST uncertain when reconciliation exhausts the budget' { + Mock Invoke-RestMethod { + throw [System.TimeoutException]::new('queue response timed out') + } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-DispatcherSleep { + throw '[dispatcher-budget-exhausted] no reconciliation time remains' + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { + throw 'Duplicate lookup must not run after sleep exhausts the budget.' + } -ModuleName QueueCiFixAzdoValidationTest + + { + Invoke-QueueTestAzdoPipelineQueue ` + -DefinitionId 302 ` + -Context $script:queueContext ` + -AuthToken test-token + } | Should -Throw '*may have been accepted*POST was issued exactly once*acceptance remains uncertain*' + + Should -Invoke Invoke-RestMethod -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { + $Method -eq 'Post' + } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + } + + It 'preserves completed results and marks all remaining work failed after budget exhaustion' { + $secondContext = $script:queueContext.PSObject.Copy() + $secondContext.PullRequestNumber = 124 + $secondContext.PullRequestId = 456913 + $secondContext.HeadSha = '3333333333333333333333333333333333333333' + $secondContext.MergeSha = '4444444444444444444444444444444444444444' + $script:queueCalls = 0 + Mock Get-AzdoDuplicateBuild { return $null } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + $script:queueCalls++ + if ($script:queueCalls -eq 1) { + return [pscustomobject]@{ + Build = [pscustomobject]@{ id = 8001 } + Reconciled = $false + } + } + throw '[dispatcher-budget-exhausted] queue budget expired before the next POST' + } -ModuleName QueueCiFixAzdoValidationTest + Mock Write-CiFixJobSummary {} -ModuleName QueueCiFixAzdoValidationTest + + $results = @( + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($script:queueContext, $secondContext) ` + -AuthToken test-token + ) + + $results.Count | Should -Be 6 + $results[0].Outcome | Should -Be 'queued' + $results[0].BuildId | Should -Be 8001 + @($results[1..5].Outcome | Sort-Object -Unique) | Should -Be @('failed') + $results[1].Error | Should -Match 'dispatcher-budget-exhausted' + $results[1].Error | Should -Match 'queue budget expired before the next POST' + foreach ($result in $results[2..5]) { + $result.Error | Should -Match "PR #$($result.PullRequestNumber) pipeline '$([regex]::Escape($result.Name))' was not processed" + $result.Error | Should -Match 'No queue POST was attempted for this work item' + $result.Error | Should -Match "while processing PR #123 pipeline 'maui-pr-uitests'" + $result.Error | Should -Not -Match 'may have been accepted' + } + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + Should -Invoke Write-CiFixJobSummary -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly -ParameterFilter { + @($Results).Count -eq 3 + } + } + + It 'preserves completed results when queue-time revalidation exhausts the budget' { + $secondContext = $script:queueContext.PSObject.Copy() + $secondContext.PullRequestNumber = 124 + $secondContext.PullRequestId = 456913 + $secondContext.HeadSha = '3333333333333333333333333333333333333333' + $secondContext.MergeSha = '4444444444444444444444444444444444444444' + $script:verificationCalls = 0 + Mock Resolve-VerifiedCiFixContext { + $script:verificationCalls++ + if ($script:verificationCalls -eq 1) { + return $NominatedContext + } + throw '[dispatcher-budget-exhausted] queue-time PR verification exceeded the shared deadline' + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { return $null } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + return [pscustomobject]@{ + Build = [pscustomobject]@{ id = 8101 } + Reconciled = $false + } + } -ModuleName QueueCiFixAzdoValidationTest + Mock Write-CiFixJobSummary {} -ModuleName QueueCiFixAzdoValidationTest + + $results = @( + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($script:queueContext, $secondContext) ` + -AuthToken test-token + ) + + $results.Count | Should -Be 6 + $results[0].Outcome | Should -Be 'queued' + $results[0].BuildId | Should -Be 8101 + $results[1].Outcome | Should -Be 'failed' + $results[1].Error | Should -Match 'dispatcher-budget-exhausted' + $results[1].Error | Should -Match 'queue-time PR verification exceeded the shared deadline' + foreach ($result in $results[2..5]) { + $result.Outcome | Should -Be 'failed' + $result.Error | Should -Match "PR #$($result.PullRequestNumber) pipeline '$([regex]::Escape($result.Name))' was not processed" + $result.Error | Should -Match "while processing PR #123 pipeline 'maui-pr-uitests'" + $result.Error | Should -Match 'No queue POST was attempted for this work item' + } + Should -Invoke Resolve-VerifiedCiFixContext -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + Should -Invoke Write-CiFixJobSummary -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly -ParameterFilter { + @($Results).Count -eq 3 + } + } + + It 'reports a nonbudget queue-time revalidation failure without losing completed results' { + $script:verificationCalls = 0 + Mock Resolve-VerifiedCiFixContext { + $script:verificationCalls++ + if ($script:verificationCalls -eq 2) { + throw 'GitHub queue-time PR refresh failed' + } + return $NominatedContext + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { return $null } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + return [pscustomobject]@{ + Build = [pscustomobject]@{ id = 8200 + $DefinitionId } + Reconciled = $false + } + } -ModuleName QueueCiFixAzdoValidationTest + Mock Write-CiFixJobSummary {} -ModuleName QueueCiFixAzdoValidationTest + + $results = @( + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($script:queueContext) ` + -AuthToken test-token + ) + + $results.Count | Should -Be 3 + $results[0].Outcome | Should -Be 'queued' + $results[0].BuildId | Should -Be 8502 + $results[1].Outcome | Should -Be 'failed' + $results[1].Error | Should -Be 'GitHub queue-time PR refresh failed' + $results[2].Outcome | Should -Be 'queued' + $results[2].BuildId | Should -Be 8514 + Should -Invoke Resolve-VerifiedCiFixContext -ModuleName QueueCiFixAzdoValidationTest -Times 3 -Exactly + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + Should -Invoke Write-CiFixJobSummary -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly -ParameterFilter { + @($Results).Count -eq 3 + } + } + + It 'keeps ambiguous uncertainty on only the submitted POST and skips every later item accurately' { + $secondContext = $script:queueContext.PSObject.Copy() + $secondContext.PullRequestNumber = 124 + $secondContext.PullRequestId = 456913 + $secondContext.HeadSha = '3333333333333333333333333333333333333333' + $secondContext.MergeSha = '4444444444444444444444444444444444444444' + Mock Get-AzdoDuplicateBuild { return $null } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + throw '[dispatcher-budget-exhausted] Azure DevOps queue request for definition 302 may have been accepted, but reconciliation expired. The POST was issued exactly once and acceptance remains uncertain.' + } -ModuleName QueueCiFixAzdoValidationTest + Mock Write-CiFixJobSummary {} -ModuleName QueueCiFixAzdoValidationTest + + $results = @( + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($script:queueContext, $secondContext) ` + -AuthToken test-token + ) + + $results.Count | Should -Be 6 + $results[0].Error | Should -Match 'definition 302 may have been accepted' + $results[0].Error | Should -Match 'acceptance remains uncertain' + foreach ($result in $results[1..5]) { + $result.Error | Should -Match "PR #$($result.PullRequestNumber) pipeline '$([regex]::Escape($result.Name))' was not processed" + $result.Error | Should -Match 'No queue POST was attempted for this work item' + $result.Error | Should -Not -Match 'may have been accepted' + $result.Error | Should -Not -Match 'acceptance remains uncertain' + } + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + Should -Invoke Write-CiFixJobSummary -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + } + + It 'isolates an unverifiable PR while processing every pipeline for the next verified head' { + $unverifiedContext = $script:queueContext.PSObject.Copy() + $unverifiedContext | Add-Member ` + -NotePropertyName VerificationError ` + -NotePropertyValue "Eligible CI-fix PR #123 head '$($unverifiedContext.HeadSha)' did not obtain a verified test merge. No Azure DevOps validation was queued or deduplicated for this PR." + $verifiedContext = $script:queueContext.PSObject.Copy() + $verifiedContext.PullRequestNumber = 124 + $verifiedContext.PullRequestId = 456913 + $verifiedContext.BaseRef = 'net11.0' + $verifiedContext.HeadRef = 'ci-fix/issue-124' + $verifiedContext.HeadSha = '3333333333333333333333333333333333333333' + $verifiedContext.MergeSha = '4444444444444444444444444444444444444444' + Mock Get-AzdoDuplicateBuild { return $null } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + return [pscustomobject]@{ + Build = [pscustomobject]@{ id = 8000 + $DefinitionId } + Reconciled = $false + } + } -ModuleName QueueCiFixAzdoValidationTest + Mock Write-CiFixJobSummary {} -ModuleName QueueCiFixAzdoValidationTest + + $results = @( + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($unverifiedContext, $verifiedContext) ` + -AuthToken test-token + ) + + $results.Count | Should -Be 6 + @($results[0..2].Outcome | Select-Object -Unique) | Should -Be @('failed') + foreach ($result in $results[0..2]) { + $result.PullRequestNumber | Should -Be 123 + $result.Error | Should -Match 'did not obtain a verified test merge' + $result.Error | Should -Match 'No Azure DevOps validation was queued or deduplicated' + } + @($results[3..5].Outcome | Select-Object -Unique) | Should -Be @('queued') + @($results[3..5].PullRequestNumber | Select-Object -Unique) | Should -Be @(124) + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 3 -Exactly -ParameterFilter { + $PullRequestNumber -eq 124 + } + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly -ParameterFilter { + $PullRequestNumber -eq 123 + } + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 3 -Exactly -ParameterFilter { + $Context.PullRequestNumber -eq 124 + } + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly -ParameterFilter { + $Context.PullRequestNumber -eq 123 + } + Should -Invoke Write-CiFixJobSummary -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + } + + It 'rejects an empty token before verified work can make any Azure request' { + Mock Get-AzdoDuplicateBuild { + throw 'Azure duplicate lookup must not run without authentication.' + } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + throw 'Azure queue POST must not run without authentication.' + } -ModuleName QueueCiFixAzdoValidationTest + + { + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($script:queueContext) ` + -AuthToken '' + } | Should -Throw '*authentication is required before processing verified PR #123*' + + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + } + + It 'stops remaining pipelines when queue-time verification observes a newer head' { + $script:verificationCalls = 0 + Mock Resolve-VerifiedCiFixContext { + $script:verificationCalls++ + if ($script:verificationCalls -eq 1) { + return $NominatedContext + } + $changedContext = $NominatedContext.PSObject.Copy() + $changedContext.HeadSha = '3333333333333333333333333333333333333333' + $changedContext.MergeSha = '4444444444444444444444444444444444444444' + return $changedContext + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { return $null } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + return [pscustomobject]@{ + Build = [pscustomobject]@{ id = 7302 } + Reconciled = $false + } + } -ModuleName QueueCiFixAzdoValidationTest + Mock Write-CiFixJobSummary {} -ModuleName QueueCiFixAzdoValidationTest + + $results = @( + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($script:queueContext) ` + -AuthToken test-token + ) + + $results.Count | Should -Be 3 + $results[0].Outcome | Should -Be 'queued' + foreach ($result in $results[1..2]) { + $result.Outcome | Should -Be 'failed' + $result.Error | Should -Match 'changed after discovery' + $result.Error | Should -Match 'No queue POST was attempted' + } + Should -Invoke Resolve-VerifiedCiFixContext -ModuleName QueueCiFixAzdoValidationTest -Times 2 -Exactly + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + } + + It 'makes no Azure request when queue-time verification observes revoked eligibility' { + Mock Resolve-VerifiedCiFixContext { + return $null + } -ModuleName QueueCiFixAzdoValidationTest + Mock Get-AzdoDuplicateBuild { + throw 'Azure duplicate lookup must not run after eligibility is revoked.' + } -ModuleName QueueCiFixAzdoValidationTest + Mock Invoke-AzdoPipelineQueue { + throw 'Azure queue POST must not run after eligibility is revoked.' + } -ModuleName QueueCiFixAzdoValidationTest + Mock Write-CiFixJobSummary {} -ModuleName QueueCiFixAzdoValidationTest + + $results = @( + Invoke-QueueTestCiFixQueueWork ` + -Contexts @($script:queueContext) ` + -AuthToken test-token + ) + + $results.Count | Should -Be 3 + foreach ($result in $results) { + $result.Outcome | Should -Be 'failed' + $result.Error | Should -Match 'no longer an eligible open automated CI-fix PR' + $result.Error | Should -Match 'No queue POST was attempted' + } + Should -Invoke Resolve-VerifiedCiFixContext -ModuleName QueueCiFixAzdoValidationTest -Times 1 -Exactly + Should -Invoke Get-AzdoDuplicateBuild -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + Should -Invoke Invoke-AzdoPipelineQueue -ModuleName QueueCiFixAzdoValidationTest -Times 0 -Exactly + } + } +} + +Describe 'event payload validation' { + It 'uses an event-neutral error for a missing payload file' { + $missingEventPath = Join-Path $TestDrive 'missing-event.json' + + $output = & pwsh -NoLogo -NoProfile -File $scriptPath ` + -EventPath $missingEventPath ` + -Repository dotnet/maui ` + -EventName workflow_run ` + -DryRun 2>&1 + + $LASTEXITCODE | Should -Not -Be 0 + $output -join [Environment]::NewLine | + Should -Match 'GITHUB_EVENT_PATH must identify a supported GitHub event payload file\.' + } +} + +Describe 'entrypoint verification failure routing' { + BeforeEach { + $script:oldStepSummary = $env:GITHUB_STEP_SUMMARY + Mock Start-Sleep {} + } + + AfterEach { + $env:GITHUB_STEP_SUMMARY = $script:oldStepSummary + } + + It 'retains one unverifiable PR while producing all dry-run payloads for the next verified PR' { + $eventPath = Join-Path $TestDrive 'mixed-event.json' + $fixturePath = Join-Path $TestDrive 'mixed-fixture.json' + $stdoutPath = Join-Path $TestDrive 'mixed-stdout.json' + $summaryPath = Join-Path $TestDrive 'mixed-summary.md' + $env:GITHUB_STEP_SUMMARY = $summaryPath + New-TestEvent | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $eventPath + + $unverifiedPullRequest = New-TestPullRequest -Number 123 + $verifiedPullRequest = New-TestPullRequest ` + -Number 124 ` + -BaseRef net11.0 ` + -HeadRef ci-fix/issue-124 ` + -Title '[ci-fix-net11] Repair CI (refs #124)' ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444' + $fixture = New-TestPullRequestFixture -PullRequests @( + $unverifiedPullRequest, + $verifiedPullRequest + ) + $fixture.commits.'2222222222222222222222222222222222222222' = New-TestMergeCommit ` + -MergeSha '2222222222222222222222222222222222222222' ` + -HeadSha '5555555555555555555555555555555555555555' + $fixture | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $fixturePath + + { + & $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -PullRequestsFixturePath $fixturePath ` + -DryRun > $stdoutPath + } | Should -Throw '*3 of 6 Azure DevOps validation pipelines failed before dry-run queue payload generation*' + + $results = @(Get-Content -Raw -LiteralPath $stdoutPath | ConvertFrom-Json -Depth 20) + $results.Count | Should -Be 6 + @($results[0..2].PullRequestNumber | Select-Object -Unique) | Should -Be @(123) + @($results[0..2].Outcome | Select-Object -Unique) | Should -Be @('failed') + @($results[3..5].PullRequestNumber | Select-Object -Unique) | Should -Be @(124) + @($results[3..5].Outcome | Select-Object -Unique) | Should -Be @('dry-run') + $summary = Get-Content -Raw -LiteralPath $summaryPath + $summary | Should -Match 'dotnet/maui#123' + $summary | Should -Match 'dotnet/maui#124' + Should -Invoke Start-Sleep -Times 3 -Exactly + } + + It 'retains all-unverifiable results without attempting OIDC or Azure work' { + $eventPath = Join-Path $TestDrive 'unverifiable-event.json' + $fixturePath = Join-Path $TestDrive 'unverifiable-fixture.json' + $stdoutPath = Join-Path $TestDrive 'unverifiable-stdout.json' + $summaryPath = Join-Path $TestDrive 'unverifiable-summary.md' + $env:GITHUB_STEP_SUMMARY = $summaryPath + New-TestEvent | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $eventPath + + $pullRequest = New-TestPullRequest + $fixture = New-TestPullRequestFixture -PullRequests @($pullRequest) + $fixture.commits.'2222222222222222222222222222222222222222' = New-TestMergeCommit ` + -MergeSha '2222222222222222222222222222222222222222' ` + -HeadSha '5555555555555555555555555555555555555555' + $fixture | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $fixturePath + + { + & $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -PullRequestsFixturePath $fixturePath ` + -DryRun > $stdoutPath + } | Should -Throw '*3 of 3 Azure DevOps validation pipelines failed before dry-run queue payload generation*' + + $results = @(Get-Content -Raw -LiteralPath $stdoutPath | ConvertFrom-Json -Depth 20) + $results.Count | Should -Be 3 + @($results.Outcome | Select-Object -Unique) | Should -Be @('failed') + foreach ($result in $results) { + $result.Error | Should -Match 'did not obtain a verified test merge' + $result.Error | Should -Match 'No Azure DevOps validation was queued or deduplicated' + } + (Get-Content -Raw -LiteralPath $summaryPath) | Should -Match 'dotnet/maui#123' + Should -Invoke Start-Sleep -Times 3 -Exactly + } +} + +Describe 'full-script offline reconciliation' { + It 'reconciles multiple realistic workflow_run PRs and ignores unrelated PRs' { + $eventPath = [System.IO.Path]::GetTempFileName() + $pullRequestsPath = [System.IO.Path]::GetTempFileName() + try { + [pscustomobject]@{ + action = 'completed' + repository = [pscustomobject]@{ full_name = 'dotnet/maui' } + workflow_run = [pscustomobject]@{ + name = 'CI Failure Fixer (main)' + path = '.github/workflows/ci-status-fix.lock.yml' + head_branch = 'main' + head_repository = [pscustomobject]@{ full_name = 'dotnet/maui' } + } + } | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $eventPath + + $fixturePullRequests = @( + (New-TestPullRequest -Number 123), + (New-TestPullRequest ` + -Number 124 ` + -BaseRef net11.0 ` + -HeadRef ci-fix/issue-124 ` + -Title '[ci-fix-net11] Repair CI (refs #124)' ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444'), + (New-TestPullRequest -Number 125 -Author attacker -HeadRef feature/unrelated) + ) + New-TestPullRequestFixture -PullRequests $fixturePullRequests | + ConvertTo-Json -Depth 20 | + Set-Content -LiteralPath $pullRequestsPath + + $output = & pwsh -NoLogo -NoProfile -File $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName workflow_run ` + -PullRequestsFixturePath $pullRequestsPath ` + -DryRun + + $LASTEXITCODE | Should -Be 0 + $results = @($output -join [Environment]::NewLine | ConvertFrom-Json -Depth 20) + $results.Count | Should -Be 6 + @($results.PullRequestNumber | Sort-Object -Unique) | Should -Be @(123, 124) + @($results.Request.triggerInfo.'pr.targetBranch' | Sort-Object -Unique) | Should -Be @('main', 'net11.0') + } + finally { + Remove-Item -LiteralPath $eventPath, $pullRequestsPath -Force -ErrorAction SilentlyContinue + } + } + + It 'makes a surviving pull_request_target event reconcile another eligible PR head' { + $eventPath = [System.IO.Path]::GetTempFileName() + $pullRequestsPath = [System.IO.Path]::GetTempFileName() + try { + New-TestEvent | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $eventPath + $fixturePullRequests = @( + (New-TestPullRequest -Number 123), + (New-TestPullRequest ` + -Number 124 ` + -BaseRef net11.0 ` + -HeadRef ci-fix/issue-124 ` + -Title '[ci-fix-net11] Repair CI (refs #124)' ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444') + ) + New-TestPullRequestFixture -PullRequests $fixturePullRequests | + ConvertTo-Json -Depth 20 | + Set-Content -LiteralPath $pullRequestsPath + + $output = & pwsh -NoLogo -NoProfile -File $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -PullRequestsFixturePath $pullRequestsPath ` + -DryRun + + $LASTEXITCODE | Should -Be 0 + $results = @($output -join [Environment]::NewLine | ConvertFrom-Json -Depth 20) + $results.Count | Should -Be 6 + @($results.PullRequestNumber | Sort-Object -Unique) | Should -Be @(123, 124) + } + finally { + Remove-Item -LiteralPath $eventPath, $pullRequestsPath -Force -ErrorAction SilentlyContinue + } + } + + It 'makes an unrelated surviving pull_request_target event reconcile eligible heads' { + $eventPath = [System.IO.Path]::GetTempFileName() + $pullRequestsPath = [System.IO.Path]::GetTempFileName() + try { + New-TestEvent ` + -Author contributor ` + -HeadRef feature/unrelated ` + -Title 'Unrelated PR' ` + -Labels @() | + ConvertTo-Json -Depth 10 | + Set-Content -LiteralPath $eventPath + $fixturePullRequests = @( + (New-TestPullRequest -Number 123), + (New-TestPullRequest -Number 125 -Author contributor -HeadRef feature/unrelated -Labels @()) + ) + New-TestPullRequestFixture -PullRequests $fixturePullRequests | + ConvertTo-Json -Depth 20 | + Set-Content -LiteralPath $pullRequestsPath + + $output = & pwsh -NoLogo -NoProfile -File $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -PullRequestsFixturePath $pullRequestsPath ` + -DryRun + + $LASTEXITCODE | Should -Be 0 + $results = @($output -join [Environment]::NewLine | ConvertFrom-Json -Depth 20) + $results.Count | Should -Be 3 + @($results.PullRequestNumber | Sort-Object -Unique) | Should -Be @(123) + } + finally { + Remove-Item -LiteralPath $eventPath, $pullRequestsPath -Force -ErrorAction SilentlyContinue + } + } + + It 'does not queue a stale event when live eligibility was revoked' { + $eventPath = [System.IO.Path]::GetTempFileName() + $pullRequestsPath = [System.IO.Path]::GetTempFileName() + try { + New-TestEvent | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $eventPath + $fixturePullRequests = @( + New-TestPullRequest -Labels @() + ) + New-TestPullRequestFixture -PullRequests $fixturePullRequests | + ConvertTo-Json -Depth 20 | + Set-Content -LiteralPath $pullRequestsPath + + $output = & pwsh -NoLogo -NoProfile -File $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -PullRequestsFixturePath $pullRequestsPath ` + -DryRun + + $LASTEXITCODE | Should -Be 0 + @($output) | Should -Be @('No eligible automated CI-fix pull request heads require reconciliation.') + } + finally { + Remove-Item -LiteralPath $eventPath, $pullRequestsPath -Force -ErrorAction SilentlyContinue + } + } + + It 'uses the live head and merge SHAs instead of a stale event snapshot' { + $eventPath = [System.IO.Path]::GetTempFileName() + $pullRequestsPath = [System.IO.Path]::GetTempFileName() + try { + New-TestEvent | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $eventPath + $fixturePullRequests = @( + New-TestPullRequest ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444' + ) + New-TestPullRequestFixture -PullRequests $fixturePullRequests | + ConvertTo-Json -Depth 20 | + Set-Content -LiteralPath $pullRequestsPath + + $output = & pwsh -NoLogo -NoProfile -File $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -PullRequestsFixturePath $pullRequestsPath ` + -DryRun + + $LASTEXITCODE | Should -Be 0 + $results = @($output -join [Environment]::NewLine | ConvertFrom-Json -Depth 20) + $results.Count | Should -Be 3 + @($results.Request.triggerInfo.'pr.sourceSha' | Sort-Object -Unique) | + Should -Be @('3333333333333333333333333333333333333333') + @($results.Request.sourceVersion | Sort-Object -Unique) | Should -Be @('4444444444444444444444444444444444444444') + } + finally { + Remove-Item -LiteralPath $eventPath, $pullRequestsPath -Force -ErrorAction SilentlyContinue + } + } + + It 'refreshes a stale list merge and queues only the verified current head pair from fixture metadata' { + $eventPath = [System.IO.Path]::GetTempFileName() + $pullRequestsPath = [System.IO.Path]::GetTempFileName() + try { + New-TestEvent | ConvertTo-Json -Depth 10 | Set-Content -LiteralPath $eventPath + $listPullRequest = New-TestPullRequest ` + -HeadSha '1111111111111111111111111111111111111111' ` + -MergeSha '2222222222222222222222222222222222222222' + $detailPullRequest = New-TestPullRequest ` + -HeadSha '3333333333333333333333333333333333333333' ` + -MergeSha '4444444444444444444444444444444444444444' + [pscustomobject]@{ + pullRequests = @($listPullRequest) + pullRequestDetails = [pscustomobject]@{ + '123' = $detailPullRequest + } + commits = [pscustomobject]@{ + '4444444444444444444444444444444444444444' = New-TestMergeCommit ` + -MergeSha '4444444444444444444444444444444444444444' ` + -HeadSha '3333333333333333333333333333333333333333' + } + } | ConvertTo-Json -Depth 20 | Set-Content -LiteralPath $pullRequestsPath + + $output = & pwsh -NoLogo -NoProfile -File $scriptPath ` + -EventPath $eventPath ` + -Repository dotnet/maui ` + -EventName pull_request_target ` + -PullRequestsFixturePath $pullRequestsPath ` + -DryRun + + $LASTEXITCODE | Should -Be 0 + $results = @($output -join [Environment]::NewLine | ConvertFrom-Json -Depth 20) + $results.Count | Should -Be 3 + @($results.Request.triggerInfo.'pr.sourceSha' | Sort-Object -Unique) | + Should -Be @('3333333333333333333333333333333333333333') + @($results.Request.sourceVersion | Sort-Object -Unique) | + Should -Be @('4444444444444444444444444444444444444444') + } + finally { + Remove-Item -LiteralPath $eventPath, $pullRequestsPath -Force -ErrorAction SilentlyContinue + } + } +} diff --git a/.github/scripts/Queue-CiFixAzdoValidation.ps1 b/.github/scripts/Queue-CiFixAzdoValidation.ps1 new file mode 100644 index 000000000000..c279109c44fb --- /dev/null +++ b/.github/scripts/Queue-CiFixAzdoValidation.ps1 @@ -0,0 +1,1144 @@ +#!/usr/bin/env pwsh + +[CmdletBinding()] +param( + [string]$EventPath = $env:GITHUB_EVENT_PATH, + [string]$Repository = $env:GITHUB_REPOSITORY, + [string]$EventName = $env:GITHUB_EVENT_NAME, + [string]$PullRequestsFixturePath, + [ValidateRange(1, 540)][int]$DispatcherBudgetSeconds = 420, + [long]$DispatcherDeadlineUnixSeconds = 0, + [switch]$DryRun +) + +$ErrorActionPreference = 'Stop' +Set-StrictMode -Version Latest + +$script:AzureDevOpsOrganization = 'dnceng-public' +$script:AzureDevOpsProject = 'public' +$script:TransientHttpStatusCodes = @(408, 429, 500, 502, 503, 504) +$script:MaxHttpAttempts = 4 +$script:RetryBaseDelaySeconds = 2 +$secondsUntilDeadline = if ($DispatcherDeadlineUnixSeconds -gt 0) { + $DispatcherDeadlineUnixSeconds - [DateTimeOffset]::UtcNow.ToUnixTimeSeconds() +} +else { + $DispatcherBudgetSeconds +} +$script:DispatcherBudgetSeconds = [Math]::Max( + 0, + [Math]::Min($DispatcherBudgetSeconds, $secondsUntilDeadline)) +$script:DispatcherStopwatch = [System.Diagnostics.Stopwatch]::StartNew() +$script:DispatcherBudgetPrefix = '[dispatcher-budget-exhausted]' + +function Get-DispatcherElapsedSeconds { + return $script:DispatcherStopwatch.Elapsed.TotalSeconds +} + +function Get-DispatcherRemainingSeconds { + return [Math]::Max(0, $script:DispatcherBudgetSeconds - (Get-DispatcherElapsedSeconds)) +} + +function Test-IsDispatcherBudgetException { + param([Parameter(Mandatory = $true)][System.Exception]$Exception) + + return $Exception.Message.StartsWith($script:DispatcherBudgetPrefix, [System.StringComparison]::Ordinal) +} + +function Get-DispatcherHttpTimeoutSeconds { + param( + [Parameter(Mandatory = $true)][string]$OperationName, + [ValidateRange(1, 300)][int]$MaximumSeconds = 30 + ) + + $remainingSeconds = [Math]::Floor((Get-DispatcherRemainingSeconds)) + if ($remainingSeconds -lt 1) { + throw "$($script:DispatcherBudgetPrefix) No time remains before '$OperationName'." + } + + return [int][Math]::Min($MaximumSeconds, $remainingSeconds) +} + +function Invoke-DispatcherSleep { + param( + [Parameter(Mandatory = $true)][string]$OperationName, + [ValidateRange(1, 300)][int]$RequestedSeconds + ) + + $remainingSeconds = [Math]::Floor((Get-DispatcherRemainingSeconds)) + $sleepSeconds = [Math]::Min($RequestedSeconds, $remainingSeconds - 1) + if ($sleepSeconds -lt 1) { + throw "$($script:DispatcherBudgetPrefix) No retry time remains before '$OperationName'." + } + + Start-Sleep -Seconds $sleepSeconds +} + +function Get-ObjectPropertyValue { + param( + [AllowNull()][object]$InputObject, + [Parameter(Mandatory = $true)][string]$Name + ) + + if ($null -eq $InputObject) { + return $null + } + + $property = $InputObject.PSObject.Properties[$Name] + if ($null -eq $property) { + return $null + } + + return $property.Value +} + +function Test-CiFixPrFingerprint { + param( + [Parameter(Mandatory = $true)][string]$Repository, + [Parameter(Mandatory = $true)][string]$HeadRepository, + [Parameter(Mandatory = $true)][string]$Title, + [Parameter(Mandatory = $true)][string]$BaseRef, + [Parameter(Mandatory = $true)][string]$HeadRef, + [Parameter(Mandatory = $true)][string]$AuthorLogin, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][string[]]$Labels + ) + + if ($Repository -cne 'dotnet/maui') { return $false } + if ($HeadRepository -cne 'dotnet/maui') { return $false } + if ($AuthorLogin -cne 'github-actions[bot]') { return $false } + if ($HeadRef -cnotmatch '^ci-fix/[A-Za-z0-9][A-Za-z0-9._/-]*$') { return $false } + if ($Labels -cnotcontains 'agentic-workflows') { return $false } + + switch ($BaseRef) { + 'main' { return $Title -cmatch '^\[ci-fix\](?:\s|$)' } + 'net11.0' { return $Title -cmatch '^\[ci-fix-net11\](?:\s|$)' } + default { return $false } + } +} + +function Get-CiFixPipelineDefinitions { + return @( + [pscustomobject]@{ Name = 'maui-pr'; DefinitionId = 302 }, + [pscustomobject]@{ Name = 'maui-pr-uitests'; DefinitionId = 313 }, + [pscustomobject]@{ Name = 'maui-pr-devicetests'; DefinitionId = 314 } + ) +} + +function Get-CiFixContextFromPullRequest { + param( + [Parameter(Mandatory = $true)][object]$PullRequest, + [Parameter(Mandatory = $true)][string]$Repository, + [bool]$RequireMergeSha = $true + ) + + $base = Get-ObjectPropertyValue -InputObject $PullRequest -Name 'base' + $head = Get-ObjectPropertyValue -InputObject $PullRequest -Name 'head' + $baseRepository = Get-ObjectPropertyValue -InputObject $base -Name 'repo' + $headRepository = Get-ObjectPropertyValue -InputObject $head -Name 'repo' + $author = Get-ObjectPropertyValue -InputObject $PullRequest -Name 'user' + + $labels = @( + @(Get-ObjectPropertyValue -InputObject $PullRequest -Name 'labels') | + ForEach-Object { [string](Get-ObjectPropertyValue -InputObject $_ -Name 'name') } | + Where-Object { -not [string]::IsNullOrWhiteSpace($_) } + ) + + $context = [pscustomobject]@{ + Repository = $Repository + BaseRepository = [string](Get-ObjectPropertyValue -InputObject $baseRepository -Name 'full_name') + HeadRepository = [string](Get-ObjectPropertyValue -InputObject $headRepository -Name 'full_name') + PullRequestNumber = [int](Get-ObjectPropertyValue -InputObject $PullRequest -Name 'number') + PullRequestId = [long](Get-ObjectPropertyValue -InputObject $PullRequest -Name 'id') + State = [string](Get-ObjectPropertyValue -InputObject $PullRequest -Name 'state') + Draft = [bool](Get-ObjectPropertyValue -InputObject $PullRequest -Name 'draft') + Title = [string](Get-ObjectPropertyValue -InputObject $PullRequest -Name 'title') + AuthorLogin = [string](Get-ObjectPropertyValue -InputObject $author -Name 'login') + BaseRef = [string](Get-ObjectPropertyValue -InputObject $base -Name 'ref') + HeadRef = [string](Get-ObjectPropertyValue -InputObject $head -Name 'ref') + HeadSha = [string](Get-ObjectPropertyValue -InputObject $head -Name 'sha') + MergeSha = [string](Get-ObjectPropertyValue -InputObject $PullRequest -Name 'merge_commit_sha') + Labels = $labels + } + + if ($context.BaseRepository -cne 'dotnet/maui') { + throw "Unexpected base repository '$($context.BaseRepository)'." + } + if ($context.State -cne 'open') { + return $null + } + if ($context.PullRequestNumber -le 0 -or $context.PullRequestId -le 0) { + throw 'Pull request number and id must be positive integers.' + } + + $isEligible = Test-CiFixPrFingerprint ` + -Repository $context.Repository ` + -HeadRepository $context.HeadRepository ` + -Title $context.Title ` + -BaseRef $context.BaseRef ` + -HeadRef $context.HeadRef ` + -AuthorLogin $context.AuthorLogin ` + -Labels $context.Labels + + if (-not $isEligible) { + return $null + } + + if ($context.HeadSha -cnotmatch '^[0-9a-fA-F]{40}$') { + throw 'Eligible CI-fix pull request head SHA is missing or invalid.' + } + if ($RequireMergeSha -and $context.MergeSha -cnotmatch '^[0-9a-fA-F]{40}$') { + Write-Warning "Eligible CI-fix PR #$($context.PullRequestNumber) has no merge commit yet; deferring validation." + return $null + } + + return $context +} + +function Get-GitHubApiHeaders { + param([Parameter(Mandatory = $true)][string]$GitHubToken) + + return @{ + Authorization = "Bearer $GitHubToken" + Accept = 'application/vnd.github+json' + 'X-GitHub-Api-Version' = '2022-11-28' + } +} + +function Get-FixturePullRequestDetail { + param( + [Parameter(Mandatory = $true)][object]$FixtureData, + [Parameter(Mandatory = $true)][int]$PullRequestNumber + ) + + $details = Get-ObjectPropertyValue -InputObject $FixtureData -Name 'pullRequestDetails' + $detail = Get-ObjectPropertyValue -InputObject $details -Name "$PullRequestNumber" + if ($null -eq $detail) { + throw "Pull request fixture has no detail metadata for PR #$PullRequestNumber." + } + + return $detail +} + +function Get-FixtureCommit { + param( + [Parameter(Mandatory = $true)][object]$FixtureData, + [Parameter(Mandatory = $true)][string]$CommitSha + ) + + $commits = Get-ObjectPropertyValue -InputObject $FixtureData -Name 'commits' + $commit = Get-ObjectPropertyValue -InputObject $commits -Name $CommitSha + if ($null -eq $commit) { + throw "Pull request fixture has no commit metadata for '$CommitSha'." + } + + return $commit +} + +function Get-CiFixMergePairDiagnostic { + param( + [Parameter(Mandatory = $true)][object]$Context, + [Parameter(Mandatory = $true)][object]$Commit + ) + + $commitSha = [string](Get-ObjectPropertyValue -InputObject $Commit -Name 'sha') + if ($commitSha -cne $Context.MergeSha) { + return "requested merge '$($Context.MergeSha)' returned commit '$commitSha'" + } + + $parents = @( + @(Get-ObjectPropertyValue -InputObject $Commit -Name 'parents') | + ForEach-Object { $_ } + ) + if ($parents.Count -ne 2) { + return "merge '$($Context.MergeSha)' has $($parents.Count) parent(s), expected exactly 2" + } + + $baseParentSha = [string](Get-ObjectPropertyValue -InputObject $parents[0] -Name 'sha') + if ($baseParentSha -cnotmatch '^[0-9a-fA-F]{40}$') { + return "merge '$($Context.MergeSha)' has an invalid first parent '$baseParentSha'" + } + + $sourceParentSha = [string](Get-ObjectPropertyValue -InputObject $parents[1] -Name 'sha') + if ($sourceParentSha -cne $Context.HeadSha) { + return "merge '$($Context.MergeSha)' source parent '$sourceParentSha' does not match head '$($Context.HeadSha)'" + } + + return $null +} + +function Resolve-VerifiedCiFixContext { + param( + [Parameter(Mandatory = $true)][object]$NominatedContext, + [Parameter(Mandatory = $true)][string]$Repository, + [AllowEmptyString()][string]$GitHubToken, + [AllowNull()][object]$FixtureData + ) + + $lastDiagnostic = 'no test merge metadata was available' + $lastHeadSha = $NominatedContext.HeadSha + $lastMergeSha = $NominatedContext.MergeSha + + for ($attempt = 1; $attempt -le $script:MaxHttpAttempts; $attempt++) { + $pullRequest = if ($null -ne $FixtureData) { + Get-FixturePullRequestDetail ` + -FixtureData $FixtureData ` + -PullRequestNumber $NominatedContext.PullRequestNumber + } + else { + Invoke-WithHttpRetry ` + -OperationName "GitHub pull request #$($NominatedContext.PullRequestNumber) detail refresh" ` + -Operation { + param($timeoutSeconds) + + Invoke-RestMethod ` + -Method Get ` + -Uri "https://api.github.com/repos/$Repository/pulls/$($NominatedContext.PullRequestNumber)" ` + -Headers (Get-GitHubApiHeaders -GitHubToken $GitHubToken) ` + -TimeoutSec $timeoutSeconds + } + } + + $context = Get-CiFixContextFromPullRequest ` + -PullRequest $pullRequest ` + -Repository $Repository ` + -RequireMergeSha $false + if ($null -eq $context) { + return $null + } + + $lastHeadSha = $context.HeadSha + $lastMergeSha = $context.MergeSha + if ($context.MergeSha -cnotmatch '^[0-9a-fA-F]{40}$') { + $lastDiagnostic = 'the refreshed pull request has no syntactically valid test merge SHA' + } + else { + try { + $commit = if ($null -ne $FixtureData) { + Get-FixtureCommit -FixtureData $FixtureData -CommitSha $context.MergeSha + } + else { + Invoke-WithHttpRetry ` + -OperationName "GitHub test merge commit $($context.MergeSha) for PR #$($context.PullRequestNumber)" ` + -Operation { + param($timeoutSeconds) + + Invoke-RestMethod ` + -Method Get ` + -Uri "https://api.github.com/repos/$Repository/git/commits/$($context.MergeSha)" ` + -Headers (Get-GitHubApiHeaders -GitHubToken $GitHubToken) ` + -TimeoutSec $timeoutSeconds + } + } + $lastDiagnostic = Get-CiFixMergePairDiagnostic -Context $context -Commit $commit + if ($null -eq $lastDiagnostic) { + return $context + } + } + catch { + if (Test-IsDispatcherBudgetException -Exception $_.Exception) { + throw + } + $statusCode = Get-HttpStatusCode -Exception $_.Exception + if ($null -eq $FixtureData -and $statusCode -notin @(404, 409, 422)) { + throw + } + $lastDiagnostic = "test merge metadata read failed: $($_.Exception.Message)" + } + } + + if ($attempt -lt $script:MaxHttpAttempts) { + Invoke-DispatcherSleep ` + -OperationName "fresh test merge for PR #$($context.PullRequestNumber) attempt $($attempt + 1)" ` + -RequestedSeconds ($script:RetryBaseDelaySeconds * $attempt) + } + } + + $verificationError = "Eligible CI-fix PR #$($NominatedContext.PullRequestNumber) head '$lastHeadSha' did not obtain a verified test merge after $($script:MaxHttpAttempts) attempts. Last candidate merge '$lastMergeSha': $lastDiagnostic. No Azure DevOps validation was queued or deduplicated for this PR." + $context | Add-Member -NotePropertyName VerificationError -NotePropertyValue $verificationError -Force + return $context +} + +function Get-CiFixEventContext { + param( + [Parameter(Mandatory = $true)][object]$Event, + [Parameter(Mandatory = $true)][string]$Repository, + [Parameter(Mandatory = $true)][string]$EventName + ) + + if ($EventName -cne 'pull_request_target') { + throw "Unexpected event '$EventName'." + } + + $action = [string](Get-ObjectPropertyValue -InputObject $Event -Name 'action') + if ($action -cnotin @('opened', 'reopened', 'synchronize', 'labeled')) { + throw "Unexpected pull_request_target action '$action'." + } + + if ($action -ceq 'labeled') { + $eventLabel = Get-ObjectPropertyValue -InputObject (Get-ObjectPropertyValue -InputObject $Event -Name 'label') -Name 'name' + if ([string]$eventLabel -cne 'agentic-workflows') { + return $null + } + } + + $pullRequest = Get-ObjectPropertyValue -InputObject $Event -Name 'pull_request' + if ($null -eq $pullRequest) { + throw 'The event does not contain pull_request metadata.' + } + + return Get-CiFixContextFromPullRequest -PullRequest $pullRequest -Repository $Repository +} + +function Test-TrustedCiFixWorkflowRun { + param( + [Parameter(Mandatory = $true)][object]$Event, + [Parameter(Mandatory = $true)][string]$Repository + ) + + if ($Repository -cne 'dotnet/maui') { + return $false + } + if ([string](Get-ObjectPropertyValue -InputObject $Event -Name 'action') -cne 'completed') { + return $false + } + + $eventRepository = Get-ObjectPropertyValue -InputObject $Event -Name 'repository' + if ([string](Get-ObjectPropertyValue -InputObject $eventRepository -Name 'full_name') -cne 'dotnet/maui') { + return $false + } + + $workflowRun = Get-ObjectPropertyValue -InputObject $Event -Name 'workflow_run' + $workflowName = [string](Get-ObjectPropertyValue -InputObject $workflowRun -Name 'name') + $workflowPath = [string](Get-ObjectPropertyValue -InputObject $workflowRun -Name 'path') + $headBranch = [string](Get-ObjectPropertyValue -InputObject $workflowRun -Name 'head_branch') + $headRepository = Get-ObjectPropertyValue -InputObject $workflowRun -Name 'head_repository' + + $allowedWorkflows = @{ + 'CI Failure Fixer (main)' = '.github/workflows/ci-status-fix.lock.yml' + 'CI Failure Fixer (net11.0)' = '.github/workflows/ci-status-fix-net11.lock.yml' + } + + return $allowedWorkflows.ContainsKey($workflowName) -and + $workflowPath -ceq $allowedWorkflows[$workflowName] -and + $headBranch -ceq 'main' -and + [string](Get-ObjectPropertyValue -InputObject $headRepository -Name 'full_name') -ceq 'dotnet/maui' +} + +function Get-OpenCiFixContexts { + param( + [Parameter(Mandatory = $true)][string]$Repository, + [AllowEmptyString()][string]$GitHubToken, + [AllowEmptyString()][string]$FixturePath + ) + + if (-not [string]::IsNullOrWhiteSpace($FixturePath)) { + if (-not (Test-Path -LiteralPath $FixturePath -PathType Leaf)) { + throw "Pull request fixture '$FixturePath' does not exist." + } + + $fixtureData = Get-Content -Raw -LiteralPath $FixturePath | ConvertFrom-Json -Depth 100 + $fixturePullRequests = @( + @(Get-ObjectPropertyValue -InputObject $fixtureData -Name 'pullRequests') | + ForEach-Object { $_ } + ) + $fixtureContexts = [System.Collections.Generic.List[object]]::new() + foreach ($pullRequest in $fixturePullRequests) { + $nominatedContext = Get-CiFixContextFromPullRequest ` + -PullRequest $pullRequest ` + -Repository $Repository ` + -RequireMergeSha $false + if ($null -ne $nominatedContext) { + $verifiedContext = Resolve-VerifiedCiFixContext ` + -NominatedContext $nominatedContext ` + -Repository $Repository ` + -GitHubToken '' ` + -FixtureData $fixtureData + if ($null -ne $verifiedContext) { + $fixtureContexts.Add($verifiedContext) + } + } + } + return $fixtureContexts.ToArray() + } + + if ([string]::IsNullOrWhiteSpace($GitHubToken)) { + throw 'GITHUB_TOKEN is required for workflow_run reconciliation.' + } + + $contexts = [System.Collections.Generic.List[object]]::new() + for ($page = 1; $page -le 10; $page++) { + $pageResponse = Invoke-WithHttpRetry -OperationName "GitHub open pull request query page $page" -Operation { + param($timeoutSeconds) + + Invoke-RestMethod ` + -Method Get ` + -Uri "https://api.github.com/repos/$Repository/pulls?state=open&per_page=100&page=$page" ` + -Headers (Get-GitHubApiHeaders -GitHubToken $GitHubToken) ` + -TimeoutSec $timeoutSeconds + } + # Invoke-RestMethod returns a top-level JSON array as one Object[] value. + # Enumerate it explicitly so pagination and per-PR validation see each PR. + $pullRequests = @($pageResponse | ForEach-Object { $_ }) + + foreach ($pullRequest in $pullRequests) { + $nominatedContext = Get-CiFixContextFromPullRequest ` + -PullRequest $pullRequest ` + -Repository $Repository ` + -RequireMergeSha $false + if ($null -ne $nominatedContext) { + $verifiedContext = Resolve-VerifiedCiFixContext ` + -NominatedContext $nominatedContext ` + -Repository $Repository ` + -GitHubToken $GitHubToken ` + -FixtureData $null + if ($null -ne $verifiedContext) { + $contexts.Add($verifiedContext) + } + } + } + + if ($pullRequests.Count -lt 100) { + return $contexts.ToArray() + } + } + + throw 'Open pull request reconciliation exceeded the bounded 1,000-PR scan.' +} + +function Test-IsTransientHttpException { + param([Parameter(Mandatory = $true)][System.Exception]$Exception) + + $currentException = $Exception + $statusCode = $null + $hasTimeout = $false + $hasNetworkException = $false + while ($null -ne $currentException) { + $response = Get-ObjectPropertyValue -InputObject $currentException -Name 'Response' + $nestedStatusCode = Get-ObjectPropertyValue -InputObject $response -Name 'StatusCode' + if ($null -eq $nestedStatusCode) { + $nestedStatusCode = Get-ObjectPropertyValue -InputObject $currentException -Name 'StatusCode' + } + if ($null -eq $statusCode -and $null -ne $nestedStatusCode) { + $statusCode = $nestedStatusCode + } + + if ($currentException -is [System.TimeoutException]) { + $hasTimeout = $true + } + if ($currentException -is [System.IO.IOException] -or + $currentException -is [System.Net.Sockets.SocketException] -or + $currentException -is [System.Net.Http.HttpRequestException]) { + $hasNetworkException = $true + } + + $currentException = $currentException.InnerException + } + + if ($null -ne $statusCode) { + $numericStatusCode = if ($statusCode.PSObject.Properties['value__']) { + [int]$statusCode.value__ + } + else { + [int]$statusCode + } + return $numericStatusCode -in $script:TransientHttpStatusCodes + } + + # PowerShell's request timeout can be a TaskCanceledException wrapping a + # TimeoutException, IOException, and SocketException. Plain cancellation + # remains non-transient so authentication and caller cancellation do not + # trigger queue retries. + return $hasTimeout -or $hasNetworkException +} + +function Get-HttpStatusCode { + param([Parameter(Mandatory = $true)][System.Exception]$Exception) + + $response = Get-ObjectPropertyValue -InputObject $Exception -Name 'Response' + $statusCode = Get-ObjectPropertyValue -InputObject $response -Name 'StatusCode' + if ($null -eq $statusCode) { + $statusCode = Get-ObjectPropertyValue -InputObject $Exception -Name 'StatusCode' + } + if ($null -eq $statusCode) { + return $null + } + + if ($statusCode.PSObject.Properties['value__']) { + return [int]$statusCode.value__ + } + + return [int]$statusCode +} + +function Invoke-WithHttpRetry { + param( + [Parameter(Mandatory = $true)][string]$OperationName, + [Parameter(Mandatory = $true)][scriptblock]$Operation + ) + + for ($attempt = 1; $attempt -le $script:MaxHttpAttempts; $attempt++) { + try { + $timeoutSeconds = Get-DispatcherHttpTimeoutSeconds -OperationName "$OperationName attempt $attempt" + return & $Operation $timeoutSeconds + } + catch { + if (Test-IsDispatcherBudgetException -Exception $_.Exception) { + throw + } + + $isTransient = Test-IsTransientHttpException -Exception $_.Exception + if (-not $isTransient -or $attempt -eq $script:MaxHttpAttempts) { + throw + } + + $delaySeconds = $script:RetryBaseDelaySeconds * $attempt + Write-Warning "$OperationName failed transiently on attempt $attempt/$($script:MaxHttpAttempts); retrying in $delaySeconds seconds." + Invoke-DispatcherSleep -OperationName "$OperationName retry $($attempt + 1)" -RequestedSeconds $delaySeconds + } + } +} + +function Get-AzdoToken { + $tenantId = $env:AZDO_TRIGGER_TENANT_ID + $clientId = $env:AZDO_TRIGGER_CLIENT_ID + if ([string]::IsNullOrWhiteSpace($tenantId) -or [string]::IsNullOrWhiteSpace($clientId)) { + throw 'AZDO_TRIGGER_TENANT_ID and AZDO_TRIGGER_CLIENT_ID must be set.' + } + + $requestToken = $env:ACTIONS_ID_TOKEN_REQUEST_TOKEN + $requestUrl = $env:ACTIONS_ID_TOKEN_REQUEST_URL + if ([string]::IsNullOrWhiteSpace($requestToken) -or [string]::IsNullOrWhiteSpace($requestUrl)) { + throw 'GitHub OIDC identity token request is unavailable.' + } + + $oidcResponse = Invoke-WithHttpRetry -OperationName 'GitHub OIDC token request' -Operation { + param($timeoutSeconds) + + Invoke-RestMethod ` + -Method Get ` + -Uri "$requestUrl&audience=api://AzureADTokenExchange" ` + -Headers @{ Authorization = "Bearer $requestToken" } ` + -TimeoutSec $timeoutSeconds + } + $oidcToken = [string](Get-ObjectPropertyValue -InputObject $oidcResponse -Name 'value') + if ([string]::IsNullOrWhiteSpace($oidcToken)) { + throw 'GitHub OIDC token request returned no token.' + } + Write-Host "::add-mask::$oidcToken" + + $body = @{ + grant_type = 'client_credentials' + client_id = $clientId + client_assertion_type = 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer' + client_assertion = $oidcToken + scope = '499b84ac-1321-427f-aa17-267ca6975798/.default' + } + + try { + $tokenResponse = Invoke-WithHttpRetry -OperationName 'Azure AD token exchange' -Operation { + param($timeoutSeconds) + + Invoke-RestMethod ` + -Method Post ` + -Uri "https://login.microsoftonline.com/$tenantId/oauth2/v2.0/token" ` + -ContentType 'application/x-www-form-urlencoded' ` + -Body $body ` + -TimeoutSec $timeoutSeconds + } + } + finally { + $body.client_assertion = $null + $oidcToken = $null + } + + $accessToken = [string](Get-ObjectPropertyValue -InputObject $tokenResponse -Name 'access_token') + if ([string]::IsNullOrWhiteSpace($accessToken)) { + throw 'Azure AD token exchange returned no Azure DevOps access token.' + } + Write-Host "::add-mask::$accessToken" + return $accessToken +} + +function Find-AzdoDuplicateBuild { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][object[]]$Builds, + [Parameter(Mandatory = $true)][int]$PullRequestNumber, + [Parameter(Mandatory = $true)][string]$HeadSha, + [Parameter(Mandatory = $true)][string]$MergeSha + ) + + $expectedBranch = "refs/pull/$PullRequestNumber/merge" + foreach ($build in $Builds) { + if ([string](Get-ObjectPropertyValue -InputObject $build -Name 'sourceBranch') -cne $expectedBranch) { + continue + } + + $triggerInfo = Get-ObjectPropertyValue -InputObject $build -Name 'triggerInfo' + $sourceSha = [string](Get-ObjectPropertyValue -InputObject $triggerInfo -Name 'pr.sourceSha') + $prNumber = [string](Get-ObjectPropertyValue -InputObject $triggerInfo -Name 'pr.number') + + if ($sourceSha -ceq $HeadSha -and + ($prNumber -ceq '' -or $prNumber -ceq "$PullRequestNumber")) { + return $build + } + } + + return $null +} + +function Get-AzdoDuplicateBuild { + param( + [Parameter(Mandatory = $true)][int]$DefinitionId, + [Parameter(Mandatory = $true)][int]$PullRequestNumber, + [Parameter(Mandatory = $true)][string]$HeadSha, + [Parameter(Mandatory = $true)][string]$MergeSha, + [Parameter(Mandatory = $true)][string]$AuthToken + ) + + $branchName = [System.Uri]::EscapeDataString("refs/pull/$PullRequestNumber/merge") + $url = "https://dev.azure.com/$($script:AzureDevOpsOrganization)/$($script:AzureDevOpsProject)/_apis/build/builds" + + "?definitions=$DefinitionId&branchName=$branchName&queryOrder=queueTimeDescending&`$top=50&api-version=7.1" + + $response = Invoke-WithHttpRetry -OperationName "Azure DevOps duplicate query for definition $DefinitionId" -Operation { + param($timeoutSeconds) + + Invoke-RestMethod ` + -Method Get ` + -Uri $url ` + -Headers @{ Authorization = "Bearer $AuthToken" } ` + -TimeoutSec $timeoutSeconds + } + + return Find-AzdoDuplicateBuild ` + -Builds @(Get-ObjectPropertyValue -InputObject $response -Name 'value') ` + -PullRequestNumber $PullRequestNumber ` + -HeadSha $HeadSha ` + -MergeSha $MergeSha +} + +function New-AzdoQueueRequest { + param( + [Parameter(Mandatory = $true)][int]$DefinitionId, + [Parameter(Mandatory = $true)][object]$Context + ) + + return [ordered]@{ + definition = [ordered]@{ id = $DefinitionId } + reason = 'pullRequest' + sourceBranch = "refs/pull/$($Context.PullRequestNumber)/merge" + sourceVersion = $Context.MergeSha + parameters = ([ordered]@{ + 'system.pullRequest.pullRequestId' = "$($Context.PullRequestId)" + 'system.pullRequest.pullRequestNumber' = "$($Context.PullRequestNumber)" + 'system.pullRequest.mergedAt' = '' + 'system.pullRequest.sourceBranch' = $Context.HeadRef + 'system.pullRequest.targetBranch' = $Context.BaseRef + 'system.pullRequest.targetBranchName' = $Context.BaseRef + 'system.pullRequest.sourceRepositoryUri' = 'https://github.com/dotnet/maui' + 'system.pullRequest.sourceCommitId' = $Context.HeadSha + 'system.pullRequest.isFork' = 'False' + } | ConvertTo-Json -Compress) + triggerInfo = [ordered]@{ + 'pr.sourceBranch' = $Context.HeadRef + 'pr.sourceSha' = $Context.HeadSha + 'pr.targetBranch' = $Context.BaseRef + 'pr.id' = "$($Context.PullRequestId)" + 'pr.title' = $Context.Title + 'pr.number' = "$($Context.PullRequestNumber)" + 'pr.isFork' = 'False' + 'pr.draft' = "$($Context.Draft)" + 'pr.providerId' = 'github' + 'pr.autoCancel' = 'true' + } + } +} + +function Invoke-AzdoPipelineQueue { + param( + [Parameter(Mandatory = $true)][int]$DefinitionId, + [Parameter(Mandatory = $true)][object]$Context, + [Parameter(Mandatory = $true)][string]$AuthToken + ) + + $request = New-AzdoQueueRequest -DefinitionId $DefinitionId -Context $Context + $body = $request | ConvertTo-Json -Depth 10 -Compress + $url = "https://dev.azure.com/$($script:AzureDevOpsOrganization)/$($script:AzureDevOpsProject)/_apis/build/builds?api-version=7.1" + + $postTimeoutSeconds = Get-DispatcherHttpTimeoutSeconds -OperationName "Azure DevOps queue POST for definition $DefinitionId" + try { + $build = Invoke-RestMethod ` + -Method Post ` + -Uri $url ` + -Headers @{ Authorization = "Bearer $AuthToken" } ` + -ContentType 'application/json' ` + -Body $body ` + -TimeoutSec $postTimeoutSeconds + return [pscustomobject]@{ Build = $build; Reconciled = $false } + } + catch { + if (-not (Test-IsTransientHttpException -Exception $_.Exception)) { + throw + } + $queueStatusCode = Get-HttpStatusCode -Exception $_.Exception + + # A timed-out or 5xx POST may have been accepted before the response was + # lost. Never blindly retry an ambiguous queue request. Reconcile the + # exact definition + PR ref + source head/merge identity first. + for ($attempt = 1; $attempt -le $script:MaxHttpAttempts; $attempt++) { + try { + Invoke-DispatcherSleep ` + -OperationName "ambiguous queue reconciliation for definition $DefinitionId attempt $attempt" ` + -RequestedSeconds ($script:RetryBaseDelaySeconds * $attempt) + $duplicate = Get-AzdoDuplicateBuild ` + -DefinitionId $DefinitionId ` + -PullRequestNumber $Context.PullRequestNumber ` + -HeadSha $Context.HeadSha ` + -MergeSha $Context.MergeSha ` + -AuthToken $AuthToken + } + catch { + if (Test-IsDispatcherBudgetException -Exception $_.Exception) { + throw "$($script:DispatcherBudgetPrefix) Azure DevOps queue request for definition $DefinitionId may have been accepted, but the shared dispatcher budget expired before exact reconciliation completed. The POST was issued exactly once and was not retried; acceptance remains uncertain." + } + throw "Azure DevOps queue request for definition $DefinitionId may have been accepted after an ambiguous transient failure (HTTP $queueStatusCode), but exact reconciliation failed: $($_.Exception.Message). The POST was issued exactly once and was not retried; acceptance remains uncertain." + } + if ($null -ne $duplicate) { + return [pscustomobject]@{ Build = $duplicate; Reconciled = $true } + } + } + + throw "Azure DevOps queue request for definition $DefinitionId may have been accepted after an ambiguous transient failure (HTTP $queueStatusCode), but no exact correlated build appeared after reconciliation. The POST was issued exactly once and was not retried; acceptance remains uncertain." + } +} + +function Write-CiFixJobSummary { + param( + [Parameter(Mandatory = $true)][object]$Context, + [Parameter(Mandatory = $true)][AllowEmptyCollection()][object[]]$Results + ) + + if ([string]::IsNullOrWhiteSpace($env:GITHUB_STEP_SUMMARY)) { + return + } + + $lines = @( + '## Automated CI-fix Azure DevOps validation', + '', + "- PR: dotnet/maui#$($Context.PullRequestNumber)", + "- Base: ``$($Context.BaseRef)``", + "- Head: ``$($Context.HeadSha)``", + "- PR ref: ``refs/pull/$($Context.PullRequestNumber)/merge``", + '', + '| Pipeline | Result | Build | Details |', + '|---|---|---|---|' + ) + + foreach ($result in $Results) { + $build = if ($result.BuildId) { + "[build $($result.BuildId)](https://dev.azure.com/dnceng-public/public/_build/results?buildId=$($result.BuildId))" + } + else { + '-' + } + $details = if ($result.PSObject.Properties['Error']) { + ([string]$result.Error).Replace('|', '\|').Replace("`r", ' ').Replace("`n", ' ') + } + else { + '-' + } + $lines += "| $($result.Name) | $($result.Outcome) | $build | $details |" + } + + Add-Content -LiteralPath $env:GITHUB_STEP_SUMMARY -Value ($lines -join [Environment]::NewLine) +} + +function New-CiFixFailureResult { + param( + [Parameter(Mandatory = $true)][object]$Context, + [Parameter(Mandatory = $true)][object]$Pipeline, + [Parameter(Mandatory = $true)][string]$ErrorMessage + ) + + return [pscustomobject]@{ + PullRequestNumber = $Context.PullRequestNumber + Name = $Pipeline.Name + DefinitionId = $Pipeline.DefinitionId + Outcome = 'failed' + BuildId = $null + Error = $ErrorMessage + } +} + +function Invoke-CiFixQueueWork { + param( + [Parameter(Mandatory = $true)][AllowEmptyCollection()][object[]]$Contexts, + [Parameter(Mandatory = $true)][AllowEmptyString()][string]$AuthToken, + [string]$Repository = 'dotnet/maui', + [AllowEmptyString()][string]$GitHubToken = '', + [AllowNull()][object]$FixtureData + ) + + $workResults = [System.Collections.Generic.List[object]]::new() + $budgetExhaustedAfter = $null + foreach ($context in $Contexts) { + $contextResults = [System.Collections.Generic.List[object]]::new() + $verificationError = [string](Get-ObjectPropertyValue -InputObject $context -Name 'VerificationError') + if ([string]::IsNullOrWhiteSpace($verificationError) -and + [string]::IsNullOrWhiteSpace($AuthToken)) { + throw "Azure DevOps authentication is required before processing verified PR #$($context.PullRequestNumber)." + } + + $contextStoppedError = $null + foreach ($pipeline in Get-CiFixPipelineDefinitions) { + if (-not [string]::IsNullOrWhiteSpace($verificationError)) { + $result = New-CiFixFailureResult ` + -Context $context ` + -Pipeline $pipeline ` + -ErrorMessage $verificationError + $workResults.Add($result) + $contextResults.Add($result) + continue + } + + if ($null -ne $contextStoppedError) { + $result = New-CiFixFailureResult ` + -Context $context ` + -Pipeline $pipeline ` + -ErrorMessage $contextStoppedError + $workResults.Add($result) + $contextResults.Add($result) + continue + } + + if ($null -ne $budgetExhaustedAfter) { + $skippedError = "$($script:DispatcherBudgetPrefix) PR #$($context.PullRequestNumber) pipeline '$($pipeline.Name)' was not processed because the shared dispatcher budget was exhausted while processing $budgetExhaustedAfter. No queue POST was attempted for this work item." + $result = New-CiFixFailureResult -Context $context -Pipeline $pipeline -ErrorMessage $skippedError + $workResults.Add($result) + $contextResults.Add($result) + continue + } + + try { + $currentContext = Resolve-VerifiedCiFixContext ` + -NominatedContext $context ` + -Repository $Repository ` + -GitHubToken $GitHubToken ` + -FixtureData $FixtureData + $currentVerificationError = if ($null -eq $currentContext) { + "PR #$($context.PullRequestNumber) is no longer an eligible open automated CI-fix PR. No queue POST was attempted for pipeline '$($pipeline.Name)' or any remaining pipeline for this PR." + } + else { + [string](Get-ObjectPropertyValue -InputObject $currentContext -Name 'VerificationError') + } + if ([string]::IsNullOrWhiteSpace($currentVerificationError) -and + ($currentContext.HeadSha -cne $context.HeadSha -or + $currentContext.MergeSha -cne $context.MergeSha)) { + $currentVerificationError = "PR #$($context.PullRequestNumber) changed after discovery: expected head '$($context.HeadSha)' with merge '$($context.MergeSha)', but live verification found head '$($currentContext.HeadSha)' with merge '$($currentContext.MergeSha)'. No queue POST was attempted for pipeline '$($pipeline.Name)' or any remaining pipeline for this PR." + } + if (-not [string]::IsNullOrWhiteSpace($currentVerificationError)) { + $contextStoppedError = $currentVerificationError + $result = New-CiFixFailureResult ` + -Context $context ` + -Pipeline $pipeline ` + -ErrorMessage $contextStoppedError + $workResults.Add($result) + $contextResults.Add($result) + continue + } + + $duplicate = Get-AzdoDuplicateBuild ` + -DefinitionId $pipeline.DefinitionId ` + -PullRequestNumber $context.PullRequestNumber ` + -HeadSha $context.HeadSha ` + -MergeSha $context.MergeSha ` + -AuthToken $AuthToken + + if ($null -ne $duplicate) { + $result = [pscustomobject]@{ + PullRequestNumber = $context.PullRequestNumber + Name = $pipeline.Name + DefinitionId = $pipeline.DefinitionId + Outcome = 'deduplicated' + BuildId = [int](Get-ObjectPropertyValue -InputObject $duplicate -Name 'id') + } + $workResults.Add($result) + $contextResults.Add($result) + continue + } + + $queueResult = Invoke-AzdoPipelineQueue ` + -DefinitionId $pipeline.DefinitionId ` + -Context $context ` + -AuthToken $AuthToken + $buildId = [int](Get-ObjectPropertyValue -InputObject $queueResult.Build -Name 'id') + if ($buildId -le 0) { + throw "Azure DevOps returned an invalid build id for definition $($pipeline.DefinitionId)." + } + + $result = [pscustomobject]@{ + PullRequestNumber = $context.PullRequestNumber + Name = $pipeline.Name + DefinitionId = $pipeline.DefinitionId + Outcome = if ($queueResult.Reconciled) { 'reconciled-after-ambiguous-post' } else { 'queued' } + BuildId = $buildId + } + $workResults.Add($result) + $contextResults.Add($result) + } + catch { + Write-Error -ErrorAction Continue "PR #$($context.PullRequestNumber) pipeline '$($pipeline.Name)' failed: $($_.Exception.Message)" + $result = New-CiFixFailureResult -Context $context -Pipeline $pipeline -ErrorMessage $_.Exception.Message + $workResults.Add($result) + $contextResults.Add($result) + if (Test-IsDispatcherBudgetException -Exception $_.Exception) { + $budgetExhaustedAfter = "PR #$($context.PullRequestNumber) pipeline '$($pipeline.Name)'" + } + } + } + Write-CiFixJobSummary -Context $context -Results $contextResults + } + + return $workResults.ToArray() +} + +if ([string]::IsNullOrWhiteSpace($EventPath) -or -not (Test-Path -LiteralPath $EventPath -PathType Leaf)) { + throw 'GITHUB_EVENT_PATH must identify a supported GitHub event payload file.' +} +if ([string]::IsNullOrWhiteSpace($Repository)) { + throw 'GITHUB_REPOSITORY is required.' +} +if (-not [string]::IsNullOrWhiteSpace($PullRequestsFixturePath) -and -not $DryRun) { + throw 'PullRequestsFixturePath is permitted only with -DryRun.' +} + +$event = Get-Content -Raw -LiteralPath $EventPath | ConvertFrom-Json -Depth 100 +$contexts = if ($EventName -ceq 'pull_request_target') { + [void](Get-CiFixEventContext -Event $event -Repository $Repository -EventName $EventName) + # GitHub concurrency preserves only one pending run. Every configured + # PR-target event therefore reconciles every live eligible head, even when + # its own PR is unrelated. The live scan is authoritative so a delayed + # webhook snapshot cannot restore revoked eligibility or queue an old SHA. + @( + Get-OpenCiFixContexts ` + -Repository $Repository ` + -GitHubToken $env:GITHUB_TOKEN ` + -FixturePath $PullRequestsFixturePath + ) +} +elseif ($EventName -ceq 'workflow_run') { + if (-not (Test-TrustedCiFixWorkflowRun -Event $event -Repository $Repository)) { + throw 'workflow_run did not originate from a trusted default-branch CI-fixer workflow.' + } + @( + Get-OpenCiFixContexts ` + -Repository $Repository ` + -GitHubToken $env:GITHUB_TOKEN ` + -FixturePath $PullRequestsFixturePath + ) +} +else { + throw "Unexpected event '$EventName'." +} + +$contexts = @($contexts | Where-Object { $null -ne $_ }) +if ($contexts.Count -eq 0) { + Write-Output 'No eligible automated CI-fix pull request heads require reconciliation.' + exit 0 +} + +$results = [System.Collections.Generic.List[object]]::new() +$verificationFailureContexts = @( + $contexts | Where-Object { + -not [string]::IsNullOrWhiteSpace( + [string](Get-ObjectPropertyValue -InputObject $_ -Name 'VerificationError')) + } +) +$queueContexts = @( + $contexts | Where-Object { + [string]::IsNullOrWhiteSpace( + [string](Get-ObjectPropertyValue -InputObject $_ -Name 'VerificationError')) + } +) +if ($verificationFailureContexts.Count -gt 0) { + foreach ($result in Invoke-CiFixQueueWork -Contexts $verificationFailureContexts -AuthToken '') { + $results.Add($result) + } +} + +if ($DryRun) { + foreach ($context in $queueContexts) { + $contextResults = [System.Collections.Generic.List[object]]::new() + foreach ($pipeline in Get-CiFixPipelineDefinitions) { + $request = New-AzdoQueueRequest -DefinitionId $pipeline.DefinitionId -Context $context + $result = [pscustomobject]@{ + PullRequestNumber = $context.PullRequestNumber + Name = $pipeline.Name + DefinitionId = $pipeline.DefinitionId + Outcome = 'dry-run' + BuildId = $null + Request = $request + } + $results.Add($result) + $contextResults.Add($result) + } + Write-CiFixJobSummary -Context $context -Results $contextResults + } + $results | ConvertTo-Json -Depth 10 + $dryRunFailures = @($results | Where-Object Outcome -eq 'failed') + if ($dryRunFailures.Count -gt 0) { + throw "$($dryRunFailures.Count) of $($results.Count) Azure DevOps validation pipelines failed before dry-run queue payload generation." + } + exit 0 +} + +$pipelines = @(Get-CiFixPipelineDefinitions) +if ($queueContexts.Count -eq 0) { + $results | ConvertTo-Json -Depth 10 + throw "$($results.Count) Azure DevOps validation pipelines failed test-merge verification; no queue requests were attempted." +} + +try { + $authToken = Get-AzdoToken +} +catch { + $authFailure = "Dispatcher authentication failed before queueing: $($_.Exception.Message)" + foreach ($context in $queueContexts) { + $contextResults = @( + foreach ($pipeline in $pipelines) { + New-CiFixFailureResult -Context $context -Pipeline $pipeline -ErrorMessage $authFailure + } + ) + foreach ($result in $contextResults) { + $results.Add($result) + } + Write-CiFixJobSummary -Context $context -Results $contextResults + } + $results | ConvertTo-Json -Depth 10 + throw $authFailure +} + +try { + $queueFixtureData = if ([string]::IsNullOrWhiteSpace($PullRequestsFixturePath)) { + $null + } + else { + Get-Content -Raw -LiteralPath $PullRequestsFixturePath | ConvertFrom-Json -Depth 100 + } + foreach ($result in Invoke-CiFixQueueWork ` + -Contexts $queueContexts ` + -AuthToken $authToken ` + -Repository $Repository ` + -GitHubToken $env:GITHUB_TOKEN ` + -FixtureData $queueFixtureData) { + $results.Add($result) + } +} +finally { + $authToken = $null +} + +$results | ConvertTo-Json -Depth 10 +$failures = @($results | Where-Object Outcome -eq 'failed') +if ($failures.Count -gt 0) { + throw "$($failures.Count) of $($results.Count) Azure DevOps validation pipelines failed to queue or deduplicate." +} diff --git a/.github/workflows/ci-fix-azdo-validation.yml b/.github/workflows/ci-fix-azdo-validation.yml new file mode 100644 index 000000000000..5ed011fdd0b8 --- /dev/null +++ b/.github/workflows/ci-fix-azdo-validation.yml @@ -0,0 +1,53 @@ +name: CI-fix Azure DevOps validation + +on: + pull_request_target: + types: [opened, reopened, synchronize, labeled] + branches: [main] + # CI-fixer PRs and follow-up pushes can be created with GITHUB_TOKEN, which + # suppresses normal event-driven workflows or leaves them approval-gated. + # Reconcile all eligible open heads after either trusted fixer completes. + workflow_run: + workflows: ["CI Failure Fixer (main)", "CI Failure Fixer (net11.0)"] + types: [completed] + +permissions: {} + +concurrency: + # Serialize all queue decisions so event delivery and workflow_run + # reconciliation cannot race the same pipeline + PR head dedupe check. + group: ci-fix-azdo-validation + cancel-in-progress: false + +jobs: + queue-validation: + if: github.repository == 'dotnet/maui' + runs-on: ubuntu-latest + # The absolute dispatcher deadline starts before checkout, preserving a + # real 3-minute reserve for summaries and explicit failure propagation. + timeout-minutes: 10 + permissions: + contents: read + id-token: write + pull-requests: read + steps: + - name: Start dispatcher deadline + shell: bash + run: echo "CI_FIX_DISPATCHER_DEADLINE_UNIX_SECONDS=$(($(date +%s) + 420))" >> "$GITHUB_ENV" + + # github.sha identifies the trusted workflow revision for both event types. + # Never fetch or execute the PR head. + - name: Checkout trusted workflow revision + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + ref: ${{ github.sha }} + fetch-depth: 1 + persist-credentials: false + + - name: Queue Azure DevOps PR validation + shell: pwsh + env: + AZDO_TRIGGER_TENANT_ID: ${{ secrets.AZDO_TRIGGER_TENANT_ID }} + AZDO_TRIGGER_CLIENT_ID: ${{ secrets.AZDO_TRIGGER_CLIENT_ID }} + GITHUB_TOKEN: ${{ github.token }} + run: .github/scripts/Queue-CiFixAzdoValidation.ps1 -DispatcherBudgetSeconds 420 -DispatcherDeadlineUnixSeconds $env:CI_FIX_DISPATCHER_DEADLINE_UNIX_SECONDS diff --git a/.github/workflows/ci-status-fix-net11.lock.yml b/.github/workflows/ci-status-fix-net11.lock.yml index 1f7d7048e00e..de7366bf296f 100644 --- a/.github/workflows/ci-status-fix-net11.lock.yml +++ b/.github/workflows/ci-status-fix-net11.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"f3f68ed16783d2dfb5b689cfa59e8497f6721744ac6da7a72eb09859d8104334","body_hash":"3d8066d926bab743eee8c4ebe6a6e2d8308fc5581d3e143adf59c953744a6a31","compiler_version":"v0.86.2","strict":true,"agent_id":"copilot","agent_model":"gpt-6.1-sol","engine_versions":{"copilot":"1.0.79"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"d683d78d7591e4c6ffd537540bf44fb64e9fba8ce2161051d4c5d4c87069ffaf","body_hash":"403c30218073a3acd2ea3bc73e7608ec7801b89ab53c93d6a4ce279e0901db2f","compiler_version":"v0.86.2","strict":true,"agent_id":"copilot","agent_model":"gpt-6.1-sol","engine_versions":{"copilot":"1.0.79"}} # gh-aw-manifest: {"version":1,"secrets":["COPILOT_PAT_0","COPILOT_PAT_1","COPILOT_PAT_2","COPILOT_PAT_3","COPILOT_PAT_4","COPILOT_PAT_5","COPILOT_PAT_6","COPILOT_PAT_7","COPILOT_PAT_8","COPILOT_PAT_9","GH_AW_CI_TRIGGER_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"6aab9e5b5c91c615506061f09bedd81a23babe3c","version":"v0.86.2"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.44","digest":"sha256:0d727725c737b58c7bdf51f640cffb928385ec46517e0917c7f1a02f1bada8b4","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.44@sha256:0d727725c737b58c7bdf51f640cffb928385ec46517e0917c7f1a02f1bada8b4"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.44","digest":"sha256:b50fbadba138f6e9aba94aca09711335c489bb3b15861220cb66f6092e042dc7","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.44@sha256:b50fbadba138f6e9aba94aca09711335c489bb3b15861220cb66f6092e042dc7"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.44","digest":"sha256:83e48bbe12c634be8c228a576832fe45f66c529ac3659db92bddbcf2eeb6d627","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.44@sha256:83e48bbe12c634be8c228a576832fe45f66c529ac3659db92bddbcf2eeb6d627"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.9","digest":"sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.9@sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196"},{"image":"ghcr.io/github/github-mcp-server:v1.9.0","digest":"sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e","pinned_image":"ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e"}]} # This file was automatically generated by gh-aw (v0.86.2). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -35,9 +35,10 @@ # caused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR # branch (never a second PR) — up to 10 attempts — then stops and defers to # humans (the open tracking issue is the hand-off surface; a dedicated -# [ci-fix-net11][needs-human] PR is planned but currently deferred — see Step 6). CI on -# the PR is kicked by a human `/azp run` for now; the loop watches, classifies, -# and re-fixes autonomously between kicks. When the SPECIFIC test a PR fixed is +# [ci-fix-net11][needs-human] PR is planned but currently deferred — see Step 6). A +# trusted base-branch workflow automatically queues all three MAUI Azure DevOps +# validation pipelines when the PR is opened and after every pushed follow-up +# fix; the loop watches, classifies, and re-fixes autonomously. When the SPECIFIC test a PR fixed is # confirmed green in that PR's own CI, the loop marks the draft PR ready for review # (a state transition only — it never approves or merges). # Never mutes tests, but @@ -1569,7 +1570,7 @@ jobs: uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: WORKFLOW_NAME: "CI Failure Fixer (net11.0)" - WORKFLOW_DESCRIPTION: "Periodic pass over open ci-scan-net11 tracking issues filed by the net11.0 CI\nfailure scanner (.github/workflows/ci-status-net11.md). This workflow targets\nthe `net11.0` branch EXCLUSIVELY: it processes only issues labelled ci-scan-net11 and\nopens every PR against net11.0. (The main branch is handled by the parallel\n.github/workflows/ci-status-fix.md — the two are split because gh-aw can\nonly transport a fix relative to ONE static base branch per workflow, and the\nmain↔net11.0 divergence exceeds gh-aw's 10 MB transport-patch cap.) The fixer\nopens ONE draft [ci-fix-net11] PR per actionable issue against net11.0, then WATCHES\nthat PR's own CI on later runs: when the fix's CI comes back red and the red is\ncaused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR\nbranch (never a second PR) — up to 10 attempts — then stops and defers to\nhumans (the open tracking issue is the hand-off surface; a dedicated\n[ci-fix-net11][needs-human] PR is planned but currently deferred — see Step 6). CI on\nthe PR is kicked by a human `/azp run` for now; the loop watches, classifies,\nand re-fixes autonomously between kicks. When the SPECIFIC test a PR fixed is\nconfirmed green in that PR's own CI, the loop marks the draft PR ready for review\n(a state transition only — it never approves or merges).\nNever mutes tests, but\nde-flakes genuinely flaky ones (deterministic synchronization, no retries /\ntimeout bumps). Always skips visual-regression / screenshot issues." + WORKFLOW_DESCRIPTION: "Periodic pass over open ci-scan-net11 tracking issues filed by the net11.0 CI\nfailure scanner (.github/workflows/ci-status-net11.md). This workflow targets\nthe `net11.0` branch EXCLUSIVELY: it processes only issues labelled ci-scan-net11 and\nopens every PR against net11.0. (The main branch is handled by the parallel\n.github/workflows/ci-status-fix.md — the two are split because gh-aw can\nonly transport a fix relative to ONE static base branch per workflow, and the\nmain↔net11.0 divergence exceeds gh-aw's 10 MB transport-patch cap.) The fixer\nopens ONE draft [ci-fix-net11] PR per actionable issue against net11.0, then WATCHES\nthat PR's own CI on later runs: when the fix's CI comes back red and the red is\ncaused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR\nbranch (never a second PR) — up to 10 attempts — then stops and defers to\nhumans (the open tracking issue is the hand-off surface; a dedicated\n[ci-fix-net11][needs-human] PR is planned but currently deferred — see Step 6). A\ntrusted base-branch workflow automatically queues all three MAUI Azure DevOps\nvalidation pipelines when the PR is opened and after every pushed follow-up\nfix; the loop watches, classifies, and re-fixes autonomously. When the SPECIFIC test a PR fixed is\nconfirmed green in that PR's own CI, the loop marks the draft PR ready for review\n(a state transition only — it never approves or merges).\nNever mutes tests, but\nde-flakes genuinely flaky ones (deterministic synchronization, no retries /\ntimeout bumps). Always skips visual-regression / screenshot issues." HAS_PATCH: ${{ needs.agent.outputs.has_patch }} GH_AW_DETECTION_CONTINUE_ON_ERROR: "true" with: diff --git a/.github/workflows/ci-status-fix-net11.md b/.github/workflows/ci-status-fix-net11.md index f2d26ab90b92..87384dce0c43 100644 --- a/.github/workflows/ci-status-fix-net11.md +++ b/.github/workflows/ci-status-fix-net11.md @@ -13,9 +13,10 @@ description: | caused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR branch (never a second PR) — up to 10 attempts — then stops and defers to humans (the open tracking issue is the hand-off surface; a dedicated - [ci-fix-net11][needs-human] PR is planned but currently deferred — see Step 6). CI on - the PR is kicked by a human `/azp run` for now; the loop watches, classifies, - and re-fixes autonomously between kicks. When the SPECIFIC test a PR fixed is + [ci-fix-net11][needs-human] PR is planned but currently deferred — see Step 6). A + trusted base-branch workflow automatically queues all three MAUI Azure DevOps + validation pipelines when the PR is opened and after every pushed follow-up + fix; the loop watches, classifies, and re-fixes autonomously. When the SPECIFIC test a PR fixed is confirmed green in that PR's own CI, the loop marks the draft PR ready for review (a state transition only — it never approves or merges). Never mutes tests, but @@ -295,8 +296,8 @@ safe-outputs: # were silently dropped, starving the workflow's #1 value (surfacing green PRs # for review). Sized to 6 (not 3) because a single green DRAFT PR that gets # flipped ready in one sweep spends TWO comment slots — the Step 3 ✅ surface - # comment (which still names the /azp-gated legs a human must kick, so it is NOT - # redundant with 🎯) AND the Step 3.6 T3 🎯 readiness comment. At max:3 the shared + # comment (which still names separately queued legs whose automatic run is + # missing, so it is NOT redundant with 🎯) AND the Step 3.6 T3 🎯 readiness comment. At max:3 the shared # bucket drained after ~1 draft flip and T3's atomicity pre-check then deferred # every further mark-ready even while the mark-ready/add_labels buckets (also 3) # sat idle; 6 lets ~3 draft flips (2 comments each) land per sweep, so the @@ -668,8 +669,9 @@ where they are more specific. requested change ONLY if YOU independently confirm it is a correct, in-bounds improvement — never merely because a reviewer asked; (b) stay within the `src/**` + PublicAPI bounds (Step 5.3) and never mute or weaken a test - (Rule 4); (c) the result is a draft PR that still needs a human `/azp run` and - a human merge; (d) the ≤ 10 attempt cap; (e) the loop only ever touches PRs it + (Rule 4); (c) the result is a draft PR whose CI is queued by the trusted + base-branch validator and that still needs a human merge; (d) the ≤ 10 attempt + cap; (e) the loop only ever touches PRs it itself created. Bot reviews — including PAT-based **User**-type automation (`dotnet-bot` / `maui-bot` / `MauiBot`, which `user.type == "User"` does NOT exclude; the R1 login denylist does) — reviews whose association is outside that @@ -1072,9 +1074,10 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: - **1b — Otherwise WAIT.** If the target test has not yet executed (pending/absent on its leg), or a completed red is (or may be) caused by the fix, or `C.dataComplete == false`, or this PR has no identifiable target test → `skipped: PR #

CI pending / - target not yet validated on ; waiting` and stop. *(Round 1: this is where a - maintainer `/azp run` is awaited — the `/azp`-gated uitests/devicetests legs will not - have run until a human kicks them.)* + target not yet validated on ; waiting` and stop. The trusted + `CI-fix Azure DevOps validation` workflow queues the three validation pipelines; if no + matching run appears, treat that as an automation failure rather than asking a + maintainer to start CI. 3. **Green → surface for review.** If `C.overallConclusion == "success"`: the checks that RAN are green. **Primary-gate check first:** the deterministic `success` verdict only certifies "at least one green check, nothing failing or @@ -1084,9 +1087,9 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: `maui-pr` is absent, `neutral`, or `skipped` (e.g. only trivial checks like `license/cla` are green while the build produced no verdict), record `skipped: PR #

primary CI gate (maui-pr) not green on ; waiting` and stop — do NOT - surface. (The `/azp`-gated `maui-pr-uitests` (def 313) / `maui-pr-devicetests` - (def 314) legs MAY still be un-run — that is expected and is named below, not a - reason to withhold the surface.) **Comment idempotency + dry-run suppress the ✅ + surface. (`maui-pr-uitests` (def 313) / `maui-pr-devicetests` (def 314) are + queued automatically for every eligible head; a missing run is named below and + treated as an automation gap.) **Comment idempotency + dry-run suppress the ✅ comment ONLY — neither skips Step 3.6.** Scan the PR's existing comments for a prior bot `✅ … validated … on ` note for THIS head SHA; if one exists, set `SKIP_SURFACE_COMMENT = true`. Resolve the attempt number from `C.effectiveAttempt` @@ -1100,21 +1103,22 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: - else `add_comment` on PR #

: `✅ Attempt /10 validated — the fix's CI is green on .` naming any `/azp`-gated legs (uitests def 313 / devicetests def - 314) that have not run and still need a maintainer `/azp run`; record `surfaced-green + ready-for-review.">` naming any separate legs (uitests def 313 / devicetests def + 314) whose automatic run has not been observed; record `surfaced-green PR #

(attempt /10)`. (Do NOT assert "ready for human review" on a PR that is still a draft — Step 3.6, not this comment, owns the draft→ready flip and posts its own 🎯 announcement when it fires.) Do NOT advance. Then — in ALL of the above cases — run **Step 3.6** (target-test - readiness gate) for this PR before stopping: its `/azp`-gated target legs may conclude - green on this SAME head SHA on a LATER sweep (an `/azp run` adds no commit), and Step + readiness gate) for this PR before stopping: its separately queued target legs may + conclude green on this SAME head SHA on a LATER sweep, and Step 3.6 is the ONLY place the draft→ready flip happens, so it MUST re-evaluate every sweep — never `stop` here before it. *(This directly attacks the real bottleneck — no reviews — so it is the highest-value outcome.)* 4. **Red → classify caused-by-fix vs unrelated-flake.** If `C.overallConclusion == "failure"`, analyze the PR's OWN failing build (NOT `net11.0`): find the AzDO - `maui-pr` build for this PR (filter builds by `branchName=refs/pull/

/merge`, - or match `sourceVersion == C.headSha`), then apply the SAME Step 4 timeline/log + `maui-pr` build for this PR (require BOTH `branchName == refs/pull/

/merge` + AND `triggerInfo["pr.sourceSha"] == C.headSha`; `sourceVersion` is the merge SHA, + not the PR-head identity), then apply the SAME Step 4 timeline/log method and Step 4.7 flake buckets to `C.failedLegs`. - **Unrelated flake only** (every failed leg is known-flaky / infra / a pre-existing baseline red NOT introduced by the fix): do NOT burn an attempt. @@ -1131,13 +1135,13 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: burns the per-run comment budget other PRs need); - else `add_comment` on PR #

: `♻️ Attempt /10: red is unrelated flake on leg(s) () on ; the fix itself is not implicated. A - maintainer re-run (/azp run ) should clear it.` record `annotated-flake + manual /azp rerun remains available if a maintainer wants to retry this same SHA.` + record `annotated-flake PR #

(head )`. Then — in ALL of the above cases — run **Step 3.6** (target-test readiness gate) for - this PR before stopping: its `/azp`-gated target legs may conclude green on this SAME + this PR before stopping: its separately queued target legs may conclude green on this SAME head SHA on a LATER sweep, and Step 3.6 is the ONLY place the draft→ready flip - happens, so it MUST re-evaluate every sweep — never `stop` here before it. *(Round 1: - human re-runs; Round 2: auto re-trigger.)* + happens, so it MUST re-evaluate every sweep — never `stop` here before it. - **Caused by the fix** (a failed leg still matches the original target signature, or the fix introduced a NEW failure): advance an attempt. - **Attempt count.** `attempt = C.effectiveAttempt` — the authoritative @@ -1205,13 +1209,36 @@ pending — the build is green on **every** platform, not just the originally-br cross-platform guard, applied to the build instead of a test). A build-only fix is undrafted ONLY on the strength of the auto-running `maui-pr` (def 302) whole-build green, which the loop CAN observe: if the PR's `[ci-scan]` issue signature or its own diff indicates the -ORIGINATING failure was in a `/azp`-gated pipeline (`maui-pr-uitests` def 313 or +ORIGINATING failure was in a separately queued pipeline (`maui-pr-uitests` def 313 or `maui-pr-devicetests` def 314) rather than `maui-pr` (def 302), do NOT undraft on -`maui-pr`-green alone — those pipelines do not auto-run on this PR (GITHUB_TOKEN cannot -trigger them), so a green `maui-pr` build is NOT evidence the gated build break is fixed; -record `skipped: build-only fix PR #

targets gated pipeline () not run — -deferring to human` and stop this gate. Otherwise, set `TARGET := "the maui-pr build -(build-only fix — no single target test)"` and proceed to **T3** to mark ready. If any `maui-pr` build leg is still unconcluded, record `skipped: build-only fix PR +`maui-pr`-green alone — a green primary build is NOT evidence the separate pipeline break +is fixed. Require the automatic matching run for that pipeline: BOTH +`branchName == refs/pull/

/merge` AND `triggerInfo["pr.sourceSha"] == C.headSha`. +If it is absent, record +`skipped: build-only fix PR #

targets pipeline () whose automatic run was not observed` +and stop this gate. Existence alone is NOT validation. For a matching def 313/314 run, +use the anonymous `_apis/build/builds//timeline` method from T2 and identify the +RELEVANT platform **BUILD** leg(s) that establish the originating build failure was fixed +(from the `[ci-scan-net11]` signature, original failed leg(s), and the PR diff). Require +every such relevant leg to have `state == "completed"` AND `result == "succeeded"`. If the +relevant evidence is absent/unknown, record `skipped: build-only fix PR #

originating +pipeline build has no conclusive relevant platform build-leg evidence +for head ()` and stop. If any relevant leg is pending/inProgress, +record `skipped: build-only fix PR #

originating pipeline build +not yet green on head (relevant build leg(s) pending)` and stop. If any +relevant leg failed/canceled/aborted, record `skipped: build-only fix PR #

originating +pipeline build still fails on head (relevant build leg(s) +: )` and stop. Do NOT require unrelated test legs or unrelated platform-test +flakes in that originating pipeline to be green for a build-only issue; only the relevant +platform BUILD legs prove the originating build break is repaired. Save +`ORIGIN_PIPELINE`, `ORIGIN_BUILD_ID`, and `ORIGIN_BUILD_LEGS` for the T3 audit, then set +`TARGET := "the primary maui-pr build plus the originating relevant platform +build legs (build-only fix — no single target test)"`. If the originating failure was +def 302 itself, no separate-pipeline evidence is required: set `TARGET := "the primary +maui-pr whole build (build-only fix — no single target test)"`. In either case, proceed +to **T3** only after the required primary and, when applicable, originating evidence is +green. +If any `maui-pr` build leg is still unconcluded, record `skipped: build-only fix PR #

not yet whole-build green (leg(s) pending)` and stop this gate WITHOUT marking ready (a green subset is not enough — a still-pending build leg could yet fail). @@ -1242,8 +1269,10 @@ test's UI-test category — its `[Category(UITestCategories.X)]` in the test/Hos visible in the PR diff or the test file — to know which leg-name substring identifies its legs; for a device test, the per-platform device-test legs. -Using the SAME build-discovery as Step 4 (filter AzDO builds by `branchName=refs/pull/

/merge` -or `sourceVersion == C.headSha`), read each build's **timeline** on `C.headSha` for the +Using the SAME build-discovery as Step 4 (require BOTH +`branchName == refs/pull/

/merge` AND `triggerInfo["pr.sourceSha"] == C.headSha`; +`sourceVersion` is the merge SHA, not the PR-head identity), read each build's +**timeline** on `C.headSha` for the pipeline(s) that RUN the target test — `maui-pr` (def 302) for unit/integration tests, `maui-pr-uitests` (def 313) for Appium UI tests, `maui-pr-devicetests` (def 314) for device tests: @@ -1275,20 +1304,21 @@ device-test platform), require ALL of: platform's category leg must have CONCLUDED (`state == "completed"`) on `C.headSha`. If a platform simply has no leg for the target's category, the test does not run there — that is fine, not a gap. But if a platform's category leg has **not concluded** (`state` is - `inProgress` / pending, or an `/azp`-gated `maui-pr-uitests` / `maui-pr-devicetests` leg that - has not been kicked), the test's status on that platform is UNKNOWN → the fix is NOT yet + `inProgress` / pending, or a separately queued `maui-pr-uitests` / `maui-pr-devicetests` leg + whose automatic run was not observed), the test's status on that platform is UNKNOWN → the fix is NOT yet validated across platforms: record `skipped: target test green on but - not yet verified on (leg(s) pending / need /azp run) on PR #

` + not yet verified on (leg(s) pending / automatic run not observed) on PR #

` and stop this gate WITHOUT marking ready. A target test whose category leg never concluded on ANY platform (**not executed** anywhere — -e.g. its `/azp`-gated pipeline has not been kicked) is likewise NOT validated: record `skipped: -target test not yet executed on PR #

( not run — needs /azp run)` and stop +e.g. its automatic pipeline run was not observed) is likewise NOT validated: record `skipped: +target test not yet executed on PR #

( automatic run not observed)` and stop this gate WITHOUT marking ready. Do NOT overclaim — a green *sibling* leg (a different category on the same platform) is not the target's leg, and a green leg on one platform is not a pass on the others. -**T3 — Mark ready + report.** If EVERY target test is VALIDATED-GREEN. The 🎯 comment, +**T3 — Mark ready + report.** If EVERY target test is VALIDATED-GREEN, or a build-only +fix passed every T1 primary/originating-pipeline requirement. The 🎯 comment, the mark-ready, and the `p/0` label are THREE SEPARATE safe-outputs — the comment existing does NOT prove the mark-ready took effect, so they are tracked independently: @@ -1327,14 +1357,22 @@ existing does NOT prove the mark-ready took effect, so they are tracked independ `p/0`; a maintainer still reviews and merges.` (For a **build-only fix** — the T1 whole-build fallback, no single target test — phrase the first clause as `🎯 Build validated green on — the maui-pr build passed on ALL platforms (buildId - ).` instead of naming a test.) + ) + build passed the relevant platform build legs + ">.` instead of naming a test. This audit must name every pipeline, + build, and relevant platform build leg whose evidence was required for readiness.) 2. `mark_pull_request_as_ready_for_review` with `reason:` a one-line justification - naming the validated test(s) and ``. + naming the validated test(s) and ``, or for a build-only fix naming the + primary build plus any validated originating pipeline/build and relevant platform + build legs. 3. `add_labels` with `labels: ["p/0"]` for PR #

— put the now-review-ready fix into the team's p/0 priority queue so it is triaged, not lost in the draft backlog. (If the PR somehow already carries `p/0`, this is a harmless no-op.) - Record `marked-ready PR #

(target green on ALL platforms on , - labeled p/0)` and stop. + labeled p/0)` for a test fix, or `marked-ready PR #

(build-only: maui-pr + whole-build green + relevant build legs green"> on , + labeled p/0)` for a build-only fix, and stop. #### Step 3.5.R — Maintainer change-request response (Track C) @@ -1558,8 +1596,8 @@ R1's idempotency guard treats this review as answered and never re-processes it. append a "previous approaches" row) + `add_comment`. The comment MUST be clearly AI-generated, embed the `ci-fix-track-c-responded: ` marker, and list, per finding, what you **applied**, and for each **PUSH BACK** state plainly why you did - not change it (technical reason). In round 1, add the `A maintainer needs to - comment /azp run maui-pr …` reminder. + not change it (technical reason). In round 1, add the `Automatic validation is + queued for this head; wait for the matching CI results before continuing` reminder. - **If everything was PUSH BACK (no commit):** emit ONLY `add_comment` on `N` embedding the `ci-fix-track-c-responded: ` marker and stating, per finding, why you did not change it. NO push, NO attempt-marker bump — a courteous decline @@ -1578,8 +1616,9 @@ This is the "is the issue actually fixed?" check. **Mode note.** In **FRESH** mode (Step 3.4) verify against the latest completed `net11.0` build (below). In **ADVANCE** mode (Step 3.5) the PR is already red — its own build IS the reproduction: run the SAME timeline/log analysis against the PR's -`maui-pr` build for `C.headSha` (filter builds by `branchName=refs/pull/

/merge` -or match `sourceVersion`), extract the still-failing signature, and carry it into +`maui-pr` build for `C.headSha` (require BOTH `branchName == refs/pull/

/merge` +AND `triggerInfo["pr.sourceSha"] == C.headSha`; do not use the merge-SHA +`sourceVersion` as PR-head identity), extract the still-failing signature, and carry it into Step 5. Skip the net11.0-build fetch in ADVANCE mode. 1. Map the issue's `Pipeline` to its definition ID (302 / 314 / 313). @@ -1910,9 +1949,9 @@ targeting `advance_pr` (the open PR number from Step 3.5): 4. Register `add_comment`, then call it once (`pull_request_number: `): a short `🔁 Attempt /10: .` Then, in - round 1, `A maintainer needs to comment /azp run maui-pr (and the gated - uitests/devicetests legs if relevant) to exercise this commit.` Keep it under + attempt, and why the previously-red signature should now clear>. The trusted + CI-fix validator automatically queues maui-pr, uitests, and devicetests for + this commit.` Keep it under 2,000 characters. > **Dry-run gate (Step 0):** if `dry_run == "true"`, emit NONE of the above. @@ -1998,7 +2037,7 @@ Flake class: test-quality - Latest verified-failing build: https://dev.azure.com/dnceng-public/public/_build/results?buildId= --- -Filed by [`ci-status-fix-net11`](https://github.com/dotnet/maui/blob/main/.github/workflows/ci-status-fix-net11.md). This is the single PR for dotnet/maui#: the workflow watches its own CI and pushes up to **10 attempts on this same PR** (it never opens a second PR). It advances only when the fix's own build settles red and that red is caused by the fix. Comments, reviews, and commits do not transfer ownership; the loop remains autonomous until this PR is closed. Eligible `CHANGES_REQUESTED` reviews are handled through Track C. After 10 attempts it stops and defers to humans. In round 1 a maintainer still needs to comment `/azp run maui-pr` (plus the gated uitests/devicetests legs when relevant) to exercise each new commit. +Filed by [`ci-status-fix-net11`](https://github.com/dotnet/maui/blob/main/.github/workflows/ci-status-fix-net11.md). This is the single PR for dotnet/maui#: the workflow watches its own CI and pushes up to **10 attempts on this same PR** (it never opens a second PR). It advances only when the fix's own build settles red and that red is caused by the fix. Comments, reviews, and commits do not transfer ownership; the loop remains autonomous until this PR is closed. Eligible `CHANGES_REQUESTED` reviews are handled through Track C. After 10 attempts it stops and defers to humans. The trusted CI-fix validator automatically queues maui-pr, uitests, and devicetests for every new PR head. ```` `Fixes #` is intentionally NOT in the body. The tracking issue is locked diff --git a/.github/workflows/ci-status-fix.lock.yml b/.github/workflows/ci-status-fix.lock.yml index 6eee3c7c2d67..9f3f2e891444 100644 --- a/.github/workflows/ci-status-fix.lock.yml +++ b/.github/workflows/ci-status-fix.lock.yml @@ -1,4 +1,4 @@ -# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"ac4777219326eb543debdc2b809019939e6a7c0a76918c3e419b307624777900","body_hash":"65f3513f049d81447ce2c166f596b4df9928d87de5bb4db52134244dea1b030f","compiler_version":"v0.86.2","strict":true,"agent_id":"copilot","agent_model":"gpt-6.1-sol","engine_versions":{"copilot":"1.0.79"}} +# gh-aw-metadata: {"schema_version":"v4","frontmatter_hash":"0a2fab5e8553740e2264d6db88e423885eb49c8aeff820d870d6b1c44b3bf1ba","body_hash":"f02e70a8b091b54d4f7fc31fc3cce44993c9eec8444b727d2857c76d19e29739","compiler_version":"v0.86.2","strict":true,"agent_id":"copilot","agent_model":"gpt-6.1-sol","engine_versions":{"copilot":"1.0.79"}} # gh-aw-manifest: {"version":1,"secrets":["COPILOT_PAT_0","COPILOT_PAT_1","COPILOT_PAT_2","COPILOT_PAT_3","COPILOT_PAT_4","COPILOT_PAT_5","COPILOT_PAT_6","COPILOT_PAT_7","COPILOT_PAT_8","COPILOT_PAT_9","GH_AW_CI_TRIGGER_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN"],"actions":[{"repo":"actions/checkout","sha":"3d3c42e5aac5ba805825da76410c181273ba90b1","version":"v7.0.1"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9.0.0"},{"repo":"actions/setup-node","sha":"820762786026740c76f36085b0efc47a31fe5020","version":"v7.0.0"},{"repo":"actions/upload-artifact","sha":"043fb46d1a93c77aae656e7c1c64a875d1fc6a0a","version":"v7.0.1"},{"repo":"github/gh-aw-actions/setup","sha":"6aab9e5b5c91c615506061f09bedd81a23babe3c","version":"v0.86.2"}],"containers":[{"image":"ghcr.io/github/gh-aw-firewall/agent:0.27.44","digest":"sha256:0d727725c737b58c7bdf51f640cffb928385ec46517e0917c7f1a02f1bada8b4","pinned_image":"ghcr.io/github/gh-aw-firewall/agent:0.27.44@sha256:0d727725c737b58c7bdf51f640cffb928385ec46517e0917c7f1a02f1bada8b4"},{"image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.44","digest":"sha256:b50fbadba138f6e9aba94aca09711335c489bb3b15861220cb66f6092e042dc7","pinned_image":"ghcr.io/github/gh-aw-firewall/api-proxy:0.27.44@sha256:b50fbadba138f6e9aba94aca09711335c489bb3b15861220cb66f6092e042dc7"},{"image":"ghcr.io/github/gh-aw-firewall/squid:0.27.44","digest":"sha256:83e48bbe12c634be8c228a576832fe45f66c529ac3659db92bddbcf2eeb6d627","pinned_image":"ghcr.io/github/gh-aw-firewall/squid:0.27.44@sha256:83e48bbe12c634be8c228a576832fe45f66c529ac3659db92bddbcf2eeb6d627"},{"image":"ghcr.io/github/gh-aw-mcpg:v0.4.9","digest":"sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f","pinned_image":"ghcr.io/github/gh-aw-mcpg:v0.4.9@sha256:e5a1569aeaf41820fa7bdee3e94468cae448133cdbf00119ad24f5b74db1ab9f"},{"image":"ghcr.io/github/gh-aw-node","digest":"sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196","pinned_image":"ghcr.io/github/gh-aw-node@sha256:0d9f1fb5fd6610c0ac1f5194a38e45a8a1e81f8a390d5142d8e4e6f26a4b3196"},{"image":"ghcr.io/github/github-mcp-server:v1.9.0","digest":"sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e","pinned_image":"ghcr.io/github/github-mcp-server:v1.9.0@sha256:881b53d6f75f69bdbc1b5b10fc2f1361717c19054143b3a8529fb5c32061a50e"}]} # This file was automatically generated by gh-aw (v0.86.2). DO NOT EDIT. To debug this workflow, load the skill at https://github.com/github/gh-aw/blob/main/debug.md # @@ -35,9 +35,10 @@ # caused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR # branch (never a second PR) — up to 10 attempts — then stops and defers to # humans (the open tracking issue is the hand-off surface; a dedicated -# [ci-fix][needs-human] PR is planned but currently deferred — see Step 6). CI on -# the PR is kicked by a human `/azp run` for now; the loop watches, classifies, -# and re-fixes autonomously between kicks. When the SPECIFIC test a PR fixed is +# [ci-fix][needs-human] PR is planned but currently deferred — see Step 6). A +# trusted base-branch workflow automatically queues all three MAUI Azure DevOps +# validation pipelines when the PR is opened and after every pushed follow-up +# fix; the loop watches, classifies, and re-fixes autonomously. When the SPECIFIC test a PR fixed is # confirmed green in that PR's own CI, the loop marks the draft PR ready for review # (a state transition only — it never approves or merges). # Never mutes tests, but @@ -1561,7 +1562,7 @@ jobs: uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0 env: WORKFLOW_NAME: "CI Failure Fixer (main)" - WORKFLOW_DESCRIPTION: "Periodic pass over open ci-scan tracking issues filed by the main-branch CI\nfailure scanner (.github/workflows/ci-status-main.md). This workflow targets\nthe `main` branch EXCLUSIVELY: it processes only issues labelled ci-scan and\nopens every PR against main. (The net11.0 branch is handled by the parallel\n.github/workflows/ci-status-fix-net11.md — the two are split because gh-aw can\nonly transport a fix relative to ONE static base branch per workflow, and the\nmain↔net11.0 divergence exceeds gh-aw's 10 MB transport-patch cap.) The fixer\nopens ONE draft [ci-fix] PR per actionable issue against main, then WATCHES\nthat PR's own CI on later runs: when the fix's CI comes back red and the red is\ncaused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR\nbranch (never a second PR) — up to 10 attempts — then stops and defers to\nhumans (the open tracking issue is the hand-off surface; a dedicated\n[ci-fix][needs-human] PR is planned but currently deferred — see Step 6). CI on\nthe PR is kicked by a human `/azp run` for now; the loop watches, classifies,\nand re-fixes autonomously between kicks. When the SPECIFIC test a PR fixed is\nconfirmed green in that PR's own CI, the loop marks the draft PR ready for review\n(a state transition only — it never approves or merges).\nNever mutes tests, but\nde-flakes genuinely flaky ones (deterministic synchronization, no retries /\ntimeout bumps). Always skips visual-regression / screenshot issues." + WORKFLOW_DESCRIPTION: "Periodic pass over open ci-scan tracking issues filed by the main-branch CI\nfailure scanner (.github/workflows/ci-status-main.md). This workflow targets\nthe `main` branch EXCLUSIVELY: it processes only issues labelled ci-scan and\nopens every PR against main. (The net11.0 branch is handled by the parallel\n.github/workflows/ci-status-fix-net11.md — the two are split because gh-aw can\nonly transport a fix relative to ONE static base branch per workflow, and the\nmain↔net11.0 divergence exceeds gh-aw's 10 MB transport-patch cap.) The fixer\nopens ONE draft [ci-fix] PR per actionable issue against main, then WATCHES\nthat PR's own CI on later runs: when the fix's CI comes back red and the red is\ncaused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR\nbranch (never a second PR) — up to 10 attempts — then stops and defers to\nhumans (the open tracking issue is the hand-off surface; a dedicated\n[ci-fix][needs-human] PR is planned but currently deferred — see Step 6). A\ntrusted base-branch workflow automatically queues all three MAUI Azure DevOps\nvalidation pipelines when the PR is opened and after every pushed follow-up\nfix; the loop watches, classifies, and re-fixes autonomously. When the SPECIFIC test a PR fixed is\nconfirmed green in that PR's own CI, the loop marks the draft PR ready for review\n(a state transition only — it never approves or merges).\nNever mutes tests, but\nde-flakes genuinely flaky ones (deterministic synchronization, no retries /\ntimeout bumps). Always skips visual-regression / screenshot issues." HAS_PATCH: ${{ needs.agent.outputs.has_patch }} GH_AW_DETECTION_CONTINUE_ON_ERROR: "true" with: diff --git a/.github/workflows/ci-status-fix.md b/.github/workflows/ci-status-fix.md index e2d5850b1d23..f0a6cd4db73f 100644 --- a/.github/workflows/ci-status-fix.md +++ b/.github/workflows/ci-status-fix.md @@ -13,9 +13,10 @@ description: | caused by the fix itself, it pushes a fresh follow-up fix onto the SAME PR branch (never a second PR) — up to 10 attempts — then stops and defers to humans (the open tracking issue is the hand-off surface; a dedicated - [ci-fix][needs-human] PR is planned but currently deferred — see Step 6). CI on - the PR is kicked by a human `/azp run` for now; the loop watches, classifies, - and re-fixes autonomously between kicks. When the SPECIFIC test a PR fixed is + [ci-fix][needs-human] PR is planned but currently deferred — see Step 6). A + trusted base-branch workflow automatically queues all three MAUI Azure DevOps + validation pipelines when the PR is opened and after every pushed follow-up + fix; the loop watches, classifies, and re-fixes autonomously. When the SPECIFIC test a PR fixed is confirmed green in that PR's own CI, the loop marks the draft PR ready for review (a state transition only — it never approves or merges). Never mutes tests, but @@ -280,8 +281,8 @@ safe-outputs: # were silently dropped, starving the workflow's #1 value (surfacing green PRs # for review). Sized to 6 (not 3) because a single green DRAFT PR that gets # flipped ready in one sweep spends TWO comment slots — the Step 3 ✅ surface - # comment (which still names the /azp-gated legs a human must kick, so it is NOT - # redundant with 🎯) AND the Step 3.6 T3 🎯 readiness comment. At max:3 the shared + # comment (which still names separately queued legs whose automatic run is + # missing, so it is NOT redundant with 🎯) AND the Step 3.6 T3 🎯 readiness comment. At max:3 the shared # bucket drained after ~1 draft flip and T3's atomicity pre-check then deferred # every further mark-ready even while the mark-ready/add_labels buckets (also 3) # sat idle; 6 lets ~3 draft flips (2 comments each) land per sweep, so the @@ -652,8 +653,9 @@ where they are more specific. requested change ONLY if YOU independently confirm it is a correct, in-bounds improvement — never merely because a reviewer asked; (b) stay within the `src/**` + PublicAPI bounds (Step 5.3) and never mute or weaken a test - (Rule 4); (c) the result is a draft PR that still needs a human `/azp run` and - a human merge; (d) the ≤ 10 attempt cap; (e) the loop only ever touches PRs it + (Rule 4); (c) the result is a draft PR whose CI is queued by the trusted + base-branch validator and that still needs a human merge; (d) the ≤ 10 attempt + cap; (e) the loop only ever touches PRs it itself created. Bot reviews — including PAT-based **User**-type automation (`dotnet-bot` / `maui-bot` / `MauiBot`, which `user.type == "User"` does NOT exclude; the R1 login denylist does) — reviews whose association is outside that @@ -1056,9 +1058,10 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: - **1b — Otherwise WAIT.** If the target test has not yet executed (pending/absent on its leg), or a completed red is (or may be) caused by the fix, or `C.dataComplete == false`, or this PR has no identifiable target test → `skipped: PR #

CI pending / - target not yet validated on ; waiting` and stop. *(Round 1: this is where a - maintainer `/azp run` is awaited — the `/azp`-gated uitests/devicetests legs will not - have run until a human kicks them.)* + target not yet validated on ; waiting` and stop. The trusted + `CI-fix Azure DevOps validation` workflow queues the three validation pipelines; if no + matching run appears, treat that as an automation failure rather than asking a + maintainer to start CI. 3. **Green → surface for review.** If `C.overallConclusion == "success"`: the checks that RAN are green. **Primary-gate check first:** the deterministic `success` verdict only certifies "at least one green check, nothing failing or @@ -1068,9 +1071,9 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: `maui-pr` is absent, `neutral`, or `skipped` (e.g. only trivial checks like `license/cla` are green while the build produced no verdict), record `skipped: PR #

primary CI gate (maui-pr) not green on ; waiting` and stop — do NOT - surface. (The `/azp`-gated `maui-pr-uitests` (def 313) / `maui-pr-devicetests` - (def 314) legs MAY still be un-run — that is expected and is named below, not a - reason to withhold the surface.) **Comment idempotency + dry-run suppress the ✅ + surface. (`maui-pr-uitests` (def 313) / `maui-pr-devicetests` (def 314) are + queued automatically for every eligible head; a missing run is named below and + treated as an automation gap.) **Comment idempotency + dry-run suppress the ✅ comment ONLY — neither skips Step 3.6.** Scan the PR's existing comments for a prior bot `✅ … validated … on ` note for THIS head SHA; if one exists, set `SKIP_SURFACE_COMMENT = true`. Resolve the attempt number from `C.effectiveAttempt` @@ -1084,21 +1087,22 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: - else `add_comment` on PR #

: `✅ Attempt /10 validated — the fix's CI is green on .` naming any `/azp`-gated legs (uitests def 313 / devicetests def - 314) that have not run and still need a maintainer `/azp run`; record `surfaced-green + ready-for-review.">` naming any separate legs (uitests def 313 / devicetests def + 314) whose automatic run has not been observed; record `surfaced-green PR #

(attempt /10)`. (Do NOT assert "ready for human review" on a PR that is still a draft — Step 3.6, not this comment, owns the draft→ready flip and posts its own 🎯 announcement when it fires.) Do NOT advance. Then — in ALL of the above cases — run **Step 3.6** (target-test - readiness gate) for this PR before stopping: its `/azp`-gated target legs may conclude - green on this SAME head SHA on a LATER sweep (an `/azp run` adds no commit), and Step + readiness gate) for this PR before stopping: its separately queued target legs may + conclude green on this SAME head SHA on a LATER sweep, and Step 3.6 is the ONLY place the draft→ready flip happens, so it MUST re-evaluate every sweep — never `stop` here before it. *(This directly attacks the real bottleneck — no reviews — so it is the highest-value outcome.)* 4. **Red → classify caused-by-fix vs unrelated-flake.** If `C.overallConclusion == "failure"`, analyze the PR's OWN failing build (NOT `main`): find the AzDO - `maui-pr` build for this PR (filter builds by `branchName=refs/pull/

/merge`, - or match `sourceVersion == C.headSha`), then apply the SAME Step 4 timeline/log + `maui-pr` build for this PR (require BOTH `branchName == refs/pull/

/merge` + AND `triggerInfo["pr.sourceSha"] == C.headSha`; `sourceVersion` is the merge SHA, + not the PR-head identity), then apply the SAME Step 4 timeline/log method and Step 4.7 flake buckets to `C.failedLegs`. - **Unrelated flake only** (every failed leg is known-flaky / infra / a pre-existing baseline red NOT introduced by the fix): do NOT burn an attempt. @@ -1115,13 +1119,13 @@ Run these gates in order — the FIRST that fires decides this cycle's outcome: burns the per-run comment budget other PRs need); - else `add_comment` on PR #

: `♻️ Attempt /10: red is unrelated flake on leg(s) () on ; the fix itself is not implicated. A - maintainer re-run (/azp run ) should clear it.` record `annotated-flake + manual /azp rerun remains available if a maintainer wants to retry this same SHA.` + record `annotated-flake PR #

(head )`. Then — in ALL of the above cases — run **Step 3.6** (target-test readiness gate) for - this PR before stopping: its `/azp`-gated target legs may conclude green on this SAME + this PR before stopping: its separately queued target legs may conclude green on this SAME head SHA on a LATER sweep, and Step 3.6 is the ONLY place the draft→ready flip - happens, so it MUST re-evaluate every sweep — never `stop` here before it. *(Round 1: - human re-runs; Round 2: auto re-trigger.)* + happens, so it MUST re-evaluate every sweep — never `stop` here before it. - **Caused by the fix** (a failed leg still matches the original target signature, or the fix introduced a NEW failure): advance an attempt. - **Attempt count.** `attempt = C.effectiveAttempt` — the authoritative @@ -1189,13 +1193,36 @@ pending — the build is green on **every** platform, not just the originally-br cross-platform guard, applied to the build instead of a test). A build-only fix is undrafted ONLY on the strength of the auto-running `maui-pr` (def 302) whole-build green, which the loop CAN observe: if the PR's `[ci-scan]` issue signature or its own diff indicates the -ORIGINATING failure was in a `/azp`-gated pipeline (`maui-pr-uitests` def 313 or +ORIGINATING failure was in a separately queued pipeline (`maui-pr-uitests` def 313 or `maui-pr-devicetests` def 314) rather than `maui-pr` (def 302), do NOT undraft on -`maui-pr`-green alone — those pipelines do not auto-run on this PR (GITHUB_TOKEN cannot -trigger them), so a green `maui-pr` build is NOT evidence the gated build break is fixed; -record `skipped: build-only fix PR #

targets gated pipeline () not run — -deferring to human` and stop this gate. Otherwise, set `TARGET := "the maui-pr build -(build-only fix — no single target test)"` and proceed to **T3** to mark ready. If any `maui-pr` build leg is still unconcluded, record `skipped: build-only fix PR +`maui-pr`-green alone — a green primary build is NOT evidence the separate pipeline break +is fixed. Require the automatic matching run for that pipeline: BOTH +`branchName == refs/pull/

/merge` AND `triggerInfo["pr.sourceSha"] == C.headSha`. +If it is absent, record +`skipped: build-only fix PR #

targets pipeline () whose automatic run was not observed` +and stop this gate. Existence alone is NOT validation. For a matching def 313/314 run, +use the anonymous `_apis/build/builds//timeline` method from T2 and identify the +RELEVANT platform **BUILD** leg(s) that establish the originating build failure was fixed +(from the `[ci-scan]` signature, original failed leg(s), and the PR diff). Require every +such relevant leg to have `state == "completed"` AND `result == "succeeded"`. If the +relevant evidence is absent/unknown, record `skipped: build-only fix PR #

originating +pipeline build has no conclusive relevant platform build-leg evidence +for head ()` and stop. If any relevant leg is pending/inProgress, +record `skipped: build-only fix PR #

originating pipeline build +not yet green on head (relevant build leg(s) pending)` and stop. If any +relevant leg failed/canceled/aborted, record `skipped: build-only fix PR #

originating +pipeline build still fails on head (relevant build leg(s) +: )` and stop. Do NOT require unrelated test legs or unrelated platform-test +flakes in that originating pipeline to be green for a build-only issue; only the relevant +platform BUILD legs prove the originating build break is repaired. Save +`ORIGIN_PIPELINE`, `ORIGIN_BUILD_ID`, and `ORIGIN_BUILD_LEGS` for the T3 audit, then set +`TARGET := "the primary maui-pr build plus the originating relevant platform +build legs (build-only fix — no single target test)"`. If the originating failure was +def 302 itself, no separate-pipeline evidence is required: set `TARGET := "the primary +maui-pr whole build (build-only fix — no single target test)"`. In either case, proceed +to **T3** only after the required primary and, when applicable, originating evidence is +green. +If any `maui-pr` build leg is still unconcluded, record `skipped: build-only fix PR #

not yet whole-build green (leg(s) pending)` and stop this gate WITHOUT marking ready (a green subset is not enough — a still-pending build leg could yet fail). @@ -1226,8 +1253,10 @@ test's UI-test category — its `[Category(UITestCategories.X)]` in the test/Hos visible in the PR diff or the test file — to know which leg-name substring identifies its legs; for a device test, the per-platform device-test legs. -Using the SAME build-discovery as Step 4 (filter AzDO builds by `branchName=refs/pull/

/merge` -or `sourceVersion == C.headSha`), read each build's **timeline** on `C.headSha` for the +Using the SAME build-discovery as Step 4 (require BOTH +`branchName == refs/pull/

/merge` AND `triggerInfo["pr.sourceSha"] == C.headSha`; +`sourceVersion` is the merge SHA, not the PR-head identity), read each build's +**timeline** on `C.headSha` for the pipeline(s) that RUN the target test — `maui-pr` (def 302) for unit/integration tests, `maui-pr-uitests` (def 313) for Appium UI tests, `maui-pr-devicetests` (def 314) for device tests: @@ -1259,20 +1288,21 @@ device-test platform), require ALL of: platform's category leg must have CONCLUDED (`state == "completed"`) on `C.headSha`. If a platform simply has no leg for the target's category, the test does not run there — that is fine, not a gap. But if a platform's category leg has **not concluded** (`state` is - `inProgress` / pending, or an `/azp`-gated `maui-pr-uitests` / `maui-pr-devicetests` leg that - has not been kicked), the test's status on that platform is UNKNOWN → the fix is NOT yet + `inProgress` / pending, or a separately queued `maui-pr-uitests` / `maui-pr-devicetests` leg + whose automatic run was not observed), the test's status on that platform is UNKNOWN → the fix is NOT yet validated across platforms: record `skipped: target test green on but - not yet verified on (leg(s) pending / need /azp run) on PR #

` + not yet verified on (leg(s) pending / automatic run not observed) on PR #

` and stop this gate WITHOUT marking ready. A target test whose category leg never concluded on ANY platform (**not executed** anywhere — -e.g. its `/azp`-gated pipeline has not been kicked) is likewise NOT validated: record `skipped: -target test not yet executed on PR #

( not run — needs /azp run)` and stop +e.g. its automatic pipeline run was not observed) is likewise NOT validated: record `skipped: +target test not yet executed on PR #

( automatic run not observed)` and stop this gate WITHOUT marking ready. Do NOT overclaim — a green *sibling* leg (a different category on the same platform) is not the target's leg, and a green leg on one platform is not a pass on the others. -**T3 — Mark ready + report.** If EVERY target test is VALIDATED-GREEN. The 🎯 comment, +**T3 — Mark ready + report.** If EVERY target test is VALIDATED-GREEN, or a build-only +fix passed every T1 primary/originating-pipeline requirement. The 🎯 comment, the mark-ready, and the `p/0` label are THREE SEPARATE safe-outputs — the comment existing does NOT prove the mark-ready took effect, so they are tracked independently: @@ -1311,14 +1341,22 @@ existing does NOT prove the mark-ready took effect, so they are tracked independ `p/0`; a maintainer still reviews and merges.` (For a **build-only fix** — the T1 whole-build fallback, no single target test — phrase the first clause as `🎯 Build validated green on — the maui-pr build passed on ALL platforms (buildId - ).` instead of naming a test.) + ) + build passed the relevant platform build legs + ">.` instead of naming a test. This audit must name every pipeline, + build, and relevant platform build leg whose evidence was required for readiness.) 2. `mark_pull_request_as_ready_for_review` with `reason:` a one-line justification - naming the validated test(s) and ``. + naming the validated test(s) and ``, or for a build-only fix naming the + primary build plus any validated originating pipeline/build and relevant platform + build legs. 3. `add_labels` with `labels: ["p/0"]` for PR #

— put the now-review-ready fix into the team's p/0 priority queue so it is triaged, not lost in the draft backlog. (If the PR somehow already carries `p/0`, this is a harmless no-op.) - Record `marked-ready PR #

(target green on ALL platforms on , - labeled p/0)` and stop. + labeled p/0)` for a test fix, or `marked-ready PR #

(build-only: maui-pr + whole-build green + relevant build legs green"> on , + labeled p/0)` for a build-only fix, and stop. #### Step 3.5.R — Maintainer change-request response (Track C) @@ -1542,8 +1580,8 @@ R1's idempotency guard treats this review as answered and never re-processes it. append a "previous approaches" row) + `add_comment`. The comment MUST be clearly AI-generated, embed the `ci-fix-track-c-responded: ` marker, and list, per finding, what you **applied**, and for each **PUSH BACK** state plainly why you did - not change it (technical reason). In round 1, add the `A maintainer needs to - comment /azp run maui-pr …` reminder. + not change it (technical reason). In round 1, add the `Automatic validation is + queued for this head; wait for the matching CI results before continuing` reminder. - **If everything was PUSH BACK (no commit):** emit ONLY `add_comment` on `N` embedding the `ci-fix-track-c-responded: ` marker and stating, per finding, why you did not change it. NO push, NO attempt-marker bump — a courteous decline @@ -1562,8 +1600,9 @@ This is the "is the issue actually fixed?" check. **Mode note.** In **FRESH** mode (Step 3.4) verify against the latest completed `main` build (below). In **ADVANCE** mode (Step 3.5) the PR is already red — its own build IS the reproduction: run the SAME timeline/log analysis against the PR's -`maui-pr` build for `C.headSha` (filter builds by `branchName=refs/pull/

/merge` -or match `sourceVersion`), extract the still-failing signature, and carry it into +`maui-pr` build for `C.headSha` (require BOTH `branchName == refs/pull/

/merge` +AND `triggerInfo["pr.sourceSha"] == C.headSha`; do not use the merge-SHA +`sourceVersion` as PR-head identity), extract the still-failing signature, and carry it into Step 5. Skip the main-build fetch in ADVANCE mode. 1. Map the issue's `Pipeline` to its definition ID (302 / 314 / 313). @@ -1892,9 +1931,9 @@ targeting `advance_pr` (the open PR number from Step 3.5): 4. Register `add_comment`, then call it once (`pull_request_number: `): a short `🔁 Attempt /10: .` Then, in - round 1, `A maintainer needs to comment /azp run maui-pr (and the gated - uitests/devicetests legs if relevant) to exercise this commit.` Keep it under + attempt, and why the previously-red signature should now clear>. The trusted + CI-fix validator automatically queues maui-pr, uitests, and devicetests for + this commit.` Keep it under 2,000 characters. > **Dry-run gate (Step 0):** if `dry_run == "true"`, emit NONE of the above. @@ -1980,7 +2019,7 @@ Flake class: test-quality - Latest verified-failing build: https://dev.azure.com/dnceng-public/public/_build/results?buildId= --- -Filed by [`ci-status-fix`](https://github.com/dotnet/maui/blob/main/.github/workflows/ci-status-fix.md). This is the single PR for dotnet/maui#: the workflow watches its own CI and pushes up to **10 attempts on this same PR** (it never opens a second PR). It advances only when the fix's own build settles red and that red is caused by the fix. Comments, reviews, and commits do not transfer ownership; the loop remains autonomous until this PR is closed. Eligible `CHANGES_REQUESTED` reviews are handled through Track C. After 10 attempts it stops and defers to humans. In round 1 a maintainer still needs to comment `/azp run maui-pr` (plus the gated uitests/devicetests legs when relevant) to exercise each new commit. +Filed by [`ci-status-fix`](https://github.com/dotnet/maui/blob/main/.github/workflows/ci-status-fix.md). This is the single PR for dotnet/maui#: the workflow watches its own CI and pushes up to **10 attempts on this same PR** (it never opens a second PR). It advances only when the fix's own build settles red and that red is caused by the fix. Comments, reviews, and commits do not transfer ownership; the loop remains autonomous until this PR is closed. Eligible `CHANGES_REQUESTED` reviews are handled through Track C. After 10 attempts it stops and defers to humans. The trusted CI-fix validator automatically queues maui-pr, uitests, and devicetests for every new PR head. ```` `Fixes #` is intentionally NOT in the body. The tracking issue is locked