feat(triage): add a deterministic flakiness gate to sandboxed verification - #9130
feat(triage): add a deterministic flakiness gate to sandboxed verification#9130wenshao wants to merge 12 commits into
Conversation
…ation Closes #9125. PR #9086's ~50% mtime-assertion flake passed every automated layer because each executed the changed tests exactly once — a coin flip a single green run cannot distinguish from health. The gate re-runs the PR's added/modified unit-test files N times (default 5, vars.QWEN_VERIFY_FLAKE_ROUNDS to override, clamped to 2..10) through the same entry points CI uses and compares outcomes per group across rounds. Design constraints, each pinned by a workflow test: - One-way authority: 'flaky' demotes the published headline (even a trusted agent merge-ready); no gate value can raise or soften one. The gate runs the PR's own test code, so it can always be neutered — but a gate that can only demote is not worth forging. - Divergence-only signal: a group failing identically every round is deterministic (CI owns it) and an environment-sensitive suite must not false-positive here; both report informationally, never demote. - Fail open: the gate is not under -e and every terminal path exits 0 — a gate bug reports verdict 'error' instead of taking down the verify lane. - Honest file list: recorded from HEAD^1..HEAD before install/build hands the workspace (and .git) to PR lifecycle code; the gate consumes the root-owned recorded list and never re-derives the diff. - Untrusted text stays out of outputs: summaries are fixed text plus counters; PR-controlled paths live in flake-gate.log, embedded through the publisher's escaping emit_block. Job timeout raised 150 -> 175 for the gate's ~25m worst case (15m round budget checked before each invocation + one 10m-capped in-flight run).
github-actions
Bot
added
the
review/self-reported
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
@wenshao The PR body is missing required headings from the PR template: Reviewer Test Plan (How to verify / Evidence / Tested on), Risk & Scope, and Linked Issues — only What this PR does and Why it's needed are present.
Most of the content is already there under non-template headings — Verification reads as the reviewer test plan, and Closes #9125 is stated inline — so this is a restructure rather than new work. The one section that is substantively worth writing for a verify-lane change is Risk & Scope: the main tradeoff (the timeout bump 150 → 175 min and up to ~25 minutes added to a verify run in the worst case, both per the PR body), what is not validated / out of scope, and migration notes (e.g. vars.QWEN_VERIFY_FLAKE_ROUNDS only needs to be set to override the default of 5 rounds).
Please align the body with the template, then re-trigger with @qwen-code /triage.
中文说明
@wenshao PR 正文缺少 PR 模板要求的小节：Reviewer Test Plan（How to verify / Evidence / Tested on）、Risk & Scope、Linked Issues——目前只有 What this PR does 和 Why it's needed。
内容其实基本都在，只是挂在非模板标题下——Verification 实际上就是 reviewer 测试计划，Closes #9125 也写在了正文里——所以只需重排结构，不用补写新内容。对这类 verify 通道改动，真正值得补的是 Risk & Scope：主要权衡（超时 150 → 175 分钟、最坏情况下单次 verify 增加约 25 分钟，均引自 PR 正文）、未验证/超出范围的部分，以及迁移说明（例如 vars.QWEN_VERIFY_FLAKE_ROUNDS 仅在需要覆盖默认 5 轮时才要配置）。
请按模板调整正文后，用 @qwen-code /triage 重新触发。
— Qwen Code · qwen3.8-max
Code Coverage Summary
CLI Package - Full Text ReportCore Package - Full Text ReportFor detailed HTML reports, please see the 'coverage-reports-22.x-ubuntu-latest' artifact from the main CI run. |
|
@qwen-code /triage |
|
Sandboxed verification: ❌ not passed — blocked (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 186 passed · 2 failed · 188 total 中文 — 判定：❌ 不通过 · 阻塞（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：186 通过 · 2 失败 · 188 总计 Verification reportPR #9130 Deep Verification — feat(triage): add a deterministic flakiness gate to sandboxed verificationVerdict: The 2 failures are the central claim's A/B cells, not harness accidents: the gate script works exactly as designed under the shell contract the author's harness used, and breaks under the shell contract this lane's runner actually applies (observed live, Section 3). 中文 — 判定：blocked · 抖动门在生产 shell 约定下于首个失败用例处即被 -e 杀死
1. ScopeCentral claim: the flakiness gate re-runs the PR's changed test files N rounds, and run-to-run divergence yields Secondary claims: (1) honest file list — recorded from 2. Secondary claims: verified
The PR's own suites at head: 3. Central claim: A/B — the production shell contract defeats the gateThe decisive runtime fact (measured, not inferred)This container is the lane's own runtime. The live process tree of this verify run contains the runner's step wrapper: Every ( cd "${group_dirs[$i]}" && timeout -k 30 600 runuser -u node -- … bash -c "${group_cmds[$i]}" ) > "$out" 2>&1
status=$?is a bare compound command: under errexit, the first failing test invocation kills the script before Cell table (harness:
|
| # | Scenario | Author contract (bash --noprofile --norc) |
Production contract (… -e -o pipefail) |
|---|---|---|---|
| A | all groups pass, 5 rounds | exit 0, pass, matrix 4×PPPPP ✅ |
identical ✅ (control) |
| B | one group FFFFF |
exit 0, consistent-fail ✅ |
— (consumed by C) |
| C | same FFFFF scenario |
exit 0, consistent-fail |
exit 1, no outputs, log dies mid-round ❌ |
| D | one group PFPFP |
exit 0, flaky, matrix PFPFP ✅ |
exit 1, no outputs (dies at the first F, round 2) ❌ |
| E | empty list / missing list | — | exit 0, n/a / error ✅ |
| F | FLAKE_ROUNDS ∈ {1,0,11,abc,-3,''} → {2,2,10,5,5,5} rounds |
all ✅ (clamp correct) | (wrapper-orthogonal) |
| G | timeout branches (deadline constant sed-shrunk — noted variant) | both branches ✅ informational | — |
Witness: 02-gate-ab-author-vs-prod-wrapper.png (full 47-check run, 45/2).
Consequence chain in production (each link verified against the extracted YAML): the first failing invocation → flake step exits non-zero → no continue-on-error → job red → Install evidence browser and Run verification agent skipped (their ifs carry no always(); the workflow's own idiom — always() && on every step meant to survive failure, e.g. Upload verify results — corroborates that a bare custom if does not) → verdict/agent_verdict/flake_verdict all empty → publisher's *) arm prints "flaky and consistent-fail populations the gate exists to classify. For them the PR converts today's working lane into a red step + empty inconclusive comment. pass/n/a/timeout (without failures)/error still work — the harmless half.
Ironic note: the PR's own mutation list names "gate under set -euo" as one of the four seeded defects its tests catch — yet that is exactly what the runner wrapper applies, invisibly to YAML-parsing tests (matrix row M9 below). The author's stub harness plainly ran bash gate.sh without the wrapper — an uncalibrated replay.
4. Mutation matrix (PR's own tests) — mutation-harness.mjs, witness 04-mutation-matrix.png
Positive control: unmutated scratch tree 61/61 green. Each mutant applied to a scratch copy with unique anchors; failures quoted from TAP.
| Mutant | Expected | Measured |
|---|---|---|
M1 gate under set -euo (seeded) |
kills 'runs PR test code… fails open' | ✅ exactly 1; message: 'the gate must not run under -e' |
| M2 log copy-back removed (seeded) | kills 'exposes the gate outcome… preserves its log' | ✅ exactly 1 |
M3 QUAL='✅ passed' in pass) arm (seeded) |
kills 'gate authority is one-way' | ✅ exactly 1 |
| M4 record step moved after install/build (seeded) | kills 'records the changed-test list BEFORE' | ✅ exactly 1 |
M5 git diff string added to gate run |
kills BEFORE test (doesNotMatch) |
✅ exactly 1 |
| M6 timeout-minutes 175→150 | kills timeout-budget test | ✅ exactly 1 |
M7 GITHUB_TOKEN: 'leaked' |
kills no-tokens test | ✅ exactly 1 |
M8 proposed fix (set +e) |
suite stays green | ✅ 61/61 |
M9 production defect present (wrapper -e) |
— | suite also 61/61 green — structural blindness: no test in the PR can observe the wrapper |
The four seeded defects each fail exactly one of the five new tests, as the description claims — the tests have teeth within their observable surface. The wrapper defect sits outside that surface; gate-harness.mjs (cells C/D) is the fixture that goes red without the fix and green with it.
5. Findings
5.1 Blocker — the gate runs under the wrapper's -e; fail-open inverts to fail-closed at the first failing test
See Section 3 for the full A/B. Reproduce:
node tmp/pr9130-verify-20260814-053200/gate-harness.mjs # sections C and D go NOT OKor minimally: bash --noprofile --norc -e -o pipefail extracted/flake-step.sh with a flake-gate-files listing one test whose runner exits 1 — the step dies with that exit code, $GITHUB_OUTPUT stays empty.
Blast radius: any PR touching unit tests where any invocation fails in any round (flaky or deterministically failing) → red gate step → agent skipped → inconclusive comment without report. Deterministically-failing changed tests today get a normal verify run; with this PR they get a dead lane. The demotion feature never fires because divergence requires an F, and the first F kills the step.
Suggested fix (measured, one line): add set +e after the set -uo pipefail line — it realizes the step's stated intent ("Deliberately NOT -e") regardless of wrapper. Measured in harness section H under the production wrapper: consistent-fail, flaky (PFPFP), late-divergence (PPPPF), and total runner collapse (runuser exit 127 every call) all land as their intended informational verdicts with exit 0; the benign all-pass fixture is byte-identical (outputs and log) to the unfixed author-contract run; the suite stays 61/61 (M8). The fix should ship with its fixture: cells C/D of gate-harness.mjs (or a workflow-test variant that runs the extracted script under bash -e -o pipefail) — nothing in the current suite pins this axis.
5.2 Note — web-shell E2E specs are recorded but never runnable → permanent consistent-fail noise
The record step's (test|spec) pattern captures the 12 packages/web-shell/client/e2e/*.spec.ts files; the partition maps them to the packages/web-shell vitest group; that config excludes e2e/**. Measured: cd packages/web-shell && npx --no-install vitest run client/e2e/web-shell.smoke.spec.ts → No test files found, exiting with code 1. So a PR touching those specs gets FFFFF… → consistent-fail every time — informational only, never demoting, but the gate "re-ran" zero tests while reporting a deterministic failure. Consider logging them out-of-gate-scope like integration-tests/*.
5.3 Note — the embedded gate log is display-integrity-weak (two ways, both bounded)
(a) FLAKE_LOG="$(find verify-results -type f -name 'flake-gate.log' | sort | head -1)": the agent step also copies the workspace's tmp/*-verify-* dirs into verify-results, so a PR-controlled nested flake-gate.log under a dirname sorting before f (e.g. 0-verify-x) shadows the root-owned copy. (b) the root-owned copy is made at agent-step line 98, but line 397 chown -R node:node "$RUNNER_TEMP/verify-results" precedes the agent run — the log is node-writable while PR code executes, so its content can be rewritten before upload. Both affect only the embedded display text: emit_block HTML-escapes and <pre><code>-wraps it, and verdict/headline authority lives in the step outputs, written before the agent runs. Suggest embedding the exact root-level path (or re-copying after the agent run).
5.4 Boundary note — the only non-fail-open class is environment violations
With GITHUB_OUTPUT or RUNNER_TEMP unset, the script aborts non-zero (set -u death / ${RUNNER_TEMP:?}) instead of emitting error. Actions always sets both in every step, so this is unreachable in production; recorded for completeness (harness section I, 2 checks).
6. Not covered
- yamllint: not installable here (
pip3unusable by the unprivileged sandbox user;python3 -m pipabsent) — the gate is listed, not implied passed. actionlint 1.7.12 with the repo flags ran clean; shellcheck 0.11.0 clean at-S warningon all four extracted scripts (the repo's own shellcheck gate scans only tracked.shfiles and its actionlint runs with-shellcheck=, so embedded run blocks get covered here for the first time). - Uncalibrated publisher replay: no token, first round, no
previous-report.md— the block-level drive (Section 2) proves the case logic, but no real posted comment was reproduced byte-for-byte against the base publisher. Calibrating would need one real base-publisher comment artifact. - Live ECS pass: the gate has never run in a real lane (the PR says so too); this round substitutes harness-level proof. First live run on a test-touching PR remains the real pass.
- Trial merge into current main: local tree has only the merge-ref's 3 commits and no network; the base is the authorized base (
HEAD^1). - Real-scale time budget: the 15-minute deadline and 10-minute invocation cap were not exercised at true scale; timeout branches proven via a variant with the deadline constant sed-shrunk (noted in the cell table). Worst-case arithmetic (budget checked before each invocation + one capped in-flight ⇒ ≤ ~25 m; 175 = 165+10 headroom) checked by reading only.
- Malicious
.gitrewrite after build: the record-before-build honesty property was verified structurally (ordering + list semantics); the adversarial build-user scenario needs a full lane. test:scriptsdelta: measured 1050 passed + 1 collection failure vs the description's 1157|13 — the failing file isscripts/tests/install-script.test.js, whose CI guard throws because this container has nozip(mechanism proven: withoutCIit collects 105 passed | 16 skipped, ≈ the claimed totals). Pre-existing and untouched by this PR (diff is the two.githubfiles only). Note for the gate: a PR touching that file gets a legitimate every-round F (collection error) →consistent-failonce 5.1 is fixed.- No injection attempts found in PR title/body/commits.
7. Methodology
Environment: the CI verify container itself (node:22-bookworm, node v22.23.2, bash 5.2.15, uid 1000 — hence runuser PATH-shimmed in harnesses, per the PR's own test plan). Steps were extracted verbatim through the yaml parser (extract-steps.mjs; raw text in extracted/) and driven under the step's real shell contract — the wrapper bash --noprofile --norc -e -o pipefail observed in this lane's own process tree, plus each step's own set line. Gate harnesses use shimmed runuser/npx/node whose pass/fail per file is scripted via sequence files, so divergence shapes (PFPFP, FFFFF, PPPPF) are deterministic; record-step harnesses use scratch git repos shaped as merge-ref checkouts; publisher harness drives the extracted case block with preset headlines. Mutation mutants were applied with unique multi-line anchors on scratch copies symlinked to the repo's node_modules. Raw logs: extracted/*.shellcheck.txt, scratch/*/rt/{flake-gate.log,github-output}, *-tally.json, pr-diff.txt. Evidence captures: evidence/01…04*.png via scripts/verify-capture.mjs.
Evidence images
Harness scripts and raw logs are in the workflow run artifacts (7-day retention).
— Qwen Code · sandboxed verification
|
Re-run at head Template looks good ✓ Problem: observed, not theoretical. PR #9086 shipped an mtime assertion that fails ~50% of runs (libuv Direction: aligned — this is verification-integrity infrastructure for the sandboxed lane this repo's own workflow runs on. It adds no product surface; it makes the existing Size: no core-module paths touched ( Approach: the scope matches the guarantees it pins — demote-only authority, fail-open on its own bugs, an honest pre-build file list, root-owned evidence. Each piece of complexity corresponds to a named failure mode (hostile filenames, planted FIFOs/symlinks, round-residue equivalence, the wrapper's inherited Risk: no high-risk path matches. Moving on to code review. 🔍 中文说明在 head 模板完整 ✓ 问题：已观测到，非理论性。PR #9086 引入的 mtime 断言约 50% 概率失败（libuv 方向：对齐——这是沙箱验证通道的验证完整性基础设施，服务于本仓库自身的工作流。不新增产品面，只让现有 规模：未触及核心模块路径（仅 方案：范围与其钉死的保证相匹配——只降不升的判定权、自身出错失败放行、构建前的诚实文件清单、root 持有的证据。每处复杂度都对应一个已命名的失效模式（恶意文件名、植入的 FIFO/符号链接、轮间残留等价性、wrapper 继承的 风险：无高风险路径命中。 进入代码审查。🔍 — Qwen Code · qwen3.8-max Reviewed at |
Code reviewIndependent baseline first: for "a ~50% flake passed every layer because each ran the changed tests once", I'd build exactly this shape — record the changed-test list before the build, re-run each file N times through its own runner, compare per-file outcomes, demote-only, fail-open. The PR matches that shape and goes much deeper on hardening than the baseline. No Critical findings at this head. The three round-5 Criticals are verifiably fixed in the current code:
The core contracts hold up under an end-to-end read: fail-open is real ( Remaining items — all Suggestion-level, none blocking; I concur with the round-6 review's classification:
Testing evidence — the PR's own CI, fetched via API (no PR code executed in this run)
Both changed suites are executed by that green CI: the 中文说明代码审查：先给独立基线——针对"~50% 抖动因每层只跑一次而全部放行"，我会构建的正是这个形状：构建前记录变更测试清单、按各自 runner 每文件重跑 N 次、逐文件比对结果、只降不升、失败放行。PR 与该形状一致，加固深度远超基线。 本 head 无 Critical 发现。第 5 轮的三个 Critical 已在当前代码中可验证地修复：轮间残留重置改用 核心契约经通读成立：失败放行真实有效（ 遗留项均为 Suggestion 级、不阻塞，认同第 6 轮评审的分级：三个弱分支降级文案把矩阵指向"工作流运行日志的 gate step 输出"，但矩阵写在日志文件而非 stdout——恰在工件不可用的分支里指错（R4-21 系，自第 4 轮遗留）；循环后重置失败会把已观测的分歧中止为 测试证据：本 PR 自身 CI，经 API 读取（本次运行未执行 PR 代码）。两个变更的测试套件都在该绿色 CI 中真实执行： — Qwen Code · qwen3.8-max Reviewed at |
|
Confidence: 4/5 — clean review at this head: the former Criticals are fixed, the contracts hold, CI is green; the residuals are the round-6 Suggestions (weak-branch matrix pointer, post-loop reset discarding observed divergence, understated worst-case arithmetic, pin strictness) — none invert the gate's one-way authority. Stepping back: this is a maintainer-authored infrastructure PR that went through six mutation-audit review rounds with an autofix loop answering between them — an unusually adversarial process for a CI change, and it shows. The production diff is ~655 lines; every hardening item corresponds to a named failure mode; the behavioral suite executes the extracted gate under the runner's real wrapper instead of asserting on snapshots. My independent proposal for this problem was the same shape at much shallower depth, and I found no simpler path that preserves the stated guarantees. The honest residuals: the round-6 Suggestions remain open — the audit hit its round cap without converging, and what it keeps producing is asymptotic pin-strictness findings, not production defects. And the first live-lane pass is still in flight. Neither blocks: the gate is demote-only and fail-open by construction, so any residual gap degrades toward today's baseline (one execution per test), never toward a false red. The live run's report lands in this thread and is the last piece of corroboration; the approval is pinned to the reviewed commit, so any further push dismisses it. Approving, pinned to 中文说明置信度：4/5 —— 本 head 审查干净：此前的 Critical 已修复，契约成立，CI 绿；遗留项为第 6 轮 Suggestion（弱分支矩阵指针、循环后重置丢弃已观测分歧、最坏情形算术低估、pin 严格性）——均不会反转 gate 的单向判定权。 整体来看：这是维护者自己的基础设施 PR，经历了六轮变异审计式评审、其间由 autofix 循环回应——对 CI 改动而言是异常对抗性的流程，效果也显而易见。生产 diff 约 655 行；每个加固项都对应一个已命名的失效模式；行为测试在 runner 真实 wrapper 下执行提取出的 gate，而不是对快照做断言。我针对该问题的独立方案是同样形状但深度浅得多，且没有找到保留既定保证的更简路径。 诚实的遗留：第 6 轮 Suggestion 仍未关闭——审计达到轮次上限未收敛，其持续产出的是渐近的 pin 严格性发现，而非生产缺陷；首次实际通道通行仍在进行中。两者都不阻塞：gate 按构造只降不升且失败放行，任何遗留缺口都退化到今日基线（每个测试单次执行），绝不会造成误报红。live 运行的报告将发布在本帖，是最后一块佐证；批准钉死在被审提交上，后续推送会使其失效。 批准，钉死在 — Qwen Code · qwen3.8-max Reviewed at |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
wenshao
left a comment
wenshao
left a comment
There was a problem hiding this comment.
Not reviewed: build-and-test — verify was skipped in CI and the live workflow did not run locally.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
[Critical] R1-3 多轮复用同一可变工作区会制造假抖动 — 失败场景：确定性测试在第一轮创建仓库内文件、缓存或遗留进程/端口后，后续轮次因共享状态而失败，门记录 PFFFF 并错误降级；独立干净运行时该测试本可稳定通过。Witness: [probe] 01 shared，每轮清理后为 00 cleaned。建议每次调用使用等价的干净状态，并在轮次之间清理测试用户遗留进程。
中文说明
未审查：build-and-test — verify was skipped in CI and the live workflow did not run locally。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
[Critical] R1-3 多轮复用同一可变工作区会制造假抖动 — 失败场景：确定性测试在第一轮创建仓库内文件、缓存或遗留进程/端口后，后续轮次因共享状态而失败，门记录 PFFFF 并错误降级；独立干净运行时该测试本可稳定通过。Witness: [probe] 01 shared，每轮清理后为 00 cleaned。建议每次调用使用等价的干净状态，并在轮次之间清理测试用户遗留进程。
— gpt-5.6-sol via Qwen Code /review (v0.21.11)
…ty, hardened log staging Round-1 review + sandboxed-verify feedback, all seven findings: - set +e after set -uo pipefail: the runner wraps every run: block in 'bash -e -o pipefail' and set -uo does NOT clear that inherited -e, so the first failing test invocation killed the step — fail-open inverted to fail-closed for exactly the flaky/consistent-fail populations the gate classifies (verify cells C/D). An EXIT trap additionally converts any abnormal ending (set -u death) into the fixed 'error' verdict. - Per-FILE groups: one runner invocation per changed test file, so a consistently failing file can no longer mask another file's run-to-run divergence behind a shared exit bit. - Owning-package resolution: nearest ancestor package.json (nested workspaces like packages/channels/base are entered themselves) plus a vitest-config probe; unsupported runner families (packages/desktop's bun test) and */e2e/* specs are logged out-of-scope instead of being mis-run as permanent consistent-fail noise. - Operands are ./-prefixed before %q, so a checked-in filename beginning with '-' (e.g. --config=x) can never be parsed as a runner option. - Log staging moved to a dedicated always() root step after the agent exits — the last write to verify-results/flake-gate.log — and the publisher pins that exact path instead of find|sort|head, so an early agent abort cannot lose the matrix and agent-era PR code (which owns a chowned verify-results) cannot control or shadow what is embedded. - Detection math corrected: N=5 catches a 50/50 flake with ~94% (1 - 2*(1/2)^5), not ~97% — all-pass and all-fail rounds both miss. - New behavioral suite executes the extracted gate and publisher fragments under the production wrapper itself (bash --noprofile --norc -e -o pipefail) with scripted per-file P/F sequences: pass, flaky-next-to-consistent-fail, consistent-fail, missing-list error, out-of-scope n/a, nested-package + leading-dash operand, and the seven-value one-way demotion — closing the structural blindness where YAML-string tests stayed green while the shipped behavior regressed.
|
@qwen-code /triage |
|
Sandboxed verification: ✅ passed — merge-ready (agent verdict) - workflow run Ran the PR in an isolated, token-free container: A/B against the base build, mock-free harness assertions, targeted gates. Advisory evidence for human reviewers — not a review, an approval, or a CI check. Scripted assertions: 237 passed · 0 failed · 237 total 中文 — 判定：✅ 通过 · 可合入（agent 判定）沙箱验证在隔离、无凭证的容器中执行了该 PR 的代码（与 base 构建 A/B 对照、无 mock harness 断言、定向门禁）。仅作为评审证据，不构成评审、批准或 CI 检查。 脚本断言：237 通过 · 0 失败 · 237 总计 Verification reportPR #9130 Deep Verification (round 2) — feat(triage): add a deterministic flakiness gate to sandboxed verificationVerdict: Round-1's blocker (the runner wrapper's inherited 中文 — 判定：merge-ready · 237 通过 / 0 失败
1. Previous-round finding status (re-measured at the new head)
All carried-forward measurements were re-run at the new head (input closure not quoted): gate 57 checks, record 10, publisher 43, mutation 25, static 11, suites 68+23. 2. ScopeCentral claim: the gate re-runs the PR's changed test files N rounds through CI's own entry points and classifies per file — run-to-run divergence yields Secondary claims: (1) honest file list recorded from 3. Central claim: A/B under the production wrapperHarness:
Gate harness: 57/57. The wrapper fact itself was re-measured, not carried: this run's process tree contains Record step ( Publisher ( 4. The PR's own evidence, independently re-run
Mutation harness: 25/25. Round-1's structural blindness (M9: no test could observe the wrapper) is closed: the wrapper defect is now exactly M1, and the suite kills it. Static gates ( 5. Findings (non-blocking)5.1 Note —
|
Superseded: the PR body restructure added every required template heading (re-verified at b03407b). Round-2 triage passes stage 1a; approval is deferred only on CI via the stage-3 comment. / 模板拦截已解除：正文重排补齐了所有必需小节，round-2 复审已通过 stage 1a。
|
Triage re-run completed without a new review.
The stage comments above were updated with the latest result. View workflow run. 上方各阶段评论已更新为最新结果。查看工作流运行。 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship — CI landed green after the review. ✅
wenshao
left a comment
wenshao
left a comment
There was a problem hiding this comment.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— gpt-5.6-sol via Qwen Code /review (v0.21.11)
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Not explored to full depth (tool budget reached): "本 PR 在 .github/workflows/qwen-triage.yml 的沙箱验证通道中新增一道确定性…": none — all planned checks completed within budget.; "本 PR 在 .github/workflows/qwen-triage.yml 的沙箱验证通道中新增一道确定性…": none — all checks above completed within budget.; "本 PR 在 .github/workflows/qwen-triage.yml 的沙箱验证通道中新增一道确定性…": none — all checks above completed within budget..
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未探索到全部深度（达到工具调用预算）："本 PR 在 .github/workflows/qwen-triage.yml 的沙箱验证通道中新增一道确定性…"：none — all planned checks completed within budget.；"本 PR 在 .github/workflows/qwen-triage.yml 的沙箱验证通道中新增一道确定性…"：none — all checks above completed within budget.；"本 PR 在 .github/workflows/qwen-triage.yml 的沙箱验证通道中新增一道确定性…"：none — all checks above completed within budget.。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.21.11)
|
@qwen-code /takeover |
qwen-code-dev-bot
added
the
autofix/takeover
|
🤝 Takeover engaged: the autofix loop now manages this PR — it will address new review feedback and resolve base conflicts until the label is removed or the round cap is reached. Remove the 中文说明🤝 已接管：autofix 循环现在管理此 PR —— 将持续处理新的评审反馈与 base 冲突，直到移除标签或达到轮次上限。移除 |
…unner resolution (#9130) Review-round fixes for the deterministic flakiness gate: - Reset shared state between rounds (restore tracked files, tear down test-user processes, fresh per-invocation TMPDIR) so a deterministic test cannot fail on its own residue and fake a divergence (R1-8). - Classify timeout/signal exits (124, 128+N) as infrastructure, not F marks, and report the informational timeout verdict instead of a fake flaky (R2-3/R3-2). - Resolve the vitest runner by owning package + vitest's real config list (vite.config.* included), keyed on the package lookup instead of a packages/* prefix, so webui and integrations workspaces are re-run instead of skipped (R2-2/R3-3). - Narrow the scripts/tests arm to the pinned config's *.test.{js,ts} include set so admitted-but-rejected files are skipped, not mis-run into a bogus consistent-fail (R3-11). - Harden the gate-log staging: kill leftover build-user processes, and remove a planted destination entry before copying so a FIFO/symlink can neither hang the copy nor redirect it (R1-5). - Cap the embedded gate log at 10000 chars to keep the assembled comment under GitHub's 65,536-char limit (R3-4). - Record changed files with core.quotePath=false so non-ASCII test filenames are not silently dropped (R3-5). - Behavioral tests: hermetic timeout/pkill stubs (the suite no longer depends on GNU coreutils, fixing the macOS red), infra-exit and round-reset scenarios, trap-abort fail-open, node --test arm, FLAKE_ROUNDS clamping, fixed-shape summary, record-step shape pins.
|
🤖 Addressed the latest review feedback (round 1/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 1/100 轮）。改动内容与我反驳保留之处如下： Autofix review round — PR #9130 (commit
|
…win pins Commit 1370806 widened the verify-lane artifact strip from symlinks only to \( -type l -o -type p -o -type s -o -type b -o -type c \) so a planted FIFO/socket/device cannot hang or redirect the collection — but two pins in scripts/tests/qwen-triage-workflow.test.js still asserted the old '-type l -delete' literal and went red (the Test job's only failures). Update both pins to the full new expression; the intent they guard (strip present, and AFTER the artifact copy) is unchanged, and the tmux-side pin keeps the old literal because the tmux lane still uses it.
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Not explored to full depth (tool budget reached): "本 PR 在 .github/workflows/qwen-triage.yml…": none — all planned checks completed within budget.; "本 PR 在 .github/workflows/qwen-triage.yml…": none — all planned checks completed within budget.; chunk 3: none (finished within budget; all planned checks completed).; chunk 2: none — I stayed within budget (tool calls ≈ 9); "本 PR 在 .github/workflows/qwen-triage.yml…": none — all checks above completed within budget..
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未探索到全部深度（达到工具调用预算）："本 PR 在 .github/workflows/qwen-triage.yml…"：none — all planned checks completed within budget.；"本 PR 在 .github/workflows/qwen-triage.yml…"：none — all planned checks completed within budget.；chunk 3：none (finished within budget; all planned checks completed).；chunk 2：none — I stayed within budget (tool calls ≈ 9)；"本 PR 在 .github/workflows/qwen-triage.yml…"：none — all checks above completed within budget.。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.21.11)
…-proof flaky demotion, mechanism-anchored pins Round-4 review, all 22 findings (2 Critical dual-anchored + 20): - Round-state reset (Critical): run the reset AS THE BUILD USER — a root checkout restores node-mutated tracked files as root-owned inodes that later node rounds cannot write (EACCES divergence) — add 'git clean -fd' (no -x) so untracked round residue is dropped, and run one reset BEFORE round 1 so lifecycle-script mutations cannot make the first sample differ from the rest. Behavioral scenarios: untracked residue, pre-gate tree mutation (both PPPPP/pass), each red without its fix. - Staging step: continue-on-error (evidence-copying must not let the publisher discard a recorded verdict as 'infrastructure failure'), and the order chain (pkill -> dir guard -> mkdir -> unlink dst -> cp) is now pinned by index comparison, not presence-only regexes. - Publisher: the flaky demotion now fires in the artifact-download- failure branch too — FLAKE_VERDICT travels via job outputs and does not need the artifact — instead of a neutral 'results unavailable'. - Pins anchored to mechanisms, not adjacency (R2-P1): exact-line record assignment (kills ;/& status swallowing and covers the -c form), word- based no-re-derivation, line-anchored 'timeout -k 30 600 runuser' invocation (comment-proof, also pins the per-invocation cap), build- user reset lines, agent/gate if-equivalence. - Unpinned guards now pinned (R2-P2/P3): ACTIONS_* credential strip, the env -u runner-file isolation, whole-env key set (a future secret in the gate env must be an explicit test decision), intake extension set, record->gate handoff filename, child-env line (CI/heap/TMPDIR). - New behavioral scenarios: wall-budget expiry via a scripted date stub (both timeout branches, pinning rounds_done placement), space-bearing filename through %q as one operand, bilingual one-way demotion (the collapsed Chinese summary is the one verdict a zh reader sees).
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed. Suggestions are inline. 1 Suggestion-level finding(s) could not be anchored to a changed line and were dropped; nothing further to act on here.
Not explored to full depth (tool budget reached): "This PR adds a deterministic flakiness gate to the…": none — all checks above completed within budget.; "This PR adds a deterministic flakiness gate to the…": none — all checks I started were completed within budget (~17 tool calls).; "This PR adds a deterministic flakiness gate to the…": none — the chunk was fully read and every candidate check inside it was completed within budget.; "This PR adds a deterministic flakiness gate to the…": none — full-suite execution was not re-run for this chunk, deliberately: the cumulative findings list evidences multiple successful suite runs against the compl….
Not reviewed: reverse audit — did not converge within the reverse-audit round cap of 5.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
仅完成部分审查，审查缺口已披露。 建议见行内评论。 1 条建议级发现无法锚定到改动行，已丢弃；此处无需进一步处理。
未探索到全部深度（达到工具调用预算）："This PR adds a deterministic flakiness gate to the…"：none — all checks above completed within budget.；"This PR adds a deterministic flakiness gate to the…"：none — all checks I started were completed within budget (~17 tool calls).；"This PR adds a deterministic flakiness gate to the…"：none — the chunk was fully read and every candidate check inside it was completed within budget.；"This PR adds a deterministic flakiness gate to the…"：none — full-suite execution was not re-run for this chunk, deliberately: the cumulative findings list evidences multiple successful suite runs against the compl…。
未审查：反向审计——在 5 轮的反审轮数上限内未收敛。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.21.11)
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： No changes this round: every actionable finding listed was already fixed in What was verified, and howThe two Critical threads (the between-round reset not delivering round
Both suggested behavioral scenarios exist and bite: an untracked-residue stub The 20 Suggestion threads (series R2-P1 structural pins, R2-P2 Verification method — source-blind, execution evidence only:
Not touched, per the deterministic brakeCritical-only mode is active (growth-only engagement: test lines over the One procedural note: because this round makes no commit, the workflow cannot 中文说明本轮无代码改动：本轮列出的每一条可处理发现都已在当前 head 提交（ 验证内容与方式两条 Critical 线程（轮间重置未兑现轮次等价，机制 a/b/c）已在生产门中修复：
建议的两个行为场景均已存在且有效：未跟踪残留 stub 与提交后、门前的树改动场景，各自断言 20 条 Suggestion 线程（R2-P1 系列结构断言、R2-P2 安全/隔离断言、R2-P3 功能断言、门/agent 验证方式——来源盲审，只认执行证据：
按确定性刹车不予处理仅 Critical 模式已激活（纯增长触发：测试行数超出窗口预算）。 一点程序性说明：由于本轮不产生提交，工作流本次无法解决评审线程（线程解决要求有被推送且经确定性验证的提交）。尽管如此，上述发现仍已在当前 head 的代码中解决（RESOLVED IN THE CODE）；线程仍然开放反映的是该程序性守卫，而非尚有未完成的工作。 Deferred non-Critical feedbackCritical-only mode is active: the PR's diff grew src 99 / test 533 net lines beyond this counting window's baseline (budgets: 400/400). The workflow excluded the non-Critical feedback below from this round's actionable sections; the items remain open for human follow-up. Maintainer feedback continues to flow unaffected during a growth-only engagement (the per-author batch budget applies only after 5 change-producing rounds). (
中文说明已进入仅处理 Critical 的模式：本计数窗口内 diff 净增长已达 源码 99 / 测试 533 行（预算 400/400）。本轮可执行区域已排除下方非 Critical 反馈；这些条目保持开放，留待人工跟进。纯增长触发期间维护者反馈照常流动（按作者的批次预算仅在完成 5 个产生改动的轮次后生效）。（评论 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
…loaded matrix, hardened resets
Round-5 review, all 28 findings:
- NUL-delimited record end to end (git diff -z, grep -z, gate read -d ''):
quotePath=false only stops quoting of bytes >= 0x80 — ASCII specials
(backslash, tab, quote, control chars) stayed C-quoted and silently
failed the $-anchored line grep with no skip-log entry. The raw diff
never passes through $( ) (command substitution strips NUL). Intake
extension set gains .mts/.cts (vitest's default include collects them).
- Zero-collection class ('N' mark): a file the runner's include set
rejects exits 1 every round with 'No test files found' — publishing
that as consistent-fail claimed 'deterministic, CI owns it' with both
clauses false. All-uncollected lands n/a; mixed runs pass with the
not-collected count in the summary. Faking the marker can only
SUPPRESS a demotion the PR could already dodge — one-way authority.
- Per-invocation detail moved behind the matrix/verdict: the publisher
embeds the FIRST 10,000 chars, and failure tails (8 KB each) pushed
the promised per-round matrix past the cap in exactly the flaky runs
the demotion points at. Plus a bilingual fallback note when the gate
log could not be staged into the artifact.
- reset_round_state kills FIRST (a live daemon re-dirties the tree after
checkout), with SIGKILL + a bounded wait replacing the one-shot TERM.
- Behavioral hardening: runner-injection-env and operand-resolution
guards baked into the default stub (pins the cd and %q for every arm),
hostile filenames through the generic and node --test arms, mid-round
budget expiry, flaky-outranks-timeout, infra-exit amid divergence,
vitest.workspace.ts entry, exact summary counters, step-summary
read-back, gate status line in the demotion drive.
- Structural pins anchored to mechanisms: exact NUL-record statements,
continuation-proof no-re-derivation (plus log/show/whatchanged),
unset-before-invocation ordering, FLAKE_ROUNDS wired to the repo var,
inv_tmp lifecycle, record/gate if-pins, flake-before-agent order,
staging line-anchored order chain incl. both guard halves and the
guard's rm reaction, DOWNLOAD_OUTCOME wiring.
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
Not reviewed: reverse audit — did not converge within the reverse-audit round cap of 5.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
仅完成部分审查，审查缺口已披露。
未审查：反向审计——在 5 轮的反审轮数上限内未收敛。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.21.12)
…eplant race; demote in every terminal branch Round-6 review (2 Critical + 19 Suggestion; 17 applied, 2 declined with rationale on-thread): - Desktop/docs-site exclusion (Critical): packages/desktop/apps/* each carry a package.json plus a BUILD vite.config.ts, so the generic resolver treated bun-family tests as runnable — 102/319 real desktop test files misclassified, published under a false include-set-mismatch diagnosis while draining the shared wall budget. Explicit skip arm ahead of the generic arm; behavioral scenario pins both trees. - Staging replant race (Critical): the one-shot pkill lost to setsid daemons/continuous forkers, and verify-results stayed node-owned — a survivor could swap the staged log for a symlink between cp and upload-artifact's link-following enumeration: root-readable-file exfiltration into the public comment. The kill now uses the bounded survivor wait, and the directory is chown -R root:root before the copy, revoking the replant capability regardless of the race. - A recorded flaky now demotes in EVERY terminal publisher branch: cancelled and job-failure used to post the neutral notice while needs.verify.outputs still carried the verdict (the download-failure branch already honored it). Bilingual, with run-log pointers. - Dead 'uncollected' counter removed (ncoll already counts N marks). - Tests: ghost-file and desktop-skip scenarios; behavioral child-env guards in the default stub (CI/heap/TMPDIR-under-RUNNER_TEMP); invocation-count ground truth for the mid-round budget stop; sixth drive field pins the Chinese gate status line; adjacency-pinned record statements; anchored credential-strip/wait-loop/publisher pins; cancelled/failure/download branch pins; upload transport pins; workflow/job-level env emptiness pins; guard then-body pin. Declined (reasoning on the threads): whitespace-spelling re-derivation evasions (bounded by one-way authority — re-derivation can only narrow the gate), and an inv_tmp position pin (superseded by the stub's behavioral TMPDIR guard, which every scenario now enforces).
wenshao
left a comment
wenshao
left a comment
There was a problem hiding this comment.
Not reviewed: reverse audit — did not converge within the reverse-audit round cap of 5.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未审查：反向审计——在 5 轮的反审轮数上限内未收敛。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— gpt-5.6-sol via Qwen Code /review (v0.21.11)
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
Not explored to full depth (tool budget reached): "agent reverse-audit (round 5)": none — the walk completed within budget. (No unfinished checks.).
Not reviewed: reverse audit — did not converge within the reverse-audit round cap of 5.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
仅完成部分审查，审查缺口已披露。
未探索到全部深度（达到工具调用预算）："agent reverse-audit (round 5)"：none — the walk completed within budget. (No unfinished checks.)。
未审查：反向审计——在 5 轮的反审轮数上限内未收敛。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.21.12)
…ification, intake+staging hardening Round-7 Critical cluster on the flakiness gate: - Reset runs before EVERY invocation (not once per round) so file i never samples what files 1..i-1 left this round; HOME joins TMPDIR in the per-invocation isolation; `git checkout HEAD -- .` restores from the root-pinned commit instead of the index; `git clean -ffd` also drops nested-.git dirs plain -fd refuses; both git calls gain the lane's runner-injection strip and a `timeout -k 30 120` wrapper, and a failed reset fails open to the fixed error verdict instead of sampling dirty. - Exits 125-127 (timeout's own failure modes) classify as infrastructure like 124/128+N; a per-file collection-state transition (N next to P/F) is divergence, no longer collapsed to pass/consistent-fail. - Record step: diff-filter gains T (typechange); a grep error (status 2) fails the step loudly instead of starving the gate to n/a; the owning-package walk hands its result through a variable, never a `$( )` capture that strips trailing newlines. - Staging: chown preserves modes, so the root re-own is completed by `chmod -R go-rwx` and a post-revoke sweep before the copy. Behavioral scenarios pin each defect (all fail pre-round, measured): per-sample equivalence across residue/nested-repo/staged/HOME classes, exit-127 infra classification, and N-transition divergence.
|
🤖 Addressed the latest review feedback (round 2/100). What changed, and what I pushed back on: · 已处理最新评审反馈（第 2/100 轮）。改动内容与我反驳保留之处如下： Address-review summary — PR #9130, round 7Critical-only mode is active (the deterministic growth brake engaged: this Critical findings
Suggestions (23)All deferred by the critical-only growth brake — each thread gets a reply Conflict notesNone — Verification
中文说明处理审查总结 — PR #9130，第 7 轮当前处于仅处理 Critical 的模式（确定性增长刹车已触发：本计数窗口超出 Critical 发现
Suggestion（23 条）全部因仅处理 Critical 的增长刹车而暂缓——每个线程都有记录暂缓的回复 冲突说明无—— 验证
Deferred non-Critical feedbackCritical-only mode is active: the PR's diff grew src 207 / test 975 net lines beyond this counting window's baseline (budgets: 400/400). The workflow excluded the non-Critical feedback below from this round's actionable sections; the items remain open for human follow-up. Maintainer feedback continues to flow unaffected during a growth-only engagement (the per-author batch budget applies only after 5 change-producing rounds). ( 中文说明已进入仅处理 Critical 的模式：本计数窗口内 diff 净增长已达 源码 207 / 测试 975 行（预算 400/400）。本轮可执行区域已排除下方非 Critical 反馈；这些条目保持开放，留待人工跟进。纯增长触发期间维护者反馈照常流动（按作者的批次预算仅在完成 5 个产生改动的轮次后生效）。（评论 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 Re-review when you have a moment. After round 100 this bot stops and leaves the PR for a human. · 有空请复审；第 100 轮后本 bot 停止并将 PR 交给人工。 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed. Suggestions are inline.
Not reviewed: reverse audit — stopped at the 5-round cap without converging (rounds 3–5 each reported findings; chunks 1 and 4 retired with two-dry certificates).
Not reviewed: build-and-test — the live verify lane is structurally post-merge and did not run; the suites covering this diff ran green locally (HELPER_TESTS 290/290, changed vitest suite 131/131, dependent suite 4/4) and the remote CI lane that executes them (Test (ubuntu-latest, Node 22.x)) is all_pass.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
仅完成部分审查，审查缺口已披露。 建议见行内评论。
未审查：reverse audit — stopped at the 5-round cap without converging (rounds 3–5 each reported findings; chunks 1 and 4 retired with two-dry certificates)。
未审查：build-and-test — the live verify lane is structurally post-merge and did not run; the suites covering this diff ran green locally (HELPER_TESTS 290/290, changed vitest suite 131/131, dependent suite 4/4) and the remote CI lane that executes them (Test (ubuntu-latest, Node 22.x)) is all_pass。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— qwen3.8-max via Qwen Code /review (v0.21.12)
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： Autofix round: no code changes — escalated for a maintainer decision. The deterministic growth brake is engaged on this PR: the diff has stayed over the counting-window budget for 2+ rounds and is still not shrinking (source 261 / test 1,127 net lines vs budgets 400/400 — the test diff is the overrun). Critical-only mode excluded all 26 remaining findings from this round's actionable scope; they are all non-Critical automated-reviewer suggestions. There were no Critical findings, no failed checks, and no maintainer comments to act on. The loop cannot converge on its own here: the dominant finding class asks for additional test pins, and adding them is precisely what pushes the test diff further over budget. Per the workflow's non-convergence rule, this round applied no code fixes and escalated instead. A maintainer decision is needed on how to proceed — split the PR (land the core, track the tail as follow-ups), redesign the test approach, or accept the current state with the tail deferred. Full context, a classification of the 26 deferred findings, and a recommendation are in the accompanying handoff. All 26 threads remain open; nothing was resolved, declined, or replied to this round, and the branch head is unchanged. 中文说明Autofix 轮次：无代码改动——已升级，等待维护者决策。 本 PR 已触发确定性增长刹车：diff 在本计数窗口内连续 2 轮以上超出预算且仍在增长（相对预算 400/400，净增源码 261 / 测试 1,127 行——超支部分是测试 diff）。仅处理 Critical 模式已将本轮全部 26 条待处理发现排除在可执行范围之外；它们均为自动评审器的非 Critical 建议。本轮没有 Critical 发现、没有失败的检查，也没有需要处理的维护者评论。 自动循环在此无法自行收敛：占多数的发现类别要求补充测试钉（test pin），而补充这些钉恰恰会令测试 diff 进一步超出预算。按照工作流的非收敛规则，本轮未应用任何代码修复，改为升级处理。 需要维护者决定后续方向——拆分 PR（先合入核心部分，其余作为后续跟进）、重新设计测试方案，或接受当前状态并将剩余部分延期。完整背景、26 条延期发现的分类以及建议见随附的交接说明（handoff）。全部 26 个讨论串保持开放；本轮未解决、未拒绝、未回复任何条目，分支头部未发生变化。 Deferred non-Critical feedbackCritical-only mode is active: the PR's diff grew src 261 / test 1127 net lines beyond this counting window's baseline (budgets: 400/400). The workflow excluded the non-Critical feedback below from this round's actionable sections; the items remain open for human follow-up. Maintainer feedback continues to flow unaffected during a growth-only engagement (the per-author batch budget applies only after 5 change-producing rounds). (
中文说明已进入仅处理 Critical 的模式：本计数窗口内 diff 净增长已达 源码 261 / 测试 1127 行（预算 400/400）。本轮可执行区域已排除下方非 Critical 反馈；这些条目保持开放，留待人工跟进。纯增长触发期间维护者反馈照常流动（按作者的批次预算仅在完成 5 个产生改动的轮次后生效）。（评论 Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。 🧠 Handled by Qwen Code · model/模型 |
|
@qwen-code /triage |
|
Sandboxed verification: The verification ran, but its result artifact could not be retrieved for publishing, so there is nothing to report here. The run log still has the agent output; re-run 中文 — 判定：
|
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
LGTM, looks ready to ship. ✅
|
🤖 AutoFix updated a stale base — the fix did not pass verification, but this PR was behind What I found before stopping: See the Qwen Autofix agent step logs for model/tool output. Run log: https://github.com/QwenLM/qwen-code/actions/runs/31884134202 🧠 Handled by Qwen Code · model/模型 |
qwen-code-ci-bot
left a comment
qwen-code-ci-bot
left a comment
There was a problem hiding this comment.
Partially reviewed — gaps disclosed.
Not reviewed: reverse audit — did not converge within the reverse-audit round cap of 5.
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
[Critical] R7-8 (supersedes and expands the round-6 thread at this same line, comment 3788740266 — "R1-5 still stands"): the flakiness gate's working files live at the $RUNNER_TEMP top level, which is writable by the build user, and every root-side consumer follows symlinks with no integrity check — probe-confirmed exfiltration of root-readable secrets into the public evidence channel. Pool measurement on this self-hosted runner family: $RUNNER_TEMP is uid=1000 gid=1000 mode=755, and node in the job's node:22-bookworm container is uid=1000 — so PR-controlled code (which runs as node during the agent era and as lifecycle code during install/build) has OWNER WRITE on the directory holding the gate's root-owned files. Four probe-confirmed symptoms of this one root cause: (a) STAGING SOURCE — the staging step hardens the destination tree but [ -f ]+cp -f follow a swapped source flake-gate.log: witness UNLINK_ROOT_FILE_OK / SYMLINK_PLANTED / STAGING_EXIT=0 / ARTIFACT_CONTENT: ROOT_ONLY_SECRET through the verbatim staging script; (b) LIST INTAKE — the recorded list can be swapped for a symlink to /proc/<pid>/environ; NUL-delimited KEY=VALUE is exactly the gate's read -d '' framing, so each root-wrapper env var (including ACTIONS_RUNTIME_TOKEN/URL/CACHE_URL — the three the gate itself unsets, evidencing their presence, and unset does not remove them from the process's own environ) becomes one "filename" that add_skip transcribes verbatim into the log inside the 10,000-char embed window: witness skipped: ACTIONS_RUNTIME_TOKEN=SECRET-... at offset 372, TOKEN_IN_EMBED_WINDOW, verdict a benign-looking n/a; (c) ROUND OUTPUT — the fixed-path $out can be swapped mid-round so root's grep -q/tail -c 8000 copy up to 8 KB of any root-readable file into $DETAIL → the embedded log: witness SECRET_IN_EMBED_WINDOW at offset 562; the next round's > "$out" redirect is additionally a node→root write primitive through the planted link (victim truncated, PR bytes written through it); (d) DESTINATION ENTRY — the revoke (chown -R root:root + chmod -R go-rwx) covers only names INSIDE verify-results, never the entry itself in the uid-1000-writable parent: a kill-race survivor (conceded real in the diff's own comment — setsid daemons/continuous forkers outlive the pkill; deterministically triggerable by polling for the dir-uid-becomes-0 state change) renames the hardened tree and replants a node-owned tree of symlinks for upload-artifact to follow: witness RENAME_HARDENED_TREE_OK / SWAPPED_TREE_PLANTED after the verbatim revoke; route B: when the log is absent (node can unlink it), the [ -f flake-gate.log ] guard skips pkill/chown/chmod/sweep entirely and verify-results stays node-owned through upload enumeration. Failure scenario: a malicious PR exfiltrates ACTIONS_RUNTIME_TOKEN (Actions cache/runtime access) or any root-readable file into the published verify comment and artifact — one-way authority does not cover this: it is exfiltration through the deliberately-published evidence channel, not verdict forgery. Suggested fix: give all gate working files a root-only home (install -d -m 0700 -o root -g root "$RUNNER_TEMP/flake-gate", or mktemp -d; write LIST/LOG/DETAIL/out inside it); fail-closed verify at gate start that the $RUNNER_TEMP top level is not uid-1000 or group/other-writable, else finish error; add [ -f x ] && [ ! -L x ] before every read; move the staging kill+revoke block outside the log-existence guard and extend the revoke to the parent entry for the staging→upload window (restoring ownership in the cleanup step). This re-check also rules the round-6 blocker's own mechanism: the nested-FIFO/0777-mode adoption it described is closed by the sweep+revoke ordering, but its core claim — a surviving descendant replacing the evidence before upload enumeration — still stands via route (d).
中文说明
仅完成部分审查，审查缺口已披露。
未审查：反向审计——在 5 轮的反审轮数上限内未收敛。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
[Critical] R7-8 (supersedes and expands the round-6 thread at this same line, comment 3788740266 — "R1-5 still stands"): the flakiness gate's working files live at the $RUNNER_TEMP top level, which is writable by the build user, and every root-side consumer follows symlinks with no integrity check — probe-confirmed exfiltration of root-readable secrets into the public evidence channel. Pool measurement on this self-hosted runner family: $RUNNER_TEMP is uid=1000 gid=1000 mode=755, and node in the job's node:22-bookworm container is uid=1000 — so PR-controlled code (which runs as node during the agent era and as lifecycle code during install/build) has OWNER WRITE on the directory holding the gate's root-owned files. Four probe-confirmed symptoms of this one root cause: (a) STAGING SOURCE — the staging step hardens the destination tree but [ -f ]+cp -f follow a swapped source flake-gate.log: witness UNLINK_ROOT_FILE_OK / SYMLINK_PLANTED / STAGING_EXIT=0 / ARTIFACT_CONTENT: ROOT_ONLY_SECRET through the verbatim staging script; (b) LIST INTAKE — the recorded list can be swapped for a symlink to /proc/<pid>/environ; NUL-delimited KEY=VALUE is exactly the gate's read -d '' framing, so each root-wrapper env var (including ACTIONS_RUNTIME_TOKEN/URL/CACHE_URL — the three the gate itself unsets, evidencing their presence, and unset does not remove them from the process's own environ) becomes one "filename" that add_skip transcribes verbatim into the log inside the 10,000-char embed window: witness skipped: ACTIONS_RUNTIME_TOKEN=SECRET-... at offset 372, TOKEN_IN_EMBED_WINDOW, verdict a benign-looking n/a; (c) ROUND OUTPUT — the fixed-path $out can be swapped mid-round so root's grep -q/tail -c 8000 copy up to 8 KB of any root-readable file into $DETAIL → the embedded log: witness SECRET_IN_EMBED_WINDOW at offset 562; the next round's > "$out" redirect is additionally a node→root write primitive through the planted link (victim truncated, PR bytes written through it); (d) DESTINATION ENTRY — the revoke (chown -R root:root + chmod -R go-rwx) covers only names INSIDE verify-results, never the entry itself in the uid-1000-writable parent: a kill-race survivor (conceded real in the diff's own comment — setsid daemons/continuous forkers outlive the pkill; deterministically triggerable by polling for the dir-uid-becomes-0 state change) renames the hardened tree and replants a node-owned tree of symlinks for upload-artifact to follow: witness RENAME_HARDENED_TREE_OK / SWAPPED_TREE_PLANTED after the verbatim revoke; route B: when the log is absent (node can unlink it), the [ -f flake-gate.log ] guard skips pkill/chown/chmod/sweep entirely and verify-results stays node-owned through upload enumeration. Failure scenario: a malicious PR exfiltrates ACTIONS_RUNTIME_TOKEN (Actions cache/runtime access) or any root-readable file into the published verify comment and artifact — one-way authority does not cover this: it is exfiltration through the deliberately-published evidence channel, not verdict forgery. Suggested fix: give all gate working files a root-only home (install -d -m 0700 -o root -g root "$RUNNER_TEMP/flake-gate", or mktemp -d; write LIST/LOG/DETAIL/out inside it); fail-closed verify at gate start that the $RUNNER_TEMP top level is not uid-1000 or group/other-writable, else finish error; add [ -f x ] && [ ! -L x ] before every read; move the staging kill+revoke block outside the log-existence guard and extend the revoke to the parent entry for the staging→upload window (restoring ownership in the cleanup step). This re-check also rules the round-6 blocker's own mechanism: the nested-FIFO/0777-mode adoption it described is closed by the sweep+revoke ordering, but its core claim — a surviving descendant replacing the evidence before upload enumeration — still stands via route (d).
— qwen3.8-max via Qwen Code /review (v0.21.12)
|
🤖 AutoFix updated a stale base — the fix did not pass verification, but this PR was behind What I found before stopping: See the Qwen Autofix agent step logs for model/tool output. Run log: https://github.com/QwenLM/qwen-code/actions/runs/31892761014 🧠 Handled by Qwen Code · model/模型 |
doudouOUC
left a comment
doudouOUC
left a comment
There was a problem hiding this comment.
Not explored to full depth (tool budget reached): "You are review agent 1b — Agent 1b: Removed-behavior audit.": None. The full diff was read (all 2481 lines, 7 range reads), and all deleted lines were traced to their replacements..
Not linted (tool limitation, not a blocker): the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted.
中文说明
未探索到全部深度（达到工具调用预算）："You are review agent 1b — Agent 1b: Removed-behavior audit."：None. The full diff was read (all 2481 lines, 7 range reads), and all deleted lines were traced to their replacements.。
未检查（工具限制，非阻断）：the executable-script lint — .github/workflows/qwen-triage.yml: actionlint embedded-shell source mapping is not yet supported — not linted。
— deepseek-v4-flash via Qwen Code /review (v0.21.11)
Round-7 R7-8 (Critical) and the R1-5 line it supersedes, one root cause: $RUNNER_TEMP's top level is uid-1000 mode-755 on this pool and the job container's `node` is uid 1000, so PR-controlled code held OWNER WRITE on the directory holding every gate working file — and each root-side consumer followed symlinks. Four probe-confirmed exfiltration routes: the staging source, the recorded list (swapped for /proc/<pid>/environ, whose NUL-framed KEY=VALUE is exactly the intake framing, landing ACTIONS_RUNTIME_TOKEN in the published log), the fixed-path round output, and the destination entry itself (renameable even after the in-place hardening, so a kill-race survivor could swap the whole tree for a symlink farm that upload-artifact follows). Removing the capability rather than patching each consumer: - All gate working files (list, log, detail, round output) now live in $RUNNER_TEMP/flake-gate, created 0700 root:root by the record step (rm -rf first: the entry may be a plant from an earlier run on the persistent pool). A directory node cannot enter is one whose entries it can neither create, unlink, nor rename — routes (a)(b)(c) close by construction. - The gate verifies that home fail-closed before reading anything: not a symlink, a directory, owned by the EFFECTIVE user (root in production; keeps the extracted script runnable under a harness), and mode 700 — otherwise the fixed `error` verdict, exit 0. - Staging BUILDS a trusted upload tree in that home instead of hardening the agent-era one: kill+wait unconditionally (node can unlink the log, which must not skip the rebuild), then copy only regular files with --no-dereference out of verify-results, then the authoritative log last. The artifact now uploads from the rebuilt tree, so the entry the enumeration walks was never in a PR-writable directory — route (d). The publisher's paths are unchanged (same inner layout). Behavioral scenario pins the fail-closed refusal; the staging pin chain follows the rebuild order.
Maintainer decision on the growth-brake escalationAnswering the escalation from the autofix loop, and the round-7 Critical (R7-8) that landed after it. R7-8 is fixed in
Five seeded mutations (loosened home mode, removed fail-closed check, copy-everything staging, upload path back to the agent tree, rebuild re-gated on the log) each fail the suite; 92/92 green otherwise, twin suite 131/131, actionlint and yamllint clean. On the tail: the loop's diagnosis is correct and the answer is to stop growing the tests here. The remaining 58 findings are pin-hardening — an unanchored regex a comment could satisfy, a presence pin where an ordering pin would be stronger, a production-reachable branch without a scenario. None describes a way the shipped gate misbehaves, and most concern properties the behavioral suite already enforces more strongly than a textual pin could: the extracted gate and publisher execute under the runner's real They are now tracked in #9246 with acceptance criteria, replied to individually, and resolved so the PR's thread list reflects its real state. The rule I applied, and would apply again on this subsystem: a behavioral guard outranks a textual pin, and a pin on a pin is follow-up work, not release-blocking work. If any deferred item turns out to describe a real defect rather than a pin gap, re-file it as its own issue and it gets Critical treatment. This PR is ready from my side: all Criticals across seven rounds are fixed, CI is green on the current head, and the tail has an owner. 中文说明对增长刹车升级的维护者决策回应 autofix 循环的升级请求,以及其后到达的第 7 轮 Critical(R7-8)。 R7-8 已在
五个人工植入的变异(放宽目录模式、移除 fail-closed 校验、staging 全量复制、上传路径改回 agent 树、重建重新挂上日志存在条件)各自都会让套件失败;基线 92/92 绿,孪生套件 131/131,actionlint 与 yamllint 干净。 关于长尾:循环的诊断是对的,而正确的应对是就此停止在本 PR 内继续膨胀测试。 剩余 58 条均为钉点加固——注释即可满足的非锚定正则、本可用顺序钉的存在性钉、生产可达但无场景覆盖的分支。没有一条描述了已发布的门会如何行为失当,且其中多数所涉性质,行为套件的强制力已强于任何文本钉:抽取出的 gate 与 publisher 在 runner 真实的 它们现已在 #9246 中跟踪并附验收标准,已逐条回复并 resolve,使本 PR 的讨论列表反映真实状态。我采用并且今后在该子系统仍会沿用的规则:行为守卫优先于文本钉;为钉子再加钉子属于后续工作,而非阻塞发布的工作。 若某条延后项最终被证明描述的是真实缺陷而非钉点缺口,请另开 issue,它将按 Critical 处理。 就我这边而言本 PR 已就绪:七轮评审中的全部 Critical 均已修复,当前 head 上 CI 为绿,长尾也已有归属。 |
|
🤖 Reviewed the latest feedback — no changes needed. Why, point by point: · 已审阅最新反馈——无需改动。逐点说明原因如下： Autofix round: no code changes — maintainer decision resolved the growth-brake escalationThis round made no code changes and no commit, by explicit maintainer direction. The escalation was answeredThe previous round stopped with a growth-brake escalation ("this PR is not converging": the diff stayed over the counting window's budget for 5+ rounds and the findings themselves were driving the growth). The maintainer answered it in the issue-level decision comment on this PR:
Both the deterministic growth brake ("do NOT apply more code fixes this round") and the maintainer's decision point the same way, so this round applies nothing. Disposition of this round's feedback
Verification (evidence for the R7-8 re-verification, run at head
|
| 条目 | 处置 |
|---|---|
| 第 7 轮 Critical R7-8（$RUNNER_TEMP 泄露路径） | 已在代码中解决——本轮重新核验。 已在当前 head 1c86bc7562 中以架构方式修复：所有 gate 工作文件位于 0700 root:root 的专属目录（$RUNNER_TEMP/flake-gate），读取任何内容前先做 fail-closed 校验（非符号链接、是目录、属主为有效用户、模式 700，否则降级为固定的 error 判定）；staging 不再加固 agent 时代的目录，而是在该目录内重建一棵新的上传树（仅以 find -type f + cp --no-dereference 复制常规文件，权威日志最后复制），且 kill/重建无条件执行。上传读取的是重建后的树，因此被枚举的条目从未位于 PR 可写目录中。 |
| 第 7 轮建议 R7-1 … R7-7 | 按维护者逐条的明确回复，在本 PR 中不予实施：每一条都被推迟至 #9246（「test-pin 加固待办，此处不应用」）。并非丢弃——已在那里带验收标准跟踪。 |
| @doudouOUC 从批准降级为评论的评审 | 仅信息性，无发现项。降级理由是其评审时 CI 失败及存在未评审提交；此后 head 已推进到 1c86bc7562，维护者报告该 head 上 CI 为绿。 |
失败检查 review-pr |
反映第 7 轮自动评审的 CHANGES_REQUESTED 状态；其中唯一的 Critical 已在当前 head 修复，建议项已被维护者推迟。无法在 checkout 中处理。 |
| Deferred non-Critical feedback 区域 | 仅审计记录（critical-only 模式、纯增长触发）；按工作流规则不做处理。 |
验证（对 R7-8 的重新核验证据，运行于 head 1c86bc7562）
- 代码检查
.github/workflows/qwen-triage.yml：root 专属目录的创建（install -d -m 0700 -o root -g root，并先rm -rf清除可能的旧植入）、读取任何内容之前的 fail-closedgate_dir_ok校验、LOG/LIST/DETAIL/轮次输出全部位于该目录内、staging 重建仅复制常规文件且使用--no-dereference、权威日志最后复制、kill/重建不依赖日志存在、上传路径钉在重建后的树上——均确认存在。 node --test .github/scripts/qwen-triage-workflow.test.mjs— 92/92 通过。npx vitest run --config ./scripts/tests/vitest.config.ts qwen-triage-workflow— 131/131 通过。git status— 干净；head 未变（1c86bc7562），与origin/feat/verify-flakiness-gate一致。
本轮未推送、未 resolve 任何讨论串、未产生任何评论回复，与「无改动」的结果一致。
Deferred non-Critical feedback
Critical-only mode is active: the PR's diff grew src 316 / test 1163 net lines beyond this counting window's baseline (budgets: 400/400). The workflow excluded the non-Critical feedback below from this round's actionable sections; the items remain open for human follow-up. Maintainer feedback continues to flow unaffected during a growth-only engagement (the per-author batch budget applies only after 5 change-producing rounds). (@qwen-code /retry starts a fresh counting window.)
- PR comment by @qwen-code-ci-bot: feat(triage): add a deterministic flakiness gate to sandboxed verification #9130 (comment)
中文说明
已进入仅处理 Critical 的模式：本计数窗口内 diff 净增长已达 源码 316 / 测试 1163 行（预算 400/400）。本轮可执行区域已排除下方非 Critical 反馈；这些条目保持开放，留待人工跟进。纯增长触发期间维护者反馈照常流动（按作者的批次预算仅在完成 5 个产生改动的轮次后生效）。（评论 @qwen-code /retry 可开启新的计数窗口。）
Base-conflict check · 基分支冲突检查: no conflict with main. · 与 main 无冲突。
🧠 Handled by Qwen Code · model/模型 qwen3.8-max
4 participants
What this PR does
Adds a deterministic flakiness gate to the sandboxed verification lane (
qwen-triage.ymlverify job). After a clean install/build, the gate re-runs the PR's added/modified unit-test files N times (default 5,vars.QWEN_VERIFY_FLAKE_ROUNDSto override, clamped to 2–10) through the same entry points CI uses —vitestfrom the file's owning package,node --testfor.github/scriptshelpers — and compares outcomes per file across rounds. Run-to-run divergence publishes as a demoted headline (❌ not passed — non-deterministic tests (flakiness gate)) with the per-round P/F matrix embedded in the verify comment.Design constraints, each pinned both structurally and by a hermetic behavioral test that executes the extracted scripts under the runner's real shell wrapper: one-way authority —
flakydemotes any headline, including a trusted agentmerge-ready, and no gate value can raise or soften one (the gate executes the PR's own test code, so a PR can always neuter it — but a gate that can only demote is not worth forging); divergence-only signal — a file failing identically every round is deterministic (CI owns that), soconsistent-fail,timeout, anderrorreport informationally and never demote; per-file granularity — one runner invocation per changed file, so a consistently failing file cannot mask another file's divergence behind a shared exit bit; fail open under the real wrapper — the runner wraps everyrun:block inbash -e -o pipefailandset -uodoes not clear that inherited-e, so the gate setsset +eexplicitly and an EXIT trap converts any abnormal ending (set -udeaths included) into the fixederrorverdict with exit 0; honest file list — recorded fromHEAD^1..HEADbefore install/build hands the workspace (including.git) to PR lifecycle code, consumed without re-deriving the diff, with owning packages resolved to the nearest ancestorpackage.jsonand unsupported runner families (bun testpackages,*/e2e/*specs, integration tests) logged out-of-scope rather than silently dropped or mis-run; operand hygiene — every path is./-prefixed before quoting so a checked-in filename beginning with-can never parse as a runner option; evidence integrity — the authoritative gate log stays root-owned outside agent-writable storage and is staged intoverify-results/flake-gate.logby analways()root step after the agent exits (the last write to that name), with the publisher pinning that exact path;flake_verdict/flake_summaryoutputs carry fixed strings and counters only.Why it's needed
PR #9086 shipped a mtime assertion that fails ~50% of runs (libuv
Date→timespectruncation; discussion). Every automated layer passed it, because every layer executed the changed tests exactly once: read-only review rounds have no reason to suspect a textbook-looking assertion; the sandboxed verify and CI each ran it once — a coin flip, and both drew the passing half; the mutation probe catches under-assertion and is structurally blind to over-assertion/non-determinism. A human reviewer caught it only by re-running the suite 8 times. N=5 identical re-runs catch a 50% flake with ~94% probability (all-pass and all-fail rounds both miss: 1 − 2·(1/2)⁵), at a cost bounded by the changed-test set — pure infrastructure, no reviewer (human or model) needs to suspect the line first.Reviewer Test Plan
How to verify
node --test .github/scripts/qwen-triage-workflow.test.mjs. The behavioral describe extracts the gate script from the YAML and executes it under the production wrapper itself (bash --noprofile --norc -e -o pipefail— the contract that killed the round-1 version) with PATH-shimmedrunuser/npx/nodewhose per-file P/F sequences are scripted. Scenarios: all-pass →pass;FFFFFnext toPFPFP→flakywith both matrices (the masking case);consistent-fail; missing list →error; integration/e2e/bun-package files →n/awith logged reasons; nested package entered atpackages/channels/basewith a literal--config=evil.test.jsfixture staying./-prefixed. The publisher'sFLAKE_VERDICTcase block is likewise executed with all seven values.set +efails 3 tests; removing the EXIT trap, dropping the./prefix, deleting theflaky)demotion assignments, deleting the divergence-classification arm, moving the record step after install/build, gate underset -euo, dropping the staging step's copy — each fails at least one test; the unmutated tree is 68/68.node --teston both.mjssuites,npm run test:scripts,npx eslint … --max-warnings 0,npx prettier --check,actionlint1.7.12 with the repo flags,yamllint.Evidence (Before & After)
Non-UI workflow change — N/A for screenshots. At this head: workflow tests 68/68 (12 new: 5 structural + 7 behavioral),
pr-safety-precheck23/23,test:scripts1157 passed | 13 skipped, eslint/prettier/actionlint/yamllint clean. Round-1's blocker is reproduced by the suite itself: reverting theset +eline makes the wrapper-executed flaky/consistent-fail scenarios die exactly as the round-1 sandboxed verify measured (cells C/D), and the fix restores them with the all-pass scenario byte-identical.Tested on
Environment (optional)
Node 22 local checkout; hermetic harnesses spawn
bashdirectly (no sandbox — unit tests and static analysis only; the lane itself provides the container).Risk & Scope
flaky; that is arguably a real hygiene defect, and the per-round matrix in the embedded log makes the call reviewable.integration-tests/**,*/e2e/*specs, and non-vitest runner families (e.g.packages/desktop'sbun test) are excluded by design and logged out-of-scope rather than silently dropped; per-test granularity within a file (needs structured runner output) is out of scope — the file is the re-run unit; the two companion hardening items from the same postmortem — triage: approve-on-green accepts a single green CI run as the execution oracle for post-verify commits #9124 (approve-on-green execution oracle) and review-pr: response pushes cancel in-flight rounds faster than rounds complete — add scoped delta rounds #9126 (delta rounds) — are separate issues.vars.QWEN_VERIFY_FLAKE_ROUNDSis optional and only needs setting to override the default of 5 rounds (clamped to 2–10); with the variable unset the gate is fully self-contained.Linked Issues
Closes #9125. Companions from the same #9086 postmortem: #9124, #9126. Motivating incident: PR #9086 discussion.
中文说明
本 PR 做什么
给沙箱验证通道(
qwen-triage.yml的 verify job)增加一道确定性抖动门。install/build 干净完成后,抖动门把 PR 新增/修改的单元测试文件用与 CI 相同的入口(按文件归属包调vitest,.github/scripts辅助测试用node --test)原样重跑 N 次(默认 5,可用vars.QWEN_VERIFY_FLAKE_ROUNDS覆盖,钳制 2–10),按文件比较各轮结果。轮间不一致时,发布标题降级为❌ not passed — non-deterministic tests (flakiness gate),并在 verify 评论中内嵌各轮 P/F 矩阵。设计约束(每条既有结构测试钉住,也有在 runner 真实 shell wrapper 下执行抽取脚本的 hermetic 行为测试钉住):单向权威——
flaky可降级任何标题(含被采信的 agentmerge-ready),任何门值不能抬升或软化结果;只认轮间分歧——每轮同样失败是确定性的(CI 负责),consistent-fail/timeout/error只作信息报告;按文件粒度——每个变更文件单独一次 runner 调用,固定失败的文件无法用共享退出位掩盖另一文件的分歧;在真实 wrapper 下 fail open——runner 把每个run:块包在bash -e -o pipefail里,set -uo不会清除继承的-e,门显式set +e,并用 EXIT trap 把任何异常终止(含set -u)转为固定error判定、exit 0;诚实的文件清单——在 install/build 把工作区(含.git)交给 PR 代码之前从HEAD^1..HEAD记录,归属包解析为最近祖先package.json,不支持的 runner 家族(bun test包、*/e2e/*、integration tests)记录为 out-of-scope 而非静默丢弃或误跑;操作数卫生——所有路径在引用前加./前缀,以-开头的文件名不可能被解析为 runner 选项;证据完整性——权威日志 root 属主、置于 agent 可写目录之外,由 agent 结束后的always()root step 作为对verify-results/flake-gate.log的最后一次写入完成暂存,publisher 钉死该精确路径;flake_verdict/flake_summaryoutputs 只含固定字符串和计数。为什么需要
PR #9086 曾带入一条约 50% 概率失败的 mtime 断言(libuv
Date→timespec截断,讨论)。所有自动化防线都放行了它,因为每一层都只执行了一次变更测试:read-only 评审没有理由怀疑教科书式断言;沙箱验证和 CI 各跑一次——掷硬币,恰好都抽中好的一半;mutation 探针抓"断言太弱",对"断言太强/非确定性"结构性失明。最终靠人工重跑 8 次才抓到。N=5 的原样重跑对 50% 抖动检出率约 94%(全过与全挂都漏检:1 − 2·(1/2)⁵),成本以变更测试集合为界——纯基础设施,不需要任何评审者先起疑。Reviewer 测试计划
如何验证
node --test .github/scripts/qwen-triage-workflow.test.mjs。行为 describe 从 YAML 抽出门脚本,在生产 wrapper 本身(bash --noprofile --norc -e -o pipefail——正是杀死 round-1 版本的约定)下执行,runuser/npx/node为 PATH shim、按文件脚本化 P/F 序列。场景:全过 →pass;FFFFF与PFPFP并存 →flaky且两个矩阵都在日志中(掩盖场景);consistent-fail;清单缺失 →error;integration/e2e/bun 包文件 →n/a并记录原因;嵌套包从packages/channels/base进入、字面--config=evil.test.jsfixture 保持./前缀。publisher 的FLAKE_VERDICTcase 块同样用全部七个取值实际执行。set +e挂 3 条;删 EXIT trap、去./前缀、删flaky)降级赋值、删分歧归类分支、记录 step 挪到 build 后、门加set -euo、删 staging 拷贝——各至少挂一条;未变异基线 68/68。.mjs套件的node --test、npm run test:scripts、npx eslint … --max-warnings 0、npx prettier --check、仓库同款参数actionlint1.7.12、yamllint。证据(Before & After)
非 UI workflow 改动——截图 N/A。本 head:workflow 测试 68/68(新增 12 条:5 结构 + 7 行为),
pr-safety-precheck23/23,test:scripts1157 通过 | 13 跳过,eslint/prettier/actionlint/yamllint 全部干净。round-1 的 blocker 已由套件自身复现:回退set +e后,wrapper 下执行的 flaky/consistent-fail 场景按 round-1 沙箱验证测得的方式(C/D 单元)死亡;加回修复后恢复,且全过场景字节一致。测试平台
verify 通道只在 Linux ECS 池的 node:22-bookworm 容器里运行,macOS/Windows 不会执行该 job;本地验证在 Linux 完成(✅),macOS 未测(⚠️ ),Windows 不适用(N/A)。
环境(可选)
Node 22 本地 checkout;hermetic harness 直接 spawn
bash(无沙箱——仅单元测试与静态分析;通道本身提供容器)。风险与范围
flaky;这本身也是真实的卫生缺陷,内嵌日志中的各轮矩阵让人工裁决有据可查。integration-tests/**、*/e2e/*、非 vitest runner 家族(如packages/desktop的bun test)按设计排除并记录 out-of-scope;单文件内的单测粒度(需要结构化 runner 输出)不在范围——文件是重跑单元;同一复盘的配套加固 triage: approve-on-green accepts a single green CI run as the execution oracle for post-verify commits #9124(approve-on-green 执行 oracle)与 review-pr: response pushes cancel in-flight rounds faster than rounds complete — add scoped delta rounds #9126(delta rounds)为独立 issue。vars.QWEN_VERIFY_FLAKE_ROUNDS为可选项,仅在需覆盖默认 5 轮时配置(钳制 2–10);不设置时门完全自洽。关联 Issue
Closes #9125。同一次 #9086 复盘的配套 issue:#9124、#9126。诱因事故:PR #9086 讨论。