Expand testing specification and close P0/P1 gaps#152
Draft
Alek99 wants to merge 34 commits into
Draft
Conversation
Greptile SummaryThis PR adds a living testing specification and gap register. The main changes are:
Confidence Score: 5/5This looks safe to merge. No blocking issues found in the changed code.
 T-Rex LogsWhat T-Rex did
Important Files Changed
Reviews (2): Last reviewed commit: "Make the testing catalog self-validating" | Re-trigger Greptile |
Merging this PR will not alter performance
Comparing 
Footnotes
|
masenf
reviewed
Jul 22, 2026
Comment on lines
+257
to
+274
| ### TST-NI-024 — Runtime DOM-XSS, CSP, and network-isolation tests | ||
|
|
||
| - Status: `NOT IMPLEMENTED` | ||
| - Current gap: text-boundary security is strong, but much of the client sink | ||
| contract is source-based rather than observed in a browser. | ||
| - Implemented when: hostile strings flow through every public text surface; | ||
| tests assert literal text, no executable nodes/dialogs, and no network request | ||
| under standalone CSP; hostile CSS is exercised; and fixed internal icon sinks | ||
| are structurally allowlisted. | ||
|
|
||
| ### TST-NI-025 — Observable or opt-in Chromium sandbox downgrade | ||
|
|
||
| - Status: `NOT IMPLEMENTED` | ||
| - Current gap: public export can retry unsandboxed after a sandboxed launch | ||
| failure without a product-level policy assertion. | ||
| - Implemented when: policy chooses explicit opt-in or an observable warning, | ||
| launch-failure tests assert exact arguments and diagnostics, no silent downgrade | ||
| occurs, and trusted CI requests `sandbox=False` explicitly when necessary. |
Comment on lines
+326
to
+333
| ### TST-NI-031 — Strict marker, skip, and xfail policy | ||
|
|
||
| - Status: `NOT IMPLEMENTED` | ||
| - Current gap: a single dependency skip can hide a suite and configured browser | ||
| failures can become skips without an enforced per-job budget. | ||
| - Implemented when: registered markers describe all layers; each job declares its | ||
| selection and allowed skips; skip reasons/budgets are enforced; required | ||
| dependency/browser skips fail; and strict xfails carry an issue and expiry. |
|
|
||
| - Status: `NOT IMPLEMENTED` | ||
| - Current gap: root hard CI floats `.[dev]` and tool versions, Python versions are | ||
| partly incidental, and the floor lane omits relevant optional test dependencies. |
Contributor
There was a problem hiding this comment.
there should be no optional test deps. i think that's a mistake.
if the test deps are overly specific to one subsystem, then we should consider splitting the tests across package boundaries. for example, reflex-xy tests can live at python/reflex-xy/tests and have their own pyproject and dev deps separate from the outer project
masenf
reviewed
Jul 22, 2026
Comment on lines
+26
to
+29
| GOOD_README = """# Testing Specification | ||
|
|
||
| See [`current.md`](current.md) and [`gaps.md`](gaps.md). | ||
| """ |
Contributor
There was a problem hiding this comment.
WTF kind of test module is this...
Alek99
force-pushed
the
agent/add-testing-spec
branch
from
July 22, 2026 03:54
8c5f886 to
a88a02a
Compare
|
Too many files changed for review. ( Bypass the limit by tagging |
Alek99
changed the title
Address review feedback on the testing specification: - add scripts/check_testing_spec.py, which validates the catalog's Markdown links and heading anchors, status vocabulary, gap-ID uniqueness/sequence/ reachability, and every referenced make target, repository path, and workflow job; the checks that were previously run by hand now run in automation - add tests/test_check_testing_spec.py with a positive case and 15 negative controls, per the register's own promotion rule - wire the checker into scripts/verify_local.py, the quick gate, make check-docs, and a new make check-testing-spec, so the root suite fails when a renamed script or retired target makes the catalog false - require an Owner field on P0 entries and enforce it in the checker - add current.md rows for TST-NI-049, 052, 053, and 054, which were defined in the register but unreachable from the inventory - move the point-in-time audit snapshot out of the evergreen README into spec/testing/audits/2026-07-21.md, recording how each number was observed - restate TST-NI-005 as the remaining scope: spec-only pull requests still receive no required result and the rest of spec/ is unchecked - order the spec/README intro to match its section order
Alek99
force-pushed
the
agent/add-testing-spec
branch
from
July 22, 2026 05:49
a88a02a to
14bf6d5
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #159 (TypeScript/vite client). Documents current testing coverage, marks missing gates as NOT IMPLEMENTED, and adds separate hard CI gates for dashboard health, browser baselines, runtime security, dependency auditing, native parity, release testing, and related P0/P1 gaps.