【No.6】 feat(metrics): scan whole response for repetition

[hoobnn]

What

Extend repetition detection from the tail of a response to its full text, and add an offline diagnostic for dumped rollout data.

has_repetition now scans the whole response with overlapping windows, keeping the existing zlib compression-ratio criterion and its threshold of 10, and keeping its boolean interface. A new scan_repetition additionally reports the hit windows' character intervals, their compression ratios and the maximum ratio. A new entrypoint reads existing rollout dumps and writes a JSON report.

Relates to #321, contributor program cohort 2, No.6 长响应全文重复检测与离线诊断. No RFC is required for this task.

• Task requirements
• Claim: Relax 开源贡献者计划第二期｜活动流程指南 🚀 #321 (comment)

Why

The previous check only looked at text[-10000:], so a response that degenerates early and then recovers a clean ending was reported as non-repetitive. The rollout/repetition_frac metric therefore undercounted exactly the failure mode it exists to track, and the metric alone could not say where in a response the repetition was.

How

• Move the detection core into relax/utils/repetition.py, which imports only the standard library, so offline diagnosis runs on a plain CPU box without torch, numpy or the Ray/metrics service stack. relax.utils.metrics.metric_utils re-exports the public names, so the historical import path and the rollout/repetition_frac call site are unchanged.
• Scan 10,000-character windows at a 5,000-character stride, the documented defaults. A nonempty response shorter than one window is scanned as a single window, and a final window is always aligned to the exact suffix so an unaligned tail is still covered. No window is silently skipped to limit cost.
• Define offsets as Unicode code points into the scanned text with Python slice semantics: the window is text[start:end], start inclusive and end exclusive. Counting code points rather than bytes keeps offsets meaningful for CJK responses. The intervals mark suspected repetitive windows, not exact repetition boundaries.
• Report covered_chars as the union of the hit intervals, so overlapping windows are never counted twice.
• Let has_repetition short-circuit at the first hit, which is all the online metric needs. The offline diagnostic keeps every window so ratios are complete.
• Add python -m relax.entrypoints.diagnose_repetition, which reads .jsonl rollout results incrementally and imports torch lazily, on the pickled .pt path alone. It emits per-sample identity (position, dataset, index, sample_index, group_index, rollout_id), hit intervals, ratios and an aggregate repetition_frac matching the online definition. Window parameters, thresholds and dump contents are validated up front, and the CLI refuses to overwrite an input dump.

Scope kept minimal. The online call site in relax/distributed/ray/rollout.py is not modified: metric_utils re-exports the detector, so rollout/repetition_frac picks up full-text scanning through the existing import. Nothing in the rollout, metrics-service or reward paths is touched, which keeps the diff reviewable and the blast radius confined to one new standard-library module.

Detection runs on Sample.response as-is. Roles are not distinguished, semantic repetition is not identified, and neither generation truncation nor reward is touched.

Metric comparability. Thresholds and window parameters are unchanged in value, but two classes of response flip from non-repetitive to repetitive, both intended. Repetition outside the final window is now seen at all. Responses of 10,000 characters or fewer are now checked at all: the previous implementation guarded on len(text) > 10000, so a fully repetitive 10,000-character response was unconditionally reported as clean. rollout/repetition_frac may therefore rise on affected runs with no change in model behavior, and values from old and new runs should not be compared directly.

Testing

• pre-commit run --all-files passes.
• Tests pass (pytest tests/utils/test_repetition_detection.py): 60 passed.
• New tests added.
• Documentation updated.

tests/utils/ was run on this branch and on its parent 2a8d2ed: both report an identical 46 failed and 42 errors, all from unrelated modules whose optional dependencies are absent on this host, while passes go from 417 to 477 — exactly the 60 tests added here. No regressions.

Fixed samples cover repetition at the beginning, in the middle with a clean ending, and at the end, plus a clean negative control. Other cases cover the empty string, a short response below one window, exactly one window, window boundaries, the unaligned final window, Chinese text with character-based offsets, a compression ratio exactly at the threshold (not a hit) and just above it (a hit), and union counting of overlapping hits.

Integration is verified through the real entrypoints rather than fixtures: four tests write dumps with the production writers (save_rollout_result_jsonl and the .pt debug dump), read them back through diagnose_dumps, and assert the offline verdict matches the rollout/repetition_frac entry produced by the real rollout metric path, for both dump formats and both train and eval dumps.

No accelerator or distributed execution was run locally, because this macOS host has no accelerator cluster. The change is CPU-only string processing and does not touch distributed code paths.

Screenshots / Logs

tests/utils/benchmark_repetition_scan.py reproduces the cost numbers quoted in the documentation. Measured on macOS 27.0 arm64, Python 3.14.7, five timing repeats, no early exit:

     chars       case  windows   wall ms    CPU ms traced MiB   RSS MiB
     10000      clean        1      0.11      0.11        0.3     25.55
     10000 repetitive        1      0.02      0.02        0.3     25.67
    100000      clean       19      2.28      2.26       0.31     25.88
    100000 repetitive       19      0.31      0.31       0.31     25.94
   1000000      clean      199     22.95      22.9       0.31     28.53
   1000000 repetitive      199      2.88      2.87       0.34     26.67

Clean text is the worst case, since every window must be scanned: a one-million-character response costs about 23 ms of CPU. Traced peak memory stays near the size of one window regardless of response length. The documentation also records a batch benchmark of the online predicate and the measurement methodology.

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)

[hoobnn]

@rai-studio-bot Please review this PR.

[SigureMo]

辛苦 merge 下最新 main，新增了两条流水线

[hoobnn]

已 merge 最新 main（#324 的两条 GPU 流水线），无冲突，新的流水线正在运行。

[hoobnn]

@SigureMo Qwen3-VL-4B-2xgpu 挂在 Create test container 步骤，报错 bind source path does not exist: /home/relax-ci/model，测试没跑到。这个 job 跑在 GPU-H20-M02-R01，M01 上是通过的，看起来是 M02 缺这个目录。

[rai-studio-bot]

审查结论

未发现阻塞问题。两条 P3 建议见行内评论：测试在缺少 torch/ray 时会失败而非跳过；has_repetition 可委托 scan_repetition 复用同一套遍历。

审查基于 292c03fd：该提交把 main 合入本分支，9 个改动文件与 883254ee 逐字节一致（仅带入 CI 文件），对两个版本都成立。

本轮独立核对过的几点：

• 检测能力是旧实现的超集：任意响应长度下，窗口列表都包含末尾那个 (len-10000, len) 窗口，旧实现能命中的新实现都能命中，不存在漏报回归；窗口并集覆盖全文且不重复计数，已用暴力枚举核对边界与末段对齐。
• 成本：全文扫描按响应长度线性增长。Linux x86 实测干净文本 1 万字符 0.36 ms、100 万字符 68 ms（文档中的 24 ms 来自 macOS）；按 512 样本、每样本约 1 万字符估算，每个 rollout step 新增约 0.2 s，属于可接受范围。
• 指标语义：rollout/repetition_frac 名称不变而判定范围变宽，新旧数值不可直接比较——PR 描述与文档已明确说明；仓库内该指标没有下游消费（无告警或阈值配置引用），因此不会改变训练行为。
• 本轮 7 项检查全部通过：pre-commit、Python 3.10/3.11/3.12、H20 单测，以及 Qwen3-4B-4xgpu-async、Qwen3-VL-4B-2xgpu 两条 GPU 集成流水线。

^{Powered by Nyanpasu with deepseek-v4.1-flash-ali medium, please check the suggestions carefully.}

[rai-studio-bot]

Nyanpasu 审查看板

审查状态： ✅ 已通过

审查版本： e3612640c15bf98aa31ebed9aa6511639c39c972

e361264 审查通过：F1–F6 均已解决，中英文开销说明已与只保存命中窗口的实现一致。本次仅修改两个文档文件，功能代码及测试未变，沿用此前 52 passed、8 skipped（缺少 torch/ray）和 84 组 JSON 等价性验证结果。当前提交尚无 CI 检查结果；本地无多节点 GPU 环境，未运行分布式集成测试。

编号	问题	优先级	状态	规则来源
F1	纯 CPU 环境下 5 个用例失败/报错而非跳过		✅ 已解决	—
F2	has_repetition 重复实现扫描循环，可委托 scan_repetition		✅ 已解决	—
F3	文档开销段落：布尔接口辅助内存 O(window_size) 已不准确		✅ 已解决	—
F4	删除未被消费的全量压缩比列表，收窄报告状态		✅ 已解决	—
F5	收窄 metric_utils 的新增公共转导出		✅ 已解决	—
F6	删除没有消费者的基准兼容别名		✅ 已解决	—

^{Powered by Nyanpasu with gpt-6-astra medium, please check the suggestions carefully.}

[hoobnn]

@rai-studio-bot F1、F2 均已在 7130c81 修复

[rai-studio-bot]

审查结论

未发现阻塞问题。上一轮两条 P3 建议在本提交均已采纳并核实解决，新增 1 条 P3 文档建议（见行内评论）。

上轮建议的处置（基于 7130c811 重新核对）：

• 测试在缺少 torch/ray 时优雅跳过 → 已解决：pytest tests/utils/test_repetition_detection.py -q 在纯 CPU 环境为 52 passed, 8 skipped（此前 1 failed, 52 passed, 3 skipped, 4 errors）。
• has_repetition 委托 scan_repetition → 已解决：在线指标与离线报告现在共用同一趟扫描、同一套阈值；等价性已重新核对（60 例全部一致，其中 33 例命中），且「旧实现命中而新实现不命中」为 0 例，不存在漏报回归。

本轮新发现：docs/en/guide/repetition-detection.md:191-193 与 docs/zh/guide/repetition-detection.md:165-166 的「开销」段落仍写着布尔接口「不保存压缩比列表、辅助内存 O(window_size)」。委托之后 has_repetition 走的就是 scan_repetition，后者为每个已扫描窗口保留一个压缩比，因此该界应改为 O(window_size + K)（干净长文本下 K 为全部窗口数，100 万字符约 199 项、约 6 KB，影响可忽略，但表述已不准确）。这条文档漂移正是我上一轮的重构建议带来的，建议随本 PR 一并修正。

CI：7130c811 目前为 pending，尚无 check-run 结果，故本轮不对 CI 作结论。

^{Powered by Nyanpasu with deepseek-v4.1-flash-ali medium, please check the suggestions carefully.}

[rai-studio-bot]

审查结论

审查通过，无阻塞项。

F1、F2 已在 7130c811 修复，F3（文档内存上界）已在 52235bdf 修复，三条均已核实关闭；详细证据见各自的行内讨论串。

• F1/F2：纯 CPU 环境 pytest tests/utils/test_repetition_detection.py -q 为 52 passed, 8 skipped（60 个用例无失败/报错）；has_repetition 委托 scan_repetition 后的等价性复核全部一致，无漏报回归。
• F3：中英文文档同批改正，改写后的表述与实现一致——提前退出下 K 在开头即重复时为 1、纯净文本为全部窗口数；实测 100 万字符为 199 个窗口、压缩比列表约 6.3 KB，与文档所述「约 199 项、约 6 KB」相符。

本轮改动仅涉及两个文档文件（+11/-8），功能代码与 7130c811 逐字节一致；CI 目前尚无 check-run 结果，故本轮不对 CI 作结论。

^{Powered by Nyanpasu with deepseek-v4.1-flash-ali medium, please check the suggestions carefully.}

[hoobnn]

@SigureMo 任务已完成，请求 review。

[SigureMo]

@rai-studio-bot 重点 review 是否过度设计，重点检查哪些代码是不必要的，可以参考 https://github.com/ShigureLab/nyanpasu/blob/main/AGENTS.md

[rai-studio-bot]

针对设计复杂度的补充审查已完成：有局部可精简之处，未发现需要阻塞合并的问题，具体建议见行内评论。

整体拆分有实际需求支撑：标准库检测核心支持无训练依赖的离线运行，既有 dump 读取、真实指标入口集成测试和 CPU／内存基准均属于任务验收范围，建议保留。优先收窄报告状态和公共导出范围，删除无消费者的兼容别名，无需引入更多抽象。

基于 52235bdf；本地检测测试 52 passed、8 skipped（缺少 torch/ray），GitHub 7 项检查通过。本地无多节点 GPU 环境，未重跑分布式集成测试。此前 F1–F3 保持已解决。

^{Powered by Nyanpasu with gpt-6-astra medium, please check the suggestions carefully.}

[rai-studio-bot]

6119bfbe 的精简修改已复核，F4–F6 已解决，未发现功能回归。仍有一处非阻塞文档遗漏：删除压缩比列表后，开销说明未同步，已在原 F3 讨论补充具体修改建议。

本地测试 52 passed、8 skipped（缺少 torch/ray）；84 组新旧 JSON 报告一致，单响应和批次基准字段验证通过。当前 CI 仍在运行；本地无多节点 GPU 环境，未运行分布式集成测试。

^{Powered by Nyanpasu with gpt-6-astra medium, please check the suggestions carefully.}

[rai-studio-bot]

审查通过。剩余开销文档问题已在 e3612640 修复，此前精简建议均已落实，无待处理审查问题。

本次仅改中英文文档，功能代码与已验证版本一致，沿用此前 52 passed、8 skipped（缺少 torch/ray）及 84 组 JSON 等价性验证结果。当前提交尚无 CI 检查结果；本地无多节点 GPU 环境，未运行分布式集成测试。

^{Powered by Nyanpasu with gpt-6-astra medium, please check the suggestions carefully.}