Skip to content

Release v0.12.0: P2 benchmark batch 2 and failure-path fixes - #16

Merged
jiangkoumo merged 9 commits into
mainfrom
release/v0.12.0
Oct 26, 2026
Merged

jiangkoumo merged 9 commits into
mainfrom
release/v0.12.0

Conversation

@jiangkoumo

Copy link
Copy Markdown
Owner

P2 第二批:共同受控基准扩至 12 项,并提供真实 Jev 的 HN / Wikipedia 双宿主测量入口。

新增

  • 保留十项 p2-controlled-v1 历史清单,新增十二项 p2-controlled-v2,覆盖缺文本来源与两字段批处理,细分预期拒绝与预算统计。
  • bench/live-benchmark/ 使用相同目标、输入、预算和独立 URL + DOM 验证器,显式 --live 才联网。
  • 分别记录 logical ask 与 HTTP 尝试;缺失、部分和未结算 token usage 传播 null 与原因。

修复

  • 超时后阻止同一次 ask 内新增 fallback HTTP 请求。
  • ego 子进程启动错误保留已有结果,并进入正常失败和清理流程。
  • staging 中途失败清理已创建目录;CDP 部分连接失败保留 target 清理错误。
  • 四条独立 Node 故障回归已接入离线测试,修复前失败、修复后通过,并经独立复审确认。

验证与复跑

  • node bench/test-release-consistency.mjs:20/20。
  • node bench/test-live-benchmark.mjs、node bench/test-common-benchmark.mjs:exit 0。
  • 更新、护栏、滚动、路由、Windows 路径及依赖不变量检查、语法检查:全部 exit 0。
  • 共同受控集:两个宿主各 12/12,错误成功与网络请求为 0,清理确认。
  • 真实测量:node bench/live-benchmark/run.mjs --live --hosts both --tasks all --rounds 3。十二次试验全部完成,10/12 达成、错误成功 0、清理失败 0,模型 jev-1.14.0。HN 两宿主各 3/3;Wikipedia CDP 1/3、ego 3/3。两次 CDP 失败均为五步预算耗尽,命令如实 exit 1,样本完整保留。

原始数据与脚本哈希:测量说明。后续异常路径修复未改写原始测量数据或重复站点测量。

已知边界

真实测量仅 macOS arm64、两任务每宿主各三轮,不能外推普遍成功率或稳定宿主性能差异。Windows/Linux、远程或用户提供的 CDP endpoint 未验证。费用和人工接管仍为 null;MCP 继续为实验性入口。

更新:ego-decision-layer update;旧版无 update 子命令的用户按原安装渠道更新。

完整变更:CHANGELOG.md。

Adds live HN/Wikipedia tasks, independent URL+DOM verifiers, the shared dual-host runner, offline policy tests, and sanitized first-batch evidence. Failed trials are retained without retry or budget relaxation.

Fixes #15
Track settled per-ask receipts, distinguish unrun and unknown usage from confirmed zero, and keep unknown token totals through failure summaries. Add offline regressions and recompute the preserved live batch without rerunning sites. Refs #15
Handle ego spawn errors without losing earlier host results, clean partial staging allocations, and propagate CDP partial cleanup problems. Add four isolated offline fault regressions with failing-before evidence. Refs #15
Copilot AI balanced review requested due to automatic review settings October 26, 2026 11:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

@jiangkoumo
jiangkoumo merged commit 791c4c0 into main Oct 26, 2026
1 check passed
@jiangkoumo
jiangkoumo deleted the release/v0.12.0 branch October 26, 2026 11:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants