Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,6 @@ __pycache__/
# 单元测试的输出:每跑一次就新建一个带时间戳的文件,属可再生产物,不入仓库。
# 注:**此前已提交的 test-* 仍在版本控制里**(报告引用它们当证据),gitignore 不影响已跟踪文件。
bench/raw/test-*

# 共同基准的本机结果(每次运行重写,属可再生产物;报告里只引用摘要)
bench/common-benchmark/RESULTS.local.json
10 changes: 10 additions & 0 deletions THIRD-PARTY.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,3 +51,13 @@ SOFTWARE.
- **来源**:https://github.com/citrolabs/ego-lite (MIT)
- **性质**:本项目的**运行基础**(通过 `ego-browser` 驱动),**未再分发**其任何代码或文档。
根目录的 `SKILL.md` 是本项目自己撰写的附加技能,装在应用包之外。

---

## @modelcontextprotocol/sdk(可选 MCP 入口的运行时依赖)

- **来源**:npm `@modelcontextprotocol/sdk`,固定版本 **1.32.2**(`mcp/package.json`、`mcp/package-lock.json`)
- **许可**:MIT
- **涉及范围**:仅 `mcp/` 目录导入(stdio server/client 传输);**未复制或修改其源码**,也不进入核心引擎、
CLI 与安装脚本的运行依赖。不装 `mcp/node_modules` 时仓库其余部分行为不变。
- **性质**:v1.x 维护线,实现到 2025-11-25 的 MCP 规范。
126 changes: 126 additions & 0 deletions bench/common-benchmark/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
# P2 共同基准(第一批,10 项)

双宿主共同验收集:同一组**目标、起始页面、输入、确定性验证器与预算**,分别在 ego lite 与
「普通 Node + 外部 Chrome(CDP)」上跑一遍,用同一份报告口径汇总。

```bash
node bench/common-benchmark/run.mjs # 两个宿主;缺宿主则 SKIP 并写明原因
node bench/common-benchmark/run.mjs --hosts both --out /tmp/p2.json
node bench/test-common-benchmark.mjs # 清单/口径/预算/清理的离线自测
```

退出码:`0` 所有被请求的宿主通过 / `1` 有宿主失败 / `2` 宿主缺失被 SKIP 或用法错误。

## 与 P1 套件的关系(复用,不另起一套)

任务本体(fixture、注入决策、`verify` 断言)**直接引用** `bench/host-acceptance/tasks.mjs` 的
10 项,不复制代码;本目录只加了三件 P1 没有的东西:

1. **任务格式与预算**(`tasks.mjs`):每项写明起始页、输入、验证器位置、步数/时间/决策请求上限;
2. **统一 runner**(`run.mjs`):两个宿主同一条执行路径,宿主缺失按 SKIP 记录原因;
3. **汇总报告**(`suite.mjs`):成功率、错误成功率、延迟、请求数、花费、护栏拒绝率、人工接管率、
失败消耗,字段与未知值口径固定。

P1 的 12 项里有 2 项(`no-text-source`、`batch-two-fields`)不在本批 10 项内:前者与
`type-text` 共享「无文本来源」路径,后者是批处理能力验证,留到第二批补齐。

## 任务格式

| 字段 | 含义 |
| --- | --- |
| `id` / `goal` / `expect` | 报告口径用;`expect=goal` 要求引擎 `success=true`,`expect=guard` 要求**不**成功 |
| `start` | `about:blank` + 就地注入的合成 fixture(`bench/host-acceptance/tasks.mjs#<id>`) |
| `input` | 注入判定器的确定性选择、`--text` 候选、是否启用响应校验 |
| `verifier` | 任务自带的 `verify(page, ctx)`:动作后真实 DOM/URL/派发记录断言 |
| `budget` | `steps`(引擎步数)、`timeMs`(单任务墙钟)、`decisionRequests`(判定器调用上限) |
| `conditions` | 登录/验证码/地域/站点变化:合成任务一律 `not-applicable`,真实站点任务另行记录 |

**预算执行方式**:`maxSteps` 直接传给引擎;`decisionRequests` 由包装的 `ask` 计数,超限即抛
`EGO_BENCH_BUDGET`(`executionStopped`),引擎停止而不是继续派发;`timeMs` 用 P1 的
单任务超时(超时关闸、后续任务记 `notRun` 并停止整套)。三项任一越界,该任务 `budgetPassed=false`
并判失败——不把「跑完了但超预算」算通过。

**网络隔离**:引擎调用期间替换 `globalThis.fetch`,任何外部请求都会记数并抛错;任务结束断言
`networkRequests === 0`。宿主的 CDP 连接与 fixture 注入在替换之前完成,不受影响。

## 报告字段

顶层:`schemaVersion`、`taskSet`、`runtime`、`hosts[]`(`host`/`status`/`reason`/`report`/`staged`/`cleanup`)、
`complete`、`status`。

`hosts[].report.summary`:

| 字段 | 口径 |
| --- | --- |
| `passed` / `failed` / `complete` | 任务通过数(`taskPassed`)、总数、是否全部跑完 |
| `successRate` | 通过 / 已跑任务 |
| `goalSuccessRate` | 只统计 `expect=goal` 的任务 |
| `falseSuccessRate` | 引擎说成功但 DOM 断言不成立 / 已跑任务(**错误成功率**) |
| `guardrailRejectionRate` | `expect=guard` 任务里被 guard/stale/控制停止拦下的比例 |
| `rejectionRateAllTasks` | 同一口径放开到全部任务 |
| `rejectionMatches` | 「是否被拦」与 `expect` 一致的**任务数**(不是比例) |
| `zeroDispatchGuardTasks` / `dispatchCountGuardTasks` | 护栏任务里 0 派发 / 已派发后才停止(`control-stop`)的数量 |
| `postStopPageCalls` | 控制停止之后引擎仍发起的页面调用总数(必须 0) |
| `decisionRequests` / `dispatches` / `latencyMs` / `networkRequests` | 计数与墙钟总和 |
| `costUsd` / `humanTakeoverRate` / `unknownMetrics` | 注入判定器下测不到,一律 `null`,不猜 0 |
| `failureConsumption` | 失败任务数及其延迟、请求数、费用(无失败时费用记 0) |

单任务还保留 P1 的 `metrics.phases`(观测/决策/执行/校验分阶段耗时)、`stopReason`、
`guardrailRejections`、`history`(紧凑化),以及新增的 `networkRequests`、`budgetPassed`。

## 宿主腿

| 腿 | 做法 | 清理与如实记录 |
| --- | --- | --- |
| `cdp` | 未给 `--endpoint` 就用 `--headless=new` 自建临时 profile 的 Chrome,`connectCdpHost` 只建自有 target | 关自有 target(`targetClosed` 必须为 true)、SIGTERM→SIGKILL 后按 PID 复查进程、删临时 profile;任一步失败即 `status=failed` |
| `ego` | 普通 `ego-browser nodejs` 子进程,脚本在 TaskSpace 里跑同一 `runCommonBenchmark` | `finish({ keep: [] })`;超时则报 `TaskSpace cleanup unconfirmed`,不算通过 |

**ego 腿为什么要暂存一拷贝**:在本工作区的 LayerFS 路径下,ego 运行时的动态 `import` 会**挂住**
(同一字节的模块放 `/tmp` 或 `~/LayerFS` 就能立刻加载,见下「LayerFS 观察」)。因此 ego 腿先把 5 个
模块(两份 `suite.mjs` / `tasks.mjs` + `scripts/decider-loop.mjs`)拷到临时目录,**逐文件校验 sha256**
后才 import,并把 `staged.files` 与原因写进报告。普通路径(仓库在主盘上)不需要这步,也没有被绕过。

## 本机实测(2026-10-25,本 worktree)

| 宿主 | 结果 | 决策请求 | 派发 | 延迟总和 | 错误成功 | 护栏拒绝率 | 清理 |
| --- | --- | ---: | ---: | ---: | ---: | ---: | --- |
| cdp(临时 headless Chrome 156) | 10/10,exit 0 | 13 | 17 | 152ms | 0 | 100%(4/4 guard) | target 关闭 + 进程退出 |
| ego(ego lite 0.6.0.20) | 10/10,exit 0 | 13 | 17 | 271ms | 0 | 100%(4/4 guard) | TaskSpace 正常 finish |

两腿都是 `networkRequests=0`、`rejectionMatches=10`、`postStopPageCalls=0`。
原始结果:`RESULTS.local.json`(本机生成,结构即报告格式)。

这些数字只说明**机制一致性**:合成 DOM + 注入判定器(`options.ask`),**不是**模型成功率,
也不是线上站点成功率。延迟受机器负载影响,不构成性能基准。

## 真实站点候选(本轮只登记,不跑)

`tasks.mjs` 的 `SITE_CANDIDATES` 只是候选清单,runner 不会执行它们。纳入条件:起始 URL 稳定、
验证器可用 URL 路径或 DOM 状态表达、登录/验证码/地域影响可记录。

| 候选 | 拟用验证器 | 已知前置 |
| --- | --- | --- |
| HN 导航 | `pathname === /newcomments` | 无需登录;站点结构长期稳定 |
| 维基搜索 | `pathname === /wiki/Jev` + 词条 DOM | 自动翻译不得参与字符串比较 |
| DDG 设置页原生下拉 | 选中值 + 设置持久化 | 需先定义「改选后如何确定性核对」 |

## 尚未验证(如实)

- 真实模型(Jev)与真实站点:本批全部离线;`jev-1.14.0` 的线上成功率未测。
- 费用 / token / 人工接管:注入判定器下不可比,报告 `null`。
- Windows / Linux 的 CDP 腿:只在本机 macOS + Chrome 156 跑过。
- 每项任务的延迟都是单次测量,没有重复轮次与方差。
- ego 腿的暂存拷贝是环境适配;在非 LayerFS 路径下未单独验证「直接 import 也正常」这条旁路。

## LayerFS 观察(本 worktree)

- **ego 运行时无法从本工作区路径 import 模块**:`ego-browser nodejs -e 'await import("<ws>/bench/...mjs")'`
在 15–25s 内无任何输出(含首行 `console.log`),同样字节的文件放 `~/LayerFS/` 或 `/tmp` 秒回。
`fs.realpath/stat` 在该路径上都是 1ms,所以不是节点元数据慢,更像 ego 加载器对该路径的处理问题。
未深挖;runner 用暂存拷贝绕过并写进报告,没有把问题当成通过。
- 常规操作没有看到 EPERM、`._` 文件、权限错误或挂载断开:`git status --short` 0.06s,
`node bench/test-common-benchmark.mjs` 5.2s,`node bench/test-host-integration-live.mjs --suite --suite-only`
6.4s,`git diff --check` 0 退出。与源仓库相比没有明显变慢的主观差异,但没有做同机对照计时。
- `ego-browser nodejs` 从 stdin 读脚本:**heredoc 与 `< file` 在本环境都不生效**(进程挂住、无输出),
管道(`printf … | ego-browser nodejs`)和 `-e 'await import("file:///…")'` 正常。runner 用
`spawn` + `stdin.end()`,等价于管道,因此不受影响。
185 changes: 185 additions & 0 deletions bench/common-benchmark/run.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
#!/usr/bin/env node
// Node owns orchestration only; both legs import the same task runner and engine.
import { spawn } from 'node:child_process';
import { createHash } from 'node:crypto';
import { access, copyFile, mkdir, mkdtemp, readFile, rm, writeFile } from 'node:fs/promises';
import { tmpdir } from 'node:os';
import { dirname, join, resolve } from 'node:path';
import { fileURLToPath } from 'node:url';
import { connectCdpHost } from '../../scripts/hosts/cdp-host.mjs';
import { runCommonBenchmark } from './suite.mjs';

const repo = resolve(dirname(fileURLToPath(import.meta.url)), '../..');
const sleep = ms => new Promise(r => setTimeout(r, ms));

export function parseArgs(argv) {
const args = { hosts: 'both' };
for (let i = 0; i < argv.length; i++) {
const name = argv[i];
if (name === '--help') { args.help = true; continue; }
if (!['--hosts', '--endpoint', '--chrome', '--ego', '--out'].includes(name)) throw new Error(`Unknown option: ${name}`);
const value = argv[++i];
if (!value || value.startsWith('--')) throw new Error(`Missing value: ${name}`);
args[name.slice(2)] = value;
}
if (!['both', 'ego', 'cdp'].includes(args.hosts)) throw new Error('hosts must be both, ego or cdp');
return args;
}

async function stopChild(child) {
if (!child?.pid) return;
const stopped = () => child.exitCode !== null || child.signalCode !== null;
if (!stopped()) {
child.kill('SIGTERM');
const deadline = Date.now() + 5000;
while (!stopped() && Date.now() < deadline) await sleep(25);
}
if (!stopped()) {
child.kill('SIGKILL');
await new Promise(r => child.once('close', r));
}
try { process.kill(child.pid, 0); }
catch (err) { if (err.code === 'ESRCH') return; throw err; }
throw new Error(`Child still alive: ${child.pid}`);
}

async function cdpLeg(args) {
const chromePath = args.chrome ?? process.env.EGO_CDP_CHROME ?? (process.platform === 'darwin'
? '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' : null);
if (!args.endpoint) {
if (!chromePath) return { host: 'cdp', status: 'skipped', reason: 'No Chrome path; supply --chrome or --endpoint' };
try { await access(chromePath); }
catch (err) {
if (err.code === 'ENOENT') return { host: 'cdp', status: 'skipped', reason: `Chrome unavailable: ${chromePath}` };
throw err;
}
}
let child, temp, host, outcome;
const cleanup = {};
try {
let endpoint = args.endpoint;
if (!endpoint) {
temp = await mkdtemp(join(tmpdir(), 'ego-p2-bench-'));
child = spawn(chromePath, ['--headless=new', '--no-first-run', '--no-default-browser-check',
'--disable-background-networking', '--remote-debugging-port=0', '--user-data-dir=' + temp, 'about:blank'],
{ cwd: repo, stdio: ['ignore', 'ignore', 'pipe'] });
cleanup.pid = child.pid;
let log = '', launchError;
child.on('error', err => { launchError = err; });
child.stderr.on('data', data => { log = (log + data.toString()).slice(-16000); });
const deadline = Date.now() + 15000;
while (!endpoint && Date.now() < deadline) {
if (launchError) throw launchError;
if (child.exitCode !== null) throw new Error('Chrome exited before CDP was ready');
endpoint = log.match(/DevTools listening on (ws:\/\/\S+)/)?.[1];
if (!endpoint) await sleep(50);
}
if (!endpoint) throw new Error('Chrome CDP startup timed out');
}
host = await connectCdpHost({ endpoint, timeoutMs: 10000, commandTimeoutMs: 15000 });
const report = await runCommonBenchmark(host.page, { host: 'cdp' });
outcome = { host: 'cdp', status: report.summary.complete && report.summary.failed === 0 ? 'passed' : 'failed',
browserVersion: host.metadata?.browserVersion, report };
} catch (err) { outcome = { host: 'cdp', status: 'failed', reason: err.message }; }
finally {
try {
if (host) {
cleanup.target = await host.close({ keepPage: false });
if (cleanup.target.targetClosed !== true) throw new Error('CDP target cleanup not confirmed');
}
} catch (err) { cleanup.error = err.message; }
try { await stopChild(child); if (child?.pid) cleanup.processStopped = true; }
catch (err) { cleanup.error = err.message; }
if (temp && (!child?.pid || cleanup.processStopped)) await rm(temp, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 });
if (cleanup.error) outcome = { ...outcome, status: 'failed' };
}
return { ...outcome, cleanup };
}

async function stageFiles() {
const stage = await mkdtemp(join(tmpdir(), 'ego-p2-stage-'));
const files = ['bench/common-benchmark/suite.mjs', 'bench/common-benchmark/tasks.mjs',
'bench/host-acceptance/suite.mjs', 'bench/host-acceptance/tasks.mjs', 'scripts/decider-loop.mjs'];
const copied = [];
for (const rel of files) {
const dest = join(stage, rel);
await mkdir(dirname(dest), { recursive: true });
await copyFile(join(repo, rel), dest);
const digest = async p => createHash('sha256').update(await readFile(p)).digest('hex');
const [from, to] = [await digest(join(repo, rel)), await digest(dest)];
if (from !== to) throw new Error(`Staged copy differs: ${rel}`);
copied.push({ path: rel, sha256: from });
}
return { stage, copied };
}

async function egoLeg(args) {
const executable = args.ego ?? 'ego-browser';
// The ego loader stalls when importing modules from this LayerFS workspace path
// (reproduced: same bytes load instantly from /tmp or ~/LayerFS). Stage a
// byte-identical copy (sha256 checked) and import that; the staged path is reported.
let stage;
try { stage = await stageFiles(); }
catch (err) { return { host: 'ego', status: 'failed', reason: `Cannot stage benchmark modules: ${err.message}` }; }
const entry = join(stage.stage, 'bench/common-benchmark/suite.mjs');
// ASCII bootstrap; module path and result file are absolute. The result goes to a
// file because the ego runtime truncates very long console lines.
const resultFile = join(stage.stage, 'result.json');
const script = `const fs=await import('node:fs/promises');
const {runCommonBenchmark}=await import(${JSON.stringify(entry)});
const space=await taskSpace('ego-p2-common-'+Date.now());
let result;
try {result=await runCommonBenchmark(space.page('p1'),{host:'ego'});}
finally {await space.finish({keep:[]});}
await fs.writeFile(${JSON.stringify(resultFile)},JSON.stringify(result));
console.log('P2_RESULT_WRITTEN');`;
const child = spawn(executable, ['nodejs'], { cwd: repo, stdio: ['pipe', 'pipe', 'pipe'] });
let stdout = '', stderr = '', combined = '', launchError, timedOut = false;
// ego-browser writes task output on stderr, so both streams are scanned.
child.stdout.on('data', data => { stdout += data.toString(); combined = (combined + data.toString()).slice(-200000); });
child.stderr.on('data', data => { stderr = (stderr + data.toString()).slice(-4000); combined = (combined + data.toString()).slice(-200000); });
child.stdin.on('error', () => {});
child.on('error', err => { launchError = err; });
const timer = setTimeout(() => { timedOut = true; stopChild(child).catch(() => {}); }, 150000);
child.stdin.end(script);
const code = await new Promise(r => child.once('close', r));
clearTimeout(timer);
const staged = { stagedFrom: 'repo root', files: stage.copied, reason: 'ego loader stalls on this workspace path' };
try {
if (launchError?.code === 'ENOENT') return { host: 'ego', status: 'skipped', reason: `ego-browser unavailable: ${executable}`, staged };
const line = combined.split('\n').find(l => l.includes('P2_RESULT_WRITTEN'));
if (timedOut || launchError || !line || code !== 0) return { host: 'ego', status: 'failed', staged,
reason: timedOut ? 'Ego process timeout; TaskSpace cleanup unconfirmed' : launchError?.message ?? `No complete result (exit ${code}): ${stderr || combined.slice(-2000)}` };
let report;
try { report = JSON.parse(await readFile(resultFile, 'utf8')); }
catch (err) { return { host: 'ego', status: 'failed', staged, reason: `Unreadable result file: ${err.message}` }; }
return { host: 'ego', status: report.summary.complete && report.summary.failed === 0 ? 'passed' : 'failed', report, staged,
cleanup: { taskSpaceFinished: true } };
} finally { await rm(stage.stage, { recursive: true, force: true, maxRetries: 5, retryDelay: 100 }); }
}

export async function run(args) {
const hosts = [];
// Sequential: both suites fence global fetch and must not race in this process.
for (const host of ['cdp', 'ego']) {
if (args.hosts !== 'both' && args.hosts !== host) continue;
hosts.push(await (host === 'cdp' ? cdpLeg(args) : egoLeg(args)));
}
return { schemaVersion: 1, taskSet: 'p2-controlled-v1', generatedAt: new Date().toISOString(),
runtime: { node: process.version, platform: process.platform, arch: process.arch },
hosts, complete: hosts.every(h => h.status === 'passed'),
status: hosts.some(h => h.status === 'failed') ? 'failed' : hosts.some(h => h.status === 'skipped') ? 'incomplete' : 'passed' };
}

if (process.argv[1] && resolve(process.argv[1]) === fileURLToPath(import.meta.url)) {
try {
const args = parseArgs(process.argv.slice(2));
if (args.help) console.log('node bench/common-benchmark/run.mjs [--hosts both|ego|cdp] [--endpoint URL | --chrome PATH] [--ego PATH] [--out FILE]\nExit: 0 all requested hosts passed; 1 failure; 2 skipped/incomplete or usage error');
else {
const report = await run(args);
if (args.out) await writeFile(resolve(args.out), JSON.stringify(report, null, 2) + '\n');
for (const h of report.hosts) console.log(JSON.stringify({ host: h.host, status: h.status, reason: h.reason, summary: h.report?.summary, cleanup: h.cleanup }));
process.exitCode = report.status === 'passed' ? 0 : report.status === 'failed' ? 1 : 2;
}
} catch (err) { console.error(err.message); process.exitCode = 2; }
}
Loading
Loading