Conversation
…h, gate publishing - pyproject reads version from polyharness.__version__ (was drifting: 0.2.5 vs 0.2.4) - ph wrap: ignore_unknown_options so agent flags like -p forward instead of erroring - publish workflows: tag<->version consistency check, tests required before build/publish - CI: pip cache, macOS matrix, wheel content verification, node --check on npm wrapper - exclude __pycache__ from wheel template data
…ups on timeout - new proposer/toolkit.py: single WorkspaceToolkit shared by the Anthropic and OpenAI proposers (the tool code previously existed as two drifting copies) - path containment via resolve() + is_relative_to (startswith let iter_1 write into iter_10 and read ws_backup siblings) - remove the bash tool: a first-token allowlist over shell=True allowed 'echo x; rm -rf', 'ls > score.json', 'find -delete'; pure-Python file_search replaces the legitimate grep uses - score.json/metadata.json write-protected from proposers (evaluator owns them) - new utils/proc.py: run_process_group — CLI proposer and evaluator children now run as session leaders and get killpg'd on timeout (agent CLIs fork grandchildren that survived subprocess.run's kill and could hang communicate); partial stdout/stderr preserved for diagnosis; explicit utf-8 decoding - proposer.timeout now configurable (was hardcoded 600s) - claw-code: drop --verbose (drowned the change summary that parse_output keeps from the stdout tail) - API proposers: temperature only sent when explicitly configured (current-gen Claude models reject sampling params); Anthropic proposer gains prompt caching, bounded retries, stop_reason recording, and a final-summary pass when the tool budget is exhausted - tests: toolkit containment/protected-file/no-shell suite; hermes factory coverage; PermissionError path
…tor contract, log resilience - search_log now records the parent actually selected (tournament/pareto lineage was silently rewritten to best_iteration) - prepare_candidate strips parent score.json/metadata.json/traces (stale copies misled the proposer and duplicated leaderboard entries); failed iterations clean up their half-built candidate dir - base harness evaluated via its iter_0 copy: traces land where the proposer is told to look and base_harness/ stays pristine - cascade: gated stage-1 scores are averaged over the FULL task list (missing tasks count 0) and flagged cascade_gated so partial scores can't win best - evaluator: bare JSON number no longer crashes; per-task mode accepts the overall_score key templates emit (was silently 0.0); duplicate task stems no longer overwrite each other - search log: corrupt lines skipped with a warning instead of bricking resume/best/apply; unknown fields tolerated for forward compat - leaderboard rebuild skips corrupt score.json - resume: patience recomputed with run-time semantics (ties count); completed runs still produce the held-out test score; delta display can be positive - workspace: exclusive flock guards against two concurrent runs - evaluator.type narrowed to 'python' (docker/custom were accepted but unimplemented); drop the docker extra
Prompt-only rules ('don't edit evaluate.py', 'don't peek at the test set')
now have mechanical backing:
- IntegrityGuard hashes the evaluate script and every search task file at
run start and re-verifies before each candidate evaluation and before the
holdout scoring; any change aborts the run with IntegrityError instead of
logging untrustworthy scores. Candidate-local evaluate.py copies are
verified against the base harness version.
- HoldoutVault moves eval_split test task files into .ph_holdout/ for the
duration of the search (crash-recoverable via manifest), restores them
hash-verified for the final holdout evaluation, so ordinary proposer
exploration never encounters them.
- Orchestrator: IntegrityError is not swallowed as a failed iteration — the
run stops and the half-built candidate is removed.
This is tamper-evidence plus accident prevention, not a jail — CLI agents
run with the user's filesystem permissions (threat model documented in the
module docstring).
…fied prompts, parallel eval Loop-engineering upgrade — the search loop now learns from its own history instead of blindly retrying: - Reflective retry: novelty rejections and iteration failures are written to PROPOSER_FEEDBACK.md in the next candidate dir (file-based, so it works identically for CLI agents and API tool loops); prompts instruct every backend to read it first. Rejections name the duplicated iteration. - LessonBook (summary/lessons.jsonl + LESSONS.md): mechanical per-iteration digest — verdict, delta, backend, change summary — grouped into 'what improved' / 'what regressed'; prompts point proposers at it before raw traces. Non-Markovian memory at a fraction of the context cost. - search_log is now a full event stream: failed and duplicate-skipped iterations are logged with status/note/backend. best/pareto only consider evaluated entries; resume recomputes patience faithfully. - Failures get their own max_consecutive_failures stop (default 5) and no longer consume early-stop patience — infra flake isn't evidence that improvement is impossible. - Bandit: canonical UCB1 (the old formula double-counted the exploration constant), rewards are max-delta-normalized fitness improvements instead of binary (late small-but-real gains stay visible), state persists to summary/bandit.json and is restored on resume, reward range validated. - Novelty filter: parent is exempt from the near-duplicate threshold (small targeted edits are what the instructions ask for; only identical copies count) — fixes the built-in conflict with 'small steps beat rewrites'; exact-match + quick_ratio pruning + text caching cut the O(N²) cost. - Unified proposer prompt builder in base.py: all three backends share layout/rules/principles; the leaderboard is injected as numbers only (raw JSON embedded previous agents' free text — a cross-iteration prompt injection channel); openai backend finally gets the improvement principles. - Per-task evaluation can run concurrently (evaluator.parallel_tasks, default 1/serial; deterministic result order). - Model defaults centralized in config.py: API default -> claude-sonnet-5, claude-code pin -> claude-opus-4-8. - Instance RNG (no global random.seed pollution).
…based shell hook, doc truth - npm wrapper: self-recursion guard (npm's ph shadowing pip's caused a fork bomb), real exit codes propagated instead of retrying the command under the next strategy (side effects ran twice), quiet import probe before running, Windows venv paths - shell hook rewritten as wrapper functions (zsh + bash): zsh preexec cannot cancel the original command, so every wrapped call executed twice and the bash branch silently did nothing; bare interactive invocations left untouched - ph wrap: streams agent output live (tee) instead of buffering to the end; stdin inherited for interactive agents - CLI errors go to stderr via a dedicated console and are immune to -q (previously 'ph -q run' failed with exit 1 and zero output) - validate_assignment on all config models (CLI overrides bypassed every Field constraint); --max-iterations/--top get IntRange; max_iterations=0 is now the documented dry-run value; ph compare accepts 'best' and rejects garbage with a clear message; ph config show masks api_key - code-generation template: comment no longer claims exec() is sandboxed - README: CI badge instead of a stale hardcoded test count, shell-hook and npm install claims match actual behavior - CHANGELOG: full 0.3.0 entry
…; clean pycache from builds - SECURITY.md documents the real threat model (evaluator is not a sandbox; integrity guard is tamper-evidence, not a jail) with a private reporting channel - Contributor Covenant 2.1 code of conduct - structured bug-report / feature-request issue forms + PR checklist - CI/publish builds clean src __pycache__ so local wheels can't ship .pyc
Black-and-white manga banner (1774x887, matching the style used across weijt606 projects): an agent evolves from a failing tangle (score 0.41) into its strongest form (0.92) along a branching search tree with a glowing Pareto-frontier path. Replaces the ASCII-art logo at the top of both READMEs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
v0.3.0 is a loop-engineering release: the search loop now learns from its own history instead of blindly retrying, the evaluation pipeline is mechanically protected against reward hacking, and a deep audit's worth of correctness and security fixes land alongside. 8 commits, 263 tests passing, full changelog in
CHANGELOG.md.Loop engineering (new capabilities)
PROPOSER_FEEDBACK.mdin the next candidate dir; every backend's prompt reads it first, so retries diverge instead of repeating the same output.LessonBook) —summary/LESSONS.md: a mechanical per-iteration digest (verdict, delta, backend, change summary) grouped into "what improved / what regressed", giving proposers non-Markovian context at a fraction of the token cost of raw traces.search_log.jsonlwithstatus/note/backend; failures get their ownmax_consecutive_failuresstop and no longer consume early-stop patience.summary/bandit.jsonand restored on--resume.evaluator.parallel_tasks), configurableproposer.timeout, workspace run lock.Integrity & security
IntegrityGuard— evaluate script + task files hashed at run start and re-verified before every evaluation; mid-run tampering aborts the run instead of laundering fake scores into the log.HoldoutVault— witheval_split, test tasks are moved out of the searchable tree during search (crash-recoverable) and restored hash-verified for the final holdout score.resolve()+is_relative_to(prefix check letiter_1write intoiter_10); the bypassable pseudo-read-onlybashtool is removed (replaced by pure-Pythonfile_search);score.json/metadata.jsonwrite-protected; leaderboard injected as numbers only (raw JSON was a cross-iteration prompt-injection channel).Correctness fixes (deep-audit findings)
score.json/traces; cascade-gated partial scores can't win best; evaluator accepts template output contracts (was silently 0.0 in per-task mode); corrupt log lines no longer brick resume/best/apply; novelty filter no longer punishes the small targeted edits the prompts ask for.ph wrapforwards dashed agent flags and streams output live; shell hook rewritten as wrapper functions (zsh preexec ran every wrapped command twice); npm wrapper no longer self-recurses and propagates real exit codes; CLI errors go to stderr and survive-q.Release & docs
polyharness.__version__(0.2.5 shipped reporting 0.2.4); publish workflows gated on tests + tag↔version consistency; CI adds macOS, wheel-content verification,node --check.claude-sonnet-5API default,claude-opus-4-8Claude Code pin); temperature unset by default for current-gen models.SECURITY.md(honest threat model),CODE_OF_CONDUCT.md, issue/PR templates; manga-style banner in both READMEs.Test plan
ruff checkclean;node --checkon both npm wrapper scriptstwine checkpassed, no.pycleakage, clean-venv install smoke test