Establish a benchmark baseline with regression comparison - #59
Merged
Conversation
Adds tooling to record a tagged PhpBench baseline and diff runs against it: - composer bench:baseline — record/overwrite the local `baseline` tag - composer bench:compare — run and diff each subject vs the baseline Stabilizes the runner via phpbench.json (5 iterations, 2 warmup, 5% retry threshold), dropping rstdev from ~25% to <4%. Iteration/warmup counts were being overridden by @Iterations/@WarmUp annotations in ParseBench.php; removed them so phpbench.json is the single source of truth. Local storage (.phpbench/) is git-ignored — wall-clock times are machine- and PHP-specific and not portable. benchmarks/BASELINE.md documents the workflow, reference figures, and host context. ROADMAP updated; CI-side baselining left as a follow-up.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #59 +/- ##
============================================
+ Coverage 92.45% 92.76% +0.30%
Complexity 380 380
============================================
Files 6 6
Lines 981 981
============================================
+ Hits 907 910 +3
+ Misses 74 71 -3 🚀 New features to boost your workflow:
|
Records a baseline from the PR base's src/ using this PR's benchmark harness, then compares the head against it on the same runner — the only apples-to-apples comparison for machine-dependent wall-clock times. Runs on pull_request only. Non-blocking (continue-on-error) with a generous assertion: it flags only catastrophic regressions (>50% slower per subject) to tolerate shared-runner noise, and never blocks a merge. Surfaces the per-subject comparison table in the job log.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds benchmarking tooling so the PhpBench suite protects against regressions, plus a non-blocking CI job that runs it on every PR.
What
composer bench:baseline— record/overwrite a localbaselinetag.composer bench:compare— run the suite and diff each subject against the baseline.phpbench.json): 5 iterations, 2 warmup, 5% retry threshold → rstdev dropped from ~25% to <4%. Removed the@Iterations/@Warmupannotations inParseBench.phpthat were silently overriding the config..phpbench/git-ignored — wall-clock times are machine/PHP-specific and not portable.benchmarks/BASELINE.mddocuments the workflow, reference figures, and host context.CI integration
New non-blocking
benchmarksjob (pull_request only):src/, using this PR's benchmark harness/config.src/and compares — same runner, same job, so machine-dependent times are apples-to-apples.continue-on-error— surfaces the comparison table and catches real cliffs without blocking on shared-runner noise.Not included (follow-up)
mb_substrin the main loop) — the baseline now makes that measurable (≈134 μs/addr at batch scale to beat).No library code changed — benchmarks, CI, config, and docs only.