Add `Basic Blocking Locust` perf test by achamayou · Pull Request #8158 · microsoft/CCF

Amaury Chamayou (achamayou) · 2026-08-14T11:23:59Z

Summary

Adds pi_basic_blocking_locust (perf label Basic Blocking Locust), which measures blocking writes (PUT /records/blocking/{key}) with locust instead of piccolo. It is intended to replace pi_basic_blocking for blocking workloads, where the piccolo client materially limits the figure it reports.

Locust is already a test dependency and is already used in tests/infra/service_load.py. No new dependencies.

All numbers below are from the bench-ab job on the CI runner, not from a developer machine.

Why not piccolo for blocking workloads

The piccolo client limits the result. submit polls a non-blocking socket without ever sleeping (src/clients/tls_client.h loops on BIO_read while BIO_should_retry). For pipelined tests a client is rarely idle, so this costs little. For blocking writes a client spends nearly all of its time waiting for commit, and spins throughout.

Both tests run in the same CI job, on the same machine, against the same build. A blocking client sends one request at a time and each returns on commit, so with a signature every 100ms the closed-loop prediction is simply the client count divided by 0.1s:

test	clients	tx/s	predicted	achieved
`Basic Blocking` (piccolo)	128	1,008	1,280	79%
`Basic Blocking Locust`	128	1,244	1,280	97%
`Basic Blocking Locust`	320	3,052	3,200	95%

At the same client count on the same machine, locust reports 23% more throughput and tracks the model to within 3%. The remaining 21% that piccolo leaves on the table is client overhead, not the service.

Its headline metric includes client startup. average_throughput_tx/s divides total requests by wall-clock from first send to last receive, so the staggered startup of 128 client processes counts as measured time. basicperf.py already records the honest figure alongside it as all_clients_active_average_throughput_tx/s. The distortion grows as a run gets faster, so the metric understates improvements, which is the wrong way round for regression detection.

Concurrency is fixed at generation time. Piccolo takes one pre-generated parquet file per client, so changing the client count means regenerating the workload. Locust takes it as an argument, which is what makes the sweep below possible.

What it runs

setting	value	why
clients	320	each holds one blocking write in flight
locust processes	10	a locust process drives all of its users from one thread, so a single process becomes the limit before the service does
signature intervals	2ms, 100ms, 1000ms	each against its own network; see below
snapshot interval	10000 transactions	matches `add_piccolo_test`; `add_e2e_test` passes nothing and `e2e_args` defaults to 10, which fsyncs a ~213KB snapshot every 10 transactions and measures the disk
measurement window	20s, after all clients have spawned

FastHttpUser (geventhttpclient) is used rather than HttpUser (requests), which cannot drive enough requests per second to saturate the service. mTLS is configured through ssl_context_factory.

The window is timed from locust's spawning_complete, and --reset-stats discards everything recorded during the ramp. Locust's own --run-time cannot be used, because it starts counting when locust starts and so includes the ramp: with a slow spawn rate a run would end mid-ramp and report a plausible looking figure for a client count it never reached. It is passed only as a backstop. The run fails, rather than reporting, if it ends before all clients have spawned, if the window achieved is short, or if any request failed.

Results

From the CI run at f23fce1:

signature interval	tx/s	p50	p99	clients / interval
2ms	24,354	11ms	21ms	-
100ms	3,049	99ms	100ms	3,200
1000ms	314	1000ms	1300ms	320

The three points measure two different things, which is why they are swept. At 1s and 100ms the workload is latency-bound: throughput is the client count divided by the interval, to within 5%, and p50 sits on the interval, so these detect regressions in the commit path. At 2ms it leaves that regime, since p50 is 11ms rather than 2ms, so the node itself is the limit and the figure measures capacity. A regression in request handling shows up there and nowhere else.

The capacity point is genuinely saturated rather than merely faster. The previous CI run measured the same configuration with a 5ms interval instead of 2ms and reported 24,388 tx/s against 24,354, a difference of 0.1%. Shortening the interval further changes nothing because the node, not the timer, is already the constraint.

CI reproduces all of this closely. Across consecutive runs, the 100ms point gave 1,241 / 1,245 / 1,247 tx/s at 128 clients and 3,049 / 3,055 at 320, and the capacity point gave 17,338 / 17,368 tx/s at 128 clients. Every one of those pairs agrees to within 0.5%, which is what makes the benchmark usable for regression detection.

Raising the client count from 128 to 320 moves the capacity point from 17,338 to 24,354 tx/s, a 41% gain for 2.5x the clients, while p50 rises from 6ms to 11ms. The service is therefore past its knee at 320 clients, which is where a capacity benchmark should sit.

Structure

tests/infra/basicperf_locustfile.py defines the load: one blocking write in flight per user, and the run's stop timing.
tests/basicperf_locust.py owns the network. Per interval it starts a node, pre-populates the key space, runs locust, and converts locust's statistics into bencher metrics.
tests/infra/key_space.py holds the key space helper previously private to basicperf.py. It has to move: basicperf.py imports piccolo, which only resolves when tests/infra is on sys.path, so it cannot be imported from a script in tests/.
scripts/perf_compare_radar.py previously dropped any benchmark with no main history, which made a new benchmark invisible on the pull request that adds it. Such a benchmark is now normalized against the branch's own earliest run instead. Long axis labels also now elide the middles of their words from the left, rather than cutting the middle out of the whole label, so that the last word stays readable longest.

Bencher output

Registered with add_e2e_test(... LABEL perf CONFIGURATIONS perf), following the commit_latency precedent for a perf test that is not piccolo-driven. bencher-ab.yml needs no change: it runs ./tests.sh -VV -L perf -C perf. One key per interval, each carrying throughput, latency and memory:

"Basic Blocking Locust (sig_ms_interval=100ms)": {
    "throughput": { "value": 3049.0 },
    "latency": { "value": 99.0, "high_value": 100.0, "low_value": 71.8 },
    "memory": { "value": 95260672, "high_value": 95260672 }
}

The locust master binds a free port per run rather than the default 5557, so concurrent locust runs on one machine do not collide.

Follow-up work

#8161 ("Isolate node and clients on separate CPUs for the locust perf test") is stacked on this branch and should merge after it. It pins the node and the load generator to disjoint CPU sets. Its measured benefit at the 100ms operating point is zero, because the node uses about a quarter of a core there. The case for it is insurance against co-tenant load on shared CI runners, which no single measurement can observe.

commit_latency and historical_query_perf_test are also registered via add_e2e_test and so also snapshot every 10 transactions. Not changed here, since correcting them would shift their published baselines, but their current figures partly measure snapshot I/O.

Testing

ctest -R pi_basic_blocking_locust -C perf passes.
ctest -R pi_basic_blocking -C perf still passes, covering the key_space.py refactor.
The radar changes were verified against fixtures: benchmarks with and without main history, and label shortening at every width from the full label down to five characters.
black, ruff, mypy, cmake-format, copyright and ASCII checks pass.

Draft while the defaults settle: the measurement window (20s), the process count (10), and whether the client ramp should eventually become a locust LoadTestShape with staged counts in one run.

Copilot

Pull request overview

Adds a Locust-driven blocking-write performance benchmark with configurable concurrency.

Changes:

Adds the FastHttpUser workload and benchmark runner.
Publishes throughput, latency, and memory metrics to Bencher.
Extracts shared key-space setup and registers the perf test.

Custom instructions used

.github/copilot-instructions.md

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`CMakeLists.txt`	Registers the new performance test.
`tests/basicperf_locust.py`	Runs Locust and publishes metrics.
`tests/infra/basicperf_locustfile.py`	Defines the blocking-write workload.
`tests/infra/key_space.py`	Provides shared key-space setup.
`tests/infra/basicperf.py`	Uses the extracted helper.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

github-actions · 2026-08-14T13:48:16Z

Description

Comparing 5 available runs from this branch (#8158) against the trend of the last 30 main runs.

Each chart plots every benchmark as an axis, with values normalized so 100 is the EWMA baseline of recent main runs, using a 7-run half-life. The 5 orange branch lines run from the oldest (faintest) to the latest (darkest and thickest); the darker blue band is the main baseline +/- 1 std dev and the lighter blue band around it is +/- 2 std dev.

Axis labels show the latest branch value and its difference from the main EWMA baseline, where 0% is on the baseline. They are coloured green where the latest run improves on the baseline, red where it regresses, and grey where the difference is within one std dev of the baseline (within noise). Higher is better for throughput and rate, lower for latency and memory.

A benchmark which does not exist on main yet has no baseline of its own, so this branch's earliest run is used as its reference and its band is measured across this branch's runs. Its axis is normalized, scaled and coloured like any other, but the comparison is against this branch rather than against main.

Throughput (tx/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-7{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(9){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(10){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 67,791 tx/s ▬ +1%"]
  axis b1["Basic Blocking: 999 tx/s ▬ -2%"]
  axis b2["Basic Blocking Locust 1000ms: 315 tx/s ▲ 1%"]
  axis b3["Basic Blocking Locust 100ms: 3,054 tx/s ▬ 0%"]
  axis b4["Basic Blocking Locust 2ms: 32,612 tx/s ▲ 34%"]
  axis b5["Basic JS: 4,820 tx/s ▬ +1%"]
  axis b6["Basic Multi-Threaded: 86,374 tx/s ▬ +1%"]
  axis b7["Historical Queries: 211,712 tx/s ▲ 5%"]
  axis b8["Logging: 61,035 tx/s ▬ -2%"]
  axis b9["Logging JWT: 10,063 tx/s ▬ -1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{106.12, 103.60, 100.42, 100.34, 130.14, 106.03, 108.90, 109.41, 107.97, 107.71}
  curve stddev1_high["main EWMA + 1 std dev"]{103.06, 101.80, 100.21, 100.17, 115.07, 103.02, 104.45, 104.70, 103.98, 103.86}
  curve stddev1_low["main EWMA - 1 std dev"]{96.94, 98.20, 99.79, 99.83, 84.93, 96.98, 95.55, 95.30, 96.02, 96.14}
  curve stddev2_low["main EWMA - 2 std dev"]{93.88, 96.40, 99.58, 99.66, 69.86, 93.97, 91.10, 90.59, 92.03, 92.29}
  curve branch_1["#8158 (3 runs earlier)"]{91.74, 98.91, 100.26, 99.79, 100.00, 97.41, 94.56, 95.76, 89.12, 100.95}
  curve branch_2["#8158 (2 runs earlier)"]{101.80, 100.93, 100.57, 99.94, 137.00, 101.58, 103.09, 108.93, 95.55, 101.91}
  curve branch_3["#8158 (1 run earlier)"]{103.46, 100.33, 100.48, 99.54, 133.00, 98.23, 101.28, 104.16, 93.41, 101.67}
  curve branch_4["#8158"]{101.39, 98.46, 100.51, 99.97, 133.90, 100.80, 101.04, 104.74, 97.84, 99.42}
  graticule polygon
  max 161
  min 46
  ticks 0
  showLegend false

Latency (ms)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-7{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#2DA44E!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic Blocking Locust 1000ms: 1,000 ms ▬ 0%"]
  axis b1["Basic Blocking Locust 100ms: 99 ms ▬ 0%"]
  axis b2["Basic Blocking Locust 2ms: 9 ms ▼ 18%"]
  axis b3["Commit Latency 16ms: 4.61 ms ▬ -16%"]
  axis b4["Commit Latency 1ms: 1.93 ms ▬ +1%"]
  axis b5["Commit Latency 256ms: 204 ms ▬ 0%"]
  curve stddev2_high["main EWMA + 2 std dev"]{100.00, 100.00, 119.81, 172.54, 110.82, 101.87}
  curve stddev1_high["main EWMA + 1 std dev"]{100.00, 100.00, 109.91, 136.27, 105.41, 100.94}
  curve stddev1_low["main EWMA - 1 std dev"]{100.00, 100.00, 90.09, 63.73, 94.59, 99.06}
  curve stddev2_low["main EWMA - 2 std dev"]{100.00, 100.00, 80.19, 27.46, 89.18, 98.13}
  curve branch_1["#8158 (3 runs earlier)"]{100.00, 100.00, 100.00, 66.56, 96.31, 102.04}
  curve branch_2["#8158 (2 runs earlier)"]{100.00, 100.00, 72.73, 95.42, 98.37, 99.78}
  curve branch_3["#8158 (1 run earlier)"]{100.00, 100.00, 81.82, 76.50, 101.32, 99.40}
  curve branch_4["#8158"]{100.00, 100.00, 81.82, 83.54, 100.60, 99.90}
  graticule polygon
  max 224
  ticks 0
  showLegend false

Memory (bytes)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-7{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(2){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(5){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(7){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(8){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(9){fill:#E5484D!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["Basic: 90.1 MiB ▲ 4%"]
  axis b1["Basic Blocking: 72 MiB ▲ 1%"]
  axis b2["Basic Blocking Locust 1000ms: 88.5 MiB ▬ 0%"]
  axis b3["Basic Blocking Locust 100ms: 89.4 MiB ▬ 0%"]
  axis b4["Basic Blocking Locust 2ms: 93.3 MiB ▲ 1%"]
  axis b5["Basic JS: 71.6 MiB ▬ +1%"]
  axis b6["Basic Multi-Threaded: 89.3 MiB ▬ 0%"]
  axis b7["Logging: 76.7 MiB ▲ 1%"]
  axis b8["Logging JWT: 69.4 MiB ▲ 1%"]
  curve stddev2_high["main EWMA + 2 std dev"]{107.05, 100.59, 101.50, 101.66, 100.63, 103.50, 102.13, 101.60, 101.53}
  curve stddev1_high["main EWMA + 1 std dev"]{103.53, 100.30, 100.75, 100.83, 100.32, 101.75, 101.06, 100.80, 100.76}
  curve stddev1_low["main EWMA - 1 std dev"]{96.47, 99.70, 99.25, 99.17, 99.68, 98.25, 98.94, 99.20, 99.24}
  curve stddev2_low["main EWMA - 2 std dev"]{92.95, 99.41, 98.50, 98.34, 99.37, 96.50, 97.87, 98.40, 98.47}
  curve branch_1["#8158 (3 runs earlier)"]{99.32, 100.07, 102.08, 101.43, 100.00, 99.38, 102.83, 101.27, 100.04}
  curve branch_2["#8158 (2 runs earlier)"]{102.00, 99.40, 100.45, 100.62, 100.78, 98.23, 99.59, 99.04, 99.37}
  curve branch_3["#8158 (1 run earlier)"]{100.53, 100.16, 100.69, 102.14, 100.68, 98.66, 100.65, 100.07, 100.24}
  curve branch_4["#8158"]{104.13, 100.62, 100.08, 100.04, 100.70, 100.70, 99.87, 100.90, 101.08}
  graticule polygon
  max 112
  min 88
  ticks 0
  showLegend false

Rate (ops/s)

---
config:
  radar:
    width: 620
    height: 620
    marginTop: 90
    marginRight: 220
    marginBottom: 60
    marginLeft: 220
    axisLabelFactor: 1.12
    curveTension: 0.08
  theme: base
  themeCSS: |
    .radarCurve-0{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-1{fill:color-mix(in srgb, #62B5E5 40%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-2{fill:color-mix(in srgb, #62B5E5 13%, var(--color-canvas-default,var(--bgColor-default,#fff)))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarCurve-3{fill:var(--color-canvas-default,var(--bgColor-default,#fff))!important;fill-opacity:1!important;stroke:none!important;stroke-width:0!important}
    .radarAxisLabel,.radarTitle{fill:var(--color-fg-default,var(--fgColor-default,#111827))!important;color:var(--color-fg-default,var(--fgColor-default,#111827))!important}
    .radarCurve-4{stroke-width:1.5px!important;stroke-opacity:0.20!important}
    .radarCurve-5{stroke-width:1.5px!important;stroke-opacity:0.30!important}
    .radarCurve-6{stroke-width:1.5px!important;stroke-opacity:0.40!important}
    .radarCurve-7{stroke-width:1.5px!important;stroke-opacity:0.50!important}
    .radarCurve-8{stroke-width:1.75px!important;stroke-opacity:1.00!important}
    .radarAxisLabel:nth-of-type(1){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(2){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(3){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(4){fill:#E5484D!important}
    .radarAxisLabel:nth-of-type(5){fill:#808A94!important}
    .radarAxisLabel:nth-of-type(6){fill:#808A94!important}
  themeVariables:
    cScale0: "#62B5E5"
    cScale1: "#62B5E5"
    cScale2: "#62B5E5"
    cScale3: "#62B5E5"
    cScale4: "#F97316"
    cScale5: "#F97316"
    cScale6: "#F97316"
    cScale7: "#F97316"
    cScale8: "#F97316"
    radar:
      axisColor: "#9CA3AF"
      graticuleColor: "#E5E7EB"
      graticuleOpacity: 0
      axisStrokeWidth: 1
      curveOpacity: 0
---
radar-beta
  axis b0["CHAMP get: 38,352,778 ops/s ▬ +1%"]
  axis b1["CHAMP put: 5,494,520 ops/s ▬ 0%"]
  axis b2["KV deserialisation: 1,631,588 ops/s ▬ +1%"]
  axis b3["KV serialisation: 1,400,364 ops/s ▼ 3%"]
  axis b4["KV s…t deserialisation: 4,169 ops/s ▬ 0%"]
  axis b5["KV snapshot serialisation: 4,787 ops/s ▬ +5%"]
  curve stddev2_high["main EWMA + 2 std dev"]{106.60, 106.17, 105.69, 104.58, 105.65, 112.07}
  curve stddev1_high["main EWMA + 1 std dev"]{103.30, 103.08, 102.84, 102.29, 102.82, 106.03}
  curve stddev1_low["main EWMA - 1 std dev"]{96.70, 96.92, 97.16, 97.71, 97.18, 93.97}
  curve stddev2_low["main EWMA - 2 std dev"]{93.40, 93.83, 94.31, 95.42, 94.35, 87.93}
  curve branch_0["#8158 (4 runs earlier)"]{99.71, 101.05, 100.54, 101.20, 98.47, 90.02}
  curve branch_1["#8158 (3 runs earlier)"]{100.59, 101.44, 100.38, 99.46, 99.77, 93.36}
  curve branch_2["#8158 (2 runs earlier)"]{99.81, 100.63, 99.73, 100.16, 100.45, 103.08}
  curve branch_3["#8158 (1 run earlier)"]{99.77, 99.90, 99.87, 100.32, 100.30, 104.71}
  curve branch_4["#8158"]{100.86, 100.36, 100.70, 97.08, 100.25, 105.20}
  graticule polygon
  max 121
  min 79
  ticks 0
  showLegend false

Adds pi_basic_blocking_locust, which measures the same blocking-write workload as pi_basic_blocking but drives it with locust rather than piccolo, so the number of concurrent clients can be varied. The load is defined in tests/infra/basicperf_locustfile.py and uses FastHttpUser, since HttpUser cannot drive enough requests per second to saturate the service. tests/basicperf_locust.py owns the network, runs locust against it, and converts locust statistics into bencher metrics (throughput, latency, memory). The key space helper shared with basicperf.py moves to tests/infra/key_space.py, since basicperf.py can only be imported from tests/infra. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

The three files added by this branch were committed with CRLF, unlike every other Python file under tests/. No functional change. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

Locust --run-time starts counting when locust starts, so it includes the ramp, and --reset-stats discards the statistics gathered during the ramp without extending the deadline. The measurement window was therefore shorter than requested, and shrank as the spawn rate was lowered, until it disappeared entirely. This mattered because varying the client count is the point of this test: at --users 128 --spawn-rate 4 the run ended mid-ramp and reported 127 tx/s instead of the ~1250 tx/s that 128 users actually sustain, and did so without failing. Start the shutdown timer from locust spawning_complete instead, so the window is the same length whatever the spawn rate is, and rename --run-time-s to --measure-time-s to describe what it now does. --run-time is kept as a backstop against a run which never finishes spawning. Also fail, rather than report, when a run ends without having spawned all users, or when the window measured is shorter than the one asked for. Both produce plausible looking figures that do not describe steady state. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

Locust workers reach the master on port 5557 by default, so a second locust run anywhere on the same machine fails to bind. Pick a free port per run instead, via the existing infra.net helper. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

Locust writes N/A rather than a number in the statistics CSV when it has too few samples to compute a percentile. float() then raised a bare ValueError, after the network had already been stopped, losing the run with no indication of what had gone wrong. Read the numeric columns through a helper which reports the column and value, and says that the run did not gather enough data. Found by a run which produced almost no samples. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

add_piccolo_test passes --snapshot-tx-interval 10000 for every piccolo perf test. add_e2e_test does not pass it at all, and e2e_args defaults it to 10, which is sensible for functional tests but not for a benchmark. This test was therefore writing and fsyncing a ~213KB snapshot every 10 transactions for the whole run, which measures the disk rather than the service, and makes the figure incomparable with Basic Blocking. Found by the vegeta comparison work, which hit the same defect. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

Blocking writes return once their transaction commits, and commit cannot outpace the signature interval, so a single interval only measures one regime. Sweeping three separates them: at 1s and 100ms the workload is latency-bound and throughput is simply the client count divided by the interval, while at 5ms the node becomes the limit and the benchmark measures capacity instead. Each interval gets its own network, since the interval is fixed in the node configuration at startup. consensus_update_timeout_ms moves with it, as in commit_latency.py, because commit cannot be observed faster than the primary sends updates. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

A benchmark added by a branch has no main runs to build an EWMA baseline from, so render_chart skipped it entirely. That made a new benchmark invisible on the very pull request which adds it, which is when it is most worth seeing. Plot such a benchmark against the branch's own earliest run instead, so its movement across the branch's runs is visible, and mark it as new. It carries no standard deviation band and is never coloured as an improvement or a regression, because there is nothing on main to compare it against. Borrowing a related benchmark's baseline was considered and rejected: an axis normalised against something which measures a different thing shows a difference which is not a change in CCF, and because the chart scale follows the largest axis, one such axis compresses every other benchmark into illegibility. Also truncate long axis labels in the middle rather than at the end. Benchmarks measured at several settings differ only in their suffix, so truncating the end left the 100ms and 1000ms axes indistinguishable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

The blocking workload is latency-bound at the longer signature intervals, so throughput there is set by the client count: 320 clients raises the 100ms point from ~1260 to ~3100 tx/s and the 1s point from ~130 to ~314, both within a few percent of clients divided by interval. Ten sending processes rather than four keeps locust from becoming the limit while driving that many clients, since each process drives all of its users from a single thread. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

The shortest interval is the point at which the node, rather than the signature timer, becomes the limit, so it is the one which measures capacity. Shortening it to 2ms pushes further past the latency-bound regime. The node ticks every 1ms in these tests, so a 2ms interval is representable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

The previous version reported a benchmark absent from main by its value alone, marked (new), with no percentage. That read differently from every other axis, and the marker made the label long enough to be truncated, which is exactly what it should not have been for benchmarks whose names differ only in a suffix. Treat such a benchmark like any other axis instead, using this branch's earliest run as its reference in place of the main EWMA baseline, so it is normalized, labelled and coloured identically. The chart description records that the reference differs. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Truncating the middle of the whole label cut out several words at once. Elide word by word from the left instead, keeping each word's first and last letter and replacing the middle with a single ellipsis character, so a label degrades gradually and the last word, which is what distinguishes one setting of a benchmark from another, stays readable longest. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

A benchmark with no main history had its standard deviation hardcoded to zero, so all four band curves collapsed to a single point at the baseline while every other axis carried a spread. That left a visible pinch in the band and, because nothing fell inside a zero-width noise threshold, any movement between branch runs was coloured as an improvement or a regression. Measure its spread the same way as for a benchmark with main history, across the runs available, which for such a benchmark are the branch's own. The radial zoom already covered these axes, since their values were always part of the data the scale is fitted to. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 2da7bd10-ceb0-41a4-b6c4-d574595d91c9

Amaury Chamayou (achamayou) added the bench-ab label Aug 14, 2026

Amaury Chamayou (achamayou) requested a balanced review from Copilot August 14, 2026 13:22

Copilot started reviewing on behalf of Amaury Chamayou (achamayou) August 14, 2026 13:23 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Comment thread tests/basicperf_locust.py Outdated

Eddy Ashton (eddyashton) mentioned this pull request Aug 14, 2026

Make blocking performance test measure throughput #8160

Open

Amaury Chamayou (achamayou) mentioned this pull request Aug 14, 2026

Isolate node and clients on separate CPUs for the locust perf test #8161

Draft

Amaury Chamayou (achamayou) and others added 7 commits August 14, 2026 17:45

Amaury Chamayou (achamayou) force-pushed the locust-blocking-perf branch from 9583136 to d4162fb Compare August 14, 2026 16:45

Amaury Chamayou (achamayou) and others added 3 commits August 14, 2026 17:56

Amaury Chamayou (achamayou) requested a balanced review from Copilot August 14, 2026 18:13

Copilot started reviewing on behalf of Amaury Chamayou (achamayou) August 14, 2026 18:14 View session

Copilot AI reviewed Aug 14, 2026

View reviewed changes

Amaury Chamayou (achamayou) and others added 2 commits August 14, 2026 19:36

Amaury Chamayou (achamayou) marked this pull request as ready for review August 14, 2026 20:31

Amaury Chamayou (achamayou) requested a review from a team as a code owner August 14, 2026 20:31

Navigation Menu

Uh oh!

Add `Basic Blocking Locust` perf test - #8158

Add `Basic Blocking Locust` perf test#8158
Amaury Chamayou (achamayou) wants to merge 13 commits into
mainfrom
locust-blocking-perf

Amaury Chamayou (achamayou) commented Aug 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

github-actions Bot commented Aug 14, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Amaury Chamayou (achamayou) commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why not piccolo for blocking workloads

What it runs

Results

Structure

Bencher output

Follow-up work

Testing

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Custom instructions used

Reviewed changes

Uh oh!

Uh oh!

github-actions Bot commented Aug 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Throughput (tx/s)

Latency (ms)

Memory (bytes)

Rate (ops/s)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Amaury Chamayou (achamayou) commented Aug 14, 2026 •

edited

Loading

github-actions Bot commented Aug 14, 2026 •

edited

Loading