:rocket: New in Pulsar master: a Docker-based perf...
# dev
l
πŸš€ New in Pulsar master: a Docker-based performance testing framework, with profiling, metrics and AI agent support Docs at tests/performance PR #26714 has been merged: github.com/apache/pulsar/pull/26714 It adds a practical way to measure a change's effect on a realistic workload during development, with a quick feedback loop: a run takes minutes on your own machine, with no separate benchmark environment. What you get β€’ Realistic scenarios on a Pulsar cluster in Docker containers, currently IoT telemetry, with producers and consumers that check every message's delivery, ordering and duplicates. β€’ A report for every run: settings, provenance, throughput, backlog, and latency charts rendered from HDR histograms. β€’ Profiling with jonoffcpu (github.com/jonoffcpu/jonoffcpu): β—¦ on-CPU time with async-profiler, and JFR events such as GC; β—¦ off-CPU time with an eBPF program in the Linux kernel, ranked by the method that waited, so lock and queue contention stands out; β—¦ flame graphs for both. β€’ Metrics out of the box: VictoriaMetrics and Grafana with Pulsar's dashboards collect broker, bookie and ZooKeeper metrics. They mark each run's timeline as annotations, and put panel images into the report. β€’ Host setup scripts that make results comparable across runs: fixed CPU frequency, and no power-management daemons during runs. AI-assisted performance work πŸ€– Finding and fixing a bottleneck is a loop: run, profile, form a hypothesis, change something, and compare over repeated runs. AI agents make progress only as fast as that loop turns. Each idea needs a measured answer before the next one. When a measurement takes minutes on a developer's machine instead of hours on shared infrastructure, an agent can try a change, check it against its baseline and move on to the next one within a single session. The run reports, the JFR recordings, the off-CPU profiles and the metrics give it the evidence to tell a real improvement from noise. Point your agent at tests/performance/AGENTS.md, hand it an experiment, and review what it found. Get started: github.com/apache/pulsar/…/performance
Copy code
./gradlew :tests:performance:launcher:run --args='--scenario tests/performance/scenarios/iot-telemetry.yaml'
Linux x86_64 is recommended for measurements. It also works on MacOS (tested with OrbStack). Feedback and improvements are very welcome! Reporting examples: β€’ Test report β€’ jonoff summary report for application's blocking callsites β€’ Off-cpu flamegraph example for application's blocking callsites β€’ *Grafana*: Integrated and pre-configured to collect metrics during the test runs β€’ VictoriaMetrics, Metrics backend for Grafana, exploring metrics example β€’ VictoriaMetrics, *PromQL* example
πŸ‘ 2
πŸ‘ 2
I'll improve AGENTS.md to actually make it useful. PR 26714 was the first one and we can evolve it from here. I'll push more improvements soon.
Prompt:
Copy code
performance test iot-telemetry-max-rate.yaml with dispatcherMaxReadBatchSize set to 500 and 1000. compare it to the baseline (value 100). Don't do profiling.
Response:
Copy code
The comparison is running in the background, so there are no results yet. It is 9 runs with no profiling, and the first baseline run started at 18:26. I'll check progress at about 18:52, and I'm
  notified as each run starts and finishes.

  How it's set up:
  - Rounds: 3 rounds of iot-telemetry-max-rate.yaml with dispatcherMaxReadBatchSize set to 100, 500 and 1000. The order rotates each round (100β†’500β†’1000, then 1000β†’100β†’500, then 500β†’1000β†’100) so
    that slow drift in the host doesn't favour one value.
  - Setting: each run sets the value with --set cluster.brokers.env.dispatcherMaxReadBatchSize=<n>. The baseline sets 100 explicitly, which is also the default in conf/broker.conf.
  - Build: all runs use one image (-Pdocker.tag=batchsize), since only a setting changes and the code doesn't.
  - Cool-down: each run waits for the CPU to cool to 50 Β°C before it starts.
  - Guards: the script stops if thermald becomes active or the disk has less than 15 GB free.
  - Where things are: runs are grouped under the names dispatcher-read-batch-{100,500,1000} in the shared reports root. The script is run-batchsize.sh and its log is runs.log, both in the
    scratchpad.

  Host: the host check first failed 7 of 11 checks: thermald was running, turbo was on, the governor was powersave, and swappiness and huge-page settings were off. I applied the performance-testing
  profile with the passwordless configure-perf-test-environment.sh start rule, and all 11 checks then passed. I'll run stop when the runs are done to restore your normal settings. It also wrote
  inttest.asyncprofiler.skipPerfEventTuning=true to ~/.gradle/gradle.properties, which stop removes again.

  When the runs finish, I'll check each report for correctness: every message delivered, no ordering violations, no throttling. Then I'll give you a table with the median and min–max of throughput,
  publish latency and end-to-end p50/p99 for 500 and 1000 against 100.
github.com/apache/pulsar/blob/…/AGENTS.md is now useful for agents and also humans
Copy code
All 9 runs completed without profiling, 3 per setting in rotating order on master f925a25884e6. Every run was valid: all 5,000,000 messages
  delivered, with no duplicates, no ordering violations and no thermal throttling. I checked in each run's resolved config that the setting took
  effect.

  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
  β”‚      Median (min–max)       β”‚     100 (baseline)     β”‚              500               β”‚              1000              β”‚
  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
  β”‚ Delivered throughput, msg/s β”‚ 90,547 (89,280–91,956) β”‚ 92,928 (89,738–93,585), +2.6 % β”‚ 92,567 (92,432–94,663), +2.2 % β”‚
  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
  β”‚ Publish p50                 β”‚               1,062 ms β”‚              1,037 ms (βˆ’2.4 %) β”‚              1,033 ms (βˆ’2.7 %) β”‚
  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
  β”‚ End-to-end p50              β”‚ 1,355 ms (1,319–1,390) β”‚              1,348 ms (βˆ’0.5 %) β”‚              1,336 ms (βˆ’1.4 %) β”‚
  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
  β”‚ End-to-end p99              β”‚ 1,921 ms (1,901–1,951) β”‚                 1,921 ms (0 %) β”‚              1,878 ms (βˆ’2.2 %) β”‚
  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
  β”‚ End-to-end p99.9            β”‚               2,023 ms β”‚              2,069 ms (+2.2 %) β”‚                 2,022 ms (0 %) β”‚
  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”΄β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

  - Throughput: all three runs at 1000 were faster than every baseline run. At 500, one run fell inside the baseline's range. However, the baseline
    runs got slower over the session (92.0k, then 90.5k, then 89.3k), so part of the gap may come from timing rather than the setting.
  - Latency: the differences are within the spread between runs. At maximum rate, latency mostly reflects the 100,000 messages each producer keeps
    in flight, so it follows throughput rather than measuring the dispatcher.
  - Next step: if you want a firmer answer, we could do more repetitions and a profiled broker comparison. That would show whether larger reads
    actually lower the dispatcher's read count or CPU per message on this workload.
that was the result of the prompt. There's also the detailed reports and Grafana dashboard snapshots etc. available.