Same-run comparison with pkl-zig

bench/compare.sh builds zig-pkl with Zig 0.17.0 and pkl-zig 6a113ab with Zig 0.16.0, runs both against the same Pkl executable, and applies the release gates in baseline.md. pkl-zig is used only through its public API.

ZIG_0_16=/path/to/zig-0.16.0/zig PKL_EXEC=/path/to/pkl bench/compare.sh 21

Method

• Both programs, bench/zig-pkl.zig and bench/pkl-zig/src/main.zig, run the same cases with the same iteration counts, backing allocator (std.heap.smp_allocator), timer, and output format.
• Codec cases decode the pkl-binary fixtures in bench/fixtures 2000 times after 200 unmeasured decodes. decode_config and decode_mixed build each library's dynamic value tree; typed_config decodes into the same Zig structs and frees them.
• Allocations count calls that reach the backing allocator, so each arena chunk counts once.
• The cold case starts pkl server, creates an evaluator, evaluates (1 + 2) * 14, and shuts down, 10 times per round after one unmeasured cycle. Each round reports the median cycle and the median of each phase. The warm case evaluates the same expression 200 times on a running evaluator after 20 unmeasured evaluations.
• Each library runs once unrecorded, then the rounds alternate which library goes first. Reported figures are medians over the rounds.
• The fixtures are evaluated through modulepath: URIs, so their bytes do not depend on the checkout path. bench/fixtures/*.pkl regenerate them with pkl eval --module-path bench/fixtures -f pkl-binary -o bench/fixtures/<name>.pklbin modulepath:/<name>.pkl.

CI runs the comparison on Linux x86_64 in the bench job, writes the summary to the job page, and uploads the raw results.

Results on macOS arm64

Apple M5, Pkl 0.32.2 native, 31 rounds. Raw data: results/macos-arm64.

Codec throughput on a laptop varies by about 20% between runs; the ratios are measured in the same run.

Results on Linux x86_64

GitHub-hosted ubuntu-24.04 runner with 2 vCPUs, Pkl 0.32.2 native, 15 rounds, from the CI bench job. Each CI run publishes its own summary and raw data.

Cold start

The cold case is bounded by work inside Pkl that both libraries pay: starting the server and, after stdin closes, the server's own exit, which takes about 2.1 ms on the Linux runner. The remaining shutdown difference on Linux is the thread that keeps the tail of the server's stderr for error reports; pkl-zig does not capture stderr.

Measured choices, on the Linux runner unless noted: