Skip to content

Performance results

The append-only record of comparative benchmark runs — what was measured, on which commit, in which environment. The numbers behind the scorecard in CANTALOUPE_PARITY.md live here; the harness contract is in specs/bench-harness.md. Per-run JSON/Markdown reports under bench/results/ are gitignored ephemera — this doc is the durable, curated record.

When you run the bench, add an entry (see Recording a run at the bottom). Don't rewrite past entries — they're a historical record of what that commit did on that hardware.

Environment (shared across the runs below unless noted)
Host AWS c7i.2xlarge — 8 vCPU (Intel Sapphire Rapids), 16 GiB, us-west-2
OS Ubuntu 24.04
Per-container limits --cpus 8 --memory 8g, identical for both servers
iiiris built from source per commit (distroless image; libvips static, Go 1.25). R1-R6 ran libvips 8.16; R7 onward 8.18.2 — see R7 for why the bump matters.
Cantaloupe 5.0.7 on Java 17 (Temurin); JP2 via Grok
Corpus master Van Gogh Irises, 9021×7122, Getty Open Content (sha256 e8db6a68…)
— large.tif vips tiffsave --tile --pyramid --compression jpeg --Q 90 → pyramidal tiled TIFF, 128 px internal tiles, ~104 MB
— master.jp2 OpenJPEG -n 6 -t 1024,1024 -p RPCL -r 20
Harness tools/bench-compare, --smoke --format jpeg,jp2,large
Metric caches-off saturation req/s (offered rate calibrated to ~100 % success); also cold (first-request) latency and peak RSS

Cantaloupe is a fixed reference (its numbers don't move with iiiris commits), so later runs are often iiiris-only (--servers iiiris) and compared against the Cantaloupe column recorded on 2026-06-07.

Results timeline — caches-off saturation req/s (iiiris)

Higher is better. Cantaloupe column is the 2026-06-07 measurement.

Cell Cantaloupe R1 baseline R2 mmap R3 +info-dims R4 +concurrency R5 +baseline-JPEG
info jpeg 82 71 512 512 512 512
info jp2 512 174 512 512 512 512
info large 512 18 512 512 512 512
full-scaled jpeg 2 9 10 10 10 15
full-scaled jp2 17 48 63 63 63 82
full-scaled large 23 17 238 238 238 176†
tiled jpeg 10 8 9 9 9 10
tiled jp2 41 35 40 39 53 85
tiled large 81 15 62 61 78 → 95 175

R4/R5 are max_concurrent: 16 (R1–R3 are 4). R5 also flips JPEG output from progressive to baseline, which lifts every JPEG-output render cell (the encode is ~3× cheaper). †full-scaled large dips because mc=16 over-subscribes that very fast op (cold 15 ms) — it's a tile-large tuning value; the cell still wins 176 vs 23. R3's win is in cost not throughput (info was already capped): see its note.

Runs

R1 — baseline (info-probe OOM fix)

  • Commit 7c663c6 · Date 2026-06-07 · Config max_concurrent: 4, both servers
  • What it isolates the OOM fix alone — the clean reference. The full caches-off matrix ran with 0 OOM / 0 restarts (prior runs crashed 3×).
  • Notable RSS info large 991 MiB, tiled large 924 MiB (whole-source buffering).
  • Finding confirms the real losses are genuine, not crash artifacts: info large 18, tiled large 15, info jp2 174.

R2 — mmap source I/O

  • Commit 0c62679 · Date 2026-06-07 · Config max_concurrent: 4, both servers
  • What it isolates mmap'ing filesystem sources (resident memory now independent of source size).
  • Finding three ❌ cells flip to the 512 cap (info large/jpeg/jp2); full-scaled large 17 → 238; tiled large 15 → 62. RSS 10–150 MiB vs Cantaloupe 0.3–2.2 GiB everywhere. tiled large 62 vs 81 — iiiris wins cold-latency (60 vs 65 ms), throughput still capped by max_concurrent: 4.

R3 — header-only info dimensions + heap O(1) LRU

  • Commit ace9886 · Date 2026-06-07 · Config max_concurrent: 4, iiiris-only
  • What it isolates reading info.json dimensions from the header (no libvips); heap cache O(1) LRU.
  • Finding info throughput was already capped, so the win is cost: info jp2 CPU 13 → 3 %, RSS 150 → 10 MiB, cold 2.7 → 1.3 ms (OpenJPEG header parse gone). info is now ~3 % CPU / 10 MiB / sub-ms for every format. Heap O(1) LRU not visible at the 512 cap / small smoke cache.

R4 — concurrency sized to the cores

  • Commit c248b34 (run) + a max_concurrent: 16 probe · Date 2026-06-07 · iiiris-only
  • What it isolates max_concurrent on the I/O-bound (EBS) tiled large decode. Product default is now NumCPU; bench config tested at 8 then 16.
  • Finding the EBS corpus makes tile reads I/O-bound, so cores idle during reads until concurrency exceeds the core count:
max_concurrent tiled large req/s CPU
4 61 ~2 cores
8 78 ~7 cores (≈ Cantaloupe 81 — tie)
16 95 ~7.5 cores (beats 81)

Bench configs set to 16. iiiris is less CPU-efficient per tile than Cantaloupe (~13 vs ~22 req/s/core) but out-throughputs it by using more of the same 8 cores, at ~12× less memory. Scorecard: 0 losing cells.

R5 — baseline JPEG (matched-output)

  • Commit 4ed49d5 · Date 2026-06-07 · Config max_concurrent: 16, iiiris-only
  • What it isolates JPEG output flipped progressive → baseline (the govips default was progressive — multiple entropy-coding passes, ~3× slower encode). Both servers now emit baseline q90, so this is a true matched-output comparison.
  • Finding the encoder was the per-tile CPU gap (parity item 2):
  • tiled large 95 → 175 req/s; per-core 13 → 25 (beats Cantaloupe's 22 — iiiris now more CPU-efficient per tile); cold 60 → 26 ms (vs 65); RSS 88–156 MiB vs 1717.
  • every JPEG-output render cell rose: full-scaled jpeg 10 → 15 (cold 553 → 363 ms), full-scaled jp2 63 → 82, tiled jp2 53 → 85.
  • full-scaled large 238 → 176 — not a regression in capability; mc=16 over-subscribes that very fast op (it's tuned for tiled large). Still 7.6× Cantaloupe's 23.
  • 0 OOM. iiiris vs Cantaloupe on tiled large: 2.2× throughput, more CPU-efficient per tile, 2.5× faster cold, ~15× less memory.

R6 — pre-1.0 parity re-validation ⚠️ INVALID for jp2/large full-scaled

Retained, not deleted — past runs are never edited, and a wrong run that is explained is more useful than one that vanished. But do not cite its full-scaled jp2 or large figures.

R6 measured iiiris returning undersized images. A bug in the shrink-on-load path (fixed in !110) made pyramidal TIFF and JP2 sources answer derived-size requests with a small pyramid level: full/!1024,1024 on the 9021×7122 master returned 70×55 for large.tif and 282×223 for master.jp2, HTTP 200, valid JPEG. iiiris was shipping ~5 KB where Cantaloupe shipped ~590 KB, so its "wins" in those cells measured it doing far less work.

Invalid: full-scaled jp2 (4.0×) and large (8.0×), both cache postures. Still valid: every tiled cell (explicit region/w,h — the bug never touched that path), every info cell, and everything on plain.jpg.

Superseded by R7. The irony worth recording: the on full-scaled jpeg latency that looked anomalous in R6 was the only correct cell in that group — jp2 and large looked faster because they were serving thumbnails.

R6 (original text) — pre-1.0 parity re-validation

  • Commit 5c25bd7 · Date 2026-08-23 · Host c7i.2xlarge, 8 CPUs / 8g per server · Config both servers, caches off + on, formats jpeg/jp2/large
  • Why the 1.0 checklist holds Cantaloupe parity as a standing release gate, not a one-time check. The previous evidence was R1–R5 (2026-06-07), which predates downscale sharpening, supersampling, and the colour-policy work in the render path. This run re-measures both servers in one pass, so the verdict doesn't depend on comparing against Cantaloupe numbers recorded eleven weeks and several builds earlier.
  • Verdict — gate holds: 6 wins, 12 ties, 0 losses on saturation throughput.
caches scenario format iiiris cantaloupe ratio
off full-scaled jpeg 15 2 7.8×
off full-scaled jp2 64 16 4.0×
off full-scaled large 128 16 8.0×
off info jpeg 1024 64 16.0×
off tiled jp2 64 32 2.0×
off tiled large 128 64 2.0×
  • Read the ties carefully. 11 of the 12 are both servers pinned at the ramp's 1024 req/s ceiling — the harness runs out of headroom, so the tie means "both exceed what this ramp measures", not "equal". Only tiled/jpeg/caches-off (8 req/s each) is a tie the benchmark actually resolved. With caches on, every cell caps for both servers.
  • Latency breaks the capped ties, and mostly for iiiris — 10 of 12 on p50, by up to 5× (on info *: 0.32 ms vs 0.90 ms). Two go the other way and are worth a look:
  • off info jp2 — 1.30 ms vs 0.94 ms.
  • on full-scaled jpeg — 4.63 ms vs 2.00 ms. Both cap on throughput, but this is a cached full-scaled render, so it should be a cache-hit serve rather than a render. 2.3× on that path is the one number in this run worth chasing. Not a gate failure; throughput tied.
  • Memory, unchanged in character. iiiris 35–199 MiB against Cantaloupe 0.76–2.4 GiB across the matrix — the same order-of-magnitude gap R2 recorded.
  • Not measured: htj2k. The corpus manifest records that its compression isn't yet tuned for size parity with jp2, so the cell wouldn't be fair; the bench host also lacks grk_compress.
  • Corpus note: the master was re-pinned on 2026-08-23 (Getty re-encoded it upstream — same artwork, same 9021×7122, new bytes; see bench/corpus/manifest.json). Comparisons with R1–R5 are therefore in-kind rather than byte-identical. The within-run comparison this verdict rests on is unaffected: both servers read the same file.

R7 — parity on correct output (pyramid fix + libvips 8.18.2)

  • Commit 5d8b9e8 · Date 2026-08-24 · Host c7i.2xlarge, 8 CPUs / 8g per server · Config both servers, caches off + on, formats jpeg/jp2/large · libvips 8.18.2
  • Why R6's pyramid cells were measuring undersized output (see above). This re-measures on correct renders, both servers in one pass.
  • Verdict — gate holds: 5 wins, 13 ties, 0 losses.
caches scenario format iiiris cantaloupe ratio
off full-scaled jpeg 14 2 7.4×
off full-scaled large 57 16 3.6×
off info jpeg 1024 64 16.0×
off tiled jp2 64 32 2.0×
off tiled large 118 64 1.9×
off full-scaled jp2 16 16 1.0× (tie)
  • full-scaled large is 3.6×, not R6's 8.0×. The honest number is lower, because R6's was measuring a 5 KB thumbnail.
  • The bug fix alone was not enough. An intermediate run on the same commit with libvips 8.16 (the version both release images shipped) scored 4 wins / 12 ties / 2 LOSSES: full-scaled jp2 at 0.06× and large at 0.20×. Correct output, but the fix's fallback gives up shrink-on-load, so those cells full-decode a 64 MP master per request. Bumping the distroless image to libvips 8.18.2 (!111) restores the fast path and turns both around. That intermediate run is the before for the bump and is not recorded separately; its numbers are quoted here.
  • Unaffected cells are stable across both runs — full-scaled jpeg 7.48 → 7.35, info jpeg 16.02 → 16.04, tiled jp2 2.00 → 2.00 — which is what makes the two moved cells attributable to libvips rather than run-to-run noise.
  • Ties are still mostly ceiling. 12 of 13 are both servers pinned at the 1024 req/s ramp ceiling; only tiled/jpeg/caches-off (8 req/s each) and full-scaled/jp2/caches-off (16 each) are ties the ramp actually resolved.
  • Not measured: htj2k (compression not tuned for size parity with jp2; host lacks grk_compress).

Recording a run

After a comparative run, append a new ### Rn — <label> section with:

  1. Commit (git rev-parse --short HEAD of the build under test) and date.
  2. Config / environment deltas from the table above (instance type, max_concurrent, caches, which servers, any corpus change).
  3. Findings — the cells that moved and why, in 2–4 lines.
  4. Add a column (or update the relevant cell) in the Results timeline if the run changes a headline number.

Keep the raw bench/results/*.json only if a number is surprising and worth re-deriving; otherwise the curated entry here is the record. Never edit a past run's numbers — add a new run instead.