Skip to content

Comparative benchmark harness

The contract for the iiiris-vs-Cantaloupe benchmark: a deliberate credibility-and-tuning instrument (not a CI gate) that drives identical IIIF workloads against both servers under identical resource limits and reports comparative throughput, latency, CPU, and memory. Operator how-to is bench/README.md; per-run results live in performance_results.md.

What it does

tools/bench-compare (a Go orchestrator) brings up one server at a time under a fixed CPU/memory envelope, drives load with vegeta as a library, samples per-container CPU/RSS over the load window, and emits a Markdown + JSON report. The bench/ tree holds the per-server Docker configs, the Cantaloupe image, the source corpus, an AWS Terraform host, and run.sh.

Surface

  • Driver: tools/bench-compare (flags below); wrapped by bench/run.sh (env: BENCH_CPUS, BENCH_MEM, BENCH_SERVERS, IIIRIS_IMAGE, CANTALOUPE_IMAGE).
  • Flags (selected): --cpus / --memory (identical per container), --scenario {info|full-scaled|tiled}, --format (csv allowlist, e.g. jpeg,jp2,large), --cache {off|on}, --servers (csv; e.g. iiiris to skip Cantaloupe), --smoke, --host (report label), --iiiris-image / --cantaloupe-image.
  • bench/ tree: iiiris/config-cache{off,on}.yaml, cantaloupe/Dockerfile + cantaloupe-cache{off,on}.properties, corpus/manifest.json (fetched by URL + SHA-256, gitignored), terraform/ (AWS host), run.sh, README.md.
  • Output: bench/results/bench-<timestamp>.{md,json} (gitignored ephemera). A header block records server versions/digests, instance type, resource limits, matched-output settings, JP2 codec, and date.

Scenarios, formats, postures

  • Scenarios: info (GET info.json — routing + concurrency ceiling), full-scaled (full/!1024,1024/... — full-region decode + downscale + encode), tiled (native-resolution deep-zoom tiles — the dominant viewer workload).
  • Formats: jpeg, jp2 (classic Part-1, OpenJPEG both sides), htj2k (Grok on the Cantaloupe side — classic OpenJPEG cannot read HTJ2K), and large (pyramidal tiled TIFF). All derive from one real public-domain master (corpus manifest.json).
  • Postures: each scenario runs caches off (isolates the raw decode→transform→encode engine) and caches on (steady state), reported separately, never averaged.
  • Load profiles: saturation (ramp to max sustainable req/s) and fixed-rate (latency percentiles + CPU/RSS at equal work).

Contracts (fairness — load-bearing)

  • Equal hardware envelope. Identical --cpus / --memory on both containers.
  • Sequential, not side-by-side. One server runs, is torn down, then the other — they never contend for the host.
  • Pinned + recorded. Cantaloupe image digest and the iiiris build/commit are recorded in every report; the corpus master is pinned by SHA-256.
  • Matched output. Both emit JPEG q90 baseline, 1024 px tiles, identical target sizes — equal-quality output, exact values in the report header.
  • Warmup discarded; only ~100 %-success rows count. A warmup pass is discarded; quote only runs whose succ column is ~100 %.
  • Not a CI gate, not a make target. Run deliberately (local for tuning, AWS c7i.2xlarge non-burstable for citable numbers) via run.sh, kept out of the curated Makefile.

Test coverage

No unit suite — it's a measurement tool, validated by its runs. Each run is recorded with its commit + environment in performance_results.md; only ~100 %-success rows are quoted, and --smoke gives a fast end-to-end sanity pass.

Out of scope

  • CI regression gating (a pass/fail perf threshold) — shared runners are too noisy; a future wrapper could add it.
  • Non-Cantaloupe servers (IIPImage, Loris, …) — Cantaloupe is the reference point.
  • Pixel-fidelity / SSIM comparison — matched settings keep quality comparable; proving pixel-equivalence is separate.
  • Continuous historical dashboards — the JSON enables it; performance_results.md is the curated record for now.
  • k6 / non-Go load generators — vegeta-as-a-library is the design.