Skip to content

Spec: Telemetry (Sentry errors + OpenTelemetry tracing)

What it does

iiiris optionally reports errors to Sentry and emits distributed traces via OpenTelemetry. Both are off by default and add no request-path overhead when disabled; the zero-config binary carries neither. A single package — internal/telemetry — is the sole consumer of both the getsentry/sentry-go and OpenTelemetry SDKs. Every other package (server, source, cache, image) is instrumented through plain-func hooks or opaque wrappers and never imports either SDK, the same "one consumer per vendor SDK" rule that governs govips (internal/image) and prometheus/client_golang (internal/metrics).

Error tracking captures recovered panics, 5xx responses, and internal source/decode/cache failures, with a privacy scrubber active by default. Tracing produces a root span per request plus child spans across source open and cache get/put, exported over OTLP; when both subsystems are on, captured errors are stamped with the active trace ID.

Surface

Configuration

sentry:                     # error tracking (opt-in)
  enabled: false            # master switch; an empty dsn also disables
  dsn: ""                   # Sentry ingestion endpoint; empty = disabled
  environment: production   # tags events + the trace resource
  release: ""               # empty derives from the build version
  sample_rate: 1.0          # fraction of error events sent (0.0–1.0)
  send_default_pii: false   # false = privacy scrubber active
  capture:                  # error sources (all default true)
    panics: true
    server_errors: true     # 5xx responses
    internal_errors: true   # source / decode / cache errors

tracing:                    # distributed tracing (opt-in)
  enabled: false            # master switch
  otlp_endpoint: ""         # OTLP/gRPC collector; empty = sample but don't export
  insecure: false           # true = plain HTTP/2 (no TLS) to the collector
  sample_rate: 0.0          # head-sampling probability for root spans (0.0–1.0)
  propagate_upstream: false # inject W3C traceparent on outbound HTTP fetches

Env overrides (flat, per the env-var policy): IIIRIS_SENTRY_ENABLED, IIIRIS_SENTRY_DSN, IIIRIS_SENTRY_ENVIRONMENT, IIIRIS_SENTRY_RELEASE, IIIRIS_SENTRY_SAMPLE_RATE, IIIRIS_SENTRY_SEND_DEFAULT_PII, IIIRIS_SENTRY_CAPTURE_PANICS, IIIRIS_SENTRY_CAPTURE_SERVER_ERRORS, IIIRIS_SENTRY_CAPTURE_INTERNAL_ERRORS, IIIRIS_TRACING_ENABLED, IIIRIS_TRACING_OTLP_ENDPOINT, IIIRIS_TRACING_INSECURE, IIIRIS_TRACING_SAMPLE_RATE, IIIRIS_TRACING_PROPAGATE_UPSTREAM. Full field reference: docs/configuration.md.

Go surface

  • internal/telemetry: New(Options, version) (*Telemetry, error), (*Telemetry).Shutdown(timeout). Methods with a *Telemetry receiver are nil-safe no-ops; a build with an empty DSN and tracing off is a fully no-op value.
  • Error hooks (plain funcs, no Sentry types): CapturePanic(ctx, rec), CaptureError(ctx, err, tags), and the server-facing serverErrorHook closure passed as server.Deps.CaptureServerError.
  • Tracing surface: TraceMiddleware() func(http.Handler) http.Handler (an opaque middleware the server applies outermost), the Traced{Source,RenderCache,InfoCache,OriginCache} decorators (parallel to the metrics Observed* wrappers), and WrapHTTPTransport(base) http.RoundTripper.
  • server.Deps fields (all optional, plain funcs / opaque values): CapturePanic, CaptureServerError, TraceMiddleware. The server installs a per-request error holder and the root-span middleware only when the respective hook is non-nil.
  • cmd/iiirisd builds the subsystem, wires the hooks and wrappers per the capture toggles, and flushes on shutdown. Nothing is active when both subsystems are disabled.

Error events

  • Panics — the server recover middleware calls CapturePanic with the request context before writing the 500; the log line is retained.
  • 5xx / internal errors — after any 5xx the countRequests middleware calls CaptureServerError(ctx, route, status, cause). cause is the error a handler recorded at its 500 site via the unexported setError(ctx, err) (source resolve/open, overlay load, pipeline execute, hook redirect), else nil. The hook reports the cause when capture.internal_errors is on, else a synthetic server error: HTTP <status> when capture.server_errors is on — one event per request, tagged route + status.

Spans

Emitted by the tracer named git.iiiris.org/iiiris. Names and attribute keys are the contract:

Span Kind Key attributes
iiiris.request server (root) otelhttp HTTP attributes
source.open client iiiris.source, iiiris.source.size or iiiris.source.found=false
cache.get internal iiiris.cache, iiiris.cache.hit
cache.put internal iiiris.cache

Resource attributes: service.name=iiiris, service.version=<build version>, deployment.environment=<sentry.environment> (schema https://opentelemetry.io/schemas/1.30.0).

Contracts

  • Off by default. sentry.enabled and tracing.enabled are false in Default(); an empty sentry.dsn disables error tracking even if enabled is true. The zero-config path installs no hooks, no error holder, and no root-span middleware.
  • One consumer per vendor SDK. getsentry/sentry-go and the OpenTelemetry SDK are imported only by internal/telemetry. The server, source, cache, and image packages take plain-func hooks or opaque wrappers and never reference either SDK.
  • Nil-safe. Every *Telemetry method is a no-op on a nil or disabled receiver.
  • Privacy: PII off by default. With send_default_pii: false a BeforeSend scrubber strips the client IP (user + request env), cookies, Authorization / auth / forwarded-for headers, and URL query strings from every event, keeping path, method, status, release, environment, and stack traces. send_default_pii: true sends Sentry's standard payload unchanged.
  • One error event per request. The internal_errors cause wins over the server_errors synthetic; both cannot fire for the same request.
  • A configured-but-broken subsystem is fatal at startup. A non-empty DSN that fails to init, or a tracer-provider build failure, aborts startup rather than silently dropping telemetry.
  • Sampling. Error sample_rate defaults to 1.0; trace sample_rate defaults to 0.0 (enabling tracing without a rate is inert). The trace sampler is ParentBased(TraceIDRatioBased(rate)), so an inbound sampled traceparent is honoured.
  • Trace export is OTLP/gRPC only, to otlp_endpoint when set (BatchSpanProcessor). An empty endpoint samples but ships nothing. Sentry receives traces by pointing otlp_endpoint at Sentry's OTLP ingestion URL; there is no in-SDK span→transaction bridge (see Out of scope). sentryotel.NewOtelIntegration() links captured errors to the active trace, not spans to Sentry.
  • Propagation is opt-in and HTTP-only. propagate_upstream injects W3C traceparent on outbound HTTP source fetches (via otelhttp.NewTransport wrapping the source client's RoundTripper). Off by default so trace IDs are not leaked to third-party origins. S3 is not propagated.
  • Global OTel state. When tracing is enabled the process sets the global TracerProvider and a composite W3C TraceContext + Baggage propagator; otelhttp and the wrappers read those globals.
  • Bounded shutdown. Buffered events/spans are flushed on graceful shutdown, capped at 3s (independent of the server shutdown budget) so an unreachable Sentry/OTLP endpoint cannot dominate process exit. Runs in both console and Windows service modes.
  • Span attributes carry no raw identifiers or keys. The request identifier already rides the root span's HTTP attributes; child spans record backend name and outcome only.

Out of scope

  • The sentryotel span processor — removed in sentry-go v0.47. Traces reach Sentry via its native OTLP ingestion; the Go SDK provides only the linking integration used here. (Historical: the design brief originally named the span processor as the bridge.)
  • S3 upstream propagation — AWS is a terminal service that does not continue the trace; source.open already captures S3 fetch latency. Adding otelaws would cost a module for no continued-trace benefit.
  • OpenTelemetry metrics and logs — the Prometheus /metrics endpoint (internal/metrics) remains the metrics surface; slog remains logging.
  • slog→Sentry bridge — logger.Error calls are not auto-forwarded; capture is explicit at the chosen sites to avoid duplicate/noisy events.
  • Fine-grained pipeline spans — decode-stage spans are not emitted; render timing is covered by source.open/root-span timing and the Prometheus pipeline histograms. Add on demonstrated need.
  • Live reconfiguration — DSN, endpoint, and enable/disable are fixed at startup; the admin config editor does not hot-swap the SDK clients.