Skip to content

RED, USE and cardinality

Two methods for two kinds of thing

RED (rate, errors, duration) describes work: requests, queries, operations. USE (utilisation, saturation, errors) describes resources: a connection pool, a queue, a cache. instrument provides ready-made sets for both, because a raw metric.Meter leaves every service to invent its own names, units and attributes.

Pull and push

go/transport-metrics serves these same helpers to Prometheus, which scrapes them. instrument records them on an OpenTelemetry meter, and the SDK pushes them over OTLP. A service usually wants one or the other. The helpers share their method names so that moving between them means changing the constructors and little else.

The two differ where each ecosystem has an opinion, and instrument follows OpenTelemetry on those:

  • Metric names are dotted, and use semantic conventions where they exist.
  • Errors are an attribute. OpenTelemetry's conventions put error.type on the duration histogram and have no error counter, so the histogram's count gives both rate and errors.
  • Methods that record an event take a context. A measurement carries an exemplar only if a context reaches it, so a counter without one could never link to a trace.

The metric name map lists every difference.

What is not here

OpenTelemetry already covers the rest of what transport-metrics builds, so instrument leaves it alone:

  • HTTP and gRPC request metrics come from otelhttp and otelgrpc.
  • Exemplars come from the SDK, with no configuration, when a sampled span is active.
  • The version comes from the Resource, so there is no build_info gauge.
  • Runtime metrics are one line, from go.opentelemetry.io/contrib/instrumentation/runtime:
err := runtime.Start(runtime.WithMeterProvider(mp))

It is not wrapped here because that package links the SDK, and instrument must stay importable from a library without doing so.

Keeping attribute values bounded

Each distinct attribute value is a new series, and series are what a metrics backend charges for in memory and money. Names passed to a helper are the caller's responsibility: use a small fixed set, never a per-call value such as an ID or a raw SQL string.

The SDK provides the backstop. Past a limit per instrument (2,000 by default), it folds further values into one series marked otel.metric.overflow=true. The limit protects the process, and it does not protect the data, because a noisy attribute can crowd out the rest of its instrument. The application that builds the MeterProvider sets it, with WithCardinalityLimit.

Errors are the one place the helper decides for you. error.type is a go/errors kind, which is a constant in code, or _OTHER. An error's message never becomes an attribute, because messages are unbounded by construction.

Metric names are a contract

Renaming a metric breaks dashboards and alerts even when no Go code changes, so emitted names change only deliberately, with a changelog note. Semantic-convention names come from a package whose import path carries the convention's version, so a name can only move when someone edits that import.