Skip to content

Instrument a queue

Create a Queue, then record items going in and coming out:

queue, err := instrument.NewQueue(mp.Meter("checkout"), "emails")
if err != nil {
    return err
}

queue.Enqueue(ctx)
queue.SetDepth(backend.Len())

// on the consumer
queue.Dequeue(ctx)
queue.SetDepth(backend.Len())
queue.ObserveWait(ctx, time.Since(item.EnqueuedAt))

start := time.Now()
err = process(ctx, item)
queue.ObserveProcessing(ctx, time.Since(start))

Pass the item's context so that a sample inside a sampled span carries an exemplar.

Zero the depth when you drop a queue

Queue has no Close, because it registers no callback, so nothing stops its depth reporting. Call SetDepth(0) when you drop a queue, or queue.depth keeps reading the last value for the life of the provider and a dashboard shows a backlog that no longer exists.

Set the depth from the queue

Pass SetDepth the real length rather than counting enqueues against dequeues yourself. Each call adds the difference from the last value to an up/down counter, so the metric reads the last value set, and concurrent callers cannot drift it.

Change the buckets for slow queues

The default buckets top out at 10 seconds. If items wait for minutes, set your own:

queue, err := instrument.NewQueue(meter, "emails", instrument.WithBuckets(1, 5, 30, 60, 300, 1800))

WithBuckets applies to both durations.

Use one name per queue

The name becomes the value of queue.name, and every distinct value is a new series, so keep names to a small fixed set, never a per-call value. RED, USE and cardinality says what happens otherwise.

Give each queue its own name per meter, because every metric carries that name as queue.name. Two queues sharing a name add together, and their depths sum.

Read the signals

Signal Query
Backlog queue.depth
Are we keeping up? the rate of queue.enqueued against the rate of queue.dequeued
Time in queue the buckets of queue.wait.duration
Time to process the buckets of queue.processing.duration

Metric names lists every metric and its Prometheus counterpart.