Instrument a queue¶
Create a Queue, then record items going in and coming out:
queue, err := instrument.NewQueue(mp.Meter("checkout"), "emails")
if err != nil {
return err
}
queue.Enqueue(ctx)
queue.SetDepth(backend.Len())
// on the consumer
queue.Dequeue(ctx)
queue.SetDepth(backend.Len())
queue.ObserveWait(ctx, time.Since(item.EnqueuedAt))
start := time.Now()
err = process(ctx, item)
queue.ObserveProcessing(ctx, time.Since(start))
Pass the item's context so that a sample inside a sampled span carries an exemplar.
Zero the depth when you drop a queue¶
Queue has no Close, because it registers no callback, so nothing stops its depth reporting.
Call SetDepth(0) when you drop a queue, or queue.depth keeps reading the last value for the
life of the provider and a dashboard shows a backlog that no longer exists.
Set the depth from the queue¶
Pass SetDepth the real length rather than counting enqueues against dequeues yourself. Each call
adds the difference from the last value to an up/down counter, so the metric reads the last value
set, and concurrent callers cannot drift it.
Change the buckets for slow queues¶
The default buckets top out at 10 seconds. If items wait for minutes, set your own:
WithBuckets applies to both durations.
Use one name per queue¶
The name becomes the value of queue.name, and every distinct value is a new series, so keep names to a small fixed set, never a per-call value. RED, USE and cardinality says what happens otherwise.
Give each queue its own name per meter, because every metric carries that name as queue.name.
Two queues sharing a name add together, and their depths sum.
Read the signals¶
| Signal | Query |
|---|---|
| Backlog | queue.depth |
| Are we keeping up? | the rate of queue.enqueued against the rate of queue.dequeued |
| Time in queue | the buckets of queue.wait.duration |
| Time to process | the buckets of queue.processing.duration |
Metric names lists every metric and its Prometheus counterpart.