Instrument a small service end to end¶
By the end of this tutorial a small HTTP service will report how many orders it served, how
many failed and how long they took, along with what its database and cache were doing. The
metrics print straight to your terminal through otelcore.Slog, so you don't need a
collector, Docker or a metrics page.
Allow about twenty minutes. You need Go at the version on the go line of
this module's go.mod or
later, and curl.
Step 1: Create the project¶
mkdir checkout && cd checkout
go mod init example.com/checkout
go get gitlab.com/phpboyscout/go/observability
go get gitlab.com/phpboyscout/go/errors
go get modernc.org/sqlite
The last one is SQLite written in pure Go, so you don't need a C compiler. It's a big package, and the first build can take a minute or so.
Step 2: Write the service with no metrics¶
Start with a service that works and reports nothing. It looks up an order's customer, keeping
each answer in a map so the database is only asked once. Create main.go:
package main
import (
"context"
"database/sql"
"fmt"
"log"
"net/http"
"strconv"
"sync"
"gitlab.com/phpboyscout/go/errors"
_ "modernc.org/sqlite"
)
// ErrNotFound has a kind, so a missing order is a queryable failure of its own.
var ErrNotFound = errors.NewSentinel("order_not_found", "no such order")
type service struct {
conn *sql.DB
mu sync.Mutex
customers map[int]string
}
func (s *service) customer(ctx context.Context, id int) (string, error) {
s.mu.Lock()
name, ok := s.customers[id]
s.mu.Unlock()
if ok {
return name, nil
}
err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
if errors.Is(err, sql.ErrNoRows) {
return "", ErrNotFound
}
if err != nil {
return "", err
}
s.mu.Lock()
s.customers[id] = name
s.mu.Unlock()
return name, nil
}
func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {
id, err := strconv.Atoi(r.PathValue("id"))
if err != nil {
http.Error(w, "bad order ID", http.StatusBadRequest)
return
}
customer, err := s.customer(r.Context(), id)
if errors.Is(err, ErrNotFound) {
http.Error(w, err.Error(), http.StatusNotFound)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
fmt.Fprintln(w, customer)
}
func main() {
conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
if err != nil {
log.Fatal(err)
}
defer conn.Close()
_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
if err != nil {
log.Fatal(err)
}
s := &service{conn: conn, customers: map[int]string{}}
mux := http.NewServeMux()
mux.HandleFunc("GET /orders/{id}", s.getOrder)
log.Println("listening on :8080")
log.Fatal(http.ListenAndServe(":8080", mux))
}
Run it, and leave it running:
In a second terminal, ask for an order that exists and one that doesn't:
If you see address already in use, something else has port 8080. Change the port in
main.go, and in every curl from here on.
Stop the service with Ctrl+C.
Step 3: Add a meter that writes to the terminal¶
Every helper in instrument takes an OpenTelemetry metric.Meter, so you need a provider to
get one from. metrics.NewProvider builds one, and otelcore.Slog tells it to write each
metric as a log line instead of sending it to a collector. Add these to the imports:
"log/slog"
"os"
"os/signal"
"gitlab.com/phpboyscout/go/observability/metrics"
"gitlab.com/phpboyscout/go/observability/otelcore"
Then add this at the top of main:
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
provider, err := metrics.NewProvider(ctx,
otelcore.Resource("checkout", "0.1.0"),
otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},
)
if err != nil {
log.Fatal(err)
}
meter := provider.Meter("checkout")
The provider exports every 60 seconds, and once more when it shuts down. That last export is
the one you'll read, so the service has to shut the provider down when you press Ctrl+C rather
than just exiting. Replace the last two lines of main with this:
server := &http.Server{Addr: ":8080", Handler: mux}
go func() {
if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
log.Fatal(err)
}
}()
log.Println("listening on :8080")
<-ctx.Done()
_ = server.Shutdown(context.Background())
if err := provider.Shutdown(context.Background()); err != nil {
log.Fatal(err)
}
provider.Shutdown is called here rather than deferred on purpose. The helpers you add next are
closed with defer, and closing one stops it reporting, so the provider has to export before
they close.
meter isn't used yet, so this won't compile until the next step.
Step 4: Wrap the operation¶
An operation is one piece of work you care about: here, serving an order. Add
"gitlab.com/phpboyscout/go/observability/instrument" to the imports, and add a field to
service:
Create it in main, after the meter, and put it in the service you build there:
Now wrap the lookup in getOrder. ObserveResult is the form that hands a value back, and it
returns the error untouched, so the rest of the handler doesn't change:
customer, err := instrument.ObserveResult(r.Context(), s.ops, "get_order", func() (string, error) {
return s.customer(r.Context(), id)
})
"get_order" becomes a label on every sample. Keep names like this to a fixed set, because each
new value is a new series. Don't build one from the order ID.
Step 5: Instrument the database¶
The database gets two kinds of metric: one for each query, and one for the connection pool.
Add a field to service:
Create it in main, and tell it how to read the pool:
db, err := instrument.NewDB(meter, "orders", instrument.WithDBSystem("sqlite"))
if err != nil {
log.Fatal(err)
}
defer db.Close()
err = db.RegisterPool(func() instrument.PoolStats {
stats := conn.Stats()
return instrument.PoolStats{
MaxOpen: stats.MaxOpenConnections,
Open: stats.OpenConnections,
InUse: stats.InUse,
Idle: stats.Idle,
WaitCount: stats.WaitCount,
WaitSeconds: stats.WaitDuration.Seconds(),
}
})
if err != nil {
log.Fatal(err)
}
Then wrap the query in customer:
err := s.db.Query(ctx, "get_order", func() error {
err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
if errors.Is(err, sql.ErrNoRows) {
return ErrNotFound
}
return err
})
if err != nil {
return "", err
}
The sql.ErrNoRows check moved inside so the helper sees ErrNotFound. That error has a kind,
order_not_found, and the kind becomes the error.type label. An error with no kind, such as
sql.ErrNoRows itself, is recorded as _OTHER.
Step 6: Instrument the cache¶
Add a field to service:
Create it in main:
cache, err := instrument.NewCache(meter, "customers")
if err != nil {
log.Fatal(err)
}
defer cache.Close()
Then record what each lookup found, and how full the map is. Here's the whole of customer with
the database and cache in place:
func (s *service) customer(ctx context.Context, id int) (string, error) {
s.mu.Lock()
name, ok := s.customers[id]
s.mu.Unlock()
if ok {
s.cache.Hit(ctx)
return name, nil
}
s.cache.Miss(ctx)
err := s.db.Query(ctx, "get_order", func() error {
err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
if errors.Is(err, sql.ErrNoRows) {
return ErrNotFound
}
return err
})
if err != nil {
return "", err
}
s.mu.Lock()
s.customers[id] = name
s.cache.SetSize(len(s.customers))
s.mu.Unlock()
return name, nil
}
Step 7: Check the whole file and run it¶
Your main.go should now look like this:
package main
import (
"context"
"database/sql"
"fmt"
"log"
"log/slog"
"net/http"
"os"
"os/signal"
"strconv"
"sync"
"gitlab.com/phpboyscout/go/errors"
"gitlab.com/phpboyscout/go/observability/instrument"
"gitlab.com/phpboyscout/go/observability/metrics"
"gitlab.com/phpboyscout/go/observability/otelcore"
_ "modernc.org/sqlite"
)
// ErrNotFound has a kind, so a missing order is a queryable failure of its own.
var ErrNotFound = errors.NewSentinel("order_not_found", "no such order")
type service struct {
conn *sql.DB
ops *instrument.Operations
db *instrument.DB
cache *instrument.Cache
mu sync.Mutex
customers map[int]string
}
func (s *service) customer(ctx context.Context, id int) (string, error) {
s.mu.Lock()
name, ok := s.customers[id]
s.mu.Unlock()
if ok {
s.cache.Hit(ctx)
return name, nil
}
s.cache.Miss(ctx)
err := s.db.Query(ctx, "get_order", func() error {
err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
if errors.Is(err, sql.ErrNoRows) {
return ErrNotFound
}
return err
})
if err != nil {
return "", err
}
s.mu.Lock()
s.customers[id] = name
s.cache.SetSize(len(s.customers))
s.mu.Unlock()
return name, nil
}
func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {
id, err := strconv.Atoi(r.PathValue("id"))
if err != nil {
http.Error(w, "bad order ID", http.StatusBadRequest)
return
}
customer, err := instrument.ObserveResult(r.Context(), s.ops, "get_order", func() (string, error) {
return s.customer(r.Context(), id)
})
if errors.Is(err, ErrNotFound) {
http.Error(w, err.Error(), http.StatusNotFound)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
fmt.Fprintln(w, customer)
}
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()
logger := slog.New(slog.NewTextHandler(os.Stdout, nil))
provider, err := metrics.NewProvider(ctx,
otelcore.Resource("checkout", "0.1.0"),
otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},
)
if err != nil {
log.Fatal(err)
}
meter := provider.Meter("checkout")
conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
if err != nil {
log.Fatal(err)
}
defer conn.Close()
_, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
if err != nil {
log.Fatal(err)
}
ops, err := instrument.NewOperations(meter)
if err != nil {
log.Fatal(err)
}
db, err := instrument.NewDB(meter, "orders", instrument.WithDBSystem("sqlite"))
if err != nil {
log.Fatal(err)
}
defer db.Close()
err = db.RegisterPool(func() instrument.PoolStats {
stats := conn.Stats()
return instrument.PoolStats{
MaxOpen: stats.MaxOpenConnections,
Open: stats.OpenConnections,
InUse: stats.InUse,
Idle: stats.Idle,
WaitCount: stats.WaitCount,
WaitSeconds: stats.WaitDuration.Seconds(),
}
})
if err != nil {
log.Fatal(err)
}
cache, err := instrument.NewCache(meter, "customers")
if err != nil {
log.Fatal(err)
}
defer cache.Close()
s := &service{conn: conn, ops: ops, db: db, cache: cache, customers: map[int]string{}}
mux := http.NewServeMux()
mux.HandleFunc("GET /orders/{id}", s.getOrder)
server := &http.Server{Addr: ":8080", Handler: mux}
go func() {
if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
log.Fatal(err)
}
}()
log.Println("listening on :8080")
<-ctx.Done()
_ = server.Shutdown(context.Background())
if err := provider.Shutdown(context.Background()); err != nil {
log.Fatal(err)
}
}
Run it, and leave it running:
In the second terminal, send some traffic. This asks for order 1 three times, order 2 once, and order 99, which doesn't exist:
Then go back to the first terminal and press Ctrl+C.
Step 8: Read the metrics¶
When you press Ctrl+C, the provider's last export prints one line per metric. Every line
starts with time= and level=INFO, left off here so the rest fits. The durations will
differ on your machine, but the counts should match:
msg=metric name=operation.duration unit=s kind=histogram count=1 sum=0.00011538 min=0.00011538 max=0.00011538 buckets="0.005=1" attrs.error.type=order_not_found attrs.operation.name=get_order
msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.00014391 min=8.6e-07 max=0.00010253 buckets="0.005=4" attrs.operation.name=get_order
msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order
msg=metric name=db.client.operation.duration unit=s kind=histogram count=1 sum=9.905e-05 min=9.905e-05 max=9.905e-05 buckets="0.005=1" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite attrs.error.type=order_not_found
msg=metric name=db.client.operation.duration unit=s kind=histogram count=2 sum=8.973e-05 min=3.087e-05 max=5.886e-05 buckets="0.005=2" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite
msg=metric name=phpboyscout.db.client.operation.active unit={operation} kind=sum value=0 monotonic=false attrs.db.namespace=orders attrs.db.operation.name=get_order
msg=metric name=db.client.connection.count unit={connection} kind=sum value=1 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=idle
msg=metric name=db.client.connection.count unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=used
msg=metric name=db.client.connection.max unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_count unit={wait} kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_time unit=s kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders
msg=metric name=cache.operations unit={operation} kind=sum value=2 monotonic=true attrs.cache.name=customers attrs.cache.result=hit
msg=metric name=cache.operations unit={operation} kind=sum value=3 monotonic=true attrs.cache.name=customers attrs.cache.result=miss
msg=metric name=cache.size unit={entry} kind=gauge value=2 attrs.cache.name=customers
msg=metric name=cache.capacity unit={entry} kind=gauge value=0 attrs.cache.name=customers
If the service ran for more than a minute, you'll also see an earlier block from a periodic export. The values are running totals, so the last block is the one to read.
Here's what those lines tell you:
- Rate, errors, duration.
operation.durationhascount=4for requests that worked andcount=1witherror.type=order_not_found, which accounts for all five.sum,min,maxandbucketsgive you the duration: every request finished inside the first bucket, 5 milliseconds. - The cache is doing its job. There were 2 hits and 3 misses, and
cache.sizeis 2, because order 99 was never stored.cache.capacityis 0 because you never set it, which means unbounded. - The database only saw the misses. The two
db.client.operation.durationlines add up to 3, the same as the cache's misses. - The pool is idle.
db.client.connection.counthas one idle connection and none in use, and nothing has waited for a connection.
These are the OpenTelemetry names, exactly as a collector would receive them. The metric names reference lists every one, and Slog record fields explains each field on the line.
What to read next¶
You used otelcore.Slog so you could read the output, but none of the instrument code
depends on it. Give the helpers a meter from any provider and they work the same way.
- To send these metrics to a collector instead, change the
Exportertootelcore.Collector{Endpoint: "http://localhost:4318"}, and follow Export all three signals to a collector you can watch. - To see metrics sooner than once a minute, or to write them to a file, read Send telemetry to the terminal or a file.
- To instrument a circuit breaker or a queue, follow their how-to guides.
- To understand the signals and why label values must stay bounded, read RED, USE and cardinality.