Skip to content

Instrument a small service end to end

By the end of this tutorial a small HTTP service will report how many orders it served, how many failed and how long they took, along with what its database and cache were doing. The metrics print straight to your terminal through otelcore.Slog, so you don't need a collector, Docker or a metrics page.

Allow about twenty minutes. You need Go at the version on the go line of this module's go.mod or later, and curl.

Step 1: Create the project

mkdir checkout && cd checkout
go mod init example.com/checkout
go get gitlab.com/phpboyscout/go/observability
go get gitlab.com/phpboyscout/go/errors
go get modernc.org/sqlite

The last one is SQLite written in pure Go, so you don't need a C compiler. It's a big package, and the first build can take a minute or so.

Step 2: Write the service with no metrics

Start with a service that works and reports nothing. It looks up an order's customer, keeping each answer in a map so the database is only asked once. Create main.go:

package main

import (
    "context"
    "database/sql"
    "fmt"
    "log"
    "net/http"
    "strconv"
    "sync"

    "gitlab.com/phpboyscout/go/errors"
    _ "modernc.org/sqlite"
)

// ErrNotFound has a kind, so a missing order is a queryable failure of its own.
var ErrNotFound = errors.NewSentinel("order_not_found", "no such order")

type service struct {
    conn *sql.DB

    mu        sync.Mutex
    customers map[int]string
}

func (s *service) customer(ctx context.Context, id int) (string, error) {
    s.mu.Lock()
    name, ok := s.customers[id]
    s.mu.Unlock()

    if ok {
        return name, nil
    }

    err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
    if errors.Is(err, sql.ErrNoRows) {
        return "", ErrNotFound
    }

    if err != nil {
        return "", err
    }

    s.mu.Lock()
    s.customers[id] = name
    s.mu.Unlock()

    return name, nil
}

func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "bad order ID", http.StatusBadRequest)
        return
    }

    customer, err := s.customer(r.Context(), id)
    if errors.Is(err, ErrNotFound) {
        http.Error(w, err.Error(), http.StatusNotFound)
        return
    }

    if err != nil {
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }

    fmt.Fprintln(w, customer)
}

func main() {
    conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()

    _, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
        INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
    if err != nil {
        log.Fatal(err)
    }

    s := &service{conn: conn, customers: map[int]string{}}

    mux := http.NewServeMux()
    mux.HandleFunc("GET /orders/{id}", s.getOrder)

    log.Println("listening on :8080")
    log.Fatal(http.ListenAndServe(":8080", mux))
}

Run it, and leave it running:

go run .

In a second terminal, ask for an order that exists and one that doesn't:

curl localhost:8080/orders/1
curl localhost:8080/orders/99
Ada
no such order

If you see address already in use, something else has port 8080. Change the port in main.go, and in every curl from here on.

Stop the service with Ctrl+C.

Step 3: Add a meter that writes to the terminal

Every helper in instrument takes an OpenTelemetry metric.Meter, so you need a provider to get one from. metrics.NewProvider builds one, and otelcore.Slog tells it to write each metric as a log line instead of sending it to a collector. Add these to the imports:

"log/slog"
"os"
"os/signal"

"gitlab.com/phpboyscout/go/observability/metrics"
"gitlab.com/phpboyscout/go/observability/otelcore"

Then add this at the top of main:

ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
defer stop()

logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

provider, err := metrics.NewProvider(ctx,
    otelcore.Resource("checkout", "0.1.0"),
    otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},
)
if err != nil {
    log.Fatal(err)
}

meter := provider.Meter("checkout")

The provider exports every 60 seconds, and once more when it shuts down. That last export is the one you'll read, so the service has to shut the provider down when you press Ctrl+C rather than just exiting. Replace the last two lines of main with this:

server := &http.Server{Addr: ":8080", Handler: mux}

go func() {
    if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
        log.Fatal(err)
    }
}()

log.Println("listening on :8080")
<-ctx.Done()

_ = server.Shutdown(context.Background())

if err := provider.Shutdown(context.Background()); err != nil {
    log.Fatal(err)
}

provider.Shutdown is called here rather than deferred on purpose. The helpers you add next are closed with defer, and closing one stops it reporting, so the provider has to export before they close.

meter isn't used yet, so this won't compile until the next step.

Step 4: Wrap the operation

An operation is one piece of work you care about: here, serving an order. Add "gitlab.com/phpboyscout/go/observability/instrument" to the imports, and add a field to service:

ops   *instrument.Operations

Create it in main, after the meter, and put it in the service you build there:

ops, err := instrument.NewOperations(meter)
if err != nil {
    log.Fatal(err)
}

Now wrap the lookup in getOrder. ObserveResult is the form that hands a value back, and it returns the error untouched, so the rest of the handler doesn't change:

customer, err := instrument.ObserveResult(r.Context(), s.ops, "get_order", func() (string, error) {
    return s.customer(r.Context(), id)
})

"get_order" becomes a label on every sample. Keep names like this to a fixed set, because each new value is a new series. Don't build one from the order ID.

Step 5: Instrument the database

The database gets two kinds of metric: one for each query, and one for the connection pool. Add a field to service:

db    *instrument.DB

Create it in main, and tell it how to read the pool:

db, err := instrument.NewDB(meter, "orders", instrument.WithDBSystem("sqlite"))
if err != nil {
    log.Fatal(err)
}
defer db.Close()

err = db.RegisterPool(func() instrument.PoolStats {
    stats := conn.Stats()

    return instrument.PoolStats{
        MaxOpen:     stats.MaxOpenConnections,
        Open:        stats.OpenConnections,
        InUse:       stats.InUse,
        Idle:        stats.Idle,
        WaitCount:   stats.WaitCount,
        WaitSeconds: stats.WaitDuration.Seconds(),
    }
})
if err != nil {
    log.Fatal(err)
}

Then wrap the query in customer:

err := s.db.Query(ctx, "get_order", func() error {
    err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
    if errors.Is(err, sql.ErrNoRows) {
        return ErrNotFound
    }

    return err
})
if err != nil {
    return "", err
}

The sql.ErrNoRows check moved inside so the helper sees ErrNotFound. That error has a kind, order_not_found, and the kind becomes the error.type label. An error with no kind, such as sql.ErrNoRows itself, is recorded as _OTHER.

Step 6: Instrument the cache

Add a field to service:

cache *instrument.Cache

Create it in main:

cache, err := instrument.NewCache(meter, "customers")
if err != nil {
    log.Fatal(err)
}
defer cache.Close()

Then record what each lookup found, and how full the map is. Here's the whole of customer with the database and cache in place:

func (s *service) customer(ctx context.Context, id int) (string, error) {
    s.mu.Lock()
    name, ok := s.customers[id]
    s.mu.Unlock()

    if ok {
        s.cache.Hit(ctx)
        return name, nil
    }

    s.cache.Miss(ctx)

    err := s.db.Query(ctx, "get_order", func() error {
        err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
        if errors.Is(err, sql.ErrNoRows) {
            return ErrNotFound
        }

        return err
    })
    if err != nil {
        return "", err
    }

    s.mu.Lock()
    s.customers[id] = name
    s.cache.SetSize(len(s.customers))
    s.mu.Unlock()

    return name, nil
}

Step 7: Check the whole file and run it

Your main.go should now look like this:

package main

import (
    "context"
    "database/sql"
    "fmt"
    "log"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "strconv"
    "sync"

    "gitlab.com/phpboyscout/go/errors"
    "gitlab.com/phpboyscout/go/observability/instrument"
    "gitlab.com/phpboyscout/go/observability/metrics"
    "gitlab.com/phpboyscout/go/observability/otelcore"
    _ "modernc.org/sqlite"
)

// ErrNotFound has a kind, so a missing order is a queryable failure of its own.
var ErrNotFound = errors.NewSentinel("order_not_found", "no such order")

type service struct {
    conn  *sql.DB
    ops   *instrument.Operations
    db    *instrument.DB
    cache *instrument.Cache

    mu        sync.Mutex
    customers map[int]string
}

func (s *service) customer(ctx context.Context, id int) (string, error) {
    s.mu.Lock()
    name, ok := s.customers[id]
    s.mu.Unlock()

    if ok {
        s.cache.Hit(ctx)
        return name, nil
    }

    s.cache.Miss(ctx)

    err := s.db.Query(ctx, "get_order", func() error {
        err := s.conn.QueryRowContext(ctx, "SELECT customer FROM orders WHERE id = ?", id).Scan(&name)
        if errors.Is(err, sql.ErrNoRows) {
            return ErrNotFound
        }

        return err
    })
    if err != nil {
        return "", err
    }

    s.mu.Lock()
    s.customers[id] = name
    s.cache.SetSize(len(s.customers))
    s.mu.Unlock()

    return name, nil
}

func (s *service) getOrder(w http.ResponseWriter, r *http.Request) {
    id, err := strconv.Atoi(r.PathValue("id"))
    if err != nil {
        http.Error(w, "bad order ID", http.StatusBadRequest)
        return
    }

    customer, err := instrument.ObserveResult(r.Context(), s.ops, "get_order", func() (string, error) {
        return s.customer(r.Context(), id)
    })
    if errors.Is(err, ErrNotFound) {
        http.Error(w, err.Error(), http.StatusNotFound)
        return
    }

    if err != nil {
        http.Error(w, "internal error", http.StatusInternalServerError)
        return
    }

    fmt.Fprintln(w, customer)
}

func main() {
    ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt)
    defer stop()

    logger := slog.New(slog.NewTextHandler(os.Stdout, nil))

    provider, err := metrics.NewProvider(ctx,
        otelcore.Resource("checkout", "0.1.0"),
        otelcore.Settings{Enabled: true, Exporter: otelcore.Slog(logger)},
    )
    if err != nil {
        log.Fatal(err)
    }

    meter := provider.Meter("checkout")

    conn, err := sql.Open("sqlite", "file:checkout?mode=memory&cache=shared")
    if err != nil {
        log.Fatal(err)
    }
    defer conn.Close()

    _, err = conn.Exec(`CREATE TABLE orders (id INTEGER PRIMARY KEY, customer TEXT);
        INSERT INTO orders VALUES (1, 'Ada'), (2, 'Grace');`)
    if err != nil {
        log.Fatal(err)
    }

    ops, err := instrument.NewOperations(meter)
    if err != nil {
        log.Fatal(err)
    }

    db, err := instrument.NewDB(meter, "orders", instrument.WithDBSystem("sqlite"))
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    err = db.RegisterPool(func() instrument.PoolStats {
        stats := conn.Stats()

        return instrument.PoolStats{
            MaxOpen:     stats.MaxOpenConnections,
            Open:        stats.OpenConnections,
            InUse:       stats.InUse,
            Idle:        stats.Idle,
            WaitCount:   stats.WaitCount,
            WaitSeconds: stats.WaitDuration.Seconds(),
        }
    })
    if err != nil {
        log.Fatal(err)
    }

    cache, err := instrument.NewCache(meter, "customers")
    if err != nil {
        log.Fatal(err)
    }
    defer cache.Close()

    s := &service{conn: conn, ops: ops, db: db, cache: cache, customers: map[int]string{}}

    mux := http.NewServeMux()
    mux.HandleFunc("GET /orders/{id}", s.getOrder)

    server := &http.Server{Addr: ":8080", Handler: mux}

    go func() {
        if err := server.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
            log.Fatal(err)
        }
    }()

    log.Println("listening on :8080")
    <-ctx.Done()

    _ = server.Shutdown(context.Background())

    if err := provider.Shutdown(context.Background()); err != nil {
        log.Fatal(err)
    }
}

Run it, and leave it running:

go run .
2026/11/13 08:23:17 listening on :8080

In the second terminal, send some traffic. This asks for order 1 three times, order 2 once, and order 99, which doesn't exist:

for id in 1 1 2 99 1; do curl -s -o /dev/null localhost:8080/orders/$id; done

Then go back to the first terminal and press Ctrl+C.

Step 8: Read the metrics

When you press Ctrl+C, the provider's last export prints one line per metric. Every line starts with time= and level=INFO, left off here so the rest fits. The durations will differ on your machine, but the counts should match:

msg=metric name=operation.duration unit=s kind=histogram count=1 sum=0.00011538 min=0.00011538 max=0.00011538 buckets="0.005=1" attrs.error.type=order_not_found attrs.operation.name=get_order
msg=metric name=operation.duration unit=s kind=histogram count=4 sum=0.00014391 min=8.6e-07 max=0.00010253 buckets="0.005=4" attrs.operation.name=get_order
msg=metric name=operation.active unit={operation} kind=sum value=0 monotonic=false attrs.operation.name=get_order
msg=metric name=db.client.operation.duration unit=s kind=histogram count=1 sum=9.905e-05 min=9.905e-05 max=9.905e-05 buckets="0.005=1" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite attrs.error.type=order_not_found
msg=metric name=db.client.operation.duration unit=s kind=histogram count=2 sum=8.973e-05 min=3.087e-05 max=5.886e-05 buckets="0.005=2" attrs.db.namespace=orders attrs.db.operation.name=get_order attrs.db.system.name=sqlite
msg=metric name=phpboyscout.db.client.operation.active unit={operation} kind=sum value=0 monotonic=false attrs.db.namespace=orders attrs.db.operation.name=get_order
msg=metric name=db.client.connection.count unit={connection} kind=sum value=1 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=idle
msg=metric name=db.client.connection.count unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders attrs.db.client.connection.state=used
msg=metric name=db.client.connection.max unit={connection} kind=sum value=0 monotonic=false attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_count unit={wait} kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders
msg=metric name=phpboyscout.db.client.connection.wait_time unit=s kind=sum value=0 monotonic=true attrs.db.client.connection.pool.name=orders
msg=metric name=cache.operations unit={operation} kind=sum value=2 monotonic=true attrs.cache.name=customers attrs.cache.result=hit
msg=metric name=cache.operations unit={operation} kind=sum value=3 monotonic=true attrs.cache.name=customers attrs.cache.result=miss
msg=metric name=cache.size unit={entry} kind=gauge value=2 attrs.cache.name=customers
msg=metric name=cache.capacity unit={entry} kind=gauge value=0 attrs.cache.name=customers

If the service ran for more than a minute, you'll also see an earlier block from a periodic export. The values are running totals, so the last block is the one to read.

Here's what those lines tell you:

  • Rate, errors, duration. operation.duration has count=4 for requests that worked and count=1 with error.type=order_not_found, which accounts for all five. sum, min, max and buckets give you the duration: every request finished inside the first bucket, 5 milliseconds.
  • The cache is doing its job. There were 2 hits and 3 misses, and cache.size is 2, because order 99 was never stored. cache.capacity is 0 because you never set it, which means unbounded.
  • The database only saw the misses. The two db.client.operation.duration lines add up to 3, the same as the cache's misses.
  • The pool is idle. db.client.connection.count has one idle connection and none in use, and nothing has waited for a connection.

These are the OpenTelemetry names, exactly as a collector would receive them. The metric names reference lists every one, and Slog record fields explains each field on the line.

You used otelcore.Slog so you could read the output, but none of the instrument code depends on it. Give the helpers a meter from any provider and they work the same way.