Skip to content

Diagnose telemetry that never arrives

NewProvider returning no error means the provider was built. It says nothing about whether the collector exists, is reachable, or accepts what you send. Work down this list; it is ordered by how often each one is the answer.

First: make the failures visible

Export happens on a background goroutine and failures are reported through the OTel global error handler, which by default writes somewhere you are not looking. Install one before building any provider:

otel.SetErrorHandler(otel.ErrorHandlerFunc(func(err error) {
    slog.Error("otel export failed", "err", err)
}))

Add the SDK's internal logger too, which is where malformed environment values and exporter warnings go:

otel.SetLogger(logr.FromSlogHandler(slog.Default().Handler()))

Most investigations end here, with a connection-refused or a 401 that was being written to nowhere.

Did you call Shutdown?

A process that exits without calling Shutdown loses whatever is buffered. For a short program — a job, a test, a reproduction — that is usually everything:

  • spans batch for up to 5 seconds before an export
  • log records for up to 1 second
  • metrics for a full export interval, 60 seconds by default
shutdownCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()

_ = tp.Shutdown(shutdownCtx)
_ = mp.Shutdown(shutdownCtx)
_ = lp.Shutdown(shutdownCtx)

Give it a timeout. Shutdown against an unreachable collector sits through the retry budget — up to 60 seconds per batch.

Is the endpoint the one you think it is?

Two paths produce very different behaviour, and the difference is whether Settings.Endpoint is empty.

Non-empty — the configured value wins and every OTEL_EXPORTER_OTLP_*_ENDPOINT variable is ignored. The request goes to <endpoint><base path>/v1/traces (or /v1/metrics, /v1/logs).

Empty — the SDK reads OTEL_EXPORTER_OTLP_ENDPOINT. If that is unset too, it does not disable itself: it defaults to localhost:4318 and fails against a machine with no collector on it. "Nothing configured" looks identical to "misconfigured".

Check the resolved value at startup rather than the configuration file:

slog.Info("otlp target", "endpoint", settings.Endpoint, "insecure", settings.Insecure)

Is it TLS, in the direction you did not expect?

http: server gave HTTP response to HTTPS client means you are speaking TLS to a plaintext collector — use an http:// endpoint or set Settings.Insecure.

The reverse is more confusing, because it does not error: a configured https:// endpoint can be sent in plaintext if OTEL_EXPORTER_OTLP_INSECURE=true is set, or if OTEL_EXPORTER_OTLP_ENDPOINT is set to an http:// URL — even though the configured host is the one actually used. Check the environment of the running process, not just the config file. The environment-variable reference has the truth table.

Traces only: is anything being sampled?

The default sampling ratio is 0.1. A low-traffic service or a short reproduction can easily produce no sampled trace at all. Set it to 1.0 while investigating:

tp, err := tracing.NewProvider(ctx, res, settings, tracing.WithSampling(1.0))

Setting OTEL_TRACES_SAMPLER=always_on does not work — this module always passes an explicit sampler, which overrides it.

If the service receives requests carrying trace context, its own ratio is not the decision anyway: the sampler is parent-based, so an unsampled upstream means nothing is recorded here regardless.

Did you register the provider globally?

NewProvider does not call otel.SetTracerProvider. Any instrumentation library — otelhttp, otelgrpc, anything reading otel.Tracer(…) — talks to a no-op provider until you do:

otel.SetTracerProvider(tp)
otel.SetMeterProvider(mp)
global.SetLoggerProvider(lp) // go.opentelemetry.io/otel/log/global

Spans created directly from tp.Tracer(…) work without this, which is why a hand-written reproduction often exports while the real service does not.

Logs only: did the call site pass a context?

An exported record with an empty Trace ID was logged through logger.Info rather than logger.InfoContext. The record still arrives — it just cannot be joined to a trace. See Correlate logs with traces.

If no records arrive at all, check that the logger is actually built from logs.Handler(lp, …) and not from a text handler left over from earlier wiring.

Traces cross a service boundary but split into two

Trace context does not propagate without a propagator, and the OTel Go default is a no-op. Set one, in every service:

otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
))

This module does not set it, deliberately — it configures export, not propagation.

Prove the pipeline with a local collector

When the checklist runs out, take the backend out of the picture. A collector with the debug exporter shows exactly what arrived, and takes two commands to start — the first-signals tutorial walks through it. If telemetry reaches a local collector and not your backend, the problem is between the collector and the backend, and none of the above.