Opentelemetry in Go

A while ago, I wrote about how to utilize OpenTelemetry and set it up with auto-instrumentation to gain quick and easy observability into your applications with the libraries you already use. I provided instructions and examples on how to set this up in Java, Python, and Node.js, but left out Go. Go is known for being very explicit and disallowing any background magic (which instrumentations in other programming languages rely heavily on, such as monkey-patching in Python). Setting up OpenTelemetry in Go requires a bit more involvement than in other languages, but can be just as powerful.

At the moment, the OpenTelemetry project does not provide an auto-instrumentation solution for Go applications like it does for other programming languages. In early 2025, a beta version of auto-instrumentation for Go was released, but it is still very limited in terms of supported libraries and frameworks. It uses eBPF to dynamically instrument your application at runtime, but this complicates application deployment. In this article, I want to stick to code-based instrumentation and show how this can still be done with relatively little effort.

Setting up OpenTelemetry in Go

First, we need to install the relevant OpenTelemetry packages:

go get "go.opentelemetry.io/otel" \
  "go.opentelemetry.io/otel/log" \
  "go.opentelemetry.io/otel/metric" \
  "go.opentelemetry.io/otel/sdk" \
  "go.opentelemetry.io/otel/sdk/log" \
  "go.opentelemetry.io/otel/sdk/metric" \
  "go.opentelemetry.io/otel/trace" \
  "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp" \
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp" \
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"

Similar to how I did it in my post Utilize OpenTelemetry with Azure Application Insights, we now need to set up our providers and exporters. I’ll do this in a separate providers.go file to keep things organized.

package telemetry

import (
  "context"
  "fmt"
  "os"

  "flomon.de/backend/src/config"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
  "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
  "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
  "go.opentelemetry.io/otel/sdk/log"
  "go.opentelemetry.io/otel/sdk/metric"
  "go.opentelemetry.io/otel/sdk/resource"
  "go.opentelemetry.io/otel/sdk/trace"
  semconv "go.opentelemetry.io/otel/semconv/v1.37.0"
)

/*************
 * PROVIDERS *
 *************/

// newLoggerProvider creates a new logger provider with the OTLP http exporter.
func newLoggerProvider(ctx context.Context, res *resource.Resource) (*log.LoggerProvider, error) {
  exporter, err := otlploghttp.New(ctx, otlploghttp.WithEndpointURL(config.GetConfig().OpenTelemetry.Endpoint+"/v1/logs"))
  if err != nil {
    return nil, fmt.Errorf("failed to create OTLP log exporter: %w", err)
  }

  processor := log.NewBatchProcessor(exporter)
  lp := log.NewLoggerProvider(
    log.WithProcessor(processor),
    log.WithResource(res),
  )

  return lp, nil
}

// newMeterProvider creates a new meter provider with the OTLP HTTP exporter.
func newMeterProvider(ctx context.Context, res *resource.Resource) (*metric.MeterProvider, error) {
  exporter, err := otlpmetrichttp.New(ctx, otlpmetrichttp.WithEndpointURL(config.GetConfig().OpenTelemetry.Endpoint))
  if err != nil {
    return nil, fmt.Errorf("failed to create OTLP metric exporter: %w", err)
  }

  mp := metric.NewMeterProvider(
    metric.WithReader(metric.NewPeriodicReader(exporter)),
    metric.WithResource(res),
  )
  otel.SetMeterProvider(mp)

  return mp, nil
}

// newTracerProvider creates a new tracer provider with the OTLP HTTP exporter.
func newTracerProvider(ctx context.Context, res *resource.Resource) (*trace.TracerProvider, error) {
  exporter, err := otlptracehttp.New(ctx, otlptracehttp.WithEndpointURL(config.GetConfig().OpenTelemetry.Endpoint))
  if err != nil {
    return nil, fmt.Errorf("failed to create OTLP trace exporter: %w", err)
  }

  // Create Resource
  tp := trace.NewTracerProvider(
    trace.WithBatcher(exporter),
    trace.WithResource(res),
  )
  otel.SetTracerProvider(tp)

  return tp, nil
}

// newResource creates a new OTEL resource with the service name and version.
func newResource(serviceName string, serviceVersion string) (*resource.Resource, error) {
  hostName, _ := os.Hostname()

  r, err := resource.Merge(
    resource.Default(),
    resource.NewWithAttributes(
      semconv.SchemaURL,
      semconv.ServiceName(serviceName),
      semconv.ServiceVersion(serviceVersion),
      semconv.HostName(hostName),
    ))

  if err != nil {
    return nil, fmt.Errorf("failed to create resource: %w", err)
  }

  return r, nil
}

This time I’m using the default OTLP HTTP exporters for logs, metrics, and traces, but you can replace these with any other exporter, for example, using gRPC instead of HTTP or using a different backend altogether. In this file, we also create a resource that describes our service with its name and version, which will be attached to all telemetry data so we can identify our service in the backend. I’m using OTLP exporters here because I’m sending the data to Grafana Alloy and utilizing Loki, Tempo, and Prometheus as backends with Grafana for visualization. I’ll explain this setup in a future blog post.

Now we can create an InitOtel function that initializes everything and can be called from our main application.

package telemetry

import (
  "context"
  "log/slog"
  "net/http"
  "os"

  "flomon.de/backend/src/config"
  "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
  "go.opentelemetry.io/otel"
  "go.opentelemetry.io/otel/log/global"
  "go.opentelemetry.io/otel/metric"
  "go.opentelemetry.io/otel/propagation"
  sdklogs "go.opentelemetry.io/otel/sdk/log"
  sdkmetrics "go.opentelemetry.io/otel/sdk/metric"
  sdktrace "go.opentelemetry.io/otel/sdk/trace"
  "go.opentelemetry.io/otel/trace"
)

var Tracer trace.Tracer
var tracerProvider *sdktrace.TracerProvider
var Meter metric.Meter
var meterProvider *sdkmetrics.MeterProvider
var loggerProvider *sdklogs.LoggerProvider
var ctx context.Context

/*************
 * INITIALIZE *
 *************/

func InitOtel() {
  ctx = context.Background()

  cfg := config.GetConfig().OpenTelemetry

  resource, err := newResource(cfg.ServiceName, cfg.ServiceVersion)
  if err != nil {
    slog.Error("failed to create resource", slog.Any("error", err))
    os.Exit(1)
  }

  tracerProvider, err = newTracerProvider(ctx, resource)
  if err != nil {
    slog.Error("failed to create tracer provider", slog.Any("error", err))
    os.Exit(1)
  }

  meterProvider, err = newMeterProvider(ctx, resource)
  if err != nil {
    slog.Error("failed to create meter provider", slog.Any("error", err))
    os.Exit(1)
  }

  loggerProvider, err = newLoggerProvider(ctx, resource)
  if err != nil {
    slog.Error("failed to create logger provider", slog.Any("error", err))
    os.Exit(1)
  }

  otel.SetTracerProvider(tracerProvider)
  otel.SetMeterProvider(meterProvider)
  global.SetLoggerProvider(loggerProvider)
  otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
  Tracer = otel.Tracer(cfg.ServiceName)
  Meter = otel.Meter(cfg.ServiceName)

  initGlobalInstrumentations()
}

func initGlobalInstrumentations() {
  // Explained later
}

func ShutdownOtel() {
  if err := tracerProvider.Shutdown(ctx); err != nil {
    slog.Error("failed to shutdown TracerProvider", slog.Any("error", err))
  }
  if err := meterProvider.Shutdown(ctx); err != nil {
    slog.Error("failed to shutdown MeterProvider", slog.Any("error", err))
  }
  if err := loggerProvider.Shutdown(ctx); err != nil {
    slog.Error("failed to shutdown LoggerProvider", slog.Any("error", err))
  }
}

As you can see, we initialize all three providers and set them as global providers so we can access them from anywhere in our application. We also create global Tracer and Meter variables that we can use to create spans and metrics. Finally, we set up a text map propagator to propagate trace context across service boundaries. With that, our OpenTelemetry setup is ready to go. Now we can start instrumenting our application by adding instrumentations for the libraries we use.

Configuring Logging

To correctly pass our logs to OpenTelemetry, we need a log bridge that forwards our logs to the OpenTelemetry logger provider. Since I’m using slog from the standard library, I use the appropriate bridge from the OpenTelemetry Go contrib repository. There are also bridges for other logging libraries, such as Zap, Zerolog, and Logrus.

go get "go.opentelemetry.io/contrib/bridges/otelslog"

I’m configuring the default logger by applying the otelslog handler to it. I’m also adding tint as an additional handler to get colored and readable output in the console during development. To apply both handlers, we also need slog-multi. Finally, I’m adding the ability to switch log levels based on configuration via environment variables. setupLogger is called from initGlobalInstrumentations in otel.go.

package telemetry

import (
  "log/slog"
  "os"

  "flomon.de/backend/src/config"
  "github.com/lmittmann/tint"
  slogmulti "github.com/samber/slog-multi"
  "go.opentelemetry.io/contrib/bridges/otelslog"
)

func setupLogger() {

  w := os.Stderr

  otelHandler := otelslog.NewHandler(config.GetConfig().OpenTelemetry.ServiceName, otelslog.WithSource(true))

  var logLevel slog.Level

  err := logLevel.UnmarshalText([]byte(config.GetConfig().Debug.LogLevel))
  if err != nil {
    logLevel = slog.LevelInfo
  }

  tintHandler := tint.NewHandler(w, &tint.Options{
    Level:     logLevel,
    AddSource: true,
  })

  slog.SetDefault(slog.New(slogmulti.Fanout(otelHandler, tintHandler)))
}

Now when writing logs using slog, they will be forwarded to OpenTelemetry as well as printed to the console in a readable format. OpenTelemetry will receive all log levels as well as source code locations, while the console output will be filtered based on the configured log level. To correlate logs with traces, it is strongly advised to always use the logging functions with Context and to ensure you are passing context from your API down through your function calls.

slog.ErrorContext(c.Request.Context(), "Unexpected Internal Server Error", slog.Any("error", err))

Adding Instrumentations

To instrument your application, you need to add instrumentations for the libraries you use. OpenTelemetry Go provides instrumentations for many popular libraries, such as HTTP servers and clients, database clients, gRPC, and more. You can find a list of available instrumentations in the OpenTelemetry registry.

I will discuss four common instrumentations here: Gin instrumentation for your HTTP APIs, GORM instrumentation for database access, HTTP client instrumentation for outgoing HTTP requests, and gRPC instrumentation for gRPC services that don’t directly support OpenTelemetry.

Gin Instrumentation

The OpenTelemetry instrumentation for Gin is a simple middleware that you can add to your Gin router. It will automatically create spans for incoming HTTP requests and add trace context to the request context. You need to install go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin and then use it as follows:

r.Use(otelgin.Middleware("flomon.de/backend"))

That’s it! Now all incoming HTTP requests will be traced automatically.

GORM Instrumentation

With GORM, it’s quite similar. You need to install gorm.io/plugin/opentelemetry/tracing, which is attached as a plugin to your GORM DB instance. It will automatically create spans for all database operations.

package persistence

import (
  "log/slog"
  "os"

  "flomon.de/backend/src/config"
  "github.com/glebarez/sqlite"
  "gorm.io/gorm"
  "gorm.io/plugin/opentelemetry/tracing"
)

var db *gorm.DB

func SetupDB() {
  // Initialize the database connection here
  if config.GetConfig().Database.Driver == "sqlite" {
    // Example for SQLite, you can replace it with your actual database initialization logic
    dbCon, err := gorm.Open(sqlite.Open("gorm.db"), &gorm.Config{})
    if err != nil {
      slog.Error("Failed to connect to database", slog.Any("error", err))

      os.Exit(1)
    }

    db = dbCon

    slog.Info("Database connection established successfully", slog.String("driver", "sqlite"))
  } else {
    // Initialize other database types (e.g., PostgreSQL, MySQL) as needed
    slog.Error("Unsupported database driver", slog.String("driver", config.GetConfig().Database.Driver))
    os.Exit(1)
  }

  // Setup Otel for GORM
  if err := db.Use(tracing.NewPlugin()); err != nil {
    slog.Error("failed to setup Otel for GORM", slog.Any("error", err))
    os.Exit(1)
  }
}

As mentioned before, make sure to always pass context from your API handlers down to your database calls, so the spans can be correlated correctly. Then you can use the WithContext method on the GORM DB instance to pass the context.

func (t *GormTodoRepository) ListTodos(ctx context.Context, ownerID string) ([]*Todo, error) {
  var todos []*Todo
  if err := t.db.WithContext(ctx).Where("owner_id = ?", ownerID).Find(&todos).Error; err != nil {
    return nil, fmt.Errorf("error listing todos: %w", err)
  }

  return todos, nil
}

Instrumenting HTTP and gRPC Clients

Some libraries do not have built-in OpenTelemetry support, but you can still instrument them using the OpenTelemetry HTTP and gRPC instrumentations if they allow setting custom transport or dial options. You can install the HTTP instrumentation via go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp and the gRPC instrumentation via go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc. Then you simply need to pass the appropriate transport or dial options when creating your clients.

// Zitadel

func NewZitadelUserRepository() *ZitadelUserRepository {
  ctx := context.Background()

  api, err := client.New(ctx,
    zitadel.New(config.GetConfig().Auth.ZitadelDomain),
    client.WithAuth(
      client.DefaultServiceUserAuthentication("./secrets/su.json", oidc.ScopeOpenID, client.ScopeZitadelAPI()),
    ),
    // Add gRPC instrumentation for OpenTelemetry
    client.WithGRPCDialOptions(
      grpc.WithStatsHandler(otelgrpc.NewClientHandler(otelgrpc.WithSpanAttributes(attribute.String("peer.service", "zitadel")))),
    ),
  )

  if err != nil {
    slog.Error("Error while creating Zitadel client", slog.Any("error", err))
    os.Exit(1)
  }

  return &ZitadelUserRepository{api: api}
}

// or Stripe
func (sp *StripePaymentService) setupPayment() {
  stripe.Key = config.GetConfig().Payment.StripeKey
  stripe.SetHTTPClient(&http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
  })
}

In order to catch other libraries that use the default HTTP client internally, you can also set the default HTTP transport to the OpenTelemetry instrumented transport globally.

http.DefaultClient = &http.Client{
    Transport: otelhttp.NewTransport(http.DefaultTransport),
  }

This way all outgoing HTTP requests will be traced automatically.

Conclusion

With this setup, you now have a solid foundation for instrumenting your Go applications with OpenTelemetry. Adding custom instrumentation on top is as simple as creating spans using the global Tracer variable and recording metrics using the global Meter variable. Just remember to always propagate context through your application to ensure all telemetry data is correlated correctly.