> ## Content Index
> Fetch the complete content index at: https://savasayik.com/llms.txt
> Use this file to discover other available public pages before exploring further.

# Gopher Holmes: On the Trail with OpenTelemetry
- URL: https://savasayik.com/gopher-holmes-on-the-trail-with-opentelemetry/
- Published: 2025-04-26T13:23:01.000Z
- Updated: 2025-04-26T13:27:57.000Z
- Description: The code runs... but do you know why? That’s the real question! Ready to lift the veil on your Go systems? With OpenTelemetry, no request will escape, no bug will stay hidden. Grab your gopher, it’s time to trace things down!
- Author: Savas AYIK
- Tags: Development, Golang, Telemetry, OpenTelemetry, Coding, #Import 2025-10-26 11:39

I have to admit, back when I started working on my first serious microservices project, I spent countless nights digging through logs trying to figure out why the system was slowing down or why some requests were vanishing into thin air. At the time, observability seemed like nothing more than logging to me. But as systems grew and became more complex, I realized just how **critical** true observability really is.

That’s when I discovered **OpenTelemetry**, one of the most valuable tools in my toolbox today. In this post, I want to share how I use OpenTelemetry in real-world **Go** applications, especially in **production environments**, sprinkled with some personal experience and practical tips.

## **What is OpenTelemetry and Why Does It Matter?**

Observability is the ability to understand what’s happening inside a system from the outside. It rests on three pillars: **traces**, **metrics**, and **logs**. Now, instead of using different tools for each, wouldn’t it be nice to have a single, unified system? That’s exactly what OpenTelemetry offers.

In distributed systems, you need **tracing** to see where requests slow down or fail, **metrics** to monitor system health, and **logs** to capture key events. But logs alone are no longer enough—without trace IDs, they lack real context.

## **Getting Started with OpenTelemetry in Go**

Integrating OpenTelemetry into a Go project is straightforward. I usually start by adding these modules:

```bash
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp

```

The first thing I typically do is set up a simple span. For example:

```go
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
    "fmt"
)

func main() {
    tracer := otel.Tracer("my-app")
    ctx, span := tracer.Start(context.Background(), "main-operation")
    defer span.End()

    fmt.Println("Hello, OpenTelemetry!")
}

```

This code starts a basic trace in my app. But at this point, it’s only local—the fun really begins when we start sending this data somewhere meaningful.

## **Sending Traces with an Exporter**

In production, you need to export your traces to a proper observability platform. I usually go with **OTLP** because it’s flexible and widely supported.

Here’s an example of setting up an OTLP exporter:

```go
package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/semconv/v1.21.0"
    "log"
)

func initTracer() *sdktrace.TracerProvider {
    ctx := context.Background()
    exporter, err := otlptracehttp.New(ctx)
    if err != nil {
        log.Fatal(err)
    }

    res, err := resource.New(ctx,
        resource.WithAttributes(
            semconv.ServiceNameKey.String("my-service"),
            semconv.ServiceVersionKey.String("1.0.0"),
            semconv.DeploymentEnvironmentKey.String("production"),
        ),
    )
    if err != nil {
        log.Fatal(err)
    }

    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exporter),
        sdktrace.WithResource(res),
        sdktrace.WithSampler(sdktrace.TraceIDRatioBased(0.01)),
    )
    otel.SetTracerProvider(tp)
    return tp
}

```

Notice the **sampler** here. I set it to 1% because logging every trace in production is costly. Start low and scale as needed.

---

## **Best Practices for Production**

1. **Sampling Strategy**: Use **TraceIDRatioBased**. 1% for high traffic, up to 100% for low traffic.
2. **Batching**: Always use **BatchSpanProcessor**. Don’t waste resources by sending spans individually.
3. **Resource Attributes**: Always include service name, version, and environment. It makes analysis way easier.
4. **Exporter Resilience**: Your app should not crash if the exporter fails. Log the error, but move on.

## **Context Propagation: Tracing Across Microservices**

One of the toughest parts of distributed systems is tracking how a single request flows through multiple services. A user’s request might hit 3-4 microservices, and each only sees a part of the story. This is where **context propagation** comes in.

OpenTelemetry uses **trace context**, passed through HTTP headers or gRPC metadata, to track a request’s journey.

In Go, this is easy. If you're using HTTP, just wrap your handlers with OpenTelemetry’s middleware:

```go
import (
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello with tracing!"))
    })

    wrappedHandler := otelhttp.NewHandler(handler, "hello-handler")
    http.Handle("/hello", wrappedHandler)
    http.ListenAndServe(":8080", nil)
}

```

Using **Gin**? Even easier:

```go
import (
    "github.com/gin-gonic/gin"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
)

func main() {
    r := gin.Default()
    r.Use(otelgin.Middleware("my-gin-service"))

    r.GET("/user/:id", func(c *gin.Context) {
        c.String(200, "User details")
    })

    r.Run(":8080")
}

```

## **Linking Logs to Traces: Shedding Light on Errors**

Traces are great, but **logs** are still essential for understanding what’s happening inside your app. The problem? In distributed systems, logs without trace context are just noise.

Linking logs with **trace IDs** is a must. OpenTelemetry gives each span a **trace ID** and **span ID**, which you can inject into your logs for instant context.

I like to add **trace IDs** to all my logs. In Go, it’s simple:

```go
import (
    "log"
    "go.opentelemetry.io/otel/trace"
)

func logWithTrace(ctx context.Context) {
    span := trace.SpanFromContext(ctx)
    traceID := span.SpanContext().TraceID().String()

    log.Printf("trace_id=%s, an error occurred: %s", traceID, "user not found")
}

```

Using **Logrus** or **Zap**? Here’s how with Logrus:

```go
import (
    "github.com/sirupsen/logrus"
    "go.opentelemetry.io/otel/trace"
)

func logWithLogrus(ctx context.Context) {
    span := trace.SpanFromContext(ctx)
    traceID := span.SpanContext().TraceID().String()

    logrus.WithField("trace_id", traceID).Error("failed to process payment")
}

```

This way, when you see an error log, you can jump straight to the trace and see the full picture.

Understanding what’s happening inside your systems and reacting to issues quickly requires observability. With OpenTelemetry in Go, it’s not just about collecting traces, but about **propagating** them across services and **linking** them to logs.

In my experience, seeing a **trace ID** in a log has saved me hours. I can instantly open the trace in my dashboard and follow every step that led to the issue. Once set up, nothing in your system is hidden from you.