Blog

Your observability pipeline is a production system too

Diana Todea’s recent piece on the CNCF blog, Is your OpenTelemetry pipeline becoming a performance bottleneck?, walks through the main ways a Collector configuration can quietly burn resources. Her central warning is worth repeating to anyone who signs off on platform budgets: “In some cases, the infrastructure responsible for observing a system can become a performance problem of its own.”

We agree, and we’d go further. Most organisations treat observability as a tool they installed. In practice it is a distributed system they operate, with its own capacity limits, failure modes and running costs. It deserves the same design scrutiny as the services it watches.

Telemetry has a cost curve nobody reviews

Instrumentation tends to grow by accretion. A team adds a span to chase a bug, auto-instrumentation turns on another library, someone attaches a customer ID as a metric attribute because it was handy. None of these decisions is reviewed for cost, and each one is multiplied by every request, every pod and every environment.

The result is a cost curve that rises with traffic but is rarely tied back to a decision. The bill shows up in three places: CPU and memory inside your application and Collectors, network egress between tiers, and ingestion or storage charges at the backend. In our experience the last one is usually what gets noticed first, often long after the instrumentation choices that drove it.

The source makes a point that teams often miss: there is no single overhead percentage you can quote for OpenTelemetry. The cost depends on throughput, attribute volume and how many spans each request produces. So the only honest answer to “how much does our observability cost?” is to measure your own workload.

Decide the failure mode before an incident decides it for you

The more important architectural question isn’t efficiency, it’s what happens under pressure. When a pipeline is overloaded, something gives. Either the application slows down, or telemetry gets dropped. For almost every system we’d recommend the second: observability should fail open, and losing some traces is a far better outcome than adding latency to a payment or a benefits claim.

That choice has to be designed in. A typical shape is a light collection tier close to the workload and a heavier processing tier behind it, with protective limits applied before any expensive work.

graph LR
  A[App SDK] --> B[Edge Collector]
  B --> C[Memory limit first]
  C --> D[Filter attributes]
  D --> E[Gateway Collector]
  E --> F[Tail sampling]
  F --> G[Backend]

The ordering matters. If enrichment and transformation run before the memory guard, the Collector does work on data it then throws away. And if you use tail sampling at the gateway, all spans from a trace need to reach the same instance, which changes how you load balance and scale that tier.

It also helps to be explicit about what each layer does when the backend is unavailable or slow.

graph TD
  A[Backend slow or down] --> B{Queue has room?}
  B -->|Yes| C[Buffer and retry]
  B -->|No| D{Memory pressure?}
  D -->|Yes| E[Refuse new data]
  D -->|No| C
  E --> F[Telemetry dropped, app unaffected]

If your team can’t draw this for your own pipeline, that’s the gap to close first.

What engineering teams should actually do

A few practices make the biggest difference, in our view:

The source’s closing advice is the right default: “treat example configurations as starting points rather than production-ready defaults.” Copying a sample config into production is how most of the problems described above arrive.

What this means if you’re buying or running software

If you commission software from a vendor or delivery partner, observability is usually treated as a line item that’s either ticked or not. That hides real risk. Some questions worth asking before go-live:

For organisations running platforms internally, the governance change is simple to state and harder to do: give observability a budget and an owner. Track telemetry volume per service the way you’d track cloud spend, and make teams aware of what their instrumentation costs. When costs rise, you want to know which decision caused it rather than discovering it in a quarterly invoice.

There’s also a sequencing lesson. Cheap wins, such as trimming unused resource attributes or relaxing infrastructure polling intervals where alerting allows, are worth taking early. Architectural changes, like splitting edge and gateway tiers or introducing tail sampling, need more care because they introduce state and new failure modes.

The takeaway

Good observability is one of the best investments a team can make in reliability. But it isn’t free, and it isn’t passive. Treat the pipeline as a production system: give it requirements, a failure mode, a capacity plan and an owner. Do that and it will stay an asset rather than becoming the next bottleneck.

If you’d like a second opinion on your observability setup, or help framing the right questions for a vendor, you’re welcome to book a free 30-minute session. We’re happy to talk it through, with no obligation.

← All posts