Your observability pipeline is a production system too
Diana Todea’s recent piece on the CNCF blog, Is your OpenTelemetry pipeline becoming a performance bottleneck?, walks through the main ways a Collector configuration can quietly burn resources. Her central warning is worth repeating to anyone who signs off on platform budgets: “In some cases, the infrastructure responsible for observing a system can become a performance problem of its own.”
We agree, and we’d go further. Most organisations treat observability as a tool they installed. In practice it is a distributed system they operate, with its own capacity limits, failure modes and running costs. It deserves the same design scrutiny as the services it watches.
Telemetry has a cost curve nobody reviews
Instrumentation tends to grow by accretion. A team adds a span to chase a bug, auto-instrumentation turns on another library, someone attaches a customer ID as a metric attribute because it was handy. None of these decisions is reviewed for cost, and each one is multiplied by every request, every pod and every environment.
The result is a cost curve that rises with traffic but is rarely tied back to a decision. The bill shows up in three places: CPU and memory inside your application and Collectors, network egress between tiers, and ingestion or storage charges at the backend. In our experience the last one is usually what gets noticed first, often long after the instrumentation choices that drove it.
The source makes a point that teams often miss: there is no single overhead percentage you can quote for OpenTelemetry. The cost depends on throughput, attribute volume and how many spans each request produces. So the only honest answer to “how much does our observability cost?” is to measure your own workload.
Decide the failure mode before an incident decides it for you
The more important architectural question isn’t efficiency, it’s what happens under pressure. When a pipeline is overloaded, something gives. Either the application slows down, or telemetry gets dropped. For almost every system we’d recommend the second: observability should fail open, and losing some traces is a far better outcome than adding latency to a payment or a benefits claim.
That choice has to be designed in. A typical shape is a light collection tier close to the workload and a heavier processing tier behind it, with protective limits applied before any expensive work.
graph LR A[App SDK] --> B[Edge Collector] B --> C[Memory limit first] C --> D[Filter attributes] D --> E[Gateway Collector] E --> F[Tail sampling] F --> G[Backend]
The ordering matters. If enrichment and transformation run before the memory guard, the Collector does work on data it then throws away. And if you use tail sampling at the gateway, all spans from a trace need to reach the same instance, which changes how you load balance and scale that tier.
It also helps to be explicit about what each layer does when the backend is unavailable or slow.
graph TD
A[Backend slow or down] --> B{Queue has room?}
B -->|Yes| C[Buffer and retry]
B -->|No| D{Memory pressure?}
D -->|Yes| E[Refuse new data]
D -->|No| C
E --> F[Telemetry dropped, app unaffected]If your team can’t draw this for your own pipeline, that’s the gap to close first.
What engineering teams should actually do
A few practices make the biggest difference, in our view:
- Instrument for questions, not completeness. Every span and attribute should exist because someone will use it to diagnose or alert on something. Unbounded values such as request IDs belong on traces or logs, not metric attributes.
- Put cardinality in code review. A new metric attribute is a schema change with a cost. Review it like one.
- Choose sampling deliberately. Head sampling is cheap. Tail sampling keeps the interesting traces, errors especially, but costs memory while it waits to decide. Pick based on the questions you need answered, not on what the example config did.
- Load test the pipeline, not just the app. Synthetic telemetry generators let you isolate Collector behaviour. Change one setting at a time and watch CPU, memory, latency, throughput and dropped data together.
- Monitor the monitors. Collectors expose their own metrics on queue depth, refused data and export failures. Alert on them, or you’ll learn about telemetry loss during the incident where you needed it.
The source’s closing advice is the right default: “treat example configurations as starting points rather than production-ready defaults.” Copying a sample config into production is how most of the problems described above arrive.
What this means if you’re buying or running software
If you commission software from a vendor or delivery partner, observability is usually treated as a line item that’s either ticked or not. That hides real risk. Some questions worth asking before go-live:
- What telemetry does the system emit, and who decided what’s collected?
- What sampling strategy is in place, and does it reliably keep errors?
- If the observability backend goes down, does the application slow down or keep serving?
- Has the telemetry pipeline been load tested at expected peak, not just the application?
- Who owns the ongoing ingestion and storage bill, and how will it grow with traffic?
- Could any attributes carry personal or sensitive information? Telemetry is data, and your privacy obligations apply to it.
For organisations running platforms internally, the governance change is simple to state and harder to do: give observability a budget and an owner. Track telemetry volume per service the way you’d track cloud spend, and make teams aware of what their instrumentation costs. When costs rise, you want to know which decision caused it rather than discovering it in a quarterly invoice.
There’s also a sequencing lesson. Cheap wins, such as trimming unused resource attributes or relaxing infrastructure polling intervals where alerting allows, are worth taking early. Architectural changes, like splitting edge and gateway tiers or introducing tail sampling, need more care because they introduce state and new failure modes.
The takeaway
Good observability is one of the best investments a team can make in reliability. But it isn’t free, and it isn’t passive. Treat the pipeline as a production system: give it requirements, a failure mode, a capacity plan and an owner. Do that and it will stay an asset rather than becoming the next bottleneck.
If you’d like a second opinion on your observability setup, or help framing the right questions for a vendor, you’re welcome to book a free 30-minute session. We’re happy to talk it through, with no obligation.
← All posts