Should we build and run the full observability stack in pre-prod? How much realism vs. waiting for prod?
Answer: Yes. You absolutely want observability in pre-production environments—local, dev, performance test, staging, CI, everywhere.
Why?
- Observability is an essential part of production. Verify your traces when you verify the rest of the system.
- Test environments are about finding problems early. Which means you’ll need to debug those problems. Which requires observability so you can answer questions like: In an integration test, which system is failing? In a performance test, what is slow? This is observability at work.
- In development and local environments, iterate on your traces. As part of development, ask “How will I know this works?” and then check whether the traces, spans, and attributes your code emits are giving you that insight.
- While you’re querying your traces in test environments, notice schema conflicts, like: “Oops, we called it status_code here but everything else uses status.code.” Getting the attributes names right during test means your production schema will be cleaner.
Observability software like Honeycomb emerged from the desperate need of managing microservices and cloud, when older ways of understanding and checking our systems just weren’t enough. We had to create something better.
And observability is better: for instance, distributed tracing with query-time aggregation is strictly more powerful than looking at logs. It is more powerful than looking only at the output of tests. It is more powerful than canned graphs and than all the older techniques in combination.
But what about when it’s technically possible to rely on output and logs, like in low-volume tests? Observability is still better. It’s our best tool and gets our work of debugging broken tests done faster. At the same time, we sharpen the tools and our skills.
Why settle for less?
Do you have a question for Ask Miss O11y? Send us an email or book some time with one of our developer advocates!
