Incident management is the way an organization reacts to any kind of outage (security, broken code, severe weather, or anything that’s disruptive to customer service). Incidents are inherently fraught, not just because they’re time consuming and costly, but because they can potentially poison the well with customers, investors, and even partners. 

In this post, we’re going to lay out the guiding principle that unifies the diverse world of CS as we see it—and show how we put it into practice.

Charity once said an off-hand sentence that became a mantra for my transition into the VP of Engineering role: “Directors run the company.” This was said in the context of thinking about how the various management roles around the company interact: line managers run teams and projects, directors run the day-to-day work of the company, and execs (including VPs) focus above all on strategy, external-facing matters, and longer-term planning for the company’s future. 

Each CWV measures a specific part of the end user experience. CWV scores can help identify gaps in web page performance. Additionally, Google uses CWV scores as one of the measures it uses to rank pages, which means they are important for SEO.

In February of 2020, I was promoted from Director of Engineering to Honeycomb’s first VP of Engineering. Although Charity wrote an extremely generous public announcement, I hesitated to talk about this new role for quite a while as I was figuring out the job. But since 2020, I’ve noticed how little candid writing there is about paths to the VPE role or what the job is really like.* 

When the Gartner Magic Quadrant Report came out in 2022, we did the professional equivalent of a spit take, then cheered wildly. NOT ONLY did they include observability for the first time ever in their newly revamped 2022 Magic Quadrant for APM & [now] Observability, but they also put us in the Leader Quadrant—our debut appearance!

Honeycomb has the ability to receive events from application. These events can take the shape of Honeycomb wide events, OpenTelemetry trace spans, and OpenTelemetry metrics. Because Honeycomb’s backend is very flexible, these OpenTelemetry signals fit in just fine—but sometimes, they have a few quirks. Let’s dive into using metrics the Honeycomb way and cover a few optimizations.

n technical support, ensuring customer satisfaction and quickly resolving issues are of utmost importance. At Honeycomb, we embrace a comprehensive approach by using our own platform—not only for engineering purposes, but to also empower our support team. 

Insightful proof-of-concepts with a tool can be difficult to undertake due to the demands on valuable resources: time, energy, and people. With a task as grand as observability, how could one truly test if Honeycomb and OpenTelemetry are right for their organization and meet their requirements? For this thought experiment, here’s a comprehensive description of the ideal product evaluation over the course of four weeks, given unlimited resources.

People seem to struggle with the idea that there are no repeat incidents. It is very easy and natural to see two distinct outages, with nearly identical failure modes, impacting the same components, and with no significant action items as repeat incidents. However, when we look at the responses and their variations, we can find key distinctions that shows the incidents as related, but not identical.

OpenTelemetry and Beelines were designed with assumptions about the types of traffic that most users would trace. Based on these assumptions, web application and API calls fit very nicely into a trace waterfall view. This is also the set of assumptions for Refinery, our sampling proxy, to manage the traces it processes. These assumptions are true for most traffic.

At Honeycomb, we are all about observability. In the past, we have proposed observability-driven development as a way to maximize your observability and supercharge your development process. But I have a problem with the terminology, and it is: I don’t want observability to drive your development.