One Year of Graviton2 at Honeycomb
A year ago, we wrote about our experiences as early adopters of Graviton2, and how we were able to see 30% price-performance improvements on one...
Interview with Honeycomb Engineer Chris Toshok: Dogfooding OpenTelemetry
We often get questions about the difference between using our Beeline SDKs compared with other integrations, especially OpenTelemetry (abbreviated "OTel"). That’s why the team decided...
Incident Report: Investigating an Incident That's Already Resolved
Summary On the 23rd of April, we discovered that an incident had occurred approximately one week earlier. On April 16, for approximately 1.5 hours we...
Dogfooding for Deploys: How Honeycomb Builds Better Builds with Observability
Observability changes the way you understand and interact with your applications in production. Beyond knowing what’s happening in prod, observability is also a compass that...
Observations on ARM64 & AWS’s Amazon EC2 M6g Instances
At re:Invent in December, Amazon announced the AWS Graviton2 processor and its forthcoming availability powering Amazon EC2 M6g instances. While the first-generation Graviton processor that...
Using Honeycomb to remember to delete a feature flag
Feature flags are great and serve us in so many ways. However, we do not love long-lived feature flags. They lead to more complicated code,...
Instrumenting Lambda with Traces: A Complete Example in Python
We’re big fans of AWS Lambda at Honeycomb. As you may have read, we recently made some major improvements to our storage engine by leveraging...
Honeycomb SLO Now Generally Available: Success, Defined.
Honeycomb now offers SLOs, aka Service Level Objectives. This is the second in a set of of essays on creating SLOs from first principles. Previously,...
From "Secondary Storage" To Just "Storage": A Tale of Lambdas, LZ4, and Garbage Collection
When we introduced Secondary Storage two years ago, it was a deliberate compromise between economy and performance. Compared to Honeycomb’s primary NVMe storage attached to...
Incident Report: Running Dry on Memory Without Noticing
On November 6, 2019, we intermittently rejected 1-3% of customer telemetry data at ingest for four periods of 20 minutes each. The trigger of the...
Working Toward Service Level Objectives (SLOs), Part 1
In theory, Honeycomb is always up. Our servers run without hiccups, our user interface loads rapidly and is highly responsive, and our query engine is...
Never Alone On Call
Does your organization have an on-call rotation? Several members of the Honeycomb engineering team recently hosted a live webcast about why they never feel alone...
All Together Now: Better Debugging With Multiple Visualizations
"Nines don't matter when users aren't happy" is something you may have heard a time or two from folks here at Honeycomb. We often emphasize...
Understand Your AWS Cost & Usage with Honeycomb
First published in August 2019. AWS bills are notoriously complicated, and the Amazon Cost Explorer doesn’t always make it easy to understand exactly where your...
Treading in Haunted Graveyards
Part 1: CI/CD for Infrastructure as Code At Honeycomb, we've often discussed the value of making software deployments early and often, and being able to...