One of the things that struck me upon joining Honeycomb was the seemingly laissez-faire approach we took towards internal SLOs. From my own research (beginning with the classic SRE book, following Google’s example), I came to these conclusions: -SLOs are strict. They aren’t as binding as an SLA, but burning through your error budget is bad. -SLOs/SLIs need to be documented somewhere, with a formal specification, and approved by stakeholders. -SLOs should drive customer-level SLAs. -Teams should be mandated to create a minimum number of SLOs for the services they own.

In the last few years, the usage of databases that charge by request, query, or insert—rather than by provisioned compute infrastructure (e.g., CPU, RAM, etc.)—has grown significantly. They’re popular for a lot of the same reasons that serverless compute functions are, as the cost will scale with your usage. No one is using your site? No problem: you’re not charged.

It may surprise you to hear, but Honeycomb doesn’t currently have a platform team. We have a platform org, and my title is Director of Platform Engineering. We have engineers doing platform work. And, we even have an SRE team and a core services team. But a platform team? Nope. I’ve been thinking about what it might mean to build a platform team up from scratch—a situation some of you may also be in—and it led me to asking crucial questions.

Recently, Honeycomb held a roundtable discussion (available on demand) with Camal Cakar, Senior Site Reliability Engineer at Jimdo; Pierre Lacerte, Director of Software Development at Upgrade; and Kristin Smith, DevOps Engineer at Campspot. We talked about using OpenTelemetry and explained some important lessons the panelists learned—and are still learning in some cases.  Here are five best practices based on these lessons and Honeycomb’s own experience with OpenTelemetry.

Today, we're announcing the launch of Honeycomb Service Map. This isn't your grandparent's version of a service map. This feature reimagines what it is that you want to know or investigate when looking at visualizations of how your services communicate with one another.

As long as humans have written software, we’ve needed to understand why our expectations (the logic we thought we wrote) don’t match reality (the logic being executed). To that end, we developed techniques to help measure reality—logging text strings, or capturing aggregated metrics—and persevered, seeking out newer and fancier logging or monitoring solutions over the intervening decades.

If you build or maintain code in GitHub, the Honeycomb Buildevents Action can help you optimize the performance of your build pipelines in GitHub Actions. This blog introduces you to the gha-buildevents Action and a new hands-on quickstart guide that will show you the inner workings of GitHub Actions workflows, the buildevents tool, and the Honeycomb UI.

Getting existing telemetry into Honeycomb just got easier! With the release of the Datadog APM Receiver, you can send your Datadog traces to the OpenTelemetry Collector, and from there, to any OpenTelemetry-compatible endpoint.

Network topology can get very complicated in the cloud, especially when you’re sending data to external SaaS providers. You will likely need to configure gateways and firewalls and keep close tabs on those points of egress. However, if your infrastructure exists within AWS, there’s a much simpler way and that’s through an AWS PrivateLink endpoint.

When you want to direct your observability data in a uniform fashion, you want to run an OpenTelemetry collector. If you have a Kubernetes cluster handy, that’s a useful place to run it. Helm is a quick way to get it running in Kubernetes; it encapsulates all the YAML object definitions that you need. OpenTelemetry publishes a Helm chart for the collector.

Today, I’d like to share with you a new community-contributed integration that helps you optimize and debug your Gradle builds. This new Gradle plugin is available today, is free to use, and you can use it immediately with a free Honeycomb account.

In this incident review, we’ll cover the outage from September 8th, 2022, where our ingest system went down repeatedly and caused interruptions for over eight hours. We will first cover the background behind the incident with a high-level view of the relevant architecture, how we tried to investigate and fix the system, and finally, we’ll go over some meaningful elements that surfaced from our incident review process.