“When things broke,” Molly explained, “you’re mad scrambling—jumping from website to website to website, trying to put the pieces together.” Molly was able to use Honeycomb to fix things up: “It makes my job easier as an SRE.”
Getting started with Honeycomb doesn’t require a lot of work: at dev.to, they used the Ruby Beeline to get it going: “I didn’t do that much,” she said. “I feel like I cheated!” We invite you along on their journey, and we’ll share precisely what lines of code dev.to used to get started.
Service Fatigue
“When I started at DEV, there was very little in the way of monitoring, in terms of observability,” said Molly. “We had six or seven little tools, each doing one little piece.”
While each tool had initially served a purpose for the team, the fact that there were so many became a pain point during incidents. It’s a familiar experience for SREs or anyone who owns code in production.
“You’d look at one service, you’d see a spike, but that was all the information you’d get. So then you’d have to go to the other services and try to line up when the spike happened.”
Even during normal site operations, that’s enough to fatigue most people. With the added stress of an outage it becomes painful, error-prone, or both. That’s why Molly took the initiative to consolidate DEV’s use of monitoring services as much as possible. Whether facing an outage or just wanting to see how the site was being used, she wanted all that information in one place.
Molly’s initial goal was to consolidate her tooling, but what she found with Honeycomb were benefits beyond easing the immediate pain of service fatigue.
“Honeycomb really gives us some great insight into how users are interacting with our platform,” she said. “It’s been super helpful not only for debugging issues, but allowing me to find places in the application where slowdowns are happening, we’re hitting the database too many times, etc.”
It’s no coincidence that this was Molly’s first project after joining the DEV team. She gave a talk at SREcon19 last year on the benefits of an easy-to-use, centralized monitoring system. “The ability to have all that information in one spot and then be able to really drill down is super convenient,” she said. “Having it all in one place has been super helpful.”
Additionally, streamlining their tooling has simplified onboarding engineers at DEV. “They no longer have to learn five, six, seven different tools,” she explained. “When they come on, I can say ‘Here’s Honeycomb, this is how you use it,’ and that’s it.”
“Sometimes people think when you’re an SRE that we just magically know, but we don’t,” said Molly. “Half of what we’re doing is literally just throwing in queries, hitting stuff. We look at a graph, ‘Ah there’s nothing there.’ Okay let’s look at a different one, ‘Ah there’s nothing there.’”
Molly makes videos for her teammates demonstrating how she uses Honeycomb to help them learn the tool, and seeing her exploration process has helped them understand that it’s more about mindset than expertise.
“I think that was a big aha moment for some people,” she said. “There’s no shortcuts to what we do, it’s basically getting all the data and finding the pieces that stick out.”
Honeycomb enables first-principles debugging. That’s when you don’t have to be an expert in the system, you just need to know where to start digging. Demystifying debugging as a process can empower an entire team.
Getting Up and Running
Molly described her experience adding Honeycomb’s Beeline gem for Ruby. It was simpler than she anticipated.
“I didn’t do that much,” she said. “I feel like I cheated!”
“I literally just took your documentation, slapped it into the project and it was like, Boom! there’s all of your data,” she said. “It was pretty awesome.” DEV is open-source, so you can see the exact change she made in her pull request on GitHub:
- add
honeycomb-beelinegem to Gemfile - add Honeycomb API key to secrets
- generate Honeycomb configuration using the Beeline’s Rails generator:
bundle exec rails generate honeycomb YOUR_API_KEY --dataset rails
Once Molly’s PR got merged, the DEV team was sending data to Honeycomb and gaining new insights.
“It was super straightforward, and honestly that got us probably 80% of the way there,” said Molly. “Any tool where you can get up and running so quickly is huge.”
By using the Ruby Beeline first, Molly reduced the overhead of starting out compared to instrumenting the entire application from scratch. “If you can get value out of it before you even have to customize? That’s a pretty big deal. If you have to go into the weeds before you can get value? That’s a blocker for a lot of people.”
Once she used Honeycomb with production data, Molly thought of specific areas she wanted to instrument in more detail. Observability isn’t all-or-nothing. The incremental approach is a lot less scary.
Using the Tool
Honeycomb has several features that encourage users to dive in and start querying. The suggested queries on the home page help users get oriented.
“That’s one of the first things I start most of the videos with,” Molly said, referring to the instructional videos she makes for her teammates. Looking at the home page, Molly said, “something sticks out and then I kind of dive into it.”
Another feature of Honeycomb that Molly appreciates is the query builder, in particular, the autocomplete. “That helps immensely in trying to figure out what fields you want to query for.”
Rather than having users type into a blank query editor, the query builder reduces friction by breaking up clauses into separate fields. There’s no bespoke query language to learn, it’s SQL but more visual and with suggestions.
(Read more about the query builder.)
With the query builder you don’t have to memorize all the fields the app is sending to Honeycomb. This evens the playing field among team members.
Molly described to her coworkers the mindset she has when investigating interesting data in Honeycomb: “Sometimes I’m not even quite sure how I’m gonna filter down into what I want, but I just start typing and I start scrolling and I start looking.”
BubbleUp is another unique feature in Honeycomb that Molly finds super helpful. “Let me tell you how many times I’ve been on other platforms where all I want is to just narrow down that graph with a drag and drop, and you can’t.”
Rather than trying to correlate system behavior and line up time periods across separate graphs, BubbleUp does the work of comparing the baseline to the outlier data you’ve selected.
Shared History
Another Honeycomb feature Molly loves is the saved queries. “Usually I’m opening like, tab after tab after tab after tab trying to save all my queries. I go down one path and it doesn’t return anything, so then I have to go all the way back.”
Honeycomb automatically saves your query history and displays them in the sidebar. “Having those saved queries, you can literally just go over to your sidebar, you can easily go back, you can go down a rabbit hole and come back to your starting point in seconds. That’s one feature that’s really stood out to me in Honeycomb that I haven’t really seen anywhere else.”
The shared history and query permalinks make it easy to collaborate and get a second set of eyes, or to prevent multiple people from hitting the same dead ends. “If you’re having an incident you can slap the link to people. They can see your history as well so then they can recreate what you did and possibly find issues with it, possibly take it down their own path.”
Level Up Your Team
If you’re a Rails developer, you can add the Beeline gem to your app and start sending data today. Go to honeycomb.io/signup to start for free.
