We’re excited to announce that we’ve closed an $11.5M Series A round with our friends at e.ventures, Storm, NextWorld, and Merian! It’s been a wild ride since we launched back in April of 2017. As we continue to grow at a rapid pace, it’s time to hire way more people and get some more hands (and hearts and heads) to help build Honeycomb.

Giving THANKS

  • THANKS to all the customers that have placed their confidence in us, all the new people who have joined our team, and to everyone in the community that’s been helping us build a better Honeycomb for you!
  • THANKS to new customers Fender Musical Instruments, Axios, Travis CI for lending their names to our announcement!
  • THANKS to our investors and their teams—Tom, Jett, Tae Hea, Ben, Alexsis, Rachel–for being as excited about this as we are!

As Christine says, “Observability is as essential for developers as for operators when it comes to understanding what’s happening with our code. It’s all about answering questions about your system using data—and during the development process we’ve got lots of questions that aren’t necessarily problems or anomalies: the questions are about what ‘normal’ is, which customer or cohorts are experiencing something specific.”

That’s why we’re here—to empower you to see what your code is doing, when it’s doing it, from development through production and back.

Honeycomb Raises $11.5 Million Series A to Make Observability Part of Every Software Engineering Lifecycle

e.ventures and Storm Ventures Lead Financing and Join Board

San Francisco, CA — Jan 31, 2018 — Honeycomb, the leading observability provider for software engineers, announces an $11.5 million Series A financing led by e.ventures with participation from existing investor Storm Ventures and new investors NextWorld Capital and Merian Ventures. This round brings the company’s total funding to $15.5 million.

“Honeycomb uniquely addresses a major pain-point for anyone debugging and trying to understand distributed systems, providing BI-like experience for software engineers. We’re excited to support the Honeycomb team and technology through their next stage of growth.” – Thomas Gieselmann, General Partner at e.ventures

Honeycomb is founded on the experience of building and operating a platform hosting millions of apps serving tens of millions of users. The product enables engineers to easily create and analyze data about the behavior of their own software, so they can instantly go from visualizing aggregate trends to pinpointing the exact shopping cart stalling on a transaction, or query starving a system, or user experiencing or causing a problem.

“We are amazed by the love that Honeycomb’s customers have for them. It’s rare to see this kind of positive feeling for commercial developer or infrastructure tools.” – Tae Hea Nahm, Managing Director at Storm Ventures

Honeycomb intends to use the funds to significantly expand sales and marketing, as well as deepen R&D focused on making high-cardinality, event-driven observability accessible to a wider market.

“A new metrics, monitoring, or logging company enters the market every month. And yet, the pain of debugging, or even understanding, distributed systems goes unaddressed. We believe an investigative, exploratory approach is required to understand the behavior of the kinds of inherently complex systems we’re building today. Honeycomb’s mission is to provide engineers a way to build and check their intuition about the behavior of their code as it gets manipulated and worked in real life. This is what we mean when we say observability.” – Charity Majors, CEO and co-founder of Honeycomb.

Honeycomb makes the process of developing and testing hypotheses about the world of your systems accessible without any specialized knowledge other than the desire to ask a question.

“Customers tell us that Honeycomb solves problems that they believed for years were impossible, or prohibitively expensive, to solve. Because that’s what their vendors told them.” – Christine Yen, CPO and co-founder of Honeycomb.

Undoubtedly, you recall the critically acclaimed (some might say “groundbreaking”) blog post titled “Knowing is Half the Battle” which literally changed the way billions of Honeycomb users viewed and experienced their Honeycomb Usage Centers; and is also literally half the tagline from GI Joe.

I’d like to share a passage from that post that I have kept close to my heart since that fateful day:

“Wouldn’t it be swell if you could manipulate some levers to alter the distribution of the existing eps and storage across your datasets? Hang in there, because we’re hard at work to bring you more control over your usage and aim to provide some key additions around managing distribution, permissions, and additional capacity.” - Peter Tuhtan, July 20th, 2017

Well, lookie lookie, we all get cookies:

star wars cookies capacity management screen, overview

Behold! Self-service capacity management! All users can still visit the Usage Center to find out if there is unused capacity to leverage, but as an account owner, if you navigate to your Honeycomb Usage Center you’ll now be able to control the distribution of your storage manually.

capacity management screen, account owner controls

Account owners can simply select “Manage” to use a dataset’s slider to alter what portion of your overall storage it will be receiving.

You’ll always be able to get a high-level understanding of how much unallocated storage you have left by visiting the Usage Center’s homepage. Exhibit A:

capacity management, unallocated storage

But if you’re like me…a loner, a rebel; sometimes you just have to bend the rules–and this is what that looks like. Exhibit B:

capacity management, overage

But! I’m also responsible 96% of the time, so I either leverage the new features of my Usage Center by using the “Manage” option to reduce my used capacity back into my storage limit, or I use the “Upgrade Plan” button to notify Honeycomb that I would like to chat about increasing my capacity plan.

Han Solo makes a call

We plan to support self-service purchasing via the “Upgrade Plan” button someday, but until then we’re here to help you with that. We’re also always here to discuss ways for you to get more out of your capacity besides spending more, so if smarter sampling, or other changes to the ways you’re sending data are of interest to you, don’t be shy :)

If you have any questions or feedback, let us know: write in to sales@honeycomb.io & support@honeycomb.io.

Cheers all, and comb on.

We’re grateful for this guest post from Tim Wilde! You can find the source code for the examples he uses in his github repo.

Strings are where data go to die

There you go; I said it.

How often have you found yourself contemplating some hair-brained regex scheme in order to extract an inkling of value from a string and wishing the data had just arrived in a well-structured package without all the textual fluff?

So why do we insist on writing prose in our logs? Take “Exception while processing order 1234 for customer abc123” for example. There are at least four important pieces of information drowning in that one sentence alone:

  1. An exception was raised!
  2. During order processing
  3. Order number 1234
  4. Customer abc123

Being an exception log message, it’s more than likely followed by a stack trace, too. And stack traces certainly don’t conform to carefully crafted log layout patterns.

I’ve been thinking about this recently. We’re currently building new systems with ambitious plans for a Microservices architecture and we’ve chosen the Elastic stack for log aggregation. First we need to get the logs from all of our services in there, so we’ve started to standardise logging layout patterns so they all conform with our Logstash grok filters.

Hold on, though. So our log layout patterns are now essentially just a data transit format?


Pretty much: we have structured data in the app which we squish into strings for logging — losing detail in the process. This is then converted back into structured data by Logstash. Lossy and redundant, right?

Structure your Logs

Then, perfectly timed, Charity tweeted this:

What does “structure your logs” mean? Simply put:

log entries should be easily machine-readable.

So instead of string interpolation, use JSON serialization. The example earlier might become:

{
   "timestamp": "2018-01-13T11:45:12.483Z",
   "level": "ERROR",
   "message": "Order processing exception",
   "data": {
      "order": 1234,
      "customer": "abc123"
   },
   "stacktrace": "... <stack trace here>..."
}

Notice how the message does not contain the data? Now we can very easily count Order processing exceptions over time. Much more valuable!

A Practical Example

Let’s put together a quick example comprised of an Elastic stack and a little .NET console app which logs to it via NLog. This will demo how to get the log messages into Elasticsearch, via Logstash, so we can view them using Kibana’s Discover function.

(TL;DR: There’s a link to the code at the end of the post.)

The Elastic stack

We’ll spin up Elasticsearch, Logstash, and Kibana in Docker to make it quick and easy to provision and clean up when we’re finished. For this we’ll need two files: docker-compose.yml which describes the stack, and logstash.conf which configures how we want our log messages to be ingested.

docker-compose.yml:

version: '3'

services:
  elasticsearch:  
    image: elasticsearch
    command: elasticsearch -Enode.name=derp -Ecluster.name=logging
    ports:
      - "9200:9200"

  logstash:  
    image: logstash
    command: logstash -f /etc/logstash/conf.d/logstash.conf
    volumes:
      - ./logstash:/etc/logstash/conf.d
    ports:
      - "5000:5000"

  kibana:  
    image: kibana
    ports:
      - "5601:5601"

While it’s far from a production-ready config, this does give us a usable set of services. The important part being that we map a logstash folder with our configuration file into the Logstash container, which saves us building a custom image, copying in the config file, etc.

logstash.conf:

input {
   http {
      port => "5000"
   }
}

filter {
   json {
      source => "message"
      remove_field => [ "message" ]
   }
   date {
      match => [  "timestamp", "YYYY-MM-dd'T'HH:mm:ss.SSS'Z'" ]
   }
}

output {
   elasticsearch {
      hosts => [ "elasticsearch:9200" ]
      index => "logs-%{+YYYY.MM.dd}"
   }
}

Compared to some Logstash configs, with mind-bending grok filters, this is a breath of fresh air. Nice and simple.

We’ll be receiving log entries via HTTP POSTs to port 5000.

When they arrive, we expect them to contain the JSON-encoded log entry in a field called message which is deserialized and all of the properties recorded in Elasticsearch. There’s a custom date formatting pattern so we can grab that correctly as well.

Finally, we want to send the result to our Elasticsearch server on port 9200 and we specify which index to use.

With those in place, we can spin up the stack:

docker-compose up --abort-on-container-exit

(I use --abort-on-container-exit while putting a docker stack together so problems will cause it to fail fast and the reason is easier to spot)

We should be able to hit http://localhost:5601 now and Kibana will ask us to set it up. We’re not quite ready to do that yet as we don’t have anything logged, so let’s get the console app done next.

Console App

This is as simple as possible. Just a .NET Core 2 console app with a couple of Nuget packages:

The NLog.StructuredLogging.Json package is by the developers at Just Eat. It provides a really handy layout which will serialize an NLog event to JSON — perfect for our needs!

We want NLog to post the log entries to our Logstash instance, so we’ll need to edit the nlog.config file as follows:

<?xml version="1.0" encoding="utf-8"?>

<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xsi:schemaLocation="NLog NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true">

    <extensions>
        <add assembly="NLog.StructuredLogging.Json" />
    </extensions>

    <targets>
        <target name="console"
                xsi:type="Console"
                layout="${structuredlogging.json}" />

        <target name="logstash"
                xsi:type="Network"
                layout="${structuredlogging.json}"
                address="http://127.0.0.1:5000" />
    </targets>

    <rules>
        <logger name="*" writeTo="console" />
        <logger name="*" writeTo="logstash" />
    </rules>
</nlog>

I’ve included a console target so we can inspect the log entries that are being sent.

Even with a centralised log aggregation system, it’s still wise to log locally too - you never know when you’re trying to report problems because the network is down and you don’t want to lose log entries!

The logstash target is a simple network target which will POST the log entries to the address specified. As our docker-compose.yml binds the logstash ports to the host (our local machine), we can just use the loopback address here.

The ${structuredlogging.json} layout will turn our log messages into JSON, something like this (formatted for readability here):

{
   "TimeStamp": "2018-01-13T00:29:50.087Z",
   "Level": "Info",
   "LoggerName": "LoggingExample.Program",
   "Message": "Extended Greetings!",
   "CallSite": "LoggingExample.Program.Main",
   "WordOfTheDay": "Jibble"
}

The last thing to do with the app is actually write some log entries!

private static void Main(string[] args)
{
   LogManager.Configuration =
      new XmlLoggingConfiguration("nlog.config");

   ILogger logger = LogManager.GetLogger(typeof(Program).FullName);

   logger.Info("Hello World!");
   logger.ExtendedInfo("Extended Greetings!",
                       new {WordOfTheDay = "Jibble"});

   logger.Factory.Flush();
}

This quickly configures NLog using the nlog.config file and grabs an ILogger instance. All standard NLog stuff.

Next we use the normal Info call so we can make sure any existing log messages will benefit from JSON encoding (which they do.) This is critically important as it means it’s not necessary find and replace every single logging call throughout the codebase straight away. Just don’t forget about them, OK?

The NLog.ExtendedLogging.Json nuget package provides a few handy extension methods, like ExtendedInfo above, which take an object or dictionary to be used as key-value pairs that are serialized to JSON as part of the log entry. These are the calls we want to be using going forward so we can send data, instead of squishing it into a string.

Now we’re ready to run the console app and then go and see what Logstash received!

cd LoggingExample
dotnet run

We should see a few lines of JSON in the console and, hopefully, we now have log entries we can play with. Let’s go back to Kibana (http://localhost:5601) and check.

Configure Kibana

Back in Kibana we’ll be asked to configure the index pattern again. It will suggest logstash-* but we’re using an index with a logs- prefix, so change this to logs-*. This will then inspect the index and grab all the field names from our log entries, which is why we had to get some log entries into the index before we could progress.

In the Time Filter field name field there are two timestamp options: @timestamp and TimeStamp. The former is when the log entry was recorded in Elasticsearch and the latter is the timestamp generated by NLog. Select @timestamp and click the Create button and we’re done.

It may seem odd to have two timestamps, and we could have Logstash remove one of them if we really wanted to, but we’ll do something interesting with these a little later on.

Playing with Data

Next we can see what Logstash has received from our app. Click the Discover link (with a compass icon), top-left. We should be able to see the log entries from our console app.

More importantly, all of their properties are separate fields and can be used to filter and search and that is why we want structured logs :)

Kibana will let us create calculated fields too. As we have two timestamp fields, we could calculate a timestamp delta which contains the delay between the moment the log entry is created in our app and when it arrives in Elasticsearch. This could be a useful statistic for monitoring network latency between the two, for example, providing us with another health check datum for free.

To quickly set that up:

  1. Click the Management cog icon on the left;
  2. Click Index Patterns on the next page;
  3. Click the scripted field tab;
  4. Click the Add Scripted Field button;
  5. Enter TimestampDelta in the Name field; and
  6. Enter doc["@timestamp"].value - doc["TimeStamp"].value in the Script field

Leave the rest of the fields as they are and click the Create button at the bottom of the page, then return to the Discover page (Compass icon, top-left) and expand one of the log entries. You should see a new field called TimestampDelta with the difference between the two timestamps of the log entry, in milliseconds.

I’m not going to describe how to use Kibana here (I’m still learning myself) but now it is a lot easier to get log events from an app into the Elastic stack and then to filter, search, summarise, and visualise the information they contain.

As the data is no longer hidden inside arbitrary strings, it’s possible to quickly ask questions of the data and uncover useful information and insights which would not have been possible before. Well, not without some gnarly regex patterns…

I will be using this to create dashboards to monitor our service statuses, and that’s just to begin with. I’m sure there is a lot more we can think of once we start exploring the data in the time we save not having to parse log messages.

And finally…

If you’d like to grab the code for this post, I’ve shared it on Github.

The docs for the Just Eat Nuget Package have some useful “best practices” for structured logs.

And thank you, Charity, for the tip and all the time and frustration this will save me in the future :)

Almost everybody hates interviewing.

You aren’t wrong to hate it: interviewing is fucking broken, in ways that tear you down and rob you of your self-respect and your will to live.

wayne's world gif of the flowbee

If interviewing were a dreaded yet rigorous process that delivered a high degree of confidence and accuracy in the results, that would be one thing. But it’s not. Predicting who will succeed and be happy is scarcely better than a coin flip, even at Facebook and Google. Your own personal enthusiasm for the hire rarely predicts your top performers or your strugglers. Meanwhile, it’s fraught with biases and risk aversion.

At Honeycomb we have dissected all the things wrong with all of the interview processes we have ever been part of, and decided to err in new and exciting ways. Maybe we can’t do better than the industry mean, but we could hardly do much worse. Why not experiment?

The Honeycomb Hiring Experiment

We believe a good process starts with attracting the kind of candidates we value, puts the candidate at ease and shows them at their best, and gives both us and the candidate a semi-realistic sense of what it feels like to sit together and work on something. We want to be respectful of their time, we want them to leave feeling valued whether we hire them or not. And we want to keep the “must-have” skill list as short as possible…no laundry lists. After all, almost any skill can be learned.

We believe bad processes make people memorize things or study to prepare. Bad processes are adversarial or unpredictable. But a more subtle point is that bad processes can be bad because they don’t produce enough signal to guess whether the person will ultimately be successful on the team. We could design a process that anyone could cruise through and come away loving us … but we’d end up having to manage a lot more people out afterwards. More heartache.

Our interview may feel “easy” to some. It’s not. We have an extraordinarily high bar, in that we don’t hire the large majority of candidates. But our bar is tuned for many criteria that are not algorithms and data structures. Here are some of our interview goals:

  • No surprises. The research is clear: unknowns cause anxiety, and people don’t perform well when they’re anxious. We willingly offer up as much detail in advance about the process as we can, even down to the questions. The hiring manager will walk you through what to expect in advance, and tell you what we’re looking for in your answers.

  • Realistic code. Obviously we don’t do whiteboard coding: it’s terrible. The coding exercises we use involve extending or improving existing code…from home, the night before, in the ease of your own dev environment and without an interviewer watching. The code should relate to actual work that you would actually do, and…that doesn’t usually mean green field, does it?

  • The communication is the interview. Plot twist…the “coding interview” isn’t really about the code. When you come in for your interview, you’ll talk through your PR with a couple of engineers, and that is the meat and bones of the interview. We believe that if you can communicate clearly about your choices and the tradeoffs, you can certainly do the work, while the reverse is not necessarily true: there are people who can do the work but can’t communicate clearly about it, and that doesn’t work for us. We need both. (We have to teach the world about distributed systems, after all!)

  • Product and production. Our other questions should explore your product sense and your production skills … your taste when it comes to shipping user experience, and the impact you will have on the team around you over the lifecycle of your code.

  • See every candidate at their best. We don’t hire people for lack of weaknesses, duh. We hire them for their strengths. We try to ask everyone in advance, “what can we do to see you at your best?” We aren’t interested in dwelling on areas where you already know you are weak. Don’t bother trying to fake it! Instead, help us learn what makes you amazing.

  • Interview us! We try to allow plenty of time for you to interview us as well. We invite all candidates to come to our engineering meetings and/or all-hands, and usually set up a “get to know us” visit separately from the interview.

How is the Experiment going? Join us and find out!

At this early stage, every single person we hire will leave a lasting imprint on who we are as a company. Who we hire is a mix of skills, interests, personality, background, and availability.

The big companies will hire anyone who “passes” the interview; for us, it’s like delicately playing Jenga with people. We have had to regretfully turn away lots of terrific people with great skills that we simply don’t need at the moment.

And we value diversity a lot. We could have hired all our awesome former coworkers instead of casting a wide net and taking a risk on unknowns…but dammit, we are building a product for everyone, we can’t risk developing a monoculture or tunnel vision.

Honeycomb is not for everyone. As our values statements try to convey, this is a high trust environment, where we operate with abnormally high ownership and autonomy. We also talk about our feelings more openly, try to give feedback regularly, receive it with vulnerability, and operate with a level of transparency that is above the norm. These are some of the most important relationships in our lives: we want to invest in them.

As Chris once put it: we’re “more than friends, less than family”. We aren’t trying to be your life — you’re an adult, you already have one! But we are trying to do something really big and hard together, and if you find all of this interesting, we would love to chat with you. 🙃🐝