Instrumenting With OpenTelemetry

Michael Sickles [Solution Architect|Honeycomb]:

Hello, my name is Michael Sickles. I’m a Solutions Architect here at Honeycomb.io. Today I’m going to be talking about instrumentation with OpenTelemetry, why we do it, how it’s useful. And really how to get that data flowing from your application into Honeycomb.    

So we’re going to begin, first, on following that instrumentation data. And using OpenTelemetry, there are multiple ways that tracing data can flow from your application to Honeycomb. One of the easiest ways to get going and started is to use the OpenTelemetry SDKs or the JVM Agent to send directly to Honeycomb. 

So we can accept OTLP data directly, that’s OpenTelemetry line protocol. In which then you’re able to right away start asking interesting questions about your application. However, as you add more information, as you get more complex, distributed services, you might instead want to have your data flow through an OpenTelemetry Collector. And the reason why you would use an OpenTelemetry Collector is it has what’s called a pipeline—so it can receive, process, and export that data, meaning that you can receive multiple formats. You might not just be using OTLP data. You might be using Zipkin or Jaeger, and it allows you to utilize that existing instrumentation you have in place to then change it into the OpenTelemetry format to send to Honeycomb. Furthermore, you might then process that data.

Maybe you have attributes in there that you want to hash or delete or maybe even add. So grab Kubernetes pod information and add it to all the spans moving through this collector. And then finally, you want to export it. Naturally, export it to Honeycomb. But you could export to multiple locations if you need to. So this really gives you full control of how your data is being received, processed, and moved around. Finally, you could also send your data through an OpenTelemetry Collector in agent mode. And so it’s an OpenTelemetry Collector; it can do the same thing in that it can receive, process, and export that data, but it’s closer to the application. 

So now maybe you want to do some sampling before it even leaves your application. That data is sampled out; it’s not going to be going towards that egress cost as it has over the wire. Or maybe using this agent, you can grab extra information. So you could grab host metrics on the application, and then send those host metrics off to Honeycomb directly, or even go through another OpenTelemetry Collector in gateway mode to, once again, kind of have this one location where all your OpenTelemetry data flows through. Gives you only one place to have to configure and send off to the vendor of your choice. (Hopefully, Honeycomb.)

3:05

Now, we have this data. We want to export it. Let’s walk through how you set that up, right? How do we export data? I keep saying export because OpenTelemetry has this concept of an exporter. So an exporter is going to take the endpoint—where is that data going—Honeycomb.io or api.honeycomb.io. And then it’s also going to take your Honeycomb-specific information, the Honeycomb team is your API key. And then the x honeycomb dataset will be the name of the data set. 

So in Honeycomb, you’re going to group very same services, maybe like prod, right? One environment, prod together could be a data set, or maybe you have multiple applications that logically group together. Once again, that’s going to be the name of your data set. So here’s an example: In NodeJS, you’re going to take the header information, that’s the metadata variable. You’re going to set up a new exporter. You’re going to export to Honeycomb’s API. And you’ll be off to the races.     

Now we also offer Honeycomb-specific distributions of OpenTelemetry. For Java and .NET currently, although we are working to add more. And what this allows you to do is really get going faster. It takes a lot of the configuration around that exporter and builds it out for you, but with Honeycomb-specific information. 

So now all you have to give it is that Honeycomb API key, that data set, the name of your service. And so for the Java agent, it’s downloading a jar, attaching to it the Java agent, and you’re off to the races. For .NET, it’s using an SDK on application start, send those three information, service name, API key, data set, and you’re going to start getting .NET services. And then information. And then they’re going to have some auto instrumentation, which I will talk about later.

With instrumentation, there are going to be two main types. You’re going to have automatic instrumentation, and you’re going to have manual. So with automatic instrumentation, it varies by language and how to set that up. Java just uses the Java agent. It has a whole bunch of libraries and frameworks built into it. This is just a small set of that. 

A lot of the other ones like Node and Go, they have what’s called instrumentation packages. Basically, you’ll use your package manager to download the different OpenTelemetry packages. It will automatically wrap those libraries and give you tracing information. So when did a specific call start and end? It will automatically grab exceptions. 

5:48

And the benefit of auto instrumentation is a lot of these libraries are commonly used by tons of different organizations, HTTP calls, database calls, AWS SDK calls, right? And you don’t have to manually set up these tracings. They’ll automatically trace for you. They’ll talk across services for you. Automatically pass tracing headers so that you don’t have to do the work. And this is great. This is going to get you going and started quickly. That’s what we want to do. Less work to get going. But then you’re going to want to ask the follow-up questions.

You’re going to want to ask something like: My customer, I want to know which of my customers is having a bad experience. So we need a customer ID field. The auto agents don’t automatically pick that up, unfortunately. This doesn’t know what variables you use, so you’ll use the OpenTelemetry SDKs. And they all follow the same pattern. You can see on the left-hand side that there are things like setting attributes. There are things like setting exceptions and adding events, which are basically logging information for a span. And we can then take that variable, customer ID, add it to a span, and questions about it. Which is great. Not only that, all the languages have different ways to grab the current auto instrumentation span and add attributes to that as well. So that way you can utilize that auto pieces for a lot of the very common libraries but then add detailed information like that customer ID. So you can then start asking follow-up questions, getting to the root cause.

So this is how it looks. We have a trace here. You’re able to see across multiple services. It could be different languages. OpenTelemetry will automatically pass and create the different tracing headers, so trace ID and span ID; it automatically does that all for you. It is not something you have to handle at all. And you’ll get these great descriptive traces to then start finding bottlenecks; e.g., where is the time spent in a given backend call?

And so we went through, started at the top. How is data flowing from my application into Honeycomb? 

We talked about how setting up an exporter works, which is exporting data from your application to Honeycomb. 

And then finally, we ended here, instrumentation. How do we set up instrumentation, how do we get going, whether that’s via automatic instrumentation or manual instrumentation? Thank you.

If you see any typos in this text or have any questions, reach out to team@honeycomb.io.

Transcript