Intro to o11y Topic 12: Customizing the Telemetry in Honeycomb

Jessica Kerr [Developer Advocate|Honeycomb]: 

Whenever I’m about to change code, I start with: what do I expect to change? I want to see the value of the index parameter on some spans. Currently, it’s there in some spans, these yellow-highlighted ones, but only as part of the value of http.url. I want to see it in a field by itself, a numeric field even. I can add the url as a field and sort-of-see the index. My goal is that I can add index is a field I’ll see five, four, three, as numbers on some of the spans.

OK, now that I’ve seen what it does now and what I want to be different, it’s time to look at code. I’m using VSCode. Any IDE will do. I’ve cloned the intro-to-olly-nodejs repository from honeycomb.io. I’ve also run `npm install` and set up my `.env` file to send data to Honeycomb.

But wait, it’s a good idea to send data to Honeycomb during development but I want to keep that separate from production data. I’ll use a different API key than for production. It needs permissions to send events and create datasets. That’s because I’ll definitely use a different dataset for my local development work. This is important. I’ll call it dev-hello-observability.

Next is to run the app and verify the existing behavior. Now it’s running, so I’ll open the app and click go and stop to generate some requests. Looks like it’s working. Now over to Honeycomb. Hit refresh. Is my new dataset available yet? There it is. And there’s a few data points!

I scroll down to recent traces, any of these for the /fib endpoint should work. I’ll click view trace. Sure enough, I’m getting spans And sure enough, I see the index only within the URL. Current behavior is verified locally. This trace also gives me a clue of where to start looking in the code. Where is the code that responds to /fib?

Over in the IDE, I’ll look for that endpoint. Hmm, there are three mentions of this string. README, that’s not code. Index.js has “app.get/fib” – that’s very promising. Indeed, this is where our program teaches the Express web framework how to respond to /fib.

And look! Here, first thing, it pulls that index parameter out of the request. This line of code is another indication that this value is important. We should definitely add it as an attribute to the span. What we want here is a span, and we’re going to set an attribute on it. The attribute gets a name and a value. 

Right now I want to prove that I can add an attribute at all, so let’s call it pizza and give it the value pizza. In Honeycomb, attribute names and values accept unicode characters, such as 1F355, slice of pizza. We need a span, not creating a new one; We want the existing span that’s active while this function is executing. 

We can get that from the opentelemetry libraries. They make some global variables available. One of them is the trace that’s in charge of creating spans in this app.  It can get us current span from a context. 

We have to give it that context, and that’s also available from the library as a global. Remember that tracing is all about putting data in context. This context object maintains this, stuff like the current trace ID and span ID. The libraries handle all that tracking.

The library to use here is the opentelemetry API. This is the only import you should need in your application code, to customize your telemetry.

In a separate file, here it’s called tracing.js, there are a lot more imports to configure autoinstrumentation and sending to Honeycomb. Someone needs to set that up once, and then everyone on the team can use the opentelemetry API to add attributes, add spans, and make the traces communicate what matters to your app.

I’ll put this Require up with all the other imports. It’s time to try it! I’ll stop the app and rerun it. I’ll open the app, then click go and stop to generate some traces. Go to Honeycomb, check our dev dataset for fresh data. It only takes a few seconds for it to show up, ready for querying. Here’s a recent trace to /fib. OK, do any fields have our new attribute? Ah, yes, some of them do. Let’s add it as a field.

Sure enough! There are slices of pizza. Haha, pizza. I can now query this dataset based on pizza. When in doubt, count. I’ll make a count visualization And group by pizza. See how the field shows up in autocomplete? The schema for this dataset is forever cluttered with this pizza fieldname.

This is why we use a different dataset for development. In development, we’re supposed to experiment with different attributes, until we find what works well, what answers our questions, and then we push that to production. Here it is, all our spans, with and without pizza. OK, enough pizza, let’s give our new attribute a real name and value.

Now back to this particular attribute. It’s time to put in the name and value we want. We know the value, that’s easy. It’s the index that was passed as a parameter to this request. What shall we name it?

Pointer: always use hard-coded strings for attribute names. They should always be constants, and sometimes you’ll want to search the code to find out where they’re set. We namespace attribute names with dots. It’s useful to have an “app” namespace for fields that are significant at an application level. It’s useful to have a namespace specific to this app, in case this same field means something different elsewhere. Personally, I want to make it clear that this attribute is incoming And finally, this attribute’s name is index.

Let’s try it out and see how it looks. Restart the app, open it, push go and stop to generate some traces. Now I want to look at them. Scroll down to recent traces, click on one from a few seconds ago. Let’s look for our new field. Aha! There it is! And now indeed, I can see the values of index, by themselves, on these spans. Looks like it came in as three, then we get this HTTP-request-to-itself with an index of two, and within that there’s another that’s a one. Before I get farther into this trace, it’s time to push this new telemetry code to production.

Transcript