Intro to o11y Topic 5: Dataset Troubleshooting

If Honeycomb says I have no data, or my dataset does not appear:

Jessica Kerr [Developer Advocate|Honeycomb]:

Let’s do some troubleshooting. You’re not seeing any data in Honeycomb for the data you set it to. Say your data set was waning harvest moon. And you’re either not seeing any data at all in Honeycomb, or as in this data, you don’t see the waning harvest moon dataset at all. It’s like you’re not seeing anything.

Okay. What could be happening? First of all, when I run the app, it prints out the API key it’s trying to use and the dataset, both of those are wrong. Okay. So my environment variables that I thought I was defining are definitely not in there. Let’s look at the code that prints this. All right. It’s this console.log here exporting to Honeycomb with API key.

And that’s coming from process.env.Honeycomb-API-key. All right. So am I defining the correct environment variable? Control C over here, control V. Yes, that part’s right, but it’s not getting in there. All right. I think it should get in there. I mean, you can set these environment variables however, But I’m trying to get it in there through .env and let’s check whether .env is being read.

Yes. The .env.config call is being made and let’s make sure I have a .env file… Oh, I don’t have a .env file. Okay. I made these changes in .env.example. Well, that’s easy to fix. I’ll create my .env file. All right, let’s run it again. Great! This time, it’s seeing my API key and my datasets go to local host:3000. Click go and stop, back over to Honeycomb.

Surely I’m going to see my dataset now. Refresh. I would definitely have to refresh the page to see it in this list. It’s still not in this list. I can go check over in this other list that shows me all my datasets. Not there either. Okay. What else could be wrong? This API key, is it right? Let’s see, it starts with 830.

I will go to my team settings and see that 830. Oh dear. My API key is not in my list of API keys. What have I done? Probably this API key goes to a completely different team or maybe it’s been deleted. All right. Let’s take one of the API keys from here and put it in. Restart the app. It should start up with a new API key.

Ah, that’s the old one. Oh, dang it. Look, I pasted it into .env.example. This needs to go into .env. Paste it there. Close that one so I don’t do that again. And let’s try it yet again, go and stop again. All right. Now, do I have my new dataset? Refresh this page. No, dang it. Check the dataset list.

Refresh. No. All right, what else could be wrong? We have the correct API. Why isn’t this going to Honeycomb? All right. Let me give you a really useful tip for working with open telemetry. Open telemetry is the instrumentation library that we’re using here. And one of the secrets is that if you have a problem, turn on logging by default open telemetries log level is silent.

It’s like, oh, don’t let my errors get in the way of your application. I’m only tracing so we can turn on logging in tracing.JS. Look for this line that says open telemetry.diag.setlocker, and commented out. This has a log level of warn. You could set that to debug, but warn is usually sufficient.

Let’s see what happens when we restart the app. Now. Okay. It starts up. All right. Now I want to trigger some telemetry. Go stop. Come back to look at the terminal and aha. I have an error message this time. Unauthenticated API key can not create datasets. Oh, all right. This data set waning harvest moon does not exist yet.

And I specifically wanted to create it, but apparently I don’t have the authority to do. Let’s go back to Honeycomb, back to team settings, six two seven. That was my API key. Aha! It only has send events access. I can create a new one and this one, I don’t need these others, but I do need create datasets because I’m trying to get my telemetry to trigger the creation of a new dataset.

Copy that one, put it into .env. The correct file this time run the app anew. Okay. The new API key is here. Cross our fingers, load the page. Push go and stop, go and stop. Go back to Honeycomb. Check our data set list again. Oh my gosh, there’s waning harvest moon! That’s wonderful. I can click on the logo to get to the home screen. I can select waning harvest. And I see a spike of data over here. Hooray!

If Honeycomb says there are no results within the specified time range:

Jessica Kerr [Developer Advocate|Honeycomb]:

All right. So you have the dataset you’re using, let’s double-check this, says hello-observability and in .env it says hello-observability. And when I run the app, it prints out that it’s going to dataset hello-observability. That all looks okay. But on the Honeycomb home screen, It says no results in this time range.

Dang it. Okay. What could be happening? Hmm. One thing I like to do is go to this datasets list and I can see here’s my dataset. It was created today and data was last received 34 minutes ago. 34 minutes ago, hmm. In this case, It has been a while. I mean, I went and had coffee since the last thing I did, which was to hit go and stop for a while.

So maybe that coffee break took longer than I thought. When I come back to the home screen, I can check this duration dropdown over here. Oh, look, I’m only looking at the last 10 minutes. Let’s try the last 30 minutes. Wait. It said 34 minutes, didn’t it? How about eight hours? That’s excessive, but yeah, sure enough, now I can see some data.

Another way to find out what’s happening is as I scroll down here, I see some recent traces from 36 minutes ago. That’s useful. If you don’t see anything there, try looking at recent events and see whether you see anything there. If you do see events but not traces. Okay. That’s out of the scope of this video. See the end of this topic for how to get more.

Transcript