About

FENDER MUSICAL INSTRUMENTS CORPORATION: Since 1946, Fender has revolutionized music and culture as one of the world’s leading musical instrument manufacturers, marketers and distributors. Fender Musical Instruments Corporation (FMIC), whose portfolio of brands includes Fender®, Squier®, Gretsch® guitars, Jackson®, EVH® and Charvel®, follows a player-centric approach to crafting the highest quality instruments and musical solutions across genres. FMIC is dedicated to unlocking the power of music through electric and acoustic guitars, amplifiers, pro audio, accessories and digital products that inspire and enable musical expression at every stage, from beginners to history-making legends.

Environment

  • AWS Lambda microservices written in Go
  • AWS Lambda sending Cloudwatch logs to Honeycomb

Goals

“Why would you enter into an online space as a guitar company? It’s to know more about players: understand who they are, talk to them, and guide them along the way.” - Ethan Kaplan, Digital GM, Fender (http://fortune.com/2017/11/19/fender-guitar-apps/)

Fender’s customer focus makes observability a key requirement: to understand and support your users, you have to have visibility into their experience of your service.

“The goal of the platform team is to create and support services that support our web and mobile applications and provide data and analytics to the business. In that way we are able to understand and support players.” - Michael Garski, Director of Software Engineering, Fender

What they needed:

  • A powerful and intuitive interface for debugging and troubleshooting problems
  • Fast search results across high-cardinality fields such as unique customer IDs

Honeycomb @ Fender

The Fender Platform team maintains their own ELK environment to aggregate logs and investigate problems, but have found the platform slow and difficult for team members to learn to use.

“We have an ELK stack, but the query language is cumbersome and the UI is not as easy to use as Honeycomb.” - Michael Garski

They installed Honeycomb specifically to get the benefit of searching across all their Cloudwatch logs from Lambda at once, and have been especially pleased with the speed at which they can zero in on an issue.

“Honeycomb and Lambda work very well together. Without Honeycomb, it was very difficult to get all the Cloudwatch logs correlated, but now we can just do a quick search by request or userID with no difficulties.” - Michael Garski

Recently, the Fender Platform team deployed an update to the subscription management service for Fender Play users. Soon afterward, they noticed errors related to the billing systems, which started to spike over the course of the morning.

Using Honeycomb, they determined that the issue was related to differences between their test and production environments and could confirm, thanks to the Honeycomb Marker they’d set, that this problem was definitely caused by their recent deployment. They were able to roll back to a stable environment within five minutes.

graph showing rollout errors

This graph shows the Lambda function status codes during the timeframe of the deploy. The spike in 500s occurred after the first Marker, then dropped after the second, which was the rollback. Subsequent Markers show that a third deploy still had the issue, but it was fixed in the 4th.

Even more critically, they were able to drill into their data and identify the exact users impacted by the issue, allowing their Customer Support team to contact those users before they contacted Customer Support to report an issue.

And all of this would have taken significantly longer using their existing ELK setup.

“If we did not have Honeycomb it may have taken an additional 30 minutes or so to determine what the issue was by poking around in ELK or Cloudwatch logs. Honeycomb’s visualization and honeymarkers made it obvious that the issue was related to a recent release. On top of that, Honeycomb allowed us to determine the affected users and pass that information on to our support team.” - Michael Garski

The team finds it easier and faster to get results with Honeycomb.

“Now that we have Honeycomb, we just use the ELK stack as the archive. “ - Michael Garski

If you think about it, mobile apps are among the production services most in need of real observability: nearly countless hardware platforms and operating systems in combination with your app’s code result in a dizzying matrix of possible sources for any given issue, which means you need the power of true high-cardinality search to solve your problems.

dizzying matrix gif

Have no fear, though–here are some tips for thinking about observing a mobile app:

Take advantage of Honeycomb’s support for high-dimensionality + high-cardinality

In our own production services, we have a standard set of metadata that we know will be useful when debugging production problems / understanding the production “normal,” so we make sure to send them on every event. (You can see some of them listed out in the first section of this dogfooding blog post.) Going through this exercise early on and standardizing among your various mobile clients will make your life easier later on by establishing a set of common field names.

Commonly-useful metadata when slicing and dicing mobile traffic include fields like: device type, OS version, app version, the SDK versions of any crucial third-party SDKs, customer ID. The question to ask yourself at this stage is: what are the attributes / measurements that tend to correlate to customer pain? For an API we measure latency of all sorts; for client-side JS, we measure a whole host of painting/navigation timers; for mobile… what matters most to you, and your use case?

(Important note: anytime you’re writing directly to Honeycomb from a client-side application, be it browser JS or mobile apps, we recommend creating a client-specific write key to retain the ability to rotate a key for a subset of your traffic — just in case a key gets compromised.)

Think through where and when you want to capture each event

We tend to think of Honeycomb events as each capturing a single “unit of work.” On an HTTP server, that often translates naively to one event = one HTTP request served. For mobile, a common pattern (e.g. the Google Analytics SDK relies on this approach) is to trigger an event per screen load — and this is a fine starting point. (Especially if you’re also able to capture timers like “how long did viewDidLoad actually take to run?“)

You may also find yourself wanting to track individual, detailed user actions like:

// User just sent a message!
event.send({
  event_type: "message_sent",
  roundtrip_dur_ms: now - start_ms,
  sender: sender.name,
  payload: message.length,
})

… But wait! There’re a couple of different angles to think about this here. On one hand, tracking an event like this directly from the client side lets you measure latencies as the user perceives them — but if this particular “interesting unit of work” already triggers a round-trip HTTP request to your servers, is it worth adding another HTTP call to Honeycomb’s? What would it look like if you just sent all the metadata along with requests to your server, then in your server’s logic, you extract out the message_sent data?

Upside: fewer HTTP requests from the client.

Downside: you lose the ability to factor in network latency, and it’s a little more complicated to reason about when you have “mobile app” events being emitted from your servers and your mobile clients into the same dataset.

While you’re still fairly low volume, I would recommend being aware of the tradeoffs :) and then blithely going ahead with the simplest solution for you at this time.

Honeycomb is in the sweet spot

When it comes to mobile, Honeycomb is a great middle ground between specialized behavioral analytics tools and specialized operations tools. We believe that life is better (and simpler) when both sides (mobile devs and ops) can view the same set of data as a single source of truth, and we feel pretty strongly that — until you need the sorts of more advanced funneling / retention graphs that mature BI tools provide you — Honeycomb provides enough of the flexibility / ad-hoc investigation that you need to understand what’s going on with your app.

Let us know if you’re thinking about using Honeycomb for mobile — we’d love to hear about your use case and point you at some helpful libraries we’ve got in the works–try us out for free!

At last, the days of having to contact us to get a Dataset rebalanced are over–you can now see and manage the entirety of your Honeycomb data across all Datasets you own!

a bouncy bee gif

From the Team Settings page, Team Owners have access to the Usage page, which shows your overall data size, and how it’s allocated among your Datasets:

overview of data usage

Below the overview, you can see your per-Dataset allocation–how your data is spread amongst your Team’s Datasets, how recently new data was sent to Honeycomb for a given Dataset, retention for each Dataset, the current size of your Fast Query Window (FQW) and other useful information:

per-Dataset allocation information

Click on a column name to sort by that column. Then, click Manage next to a Dataset to see the provisioning sliders to rebalance/re-allocate the Fast Query Window size vs Secondary Storage for that Dataset.

We’re happy to be able to put more visibility and control into the hands of our users, keep checking back to see what we do next–and give Honeycomb a try for free for 30 days!

Learn By Example

Honeycomb is next generation DevOps observability and debugging. It’s incredibly flexible and there’s a lot to learn, so getting started sending data can seem overwhelming.

We’re toiling like good worker bees to make the path clearer, and one of the best approaches is near and dear to my heart - teaching by example. As a hands on learner, I couldn’t be more pleased to introduce you to the shiny new Honeycomb Examples Repo today. Containing four examples in its early stages, I hope to someday have it serve as a reference of fully shrink-wrapped ready to rock examples allowing you to not only send data in from simulations of “real world” apps and logs, but to rip off the best bits and re-use them in your own instrumentation and Honeycomb usage.

Structure

Inside the examples repository there are four examples today:

  • golang-webapp - A two tier Twitter clone written in Go
  • honeytail-dockerd - A demonstration of Honeytail’s keyval parser to ingest Docker daemon logs
  • honeytail-nginx - A demonstration of Honeytail’s nginx parser to demonstrate starting at the edge
  • python-api - A Flask app demonstrating HTTP instrumentation with Python

All of the examples can be run locally on your laptop, or rapidly bootstrapped using Docker Compose. Simply set HONEYCOMB_WRITEKEY environment variable to your Honeycomb write key and run docker-compose up in the example’s directory:

$ docker-compose up

As soon as data starts flowing in (e.g., from making HTTP requests to the application), you should be able to see the created datasets in your Honeycomb account. For instance, here you can see the output of the Golang web app containers, including the sent events being printed to the terminal for debugging purposes.

We want to make these examples as easy as possible to run - you should be able to see how Honeycomb instrumentation works and poke at examples before venturing off into the big unknown to do it yourself!

Participate

Where would you like to see the examples repository go?

Is there something on your mind that you’d like to observe but can’t today? Is it serverless apps, slow query logs, job queues, or something else entirely? Tweet at us @honeycombio or message us in the Intercom bubble on our website and let us know!

The sky’s the limit with Honeycomb. And as always, don’t miss out on trying a free Honeycomb trial today!