Controlling Features in Production: Observability + Feature Flags

Dawn Parzych [Developer Marketing Manager|LaunchDarkly]:

Hi, everyone. My name is Dawn Parzych, and I’m the Manager of Developer Marketing at LaunchDarkly. And I’m here today to talk about how you can control your features in production with feature flags. I’m going to kick things off with two questions that I want you to think about over the course of the talk. First, do you know how often you deploy? If you’re listening to this live, drop the answer to that in the chat. And this next question, just keep it in the back of your mind as I go on with the talk. What happens when the deployment causes problems? Frequent deploys are the goal of many organizations. But there are a couple of things that get in the way of that. The first thing is using the term deploy and release interchangeably. These are not the same thing. A release is a business decision. And it’s expensive. Rolling back a release takes time, energy, unhappy customers. 

Deployment is testing and verification that your code is ready to go and be released. You can deploy a lot without releasing. And different teams can own these two parts. So I’m encouraging you to go ahead and deploy broken code. But please, try not to release broken code. Put your code into production and make sure it works there without showing it to customers. The trick to doing this, to deploying broken code is to use feature flags and progressive delivery. But before we go into what is progressive delivery, I’m going to provide a quick definition of feature flags. Just to make sure that we’ve got the foundations in place because feature flags are one of the foundations of progressive delivery. So a feature flag is a control point in your code whose behavior can vary during Runtime based on external input. And this is incredibly important as we move through this talk, and we get into our demo.

Feature flags are key to enabling progressive delivery. Progressive delivery is about delivering changes first to small, low-risk audiences and then expanding this to larger and riskier audiences. Every step of the way you’re validating these results as you go. The goal of this is to get users what they want, on budget, within a reasonable time, all while knowing what happens afterward. Sounds like a miracle, right? This is done through processes of release progression, validation, and delegation.

3:11

Release progression is this iterative cycle that you go through. When you think about progressively releasing a feature, you want to go through a set of stages. First, you release a feature behind a flag. Make it available to internal teams for testing in a real production environment, without impacting all of your users. Collect data during this time, make sure that your observability data is happy. If problems arise, you could iterate, get them resolved, go back, and kind of start over again through this process. As code is tested through these iterations, you build confidence. You’re gradually turning this flag on for more end user segments. As you’re doing this, you’re continuing to test and validate along the way until you’ve released this to 100% of your users. Throughout this process, if an error occurs, you can instantly roll back the feature by turning the flag off. Again, without end user disruption or needing to redeploy code.     

What progressive delivery does is it allows you to adjust the number of users that are able to see new features at a pace that is appropriate for your business. And only you and your team can determine what that pace is. What this allows is making smaller changes, smaller changes, smaller bets. There’s less chance of something going wrong enough to be a catastrophic break. Think of this a little bit like playing penny slots. It takes a lot of effort to lose a thousand dollars at penny slots. Compare that to going to a $100 Blackjack table. That thousand dollars is going to go very quickly. Larger bets, larger opportunities to kind of have things go array.  

Most people aren’t going to notice a single loss playing penny slots. But at that Blackjack table, you’re going to feel that a lot quicker and a lot harder. Making your deployments more granular and smaller means that even if you lose a bet about the feature being ready, it’s really not that big of a deal. Progressive delivery helps companies stay competitive by delivering these really small increments of change faster and safer. You’ve created more Checkpoints during your rollout. You have more opportunities for testing and experimenting to improve the quality.    

Delivering features though isn’t just about rolling them out to customers. It also includes empowering the right people in your organization who are most closely responsible for the outcome of that feature. When multiple people have the ability to toggle flags on or off, it can be a challenge to know if a new feature has caused that change in performance or operational instability. The LaunchDarkly and Honeycomb integration helps you take the guesswork out of that equation. When the user toggles a feature on or off, a marker is sent to Honeycomb displaying who toggled which flag. Let’s go see this in action.    

All right, so I’m about to go and do a live demo. And it’s been many years since I’ve done a live demo on stage. So before we begin, I need to get and provide an offering to the demo gods. So I’m going to put on a trusty headband, and we’re going to cross our fingers and hope this demo goes as planned.

6:59

For the Honeycomb and LaunchDarkly integration, to get the marker sent when you’re creating flags, it is a pretty quick process. Within your LaunchDarkly account, your environment, you want to click into integrations. And scroll down and find the Honeycomb section. And we’re going to go ahead and click “add integration.”  We’ll provide a quick name for this. What this requires for this from Honeycomb is two pieces of information. One is your Honeycomb data set, and the second is the Honeycomb API key. So we are going to go into Honeycomb and grab our API key. We are calling this data set in Honeycomb. This is called Dev Week demo. Within the policy, for this demo, we don’t want to use all of our projects and all of our production environments. I want only the flag changes that are taking place in this test environment that we have set up. So I am going to change what we have for our resources for the policy statement. This is not a step that you necessarily need to go through, but for our demo today, I only want the flags within the Dawn environment in the support service project to be toggling any changes here.      

So we’re going to update the statement. Yes, I understand the terms and conditions of this. Save the configuration. And then we’re going to cross our fingers, go back into here, and send a validation request. And we see within LaunchDarkly that our test event was set successfully. What this should be providing in our Honeycomb data is a marker that indicates that a test event was set. So if I mouse over this little data point here, we can see that Dawn Parzych generated a test event. So we’ve got an integration working. We have information going from LaunchDarkly into Honeycomb. But generating the test event isn’t what’s the meat of the integration. What’s more important is getting the markers when a feature is enabled or disabled. We’re going back into LaunchDarkly for a moment. And I click on our feature flags.

I have a feature flag that I’ve created just a little while ago called DevWeek demo. This is currently off. I can toggle this feature on. Save changes. Flag’s on. I should now have a marker that has appeared within Honeycomb as well. So this is now showing me that Dawn Parzych turned on a flag and turned on which flags. So when you’re delegating enabling and disabling features to multiple users within Honeycomb, you’re very quickly able to see if some operational instability, if there are things that don’t look right, hey, wait a minute, this flag was just turned on. And this person turned it on. So you can go back and have them disable that flag or maybe do some additional troubleshooting.

11:11

The same thing happens if I was to go in and turn this off. I will see that same marker. Not the same marker. Here we will see a different marker come in shortly after it that flag has been turned off. So again, you’re able to see, okay, here’s when a flag was enabled or disabled, and does it correlate with any anomalies that you’re seeing within the Honeycomb data.   

Markers aren’t everything. You also have to deal for failures situations. So we’re going to go back in and talk about what you do in the case of a failure. So even with all the advance planning and feature flags, failures are still going to be encountered. Because failure happens. What you need to determine, and this is different for every organization is how much failure is tolerable within your application. You need a plan in place to figure it out. And part of that failure tolerance depends on how crucial your product is and how many people are using it.    

Now an electronic health records companies may have a much different tolerance than a tower defense game. The amount of downtime that’s a deal-breaker in those situations can be very different. You can have as many nines of uptime as you want, as long as you’re willing to pay for it.     

What feature flags help you do is turn off portions of a release. Rolling back a release takes time. And that cuts into those valuable minutes that can put your SLOs on the line. Turning the flag off can be done in a few seconds. And flags give you the ability to turn off just a portion of the release, the thing that’s having problems. They’re kind of a built-in safety valve that allows you to roll forwards not back. We call these circuit breakers. Humans are slow at reacting to emergencies. And that’s why we have these circuit breakers that exist. When something goes wrong, you can have the system fail to a safer state rather than causing harm. With launch darkly and Honeycomb integration, you can configure LaunchDarkly triggers to automatically disable a feature once thresholds are exceeded in Honeycomb.

13:53

I’m going to go through and show you how to go about doing that. So we want to set up a trigger, and that requires some configuration in both LaunchDarkly and in Honeycomb. We want to go into this DevWeek demo flag that I’ve had created. And head over into the settings section. But midway, a little less than midway down the page, we have triggers for the environment that we’re in. And we’re going to go ahead and add a trigger here. Default comes up, says generic trigger. That’s not what we want. We’re actually wanting to create a Honeycomb trigger. And you have two options here. The trigger can be created when the flag targeting is turned off. Or when the flag targeting is turned to on. So we’re going to go in and say update the flag targeting to off. And save the trigger. This is a very important step. 

A URL is created here. Do not navigate away from this page without copying this URL. As soon as you navigate away from this page, you are no longer going to be able to get this URL back. Now you can go ahead and delete the trigger and recreate a new one, but this is obfuscated after the initial configuration is set up. And this is listed on the page, but I also want to make sure that I’m sharing that information with you here.  

So we have this URL created. We are ready to go back into Honeycomb and provide Honeycomb with this information. So within Honeycomb, we want to go into our team settings, and over to integrations. And we want to add a trigger recipient within Honeycomb. So we’re going to select “add integration.”  We’re going to specify a webhook. We’re going to call this DevWeek demo. Paste in that long URL that we copied over from LaunchDarkly. And oh, forgot to add a secret. And we’ve gone ahead and added that into here. Then take a look at our flag to check the state of it. It’s currently off. The trigger we just created is to notify us if it is turned off. I’m going to go ahead and enable this right now. Set it to on. And again, cross your fingers, let’s hope this works. 

If I click on “test” here, this is going to send a test notification. And if we go back. So we now see that that was toggled from the on state into the off state through that test that had been done. So what this is doing is if a trigger is configured within Honeycomb, metrics have been passed. Something’s not going right. A flag can automatically be turned off. So we’re eliminating that kind of intermediary, you know, a human has to go in and disable it. You can also do the opposite and have Honeycomb turn a feature on as well. So just like we set up here, we would instead of selecting turn a feature off, select turn a feature on the component. And you can have Honeycomb turn features off and on within LaunchDarkly for days on end.

18:25

These simple steps are all you need to get your feature flags up and running within LaunchDarkly and getting that marker and trigger configured within Honeycomb. Thank you all for taking the time today to learn about LaunchDarkly and feature flags. And our integration with Honeycomb. If you are interested in getting the slides from this as well as a LaunchDarkly T-shirt, you can go to LaunchDarkly.com/Dawn from the dropdown list, select Honeycomb, and we will send you the slides that include links and to our documentation on how all this stuff works as well as a T-shirt if you’re interested. Thank you all very much for your time.

If you see any typos in this text or have any questions, reach out to team@honeycomb.io.

Transcript