Progressive Delivery: Using Feature Flags & Observability to Ship Confidently

Dawn Parzych [Developer Advocate|LaunchDarkly]:

Hi, everyone. Welcome to today’s webinar. We’re going to be starting in about two minutes, to give people a few minutes to sign in and get settled.

Charity Majors [CTO & Co-founder|Honeycomb.io]:

Well, I am wildly looking for more graphics to add to my very shitty slides. Happy Thursday. Hi, Dawn.

Dawn Parzych:

Hi, Charity.

Charity Majors:

Is it your birthday? Did I hear it was your birthday?

Dawn Parzych:

It is my birthday. I thought what better way to celebrate my birthday than to present with you today?

Charity Majors:

Where are the cupcakes?

Dawn Parzych:

They aren’t here. I’m assuming they’re coming later because I was promised cupcakes.

Charity Majors:

Well, that’s no joke. If you were promised cupcakes, there better be cupcakes.

Dawn Parzych:

I better have cupcakes.

Charity Majors:

Dear God.

Dawn Parzych:

Or they may be at my front door. I haven’t looked outside yet. And I’m looking… Okay. The recording is going. All this fun, do I have things in the right place? It says it’s recording. So for those of you joining us, this is being recorded today.

Charity Majors:

This will go down in your permanent record.

Dawn Parzych:

Yes. Now when people listen to it outside of this, on-demand, they’ll wonder, “Is everyday Dawn’s birthday?” Yes.

Charity Majors:

Every day is Dawn’s birthday in my heart.

Dawn Parzych:

I will totally do that.

Charity Majors:

Oh. We have chat bubbles. Oh… people.

Dawn Parzych:

Yay. I guess that’s a sign that this is actually working and people can hear us.

Charity Majors:

It seems to be working. Fantastic.

Dawn Parzych:

Excellent. All right. We’re going to kick off. I see people coming. I’m sure more people will come in over time because what is time anyway, these days?

Charity Majors:

This is the first webinar that I have done since… I want to say a year. I would say it’s been a year and a half since I’ve done a webinar.

Dawn Parzych:

Wow.

Charity Majors:

Maybe two. It’s been a long time.

Dawn Parzych:

That is a long time. I have just sent things over. Wait a minute, wrong way. Wrong-way.

Charity Majors:

Somebody just said, “I hope the cupcakes aren’t a lie.” We all hope the cupcakes aren’t a lie. The cake is a lie.

Dawn Parzych:

I would be very upset if they were a lie. Yay. Okay. We’re in there now. Move my mouse off that screen. Great. So, welcome to the Progressive Delivery, otherwise known as my birthday party webinar. Thank you for joining Charity and me.

Before we get started, I’m going to go through a little bit of housekeeping stuff. Today’s webinar is being recorded. After the webinar, you’ll get an on-demand link emailed to you so that you can share this with other people. The conversation aspect of this will run about 25 minutes, and then we’re going to do a Q&A at the end. Please ask questions. We love questions. Drop them into the Q&A button, Zoom toolbar to ask questions. Those are the best way for us to see them.

Charity Majors:

Yes, we are hoping for lots of questions. This is… I think the material we have is pretty compact and pretty dense. And we’re hoping to just be able to talk through anything you guys have to bring up.

Dawn Parzych:

Yeah. So we already have one question asking if we get to eat cupcakes every day. Yes. You can have cupcakes every day. Feel free to upvote questions so we know who wants to hear what. And I was going to check in to make sure you all can see and hear us, but I’m seeing you respond to us, so we know you can see and hear us. So that’s excellent.

Quick code of conduct before we kick-off. So we can all focus on what brings us together today, and that’s learning. We all assume good intentions here. When you’re chatting or communicating in the Q&A, please avoid any shaming language in the chat and questions. If anyone has any issues or concerns, you can say so in the chat here. Kelly from Honeycomb and Alex from LaunchDarkly are in chat and they’re supporting and listening to us. If you want to reach out more privately, you can select just panelists, and I believe that the code of conduct has been pasted into the chat with an email address for you to access as well.

So now that we’re done with all of that fun housekeeping stuff, let’s get into the rest of it. Hi, let me introduce myself. I’m Dawn. I am a Developer Advocate at LaunchDarkly, and I’m super excited and happy to be here today with Charity to talk about progressive delivery and feature flags.

Charity Majors:

And birthday girl. I am Charity Majors with my oops, it’s misspelled, @mipstytipsy, which fun fact, was actually my EverQuest Enchanter’s name. That’s where that name comes from. Once upon a time, I played way too much EverQuest. I am the CTO/Co-Founder of Honeycomb and I’m here too.

6:12

Dawn Parzych:

Excellent. So I’m going to attempt to launch a poll that you can answer anytime during the webinar, as we talk about progressive delivery. We’d like to hear how you’re using progressive delivery and probably should go ahead and define what progressive delivery is.

Charity Majors:

Oh man, the host and panelists can’t vote.

Dawn Parzych:

I’m assuming you use progressive delivery.

Charity Majors:

I say I do. Just like everyone says they do observability. So, there’s nobody checking. No lie detectors there. Yeah.

Dawn Parzych:

So for us at LaunchDarkly, I see progressive delivery helps companies stay competitive by delivering small increments of change faster and safer. Everyone wants to move fast, but you also want to be safe as you’re moving fast. And that’s one thing that progressive delivery is able to do because it inserts more checkpoints during the rollout so that you have more opportunities for testing, experimenting, and gathering user feedback to improve quality before you deliver.

Charity Majors:

It lets you move with confidence I would say. Lets you move swiftly with confidence because at each point you’re checking yourself and you’re actually looking at it. Am I doing what I think I’m doing? Am I doing what I set out to do? Does anything else look weird? Right. I think it was Monkchips, James, who coined the term progressive delivery, or maybe popularized it, but I’m a big fan because I think it’s a term that we’ve needed a word for. We needed a word to describe this for a while. I used to talk about canaries a lot, but it’s not just canaries. It’s a whole constellation of stuff that’s around deploying in a way that is not a big bang. It’s the anti-big bang.

Dawn Parzych:

Right, so it allows you to deliver first to smaller, low-risk audiences before you deliver to everybody and deal with that, “Oh no, we’ve really broken something for really important customers.” Not that low-risk projects aren’t important as well.

Charity Majors:

If you’ve ever deployed something and then it took five times as long to recover from the deploy then the actual deploy itself, and now your week is toast. And you’re just like, “No deploys on Friday”, which is the wrong lesson to take.

Dawn Parzych:

Exactly, and progressive delivery allows you to deploy on Fridays.

Charity Majors:

Exactly.

Dawn Parzych:

Because as we’re delivering those small increments of change, you can feel confident. You can deliver it safer. You also have this concept of software ownership. Charity, I love the way that you described this, it closes the loop between dev and ops. It makes DevOps.

Charity Majors:

Yeah. I feel like the DevOps split was a thing that should never have happened. It was the wrong road that we went down. We were like, “Okay, this is getting hard and it’s too much for one person to do. So we’re just going to make the people who build something, not the people who have to run it.” But you have to have that original intent in your head if you’re going to have any hope of debugging or understanding or maintaining. They have to exist in the same brain. And so I feel like we’re finally starting to heal that rift. It fulfills the promise of DevOps, which is you write it, you run it, you build it, you maintain it.

I believe that this shouldn’t feel like a death sentence. It shouldn’t feel like a punishment. It should actually… I know that ops has a lot of, there’s a lot of blame to go to ops. Because we’ve been the martyrs and the masochists and just taken pride in our, “I got woken up five times last night, there’s hair on my chest.” That’s not fun. Nobody wants to join us in that world. But I’m over 30, I don’t want to live in that world either anymore.

And in fact, the only hope that we have as an industry of dragging ourselves out of that world and into a world where we build systems humanely in a way that you can be over 30 or 40 or 50 and still writing and supporting your own software, is if we actually learn to do this, because this is what makes our systems better.

If you’re shipping new bullshit every day out to that hairball of a system that you have, it’s like something a cat coughed up, that nobody’s ever understood. Nobody’s ever understood it, and you just keep deploying more shit that nobody understands to it. No wonder everything’s on fire, no wonder people are miserable, and no wonder nobody wants to be on call. But, I feel like, I’m almost done ranting I promise, but I feel like the biggest enemy that we have to face is the fact that we just accepted that this is how it is building software. It just has to suck. There just has to be shitty parts that somebody has to suck it up and take. It doesn’t have to be like that. It genuinely is possible to build and ship and run systems that are well understood. That are clear, that are intuitive. That you don’t have to fear deploys because you have control. All right, I’m done ranting. You’ve all heard it before. We can move along.

Dawn Parzych:

We’ll move on.

Charity Majors:

Except for ownership. Ownership is important. And honestly, this is what makes us enjoy our work. Right? What is it that Daniel Pink said? The things that we all crave in our work are autonomy, mastery, and meaning. And I feel like this is the element. This is the missing link that lets us get actual autonomy and ownership over at work.

Dawn Parzych:

I totally agree. And that humane aspect is part of this progressive delivery. It’s slowing down how you roll features out. It’s using this notion of release progression to see who gets impacted. Just because it used to be done with these big bangs, it doesn’t mean it has to stay that way. You need to roll up features at a pace that’s appropriate to your business. And what’s appropriate for one business does not make it appropriate for all businesses.

Charity Majors:

We’re not all Google.

11:46

Dawn Parzych:

Just because Facebook or Google or somebody else does it this way doesn’t mean that you have to. Think about the humans that work at your company and the humans that are using your software and what works for them.

So when you release and think, “Do we want to do a targeted rollout? Are we doing Canary launches, ring deployments, percentage-based rollouts? Or do you want to do a big bang?” Sometimes you have to do a big bang.

Charity Majors:

Sometimes you have to.

Dawn Parzych:

But for the majority of things, you don’t. You can take it in small increments.

Charity Majors:

Yeah. I feel like every single one of these socio-technical systems that we work on, every single company, every single one of them is different. Right? We can learn from each other, but there is no substitute. At the end of the day, somebody has to understand and debug and fix and tune your socio-technical system, and it’s not the same as anyone else’s in the world. This is why it’s fun and interesting. Right? There’s no recipe book. There’s no playbook, just follow these five things, because every single one of these systems are different and too complex to be reduced to just a playbook. But that’s why it’s fun.

Dawn Parzych:

Yeah. And part of that as well is figuring out, who’s responsible? Empowering other people in the organization to be able to turn features on and off. This notion of progressive delegation, who most closely is responsible for the outcome of that feature? I heard a talk, much like when we did a customer trajectory nano series recently, and Trade.IO talked about how they use feature flags with Slack to allow sales engineers to, in the middle of a demo, enable a feature for a customer or disable that feature for a customer.

Charity Majors:

That’s badass.

Dawn Parzych:

They’re doing it 20 times a day. Can you imagine having to open a ticket or call an engineer and say, “Hey, can you turn this on?” The sales engineer needs it in that moment.

Charity Majors:

Everyone does it. You want to empower the person who’s closest to doing the work because they’re the ones who know what needs to be done. Everybody belongs in production. We’ve erected these massive gates to keep everyone out of production because it’s so scary. Because we were so used to it being so fragile, right? It’s just like “Stay away. I’m pretty sure as soon as someone touches it, everything is going to go up in flames.” But there is a serious competitive advantage in making production a welcoming place, making it a playground with guard rails and with safety features, but a place that welcomes everyone in, because everyone does a better job when they’re close to production.

Dawn Parzych:

Exactly. So I did mention a little while ago this notion of feature flags and how they were used within Slack. I do want to make sure everyone is aware of what a feature flag is in case you haven’t heard of them. Feature flags, from LaunchDarkly’s perspective and my perspective, are control points that allow you to alter the behavior at runtime based on targeting rules or external input. So this notion of progressive delivery is enabled because you insert these decision points or fancy “if statement,” if you will, into your code to say “These people have access”, or “I want 5% of my users to have access to this too.”

Charity Majors:

What I love about this is, this is what decouples releases from deploys.

Dawn Parzych:

Absolutely. That whole notion of releasing on Friday, you’re not releasing a feature on a Friday, you’re deploying the code.

Charity Majors:

Yeah, there’s a difference.

Dawn Parzych:

Put it out there. Don’t show it to anybody for Friday, but just put it up there and see what happens in the system over the weekend.

Charity Majors:

You should be able to ship your code continuously. Every time you save your work, you ship a diff, you merge it back. I’m a big fan, every time that you merge back to main, it should automatically get picked up, run through tests, and deployed the master within minutes. That does not mean that it is necessarily live for all your users to use it. That would be terrifying and stupid. Nobody’s suggesting that. And feature flags are the special sauce and the magic that let you have your cake and eat it too.

16:09

Dawn Parzych:

Absolutely. And it’s great to see in the poll results that about 20% of the people on here today say they’re already a pro with feature flags, and 37% have just started using them. Great. 36% haven’t yet, and 7% used to use them, but now are on the team that doesn’t. So you’re about a 50/50 split between those using them and those that aren’t using them.

Charity Majors:

They’re addictive, aren’t they? Like once you start using them, it’s just like, “Oh, I could use it for this. I could use it for this and they can use this.” It unblocks you in so many ways. It makes it so much easier, and it lowers the barrier while improving the safety threshold. It’s a pretty catchy concept, I think.

Dawn Parzych:

Yeah. I just took over the website for the school’s PTA. And I was trying to do something, I’m like, I just need a feature flag. Turn this stuff off. I was like, “Why can’t I have a feature flag?”

Charity Majors:

It’s so hard to go back. You go back to a road where it’s all on or all off for everyone. It’s just, how did we ever survive?

Dawn Parzych:

Not possible. So we’ve talked about, feature flags and progressive delivery. And the other aspect of all this progressive delivery is observability. So Charity, do you want to chat about this?

Charity Majors:

Yeah. it’s kind of… most of us, I’m an ops engineer and the entire, and the experience that I had, capsule summary, I was at Parse, I was on fire and I couldn’t figure out what the fuck was going on. We were doing microservices, if you’re familiar with microservices, we were doing a lot of shit. We also had like a million mobile apps, and they’re going down all the time. Or they were complaining about being down, whether they were down or not. And there were shared pools of services, shared pools of databases.

And so they could go down when someone who shared any one of those pools with them, had to get featured the iTunes Store. It was just literally impossible. Y’all are flying blind. If you have monitoring tools, if you’re using metrics and monitoring, you do not realize just how blind you are flying. The ability to slice and dice to break down by high cardinality dimensions, like build ID, and then see the impact of what you have shipped at a very granular level… I started Honeycomb because the idea of going back to… I’m not going to name competitors’ names. The world of metrics monitoring, it felt like going out to drive down the freeway without my glasses on, which if you know how blind I am, you’d know it is a terrible idea. It’s like, you have a vague sense of what’s going on, but you can’t actually isolate or pinpoint or compare or see, the context or correlate.

All right, anyway. TL;DR. It matters a lot. And it matters because this is not just about being a better way of operating your software, which it is, but it’s about pulling that whole act of validating way back into the development cycle so that you’re practicing instead of just TDD, test-driven development, you’re practicing. TDD is great, right? But it abstracts away everything interesting about the world. It’s just like, anything that could possibly be chaotic or random or concurrency, well that doesn’t exist. Now let’s run our code. Which is not useless, but it’s also not that useful. Right? And when you’re practicing observability driven development, what that means is, as you’re writing your code, you’re instrumenting it. You’re aware that the future you is going to need to understand, is this doing what I want it to or not?

So you’re writing your code, you’re instrumenting it so that you can answer that question, you ship it, and then you go and you look at it through the lens of your instrumentation, and you ask yourself, “Is it doing what I’m trying to do? Is it doing what I broke the code to do? And does anything else look weird?” And I know that last question sounds very fuzzy and like, “wouldn’t it be nice if it could be more specific?” But it can’t. Weird is… it can’t be defined lower than weird. And this is why it’s important to be in production every single day. Because if you aren’t looking at production when it’s not broken, you don’t know what healthy looks like. And then you can’t trust your gut instinct, right? Like your gut instinct is a powerful tool. If you ship some code and you go and you look at, and you’re like, “what’s that? That sure showed up at a funny time to just the subset posts that I deployed to.”

Well, that moment right here is the most powerful time in all of creation for you find the bugs in your code, because it’s fresh in your head, you know exactly what you’re trying to do. You know exactly what you just did. You haven’t forgotten it yet. You haven’t paged it out to the… moved on to another problem, and you’re looking at it right there.

When people start practicing observability driven development and regularly looking at their changes, minutes after they make it, maybe they find 80 to 90% of bugs. I know that sounds like I’m pulling that out of my ass, because I am, but also experience. We find shit so fast before our customers ever have to experience it. Right? Because you’re there, you’re looking at it. It doesn’t have time to curdle. It doesn’t have time to fester over months. Doesn’t have time to become the new normal, right? That time right there is the most powerful time in all of history for finding the bugs in your code. But it requires that you be instrumenting. It requires that you have that tight feedback loop, and it requires that you be able to write down the high cardinality dimensions when you have high dimensionality, all the stuff that I’ve been yammering on and on and on about for years on Twitter. So I’m not going to repeat myself here. Cool. We can move on.

22:07

Dawn Parzych:

Yeah. And looking at the poll results as well, similar to feature flags, about 50% of the people are… actually, no, I’m doing the math wrong. 47% have just started. 17% are a pro. So 60-40 split.

Charity Majors:

This is a very educated crowd. I feel like we’re in very elite company.

Dawn Parzych:

Excellent. Yes. And it’s one thing to talk about all these definitions, but my favorite thing is like hearing about, how are they used in practice?

Charity Majors:

Yeah. Honestly, I am amazed that there are this many people here, because like I’m so cynical about vendors. I don’t know that I’ve ever gone to show up for a vendor’s webinar. Because they’re just going to tell you a bunch of bullshit. Right?

Dawn Parzych:

They’re here for my birthday. I mean, let’s be honest. They’re here for my birthday. This is an awesome birthday party.

Charity Majors:

That makes sense. But it’s weird to me that I find myself in the seat of being a vendor. It really is. But I’ve experienced engineering teams before and after observability. One more small detour. I feel like anytime you’re asking someone to adopt something new, some new tools, some new practice, what you’re offering them has to be an order of magnitude better than what they have today. Right? In order to be worth their time, their energy, that energy, their team, replace. Because none of this is your core business, right? You have a business, it is shipping value to users and it’s not this. The best you can hope for from this is that it gets out of your way and helps you do your job faster. You don’t want to do dedicate a lot of time. So it really has to be an order of magnitude better than what you’ve got.

And I feel like it’s just in the last year or two that I feel like I can say full-throatedly, honest to God, that I feel like not any one of these tools alone, but this collection of tools is solidly an order of magnitude better than what came before in terms of enabling teams to move faster, helping teams move safely, helping them not get paged in the middle of the night, helping them find the bugs when they’ve shipped them. I feel like if you look at the Dora report year over year, the Elite category just showed up two years ago as a little seven percenter. And then the next year it’s like 22%. And it’s like achieving liftoff velocity. And those are the people who are adopting these best practices. God bless the people who suffered through earlier versions that were twice as good.

Anyway. So progression delivery, you were just talking about getting very concrete and specific. And when I say progressive delivery, these are some of the things that… Can you go back one, or a couple…

Dawn Parzych:

Sorry.

Charity Majors:

Yeah. Some of the things that come to mind for me are… so one of the first times that I really had to invest in this was when we did a rewrite at Parse from Ruby API to GoLang API. It’s like replacing the engines in a jet while it’s flying. It was insanely hard. Also, you can set things up so that you can fork the output, right? So that you ship your code to a node that forks it, and then returns a good result to the user, dips the output. So you can see, is your new code actually, is it the same? Is it different? Is it faster? Is it slower? It’s really useful for dark launching shit. It’s useful for ruling out two groups of people.

Like this is a different thing than rolling out to percentages, rolling out to changes in waves. A lot of people miss this one and they’ll deploy one host, and then a few hosts, and then 10%. And then 25%, and then they get to 50% and they’re like, great, I’m done. And they turn on the rest. And that is when your backend falls over. Right? Like you can’t forget that last 20, 30%. You actually do want, when you get to the stage where you’re running fast, you want to roll it out progressively in percentages because that lock percentage isn’t actually going to kill you until you’re… what’s that noise? And decoupling deploys and releases. Yeah. Okay. You can move on.

Dawn Parzych:

Excellent. Yay.

26:23

Charity Majors:

And the first one that I wanted to talk about here was the Parse/Facebook thing. Which I have a blog post about. There’s this gnarly-ass… well, we used Scuba, which was the forerunner of Honeycomb, right? This is when I first fell in love with observability and realizing how different it was because the shit was literally impossible with Ganglia. Sorry, not mentioning competitors. Literally impossible to do well until we got into Scuba. Facebook, interestingly, so what Facebook does is they have this internal cluster of Facebook that they roll everything out to first. So like Facebook would go down for employees a couple of times a week. It would be like, “Oh, Facebook’s down.” Then after that, they roll it out to Brazil. I don’t know what Brazil ever did to them, but Brazil gets it first. But they just start geographically going around the world. So this is why, again, you have to know your product, you have to know your users. You can’t take anyone else who wrote that for blowing the shit out because it won’t actually work for you. You have to understand your stuff.

Oh, yes. I was going to talk about the Honeycomb setup. So Honeycomb is actually deployed from a very small shell script. It runs from cron once an hour. And so we have a dogfood cluster, and then we also have a kibble cluster. So all of our production traffic gets snipped. And all of the analytics from our users go into the dogfood one. And then because we also want to be able to understand our dogfood cluster, we have sort of a meta-dogfood, which is called kibble.

So we deployed a kibble first, and then we wait like 30 minutes, and then we wait another 30 minutes, and then we deployed production. And again, it is fully automated. So as soon as somebody merges their diff to me, it kicks off, does it once an hour, but it kicks up everything so that you know that within an hour, your changes will be live in production every time, which, I would like about 15 minutes, but like the goal is for it to be short enough so it’s like muscle memory, right? It should just be a habit. There should not be enough time in there for you to go off and get distracted doing something else. That’s what I wanted to share.

Oh, yes. Oh, this is just a graph of the Honeycomb deploy. Well yeah, the API server version changing there. And then example three. I thought that we had interspersed one of yours.

Dawn Parzych:

I thought we had, we can move them. I can…

Charity Majors:

Let’s skip to yours and then go back to AB testing signup flow.

Dawn Parzych:

All right. So, one of the ways that we do progressive delivery and use feature flags and observability data is by running experiments or doing AB testing. One thing that we had heard from some of our customers that had a very large number of flags is that the main feature page with all the feature flags was taking way too long to load because there was no pagination on the page. So obviously, the more flags you have, the slower it’s going to be to load all of that. So when we were building up pagination, we wanted to make sure that we weren’t hurting one group of customers that didn’t have a large number of flags, but we were helping those that did.

So we ran an experiment where we bucketed out our customers and looked at page performance. We looked at backend stats and data as we were rolling this out to see how pagination would impact users if they had a large number of flags or not. Thankfully, those with a small number of flags didn’t get negatively impacted, so we ended up rolling this out to production. But having that feedback loop, knowing that we weren’t just trusting our gut, we wanted to know that yes, all of our customers were going to either benefit this or not be harmed by it when we were rolling it out.

The other really unique way that I think we use some feature flags and observability is to automatically adjust our tracing on the fly. So we have these very granular targeting roles within LaunchDarkly, and we’ve used them to tweak the experience of a user, but where the user is the trace.

So we’ve added the trace attributes to what normally is defined as a user attribute. And so what we want to do is, in one instance, we wanted to trace the initialization code path for our SDKs. And we only wanted the initialization traced. We didn’t want all of it traced. So we set up targeting attributes that would only look at various attributes within that initialization path. And our targeting rules operate from top to bottom. So if you see here, the items on the top would execute first. So if it matches this, we’re going to take the trace. If it didn’t match those initialization patterns, then we wouldn’t. This has been kind of a fun way that we use feature flags to help with observability, to either collect more information. Sometimes we’ll say we want to trace just a single customer because a single customer’s having a problem. And so we’ll just go and grab complete traces of everything for those customers. So that’s a fun way of doing that.

Charity, do you want to go back to your other example and then…?

Charity Majors:

Yeah. Sure. Can you hear that squeaking behind me? Is it terrible?

Dawn Parzych:

It’s not terrible.

32:27

Charity Majors:

Okay, great. Okay. Yeah. So this is the thing that Alyson was just showing me this morning. She’s been working on in the signup flow, and this is very simple, incredibly powerful, and effective. This is a screenshot of the LaunchDarkly feature flag where it’s just, you set it up to be 50% the new one, 50% the old one. And if you go forward one… oop, these are out of order. Go forward one more.

Dawn Parzych:

I hit the arrow too quickly.

Charity Majors:

There we go. This is just, we don’t have like the constant stream of hundreds of signups, but you see you can see them being broken down by the feature flag down below, and then the other one, the one we just passed, is AB testing the signup flows so you can see the results. Boom. It’s so easy. It’s so powerful, it’s so much fun.

Dawn Parzych:

Yeah. And I love the fact that we can like run these experiments to confirm that gut feeling. So, I think this is going to go better, but I don’t want to just base a decision on, I thought. I want to have that data to back it up.

Charity Majors:

You want to know.

Dawn Parzych:

Yeah.

Charity Majors:

And in the battle days, what we used to do was AG proxy rules, giant regular expression, where we would set based on the hash prefix or something, we’d be like, “Well, we’re randomly assigning you to this percentage.” And then managing those rules just about robbed me of my sanity. And of course, we were generating them from chef cookbooks. And so it was just scary.

Dawn Parzych:

All right. So, in the interest of time here, I’m going to jump in.

Charity Majors:

Kelly is rushing us along.

Dawn Parzych:

I know. I’m going to jump into some Q&A.

Charity Majors:

We’re about done.

Dawn Parzych:

Yeah. Actually, you know what? I think I only had one more example. I did. It’d be remiss of me not to talk about this. Because this is something really cool that LaunchDarkly and Honeycomb just did together, where we’ve implemented flag triggers. This was just launched last week by LaunchDarkly where you can have a private or a secret URL where when a threshold is passed in Honeycomb, it will automatically disable the flag in LaunchDarkly. So it really puts in those safety mechanisms that we’ve been talking about. And instead of trying to go and be like, “Oh, I’ve got to go turn that off,” it’s all just done magically, between Honeycomb and LaunchDarkly. We’ve got more information about flag triggers on the site. So I’m not going to spend more time, but I wanted to let you know that that information is up there.

Charity Majors:

Super cool.

Dawn Parzych:

And now we’ll jump into the Q&A.

Charity Majors:

Questions.

Dawn Parzych:

Questions. So the first one that came in from Adam says, “What are the guardrails that you would put in place before letting everyone enable, disable feature flags in production?”

Charity Majors:

It depends. It depends on many things. Like how large is your team? How much do you trust them? How literate are they? Have they ever used them before? Are they… Dawn, why don’t you take this?

Dawn Parzych:

Yea, I can help with that. What we do is, we have the ability to set tags and control who in an organization can disable flags. So you may have those operational feature flags that you don’t want somebody in marketing or sales engineering to toggle on and off, but you want a sales engineer to be able to toggle a feature on and off in the middle of a demo to show, “Hey, here’s how we do this.” That example that I gave earlier from one of our customers, trade out IO. And so you put those safeguards in place where people only have the ability to toggle the flags that are relevant to them, through tagging our backs, things along those lines.

Next question. Charity.

So it’s like, “What are your thoughts on trunk-based development and constantly merging behind a flag versus working in a branch and only merging when done?”

Charity Majors:

I mean, this is just one of those religious practices. You just pick one. I honestly think my preference is to be constantly merging behind a flag. You want your code… You don’t want it to get stale. You don’t want it to get moldy. Code gets moldy. You need to get it into production, where it stays fresh.

Dawn Parzych:

Absolutely. Another question here, let’s see. I’ve never used for known anyone who has used feature flags or the observability that Honeycomb provides. I struggle to imagine how that could work without extensive reroutes to our code. Can you talk a little bit about how that integration actually happens?

Charity Majors:

Yeah. There’s no rewriting at all, just add the library and you’re done. That’s it.

Dawn Parzych:

Yeah. So from the LaunchDarkly perspective, it’s install an SDK, and then it’s a few pieces of code that you wrap around the features that you want to control via flags.

Charity Majors:

I mean, a lot of work has gone into making this easy for you. Let’s put it that way. But it used to be much harder. But now it’s quite easy for both of them. There’s plenty of copy-paste. You just install the library and it’s all done for you.

Dawn Parzych:

Absolutely.

38:14

Charity Majors:

If you feel the company should already have some prerequisites before embracing the feature flag concept, for it to be successful from day one, particular automation that should be in place? This is not really one of those, you must be this height to ride this ride sort of scenarios because it is one of the things that helps protect you from the bad things. Extensive test coverage is never the wrong answer. That’s always a great thing to have, but feature flags are a thing that protects you from catastrophes. And so it’s kind of like saying, should you make sure that the doctor has bandaged you before he does your arm sling? They achieve the same goal of making you safer and productive.

Dawn Parzych:

And what I would add to that is, if you’re starting with feature flags, like we would love for you to go and feature flag all the things, but start small. Start with a lower risk project, a lower risk feature.

Charity Majors:

I would say actually, I slightly disagree. Instead of lower risk. I would say start with something that matters. Start with something that is important that you’re working on actively every day. Because you don’t want it to just be something that is low impact, not important, and just off there where people are going to forget about it. Same with instrumentation, right? People are always asking me, how do you roll out better instrumentation? Imagine it’s a headlamp, and everywhere you go to fix something, the first thing you do is you add instrumentation and you add feature flags. And that way, it’s being actively… it’s not going to get the code rotten with smell. It’s going to be actively used because that’s what it’s there for. Right? It’s there to help you while you’re actively changing things so that you’re changing things safer. It’s not to get put out to pasture. That’s not the purpose of these tools.

Dawn Parzych:

Absolutely. Another question that I’m seeing here is… I think this came up different ways, so I’m going to paraphrase a question. And it’s about the hierarchy of flags and controlling who sees what, and if there are dependencies. So, is there a shared view, things along those lines? So within LaunchDarkly, we have this notion of like a flag hierarchy where you can have a prerequisite. So if you’re building a new home page and there are four widgets on it, you would have five flags, one for the new homepage, and one for each of the four widgets, and the four widgets would depend and have a prerequisite of that homepage flag be enabled. So you do have this notion of, are features going to accidentally be shown when like they shouldn’t be shown? So you have these hierarchies and permissions within there.

Charity Majors:

Somebody said, we struggle to sell developers outside our platform team on flags and different rollout strategies. What would effectively encouraging teams to adopt these practices? My first answer is to put them on call because usually exposing them to some pain will do it. And he says they are, and these specifically are the ones dealing the old ways. Oh, that’s terrible. Hmm. So they are on call. Is it miserable? Are they in pain when they’re on call? They’re suffering, right?

So honestly, depending on who you are, what your position is, what your appetite for battle is, you can try enlisting managers. Because one way to get this stuff to take root is to make sure that it is clearly written into your, say, leveling documents that senior engineers are expected to adopt new practices, to do things the right way. These are the tools that managers have. So if you’ve got the managers on your side, you can enlist them. And that’s a slow way sometimes of bending that curve. Because really, the senior engineer, you’re going to be most effective at embracing new ways, doing better ways if your senior engineers are not only on board, but understand why that matters. I’m sorry, there’s no magic bullet. It’s going to be an uphill battle without that. The advice that I often give people in this situation is to quit. Find a company that deserves you. Seriously. You shouldn’t have to go down with the boat.

Dawn Parzych:

Yeah. It’s not an easy thing to do all the time, but sometimes find the companies that are more willing to embrace these types of practices versus, this is the way we’ve done things. These are the way we will always do things.

Charity Majors:

There are companies that are hungry for this kind of transformation. There are companies who would embrace you, who would welcome these changes, who would look at you as a leader for bringing them. The question of how much of your life force do you want to try to extend to dragging people into the glorious future kicking and screaming?

Dawn Parzych:

Yeah. Shameless plug, LaunchDarkly is hiring. I don’t know if Honeycomb is, but I’ll throw that out there. Other question.

44:00

Charity Majors:

If you ave specific questions, just general, I am always happy to take… If people send me emails with their situation, I will try to answer them in my blog in more of a thought out long-form. Done that a couple of times, and I’m happy to try. If you send me all the details you have, I’ll take a swing at it. Sorry, I can’t do more.

Dawn Parzych:

That’s great. And another question that I really like here, and it’s been upvoted a lot, so we’re going to get this one is, how would you recommend using feature flags for infrastructure changes? For example, rolling out new monitoring infrastructure rollouts or things along those lines. Charity, has Honeycomb done any of that for infrastructure changes?

Charity Majors:

I don’t know that we’ve done that much with infra… We try not to do much with infrastructure. We do pretty boring stuff. Keep it simple.

Dawn Parzych:

Yeah, so we’ve done a number of infrastructure changes. We have a number of blogs on our site talking about how we’ve done database migrations using feature flags.

Charity Majors:

Oh yeah. We use feature flags for database migrations too, but I think it’s just… it’s all in the codebase. There isn’t anything outside of that.

Dawn Parzych:

Yeah. But the infrastructure changes is definitely not an area a lot of people talk about, or think about when it comes to feature flags, but it definitely can be used. We have a number of resources on the LaunchDarkly site about that. And I’m happy to, if you message me, ping me on Twitter, I’ll find them and post them. I don’t have them off the top of my head right now.

I’ll go to this question too because I love talking about technical debt. Because who doesn’t like talking about that? I could talk about what we do for best practices and to clean up feature flags, to make sure they don’t turn into technical debt. Charity, can you share like what Honeycomb does to make sure that flags don’t become stale debt?

Charity Majors:

Sorry. I’m busy feeding leads to somebody in the text answer. So you go for it.

Dawn Parzych:

Great. No problem. So some of the things we like to do are, we suggest tagging all of your flags with a sprint date or a release date. And then you can search through and find the older ones. We also have the ability in the interface to say when a flag’s been completely rolled out to everybody, and if it’s been completely rolled out to everybody and it’s not an operational flag that’s being used as a kill switch, then that’s a prime candidate for turning that flag off and removing it.

We have code references. So you can very easily, in the interface, see all the places in your code that refers to that flag to make it a less labor-intensive process to get that removed. And one of my favorite ways of doing a feature flag cleanup is, we’ve heard of companies that have gamified it, and they hold what they call like capture the flag.

Charity Majors:

Oh my God.

Dawn Parzych:

Come on. Everything has to be a game.

Charity Majors:

Somebody else asked a question that got threaded off of it. It says, my company expires feature flags after a couple of minor versions. Is that a good idea? Is there a better way?

Dawn Parzych:

That works. If it’s rolled out doing it at a sprint, going back and saying, “Okay, these versions have all rolled out. We know these are fully rolled out,” then take them out. You do need to make sure when you are going around that, that after those few versions, that flag has rolled out to 100% of users, or you’re serving the same version to all of your users.

Charity Majors:

What sort of latency is involved in fetching flag values and states? Tens of milliseconds?

Dawn Parzych:

Very small. Yes.

Charity Majors:

Very, very, very fast. Yeah. I’m just typing answers right now.

Dawn Parzych:

That’s fine. Yeah. There’s been a ton of questions out there. We will make sure they are all answered afterward. And if we didn’t, as I said, ping me on Twitter. I’m happy to strike up a conversation about any of this further. I love talking about these types of topics. So there we go.

Oh, one thing that I will bring up that I saw here is… I lost the question. There we go. Failure cases for a feature flag system. So you depend on a feature flag system to roll out a feature. That system fails. Now the feature is deployed to a larger or smaller target population. How do you guard for that?

So for LaunchDarkly, we have the functionality where you can put in a default value. So if for some reason, your application is unable to communicate with LaunchDarkly, the default value is, do you want that on or off? So you would specify in this case, you would probably want that default value to not serve the flag. So you have the options to have those safeguards in place. So we have built that into our system.

We have a few minutes left. I think I want to make sure everyone knows that we will have this recorded. We’ll be sending out the slide deck, which has some links to information. I will go and find the infrastructure links and get those posted to Twitter for people, and any other questions? Like I said, please feel free to reach out to me on Twitter. Charity, do you have any parting thoughts?

Charity Majors:

I’m an ops nerd. I don’t usually recommend software, because I think you should run less software. Software’s usually the problem, not the solution. But the highest performing teams that I’ve ever seen embrace this shit. It’s worth it. That’s all.

Dawn Parzych:

Great.

Charity Majors:

Thanks, everyone.

Dawn Parzych:

Thank you all so much. It was a pleasure having you here, and Charity, thank you for having me chat with you. This was a lot of fun.

Charity Majors:

Ah, man, there was stuff in chat that I didn’t see that people were asking questions over there too.

If you see any typos in this text or have any questions, reach out to marketing@honeycomb.io.

Transcript