Landing Observability in the Enterprise

John Feminella [Advisor on Software, Technology, and Transformation]:

Hi, everyone. I’m John Feminella from Charlottesville, VA where I serve as a technology advisor. Today we’re talking about landing observability in the enterprise. My job is to help transform the way enterprises write, operate, and deploy software so that their businesses are more effective. More than that, I also try to think about how technology organizations need to behave and operate if they want the people in them to be effective.

In particular, what I would like to talk about today is the striking discontinuity between how individual teams land observability and how an entire firm or technology organization does. We will also talk about why it can sometimes be hard to see that difference in a large organization, what we can do to set our teams up for success, whether you are an executive or inline member.

At the end of the talk, I would like to leave you with a toolkit that you can use for bringing observability to a broader audience. Right now, that toolkit is empty so let’s start building.

One thing that’s difficult for some to grasp is how varied the size, scale, and complexity of different organizations can be. Small firms look like small teams where everyone works together toward common goals, and few people are outside your own orbit. Large firms look totally different. Any success we may have had in introducing technology into our own team might not be repeatable if we don’t change tactics for the wider world beyond.

To put that in perspective, consider this graph that represents the number of firms of a particular size that are incorporated somewhere in the United States of America. Now the vast, overwhelming majority of businesses in the U.S. are small—sole proprietorships, independent consultants, and teams of one or two or three people. This encompasses a wide variety of different businesses, many of which don’t produce software like construction firms or restaurants. 

Let’s instead consider firms that employ at least two technical staff. We’ll call these firms “with engineers” for lack of a better term. When we consider firms “with engineers”, the number shrinks to about one-fourth of the total size but the number of employees in the median has quadrupled to about 13. Most businesses look like this. A small team or two working together to accomplish something and grow over time. Some of these businesses will fail and disband. Some will stay about the same size for a while. And some will go on to become bigger.

What about at the top end of the scale? What do the biggest businesses look like? There are a lot of ways to count who is the biggest. If we confine to the Fortune 500, the median business there has about 26,190 employees. There’s a huge difference between the median firm that employs software developers at all and the top few firms that employ thousands, or tens of thousands of them. In fact, this difference is over 2,000 times bigger.

Statistically speaking, most people are used to working in small teams of less than a dozen or two to get things done but we see the size and scale of the biggest organizations are so vast that our intuitions and experience about how to drive change might not apply. So when champions from teams emerge to try and shepherd observability beyond the confines of their team, they often bump into frustrating walls that prevent them from making meaningful progress. Let’s examine some of those intuitions and build up a toolkit for understanding how we can get observability into complex organizations.

3:43

I think there are two fundamental ways in which bringing observability into a broader category of people looks really different than doing it for small teams. Because it’s so different, I think it’s important that we understand how we’re going to approach it if we want to be successful. 

If we would like to take an improvement from inside our team out to the broader organization, I think there are two things that we need to be able to understand. I’ll summarize the concepts by calling them structure and outcomes. 

By structure, I mean the way that the organization is laid out to accomplish its operations and how people collaborate to get things done—who reports to whom, how the people communicate and work together, and what responsibilities everyone has. These are all questions about the structure of the organization.

By outcome, I mean the goals and objectives that people care about and how the individual objectives line up to something bigger and broader that the overall company cares about.

Let’s talk about structure first. In a small team, everyone has clearly defined roles that are the same level of abstraction. In an enterprise, the roles should also be clearly defined, but the scope and the scale of what each person is responsible for may differ wildly and may encompass a greater number of people or functions. If we want to build a coalition of people who might be interested in observability and grow support for this idea within a firm, it makes sense to understand the structure of the organization so we can identify who the right people will be.

Here, for example, is Carol, the Executive Director of Treasury Operations. She cares a lot about the uptime and availability of our fictional firm’s payment software because the business can’t make money if the website is down. Carol might be very interested in the outcomes generated by observability and is likely also interested in what kinds of priorities need to shift around to implement it.

But Carol may not care about how this happens or what engineering tools and technologies are used to bring it about. For leaders like Carol, observability isn’t about what technology we use, it’s about the fact that we can tell them the business will be more robust and that we can resolve problems faster. Will Carol have to wait for a sprint or two longer for the integration of a new payment to land while we instrument a couple of important services? These are the kinds of questions that are important to Carol. And if they want their support, we need to understand what they care about.

A tactic that new champions often try is to attempt to find people who have similar goals to their own team and then convince them to adopt the new approach. In this example, for an individual contributor, that would be like looking for other people who have goals similar to Dinesh, Emma, and Farah. This can work but it’s slow and plodding. You have to work with one team at a time, and chances are good that you will still have your own commitments to your own team to handle. So effectively you’ve given yourself a second job, and you might have to stick with this approach for a long time before you get good penetration throughout your organization.

6:40

Our first step in bringing this to a wider audience is going to be to think about what potential supporters might exist for our cause. Accordingly, the first thing we’re going to add to our toolkit is this exact tactic—identifying potential supporters. If we can use our knowledge of the structure of the organization to locate people who will be sympathetic to the benefits of observability, given its trade-offs, regardless of how it gets implemented, we can start building a coalition of interested folks.

How do we use our understanding of the structure of an organization to identify potential supporters? Well, because enterprises are big, they can’t afford to let the overhead of managing all those people spiral too far out of control. There are a lot of ways to try to manage complexity but the two levers you see big companies reaching for the most are hierarchy and standardization.

By hierarchy, I mean the way the responsibilities and functions of the business are distributed vertically in the org chart. To see how different these hierarchies are, let’s look at the organizational chart of a typical enterprise firm. In the following examples, I’ll use the anonymized structure of an actual Fortune 500 company that is also one of my clients. This is a large bank with thousands of thousands of physical branches and an expansive international payments network.

If you’re an individual contributor, it all starts with you. You and your teammates are the beating heart of the company’s productivity. You belong to a team. The median team size is 9 to 15 people, depending upon how expansively a team is defined. In this example, you belong to a product team that works on a mobile application for accessing bank accounts. Your job centers around owning three different microservices that support the backend of the application. Your team has an engineering manager who is your people leader. They provide feedback up and down the org hierarchy, they’re responsible for your personal and professional growth and wellbeing, and they help implement the organization’s policies at the level of each team for consistency. 

Your manager might manage more than one team but in this example, they just manage yours. You have peers and other teams with different managers with whom you might also collaborate. For example, the customers of the three microservices you manage might be represented by other teams. You also have peers you don’t really collaborate with, either because they’re not your customers, or they work with a different part of the technology stack that you don’t overlap with much.

Together, you and your peer teams are overseen by an engineering director who is responsible for shaping the effectiveness and growth of the organization, and ensuring that the engineering organization’s commitments to everyone else are met.

9:15

The director owns the engineering organization, but the engineering organization is just one of many functions of the business. These collective functions are carried out by the rest of the peer organizations, which might be similar in size or even larger than your own.

Those peer organizations then form a line of business. And another level up the stack, we can zoom out to see all the lines of business in the entire company, and the presidents that lead them. Ultimately, those lines of business roll up to the CEO, the leader of the organization.

That’s a lot of levels. And I actually think this example is a fairly flat organization compared to most enterprises. There are only five levels between a line member and the CEO of the entire company. Even so, you can see how the responsibilities look really different from each other, especially at the top and bottom of this hierarchy. That tells us that if we’re looking to identify the right people, we need to target the right level of abstraction. Not everyone is going to need or care about observability. But there might be a lot of people who care about increased uptime, better developer effectiveness, faster time to repair outages, and so on.

That also means we need to tailor our message for why people should support our proposal in the right way. Simply saying that observability is objectively good or makes operations easier and more productive might not be interesting to some folks, especially outside the engineering space. What we need to do is be specific about the kinds of objectives we hope to achieve with observability and to make sure that we bring that message to the right people. In this example, bringing this to the CEO is probably not a path for success for landing observability more broadly. They’re not likely to have the right context. So even if they’re willing to hear you out, a good CEO is going to understand that someone else in the organization besides them is the right decision-maker.

Conversely, going up a few levels might be the right thing to do if the firm’s overriding objective is something observability can directly or indirectly help with, like the stability of services and production environments. Perhaps, for example, the firm has been plagued by production outages and customers are starting to lose confidence. One approach could be to try to position observability as being an important part of the broader strategy. That brings us to the next tool in the toolbox. Tailoring our message to a level of the structural hierarchy that we’re building support from is key to growing our coalition.

The other piece of structure is standardization. By standardization, I mean the way that the responsibilities and functions are distributed horizontally in the org chart. One pattern we saw before was the CEO presiding over different lines of business including yours. But did you notice, in our example, the functions of that line of business weren’t specific to that business? 

For example, we have a sales organization, a legal organization, an engineering organization, and so on. Other lines of business are likely to have similar functions. If you’re a technologist, you might be tempted to call that duplication that should be factored out. And that’s often what many enterprises do, especially with functions related to technology. 

12:22

The most common way that pattern manifests is the creation of a shared services organization. Different enterprises will call this different things, but it usually has some kind of name that evokes a cross-cutting feel to it like “Platform Engineering” or “Shared Services”. 

Some orgs even lump this entire idea directly under the IT organization. When this happens, it can be a challenge to introduce valuable or innovative new technologies like observability because this organization may effectively be a gatekeeper to new ideas. It will often have the responsibility of evaluating and sometimes operating new technology but because they don’t have the same product goals that your team might have, they can be somewhat divorced from the outcomes that your coalition might care about.

That means that if we’re operating in an environment that has shared shares, observability is likely to become one of the services. In an enterprise, it’s also likely that someone else besides our team is going to be in charge of deploying Honeycomb across a hundred engineering teams. We need to understand how the shared services organization works and what its objectives and goals are. But we shouldn’t be afraid to have our supporters go to bat for us and help push something important through. They’ll be especially willing to do that if we line up to the goals and outcomes that they care about.

Let’s talk about outcomes next and how we can support our coalition. On a small team, you might be given broad latitude to try out different ideas and work on different things because it’s reasonable to make the assumption that we’re all headed in the same direction and working towards the same things. In other words, you share common goals with the entire team. But in a larger organization, the goals and objectives of the company may differ radically from that of a single team so understanding who to talk to and how to align your goals with theirs is key. People will want to know what they’re going to get out of a proposed experiment with you. The question we need to answer here is “What kinds of success are you interested in?”

In the example, the firm cares about a different set of things than individual teams do. Ideally in a well-functioning enterprise, it should be clear to teams how their work connects with the objectives of the firm. For instance, if sales leaders care about improving something at the entire level of the firm, like the cost of goods sold, then one thing that your team is doing is contributing to some slice of a measurable improvement of that overall goal. 

In a less-well-functioning enterprise, it might be hard to see how your team’s work connects to the objectives. You will have to do more work to explain to supporters why observability supports those objectives, or even argue to leaders that they need to include it as an objective.

With all those levels of hierarchy, we have introduced people who look really different from each other. The CEO may care about one thing while a developer cares about something else entirely. If you’re hoping to introduce new technology to a decision-maker who is two or three or five levels above us, we better be targeting the right person in the structure. That requires knowledge of the motivations, goals, and objectives of the folks we’re working with. It also required thinking about how they would respond to a proposal for better observability. If we bring our proposed experiment to a director, we’re going to be likely to have a good conversation about improving production stability and resolving problems. If we brought the same proposal to someone higher up the stack, the questions might be more generic because they’ll have less context about our problems. 

When we identify these decision-makers, we are looking for budgetary authority to make a decision. If they don’t have or control the money to run a useful technology experiment, they might still be a good entry point for a good conversation but they’re not going to be able to deliver for us without someone else’s help. 

16:06

Getting decision-makers onboard is a key part of our coalition-building toolkit. If we don’t have the right people with the right authority on board, it doesn’t matter how great or powerful our ideas are. Not only are the goals and objectives that each person has in an enterprise likely to be different across the hierarchy, the results they’re accountable for delivering will also be different. Your team cares about shipping a specific system, while the CEO cares about abstract ideas that span multiple lines of business and provides sweeping benefits to lots of people.

Users of technology, in addition, often aren’t the buyers of that technology. It’s even more important that we line up to shared outcomes. Our team isn’t going to be the group that buys observability for the entire company. So without the discretionary budget to run technology experiments ourselves, it can sometimes fall to individual teams to convince others to go to bat for us.

The last thing we’re going to add is the importance of lining up to shared outcomes. If we want our experiment to have a chance at being propagated beyond our team, we need to be able to say why other people outside of our team should care. They have their own goals and plans. So if others support us, it’s going to make success more likely when we can clearly explain how observability supports them too.

I would like to close out our talk by reviewing our toolkit and giving you some final words. First, if you’re coming from a small, close-knit team, our toolkit might seem strange. I would be remiss if I didn’t add that a large part of how easy it is to answer the questions and come the understanding necessary comes from the maturity of the culture and communication style of your organization. 

Like many problems in technology, it’s not really about computers at all. Instead, it is about how people communicate with and trust each other. The best kind of cultures for landing demonstrated improvements like observability to a wide user base will have a pre-exchange of ideas for improving the culture, openness to receiving and giving feedback, willingness to run experiments and change things, and transparency on important matters with members.

Here’s what our toolkit looks like again all put together. First, identifying potential supporters is how we build an initial base of people who can go to bat for us. Some people will be a good fit, and some won’t, either because they don’t have the right authority or because they’re not at the right level of abstraction. We do that by tailoring our message about why observability is good to each person we speak to. Ideally, they care about the same things that we care about but that won’t always be the case in an enterprise. We want to line up to important and shared outcomes held by key decision-makers. Who has the budgetary authority to say yes to our ideas? Who will benefit from a successful experiment? And what does success look like for them? Those are all key for helping our proposal land.

Finally, if we have the presence of shared services in our firm, this is a complicating factor that means we’re going to need their cooperation too. In those cases, we either need to have our champions go to bat for us,  or we need to bring shared services onboard directly. If we can leverage these five tactics to build a coalition, advance an interesting proposal for observability on a correctly sized experiment, we’re more likely to be successful in the enterprise. 

I have used this playbook in all the work that I do for all firms of all sizes and for all kinds of technologies. I hope it serves as a useful model for you too. I would love to hear more about your stories and experiences introducing observability and new ideas into enterprises. Feel free to reach out to me after the conference and I hope you enjoy the rest of o11ycon.

19:42

Corey Quinn [Chief Cloud Economist|The Duckbill Group]:

John, thank you very much for doing that. The first piece of feedback, and I imagine we’re going to get more questions coming in the channel presently, is from Liz saying that this is such a great talk about all the layers in the organization and how software gets adopted at scale. This means that it’s a near certainty when you say things like this, you get engineering types who are more passionate about the technology than the adoption of that technology, quoting The Empire Strikes Back with “That’s not true, that’s impossible!” Accurate? Not so accurate? Am I mis-categorizing wildly and about to get emails?

John Feminella:

I’ve never heard anyone quote The Empire Strikes Back to me in a business context.

Corey Quinn: 

You need to work with different companies, then, my stars.

John Feminella:

I clearly haven’t tried hard enough with enough enterprises. I think there’s a huge difference in the range of experiences people can have. And that’s part of why I made the mention about culture. How easy or hard this is to do, culture is really a multiplier on that. 

If you’re part of an organization that’s got a great culture around, “You ought to be doing something different” and that’s the feedback you can give freely and have it be received well, then you’re halfway there already. 

But if you’re part of an organization where everybody is “stay in your lane, this person does this, don’t talk to me about something that’s not about that” it becomes a harder challenge. 

Ironically, some of my work is about getting to be an outsider. I get to sidestep, as I’m sure you’ve had this experience, all of the silos that might exist because they’re paying me to have an opinion. And when I’m being paid to have an opinion, it becomes a lot easier to tell people what they probably are getting told already from five levels down but hasn’t propagated up to the right people. So I think it’s a really hard problem in general. There are no easy answers. I hope that the talk gave people pointers about how to navigate that effectively.

Corey Quinn: 

When I wind up talking to “big E” enterprises, the challenge I keep smacking into is that the bigger the company gets, the greater the organization distance is between the various constituencies. Let’s be clear. I’m talking about advisory consulting services but I hear the same story with people selling vendor solutions as well. And I have to assume that is true when you’re starting to deal with things such as shifting over to observability, onboarding it, and demonstrating the value of it. 

There is the person who has to sign off on the investment. Even if it is an internal effort, it still costs money. Then there’s the people’s time that by saying yes to one thing you’re saying no to something else. Then there are the people who are championing that. It seems like you wind up with this incredibly disconnected group of stakeholders. Is that something you’re seen? I’m just trying to understand, is this something that you see too? Or am I just dealing with really interesting companies that are different enterprises?

John Feminella:

Well, a cross-cutting concern like observability is something that can benefit people up and down the stack, whether you are a product person, or engineer, or operations, or anywhere in between or adjacent. You all get to benefit from observability. Organizations aren’t set up necessarily to think that way. They haven’t been designed to deliver that kind of outcome. 

The closest nexus that you get is shared services but then they’re so far away from the work that the engineers do that they don’t provide useful outcomes. That is when somebody has to do the glue work of bringing people together and saying “You’re going to operate this. The firm wants this benefit and you’re responsible for building it. And collective we will produce that outcome.” 

But those people are not necessarily used to working that way together, so they don’t have the muscle memory of  “We’re all in the same boat together”. But we’ve never been in the same boat together. That’s really hard for enterprises to be constructionist about teams in that way. A lot of time they take a sledgehammer to the way you might have been doing things so we can land this crack team of smart people to work on a specific problem like observability.

Corey Quinn:

Matt chimed in to say that he’s so happy to be hearing your voice again which tells me that he’s probably related to you on some level, or you should pay him his commission on that. 

24:10

And Erik asked, “What are good ways to grease the gears in organizations that are  ‘stay in your lane” and have difficulty changing? And what kind of advisor should Eric hire to know that you grease wheels, not gears? And lastly, finding high-level champions helps, but what else can you do?

John Feminella:

Sure. There are a few different questions there. The best way to “grease the gears” is to think about how people are aligned to outcomes that they want to get done. They’re incentivized to behave in certain ways because their bonus is attached to some outcome. Or they care about customers and they want to make sure some things are true. Those are the things to look for. People that have the kind of outcomes lined up with the benefits of observability, are the people you should probably talk to first. So much the better if they’re lined up with a discretionary technology budget that will let you run an experiment. 

Even when an organization is a “stay in your lane” organization, they will still probably appreciate hearing about, “Hey, I know you care about this kind of thing that I saw on our OKR tracker, that you are responsible for these outcomes. Here’s how I think I can help.” I think few people would refuse an offer of “I can make your life easier.” I certainly wouldn’t. 

The high-level champions part is a way of doing an end-run around whatever organizational structure you might have. Having someone high enough up who can go to bat for you, is the cheat code for not having to build a coalition at lower levels. Skip to the CIO or VP of Technology. They say “I don’t care whatever whoever said, I want you to do this” and sometimes that’s sufficient. That’s not coalition-building, of course, and in the wrong kind of culture that can obstruct your ability to land the experiment.

26:00

Corey Quinn:

Culture is always the hardest part. It’s not great to say it in many contexts, but the truth is that the unofficial org chart outweighs the official one. You find people with a demonstrated track record of success in winning hearts and minds in an organization because they see it in a way other folks don’t. There are other folks who are super passionate about that but who aren’t effective champions of their ideas are often not the right pick.

John Feminella:

I’m sure you’ve seen this too. I’ve seen a lot of people talk about why some random enterprise SaaS vendor doesn’t publish their pricing on their webpage. It is because they literally have no idea how much it’s going to cost to sell to you until they figure out who they need to talk to and how much it is. The cost of goods sold for one firm might be really easy because they have a great culture that lets you bring ideas in, but another firm might require, for the exact same thing, 12 months of work to find who the right people are.

That does not completely absolve people of the responsibility of having transparent pricing but that gives you a hint of how much complexity in these organizations. There is so much difference in how they are set up, who cares about which things in the org chart, how different are things from what is actually represented, and on. I would be surprised if I would ever be characterized as a person who wins hearts and minds, but I think I’m okay at telling people what they’re doing wrong and helping them to do it better.

Corey Quinn:

You call yourself an advisor. If that’s what you’re doing professionally, great. That’s the role. If not, if you’re one of those “I yell advice to people”, there’s a place for that. it is called Twitter. It comes down to effectively winning ideas. Ideas carry the day but that’s only a partial truth. Ideas have to be presented in a palatable way.

To that end, we have a question from Lee. How do you broach the subject around breaking down silos and getting teams to believe that actually talking to each other and working together is going to make them much better?

John Feminella:

I think one thing that I try to look at there is like what is the cause or origination of the silo? Sometimes it’s inadvertent.  Sometimes it’s just that the other team can’t see what your team is doing or vice versa so there’s no awareness. The silo isn’t deliberately constructed. It’s just, “Hey, have you talked to this person before?”

Part of the job of an outsider or leader is to say, “Have you met? Have you talked to this person? Do you understand what they’re doing?” Sometimes it’s strictly a communications problem and no awareness that that’s happening. 

Other times, the silo is a deliberate organizational construction. We choose to have a firewall between this group and that group for structural reasons, or legal reasons, and we don’t want these groups talking together. When it’s in that bucket, that’s when going one or two levels up becomes more important. Because you’re not going to be able to discover other teams like you or who are interested in this proposal, it’s probably time to have a conversation about what the experiment actually is. And say, “OK, my director, what can you do to help me find other people who might be interested in this too so we can run this experiment?”

Corey Quinn:

Last question, how do I become an observability champion and push my org forward as an IC?

John Feminella:

Great question. I think probably the easiest way is to work for an observability company in the pre-sales role.

Corey Quinn:

That sounds like it’s not a champion so much as someone with a vested interest in it.

John Feminella:

That’s true. You can come back to the same company you left, you can probably make the sale easier. I think in my short time remaining, it’s really about understanding the structure and where are you going to line up to outcomes. Understand who cares about what, and go after the things that line up best with observability.

If you see any typos in this text or have any questions, reach out to team@honeycomb.io.

Transcript