Observability Maturity
Community Research Findings
Conducted by Clearpath Strategies
Sponsored by Honeycomb
Hypothesis
Adopting observability tools, site reliability engineering (SRE) practices and a culture of shared ownership translates to efficiencies across the software engineering cycle, better end-user experiences and ultimately Production Excellence.
Key findings
Observability enables production excellence.
Advanced observability enables better outcomes for engineering and DevOps / SRE teams. Survey respondents with advanced observability are more aware of where their tech debt lies, are more proactive about paying it down, have more confidence in their ability to detect bugs in production, and have more satisfied end-users.
Three in four teams have yet to begin or are early in their observability journeys.
Around a quarter of respondents exhibit advanced observability practices. A plurality of teams are actively on their journeys but still early, with some Observability Tooling or processes—but not both. The remainder of respondents are generally aware of observability and its benefits—though some conflate observability with tangential practices like metrics monitoring and log management tooling—but have not developed plans for or prioritized observability that encompasses practices and culture in addition to tooling.
There is momentum behind the shift toward achieving more observable systems.
As the benefits of observability become clearer, most teams not already practicing at an advanced level have near-term plans to move in that direction. Of the 47% of respondents whose teams are not currently practicing observability, three in four have a plan to do so within the next two years.
Advanced Observability practitioners focus on outcomes.
By framing observability objectives around desired outcomes including higher quality instrumented code, predictable release cadence, confidence in detecting bugs, ability to resolve incidents and maintain resilient systems, teams can better position themselves to achieve production excellence.
Of the 47% of respondents whose teams are not currently practicing Observability, three in four have a plan to do so within the next two years.
The Observability Spectrum
Observability is an iterative process, or a journey. That process, however, is never truly finished and can often be fragmented. Most companies have adopted some–but not all–observability practices. Which practices they have adopted, however, are not necessarily consistent. The Observability Spectrum identifies 5 distinct groups:
More than one in three respondents fall in our Working toward Clarity (Evolving) category
Enlightened & Sees the Light Groups
Short of 10% of those surveyed report a combination of practices and tooling that reflect a highly advanced observable system (Highly Advanced). Another 17% (Advanced) of participants report practices, tooling, and outcomes consistent with relatively sophisticated observability practices. These teams are leveraging Observability Tooling and/or processes on a team or organization-wide basis and also prioritize key practices to achieve performance excellence.
Defining characteristics
Observability is a high priority for their teams. Cohort members come from a mix of company sizes. Companies represented by this cohort tend to have a high number of developers (42% have 100+ developers).
Working Toward Clarity Group
Most members of our community fall in the evolving middle. Participants report observability processes or tooling–but rarely both–and report some, but not most, key capabilities.
Defining characteristics
Mostly represent small <100 employee companies. A majority within this group report observability is mostly practiced on a team-by-team basis (64%), roughly half (46%) do not report using Observability Tooling.
Searching for the Path
Approximately one in five respondents do not currently practice or use Observability Tooling, but have plans to practice observability within the next year. Among these teams, reported practices and tooling suggest production excellence is a priority.
Defining characteristics
High Observability awareness (92%); more DevOps and Managers; most from smaller, <100 employee companies (45%).
In the Dark
17% of respondents are not practicing observability and do not currently use tools with no plans in the near future to adopt.
Companies represented by Enlightened & Sees the Light Groups tend to have a high number of developers. 42% of them have 100+ developers.
Advanced observability drives teams
to production excellence
We found that advanced observability tooling and practices go hand-in-hand with achieving better outcomes: Advanced teams realize better outcomes, compared to their Evolving and Planning counterparts.
Ability to Detect Bugs in Production
Participants on Advanced teams are more confident in their ability to detect bugs after code is deployed to production. Advanced teams also possess the confidence to solve problems when crisis strikes–52% are very confident in their capacity to detect bugs in production and another 40% are somewhat confident. Confidence levels drop significantly among Evolving teams to only 18% very confident and 50% somewhat confident–comparable to the levels of those still in the Planning stage.
How & Where Technical Debt Costs Your Organization
Advanced observability teams also have a much stronger handle on their technical debt. Over half (52%) can show how and where technical debt costs their organization, and 85% report a good understanding of their technical debt at some level within the organization. Moreover, 36% of these Advanced teams have a process in place to deal with their technical debt.
52% of developers on Advanced teams are very confident in their capacity to detect bugs in production and another 40% are somewhat confident.
Advanced observability capabilities report customers who are “Always satisfied” at twice the rate of less-advanced teams.
Over half (52%) can show how and where technical debt costs their organization, and 85% report a good understanding of their technical debt at some level within the organization.
Company culture and priorities lay the
foundation for observable systems
Culture has as much influence on outcomes as technology practices and capabilities. To achieve observability, teams must prioritize setting Service Level Objectives (SLOs), measure, track and iterate to improve. This is a central tenet to observability, which our research bears out. To better understand the theory of SLOs and their criticality to observability, read InfoQ article “SLOs are the API for Your Engineering Team”.
Among Advanced teams, a 90% supermajority reports their team sets and measures SLIs, with nearly one in four (24%) saying their team always does so.
Moreover, 78% of Advanced teams report setting and measuring SLOs, versus only 38% of those with Less Mature observability practices.
Advanced teams are also more likely to practice DevOps (94% currently practice) and CI/CD (94% currently practice) than less advanced teams.
Trending toward more observable systems
To those who practice observability, the value is clear. Fully 90% of those who use observability tooling say those tools are important to their team’s software development success, including 39% who say observability tools are very important. While many more participants currently use tooling tangential to observability, e.g., monitoring (84% use), log management (79% use), and error reporting (71% use)–each is perceived as less impactful to their software development success and achieving production excellence.
Recommendations and Resources
Accelerate the State of DevOps DORA Reports (2018 & 2019)
Accelerate book (The Science of Lean Software and DevOps: Building and Scaling High Performing Technology Organizations)
O’Reilly Site Reliability Engineering (SRE) Handbook
Follow Charity Majors & Liz Fong-Jones on Twitter
Charity Majors personal blog
So you want to build an observability tool (blog by Charity Majors)
eGuide on Observability for Developers – team Honeycomb
eGuide on Achieving Observability – team Honeycomb
Cultivating Production Excellence for complex & distributed systems (DevOpsDays Presentation)
90% of those who use observability tooling say those tools are important to their team’s software development success.
Conclusion
Our combined decades of field experience have witnessed it first hand working directly with software engineering teams of all sizes. These research findings confirm our qualitative insights.
Observability is a rapidly evolving field. Organizations with teams on the journey—especially those in the Evolving and Planning stages—are seeking guidance and best practices. While there is no one-size-fits-all approach to observability or a specific starting place, advanced teams share the outcomes that underpin production excellence.
As you determine your current place on the Observability Spectrum, we recommend conducting up-front research to fully understand the landscape of tooling and all options available. Don’t focus on only tooling or internal practices. Adopt them together as they will not only support but also reinforce your investment and payoff.
Survey Respondents – who we heard from
Through this Observability Maturity research study, 406 respondents come from teams with a variety of observability practices and adoption levels. A majority of our respondents (57%) were DevOps/SRE and developers, and roughly one-in-three (31%) work at large enterprises (1000+ employees).
