2021 Observability Maturity Community Research Findings
20 minute read
Executive summary
The 2021 Observability Maturity Community Research Findings report is a follow- up to the inaugural report published in Q1 2020. The survey was commissioned by Honeycomb, which provides full-stack observability designed for high-cardinality data and collaborative problem-solving. Founded by Christine Yen and Charity Majors, two engineers with experience debugging problems at scale for tens of millions of users, Honeycomb empowers every engineer to interrogate application telemetry data in granular and arbitrary ways, so they can deeply understand and debug the behavior of their production systems.
The survey was conducted by ClearPath Strategies, with the goal of understanding community perceptions and awareness of observability, how engineering teams are approaching observability, and mapping an observability maturity scale that updates last year’s findings.
The 2021 survey found that, overall, observability adoption is on the rise. Organizations on the higher end of the observability maturity spectrum are realizing benefits, such as:
• Higher productivity
• Improvement in code quality
• End-user satisfaction
• Software developer retention
While overall adoption is on the rise, the report also finds that a majority of teams are at the earliest stages of observability maturity. Teams on the lower end of the spectrum are not realizing the same outcomes we see from organizations at the higher end, such as deploying more frequently, finding bugs more quickly before and after pushing to production, and reduced burnout.
Teams further along the observability maturity spectrum not only realized additional benefits; they also realized those benefits with higher-quality business outcomes—for example, virtually all the Advanced and nearly 9 in 10 of Intermediate respondents provide customer experiences that leave their customers either “Always Satisfied” or “Sometimes Satisfied” with their services.
In short, teams on the higher end of the observability spectrum realize clear value in several areas, including an improved ability to identify problems and solve those problems faster, more satisfied customers, and more confidence deploying frequently to production when compared to their less-mature counterparts.
The 2021 report details how observability can help your organization achieve production excellence and gain a competitive edge.
Key findings
The second wave of our Observability Maturity Community Research Findings verifies that observability adoption is on the rise, with more survey participants practicing observability today than they were in the 2020 report. However, there are signs that confusion may exist around how observability is implemented, especially among groups lower on the maturity spectrum. Teams that score lower on the maturity spectrum report less overall comprehensive understanding of their production systems and also realize fewer benefits than their more mature counterparts.
While observability adoption increased this year, shifts in maturity were only significant at the upper and middle parts of the spectrum (a 4% increase in Advanced teams and a 2% increase in Intermediate teams). Respondents indicated several barriers to progressing their goals, but those in the middle of the maturity spectrum disproportionately cited lack of skills to implement as a barrier to adoption.
Key findings for the 2021 report include:
- Observability is gaining traction. More organizations are practicing observability today than in the 2020 report. Sixty-one percent of respondents reported that their teams are currently practicing observability, an increase of 8% from last year. That increase is sharply reflected across individual teams (up 7%) as opposed to entire organizations (up only 1%).
- Teams on the higher end of the maturity spectrum realize more benefits than their less-mature counterparts. Teams are attracted to observability’s promise to help them understand problems better and solve problems faster—and teams that are mature in their observability practice realize even more impactful business outcomes, including deploying more frequently, being able to find bugs more quickly before and after pushing to production, and reduced burnout.
- More-mature teams are also 3X more likely to deliver higher customer satisfaction. Teams that have achieved Intermediate or Advanced-level maturity reported their end-user customers are “Always Satisfied” with their service quality and capability at a rate of three times more than teams that do not practice observability.
-
Lack of implementation skills is a disproportionate barrier for observability adoption. While interest in observability has gained significant momentum, some organizations face barriers to reaching higher levels of maturity, such as lack of implementation skills, difficulty scaling to other teams, and competing with other initiatives.
Observability adoption is primarily being driven by software developers, followed closely by DevOps engineers, and site reliability engineers (SREs). These groups are driving interest in observability, both in terms of introducing observability to their organizations and persuading decision-makers to implement the practice. However, this grassroots adoption pattern results in most teams who planned to adopt observability doing so on a team-by-team basis, rather than driving adoption across the entire organization.
The teams that have adopted observability practices reported increased confidence in catching bugs before and after production, in addition to a number of other benefits, including the ability to:
- Understand what’s happening inside their system at any time, without shipping new code (70% of respondents)
- Immediately identify the problem when something breaks and the impact it has on other systems (69%)
- Understand their entire system, from high-level trends to specific events and outliers (63%)
- Immediately identify the solution to a problem (51%)
However, while interest in observability has gained significant momentum,
we also found that maturity is shifting at different paces: The Intermediate and Advanced groups have shifted toward higher maturity over the past year. Overall, the proportion of these two higher-spectrum groups remained constant. This could suggest that groups who are well-versed in observability practices are accelerating their skills and may be pulling away from their lower-maturity counterparts.
In contrast, the Novice group has remained stagnant year-over-year, despite more respondents reporting that they have started their observability adoption journey. We would have expected to see a shift from the Novice group toward the Intermediate group if progression through the maturity curve was happening at a consistent pace. The stagnation suggests that some groups may get stuck in the middle of their adoption journey. When asked about adoption barriers, the Novice group disproportionately cites lack of implementation skills in comparison to all other groups.
Most respondents fall in the evolving middle, where they practice observability processes or tooling, but rarely both. They also report some, but not most, key capabilities, such as identifying and resolving bugs before and after deploying to production. Lastly, at the time of the survey, approximately one in five respondents were not practicing or using observability tooling, but have plans to do so within the next year.
The Observability Spectrum
Observability is measured as a set of capabilities that help engineering teams gain comprehensive understandings of their production systems. Those capabilities are sociotechnical in nature—they have a technical component, as well as a component of social culture and practice. Our observability maturity model reflects the ongoing nature of the continuous learning and improvement needed to unlock these capabilities. This year’s survey found that most respondents have adopted some, but not all, of the practices required to achieve the biggest benefits associated with observability.
Our observability maturity model evaluates the extent to which teams practice and realize the benefits of observability, and scores them along an observability maturity spectrum. The observability maturity spectrum identifies five distinct groups:
1. No Plans. Those who do not have plans to adopt observability or whose behaviors and capabilities suggest barriers to adoption.
2. Planning. Those who indicate an intention to practice observability in the next 1–2 years and whose behaviors and capabilities suggest their organization is well positioned to succeed in observability adoption.
3. Novice. These respondents report practicing observability, but their practices and processes suggest they are in the earliest stages of adoption.
4. Intermediate. Respondents who are practicing observability and are beginning to achieve a comprehensive understanding of their production systems.
5. Advanced. Respondents who are practicing observability and realizing both comprehensive understandings of their production systems and benefits associated with that practice.
Advanced and Intermediate
In the Advanced and Intermediate groups, over 10% of those surveyed reported a combination of practices and tooling that reflect a highly observable system. Another 13% of the Intermediate respondents reported practices, tooling, and outcomes consistent with relatively sophisticated observability practices, down from 17% last year—but that 4% shifted to the Advanced group on the observability spectrum. In other words, we’re seeing groups higher on the maturity scale continue to improve.
These two groups highly prioritize observability: 50% practice observability across the organization and 43% on a team-by-team basis. Respondents also reported high public cloud use and were from a mix of company sizes, but most work at large enterprises (57%) and in the tech industry (46%).
Novice
Most survey respondents fall into the Novice group. Participants report observability processes or tooling, but rarely both. There may be several reasons for this misalignment, but we believe this may reflect mismatched expectations among practitioners in this group.
Analyzing responses, this group is more likely to self-report that they are practicing observability because they are using tools like logs, metrics, and traces. However, they also do not report having the key capabilities associated with an observability practice. They report that they do not have a comprehensive understanding of their systems. They also report some, but not most, key capabilities. Additionally, one in two respondents in this group indicated that practices like observability and DevOps are siloed within their organization or only practiced on a team-by-team basis.
Planning
Teams in this group reported practices and tooling that suggest identifying and resolving problems faster are top priorities. More specifically, in this group, one in four are at the very beginning of their observability journey and are starting to practice on a team-by-team basis (26%), while one in three respondents plan to practice observability in the next 12 months (33%). Approximately one in five respondents do not currently practice or use