The 2024 State of Testing in DevOps Report

Understanding how companies are progressing in their journeys to DevOps transformation demands a roadmap. Though DevOps adoption and maturity will look different at every organization, we broke the road to DevOps into four stages: aspiring DevOps teams, striding towards DevOps teams, mostly DevOps teams, and fully DevOps teams.

Once an organization has reached full DevOps maturity, they have adopted CI/CD and automated most (or all) workflows in their development pipelines, created seamless processes for collaboration, learned to measure their performance with more effective metrics, and have established the foundation for continuous improvement.

Despite emerging in 2009, DevOps is still a work in progress for most organizations: just 1-in-3 teams (33%) consider themselves mostly or fully DevOps, with an equal number making early strides on their DevOps journey.

The slow journey to full DevOps maturity, however, isn’t due to lack of desire from companies. When asked about their biggest DevOps obstacles, just 11% of respondents said that their primary challenge is a lack of prioritization from their organizations.

Instead, respondents shared that their DevOps transformations are more likely to be slowed by technology limitations and budget constraints. 2024 marked the first time since mabl began running the Testing in DevOps Report that technology limitations were the most common DevOps obstacle, narrowly beating budget concerns. This is a notable shift from mabl’s previous report, which found that 82% of teams were slowed by non-technological challenges. 

The rise in technological challenges can largely be attributed to mature DevOps teams in the “mostly DevOps” or “fully DevOps” stages of transformation. 40% of fully DevOps teams say that technology limits are hindering their DevOps practices, compared to just 20% of fully DevOps teams who say that budget is their primary challenge. In contrast, early stage DevOps companies are much more likely to be held back by budgets, with approximately 1-in-3 aspiring DevOps teams reporting budgets as their primary concern. 

The shift between budget constraints and technological limitations as teams reach DevOps maturity underscores the importance of early investments in DevOps tools. If they choose the less effective or useful solutions, an aspiring DevOps company will see a worse return on investment and set themselves up for ongoing challenges on the road to DevOps. If they identify tools and capabilities that support long-term success, however, the odds increase that they’ll join the elite 16% who have succeeded in their DevOps transformation. 

One significant trend identified in the State of Testing in DevOps Report: 83% of teams have accelerated deployments in the last 12 months. This trend was consistent across mobile app development teams and web app development teams, signaling a growing emphasis on delivering new innovations to market at an increased pace.

That said, the survey also revealed that deployment frequency improvements are still needed by a vast majority of software development teams, as 80% of respondents shared that their organizations have release cycles of a week or longer. This includes the two-thirds of both mobile app development teams and web app development teams shipping on similar cadences. 

The fact that almost two-thirds of mobile app and web app development teams are shipping software once per month or less is striking, but particularly notable for mobile applications, which can face penalties from app stores if their applications aren’t updated frequently enough or fail to meet certain performance, accessibility, and compatibility standards. More frequent updates also improve an app’s standing in search rankings in app stores, building a competitive edge for agile companies who are able to ship software more often than their competition. Being able to consistently deliver new features, important security updates, and performance improvements are essential for acquiring and retaining customers, yet the majority of development organizations are currently unable to do so. 

The lack of high-velocity teams mirrors the large proportion of companies still struggling to automate their development pipelines: 87% of teams have yet to fully adopt CI/CD, with another 4% of respondents unsure of their automation capabilities. 

Low pipeline automation impacts teams on multiple fronts. For one, teams with less automated pipelines face an uphill battle as they accelerate deployment frequency since their team’s workload is compounded by repetitive tasks, which increases burnout and turnover. The costs of low automation then multiply across the organization as workloads increase, burnout builds, and employees quit. Unless these organizations hire more employees or find tools to reduce the burden on their current workforce, there’s a human limit and financial cost to how fast their teams can deliver new innovations to market.

DevOps success, though a priority for 89% of companies, is still an ongoing challenge for two-thirds of companies. With technology limitations rising 44% since 2022, it’s clear that early investments in automation and tooling are critical for teams in the early stages of their DevOps transformation. Slow progress in pipeline automation indicates an opportunity to surpass competitors through smart investments in technology that further streamlines development and supports continued gains in deployment frequency. 

Software testing teams, despite being the connective tissue between how apps are built and how apps are used, are managing a rising tide of manual testing and legacy test automation challenges.

Manual testing dominates the current testing landscape, with 74% of teams using it in their software testing strategies. A majority (60%) also reported using open source test automation tools like Selenium or Playwright, with slightly less than half of companies relying on commercial test automation tools to support quality.  

The reliance on manual testing extends to mobile application teams. Almost two-thirds of these teams report using manual testing in their quality strategy, followed by 56% of teams leveraging automated testing. Slightly less than one-third of mobile app development teams say they’re using third party device farms. 

The widespread reliance on manual testing is unsurprising, as manual testing is essential for exploratory testing for new features and edge cases. But if manual testing makes up the bulk - or even the entirety - of an organization’s software testing strategy, there’s a greater risk of bottlenecks as deployments accelerate. 

When it comes to automated testing tools, teams are more likely to leverage open source tooling until they reach the final stage of DevOps maturity. As organizations progress on their DevOps journey, the likelihood that they utilize commercial test automation tools increases significantly. 

The impact of open source test automation can be seen in how development teams spend testing time. When asked about the most time-consuming testing tasks, a plurality of respondents (21%) ranked test maintenance as their top choice, followed closely by test execution (19%).  

Test maintenance rose from a distant second place in 2022 to earn the dubious distinction of being the most time-consuming testing task in 2024, with 1-in-5 teams reporting it as the largest time commitment needed for testing. 

The most time-consuming testing tasks closely corresponded with the most significant testing headaches, with one-third of respondents saying that test maintenance was their biggest pain point. This percentage is almost double the amount of teams who said that test maintenance was their biggest testing challenge in 2022, a massive shift that signals the challenges of flaky, unreliable automated tests as deployment frequencies accelerate. 

The rise of test maintenance as the #1 testing challenge corresponds to mature DevOps teams empowering more people to contribute to testing, investing in more testing tools (below), and identifying technology limits as their main DevOps obstacle. Software testing and test automation are clearly important to DevOps transformation, as illustrated by a growing investment in automated testing tools, but unless test maintenance is sustainably managed, DevOps teams will struggle to reach their full potential as flaky tests cause delays in delivery cycles.

Collectively, these changes signal that DevOps teams embrace a culture of quality that engages everyone in testing, which is further supported by the fact that fully DevOps teams are the most likely to report that everyone on the team has a role in quality. Approximately 1-in-3 fully DevOps teams responded that testing is a shared activity across the organization, versus roughly 1-in-5 mostly DevOps teams or striding DevOps teams. 

Fully DevOps teams reported a rise in testing contributions from QA professionals and manual testers, automation engineers, and software engineers, signaling that testing needs and testing collaboration increase with DevOps maturity. 

The testing needs and concerns of these different groups vary significantly, as do their areas of expertise. Meeting these varying needs, therefore, requires a diverse set of testing tools. 

The increase in roles contributing to testing coincides with an increased investment in software testing tools as organizations mature their DevOps practices. Virtually zero respondents (5% total) said their organizations were using no testing tools, while a similarly small percentage were using one testing tool (8% of aspiring and mostly DevOps teams, 7% of striding DevOps teams, and 6% of fully DevOps teams). A majority of companies (58%) are using 3+ testing tools, despite the fact that many teams are struggling to maintain sufficient test coverage and the fact that budget issues and technology limitations are the two most common DevOps obstacles. Software testing is clearly important for DevOps maturity, yet complex testing technology stacks aren’t solving the core challenges of testing in DevOps.

Once teams reach three or more testing tools, a pattern emerges: fully DevOps are more likely to be using a greater number of testing tools. A full quarter (26%) of fully DevOps teams are using at least five tools, with another 12% using four tools, and 31% using three. 

The growing testing tech stack illustrates the growing complexity of managing testing in DevOps. Not only do DevOps teams need automated testing tools that can be used by a variety of roles and skills, as seen above, but they also need to validate functional and non-functional quality, multiple devices and browsers, plus APIs.

Testing across each of these aspects is essential, but a growing testing tech stack means increased costs, increased silos, and increased onboarding time. Testing is an important priority for DevOps maturity, but teams need to consider what automated testing tools can help them avoid duplicative work, unnecessary costs, and siloed workflows. 

Given the impact of testing tools, it’s no surprise that a plurality of survey respondents reported that their organizations plan to invest in new automated testing tools to improve their approach to quality.

Underscoring the need to further improve testing tools on the road to DevOps is that new test automation tools are the #1 way companies plan to invest in testing, even as budgets are being limited. Aspiring DevOps teams are even more likely to buy new software testing tools, with 48% planning to make new purchases in the near future. This trend indicates that these organizations are prioritizing bringing their testing tech stack up to industry standards. 

The right testing tools can help address other quality priorities ranking high on companies’ to-do lists, which likely explains why new testing tools took the lead. 

New testing tools are also essential to support the wide remit of testing types being handled by in-house QA and development teams. The 2024 Testing in DevOps survey found that the vast majority of tests are being planned and executed by employees, not third-party consultants. 

Across web applications and mobile applications, functional testing and non-functional testing, teams are being asked to handle a greater share of testing strategies. This is a marked shift from mabl’s previous Testing in DevOps Report, which found that 61% of QA teams relied on some level of third-party support. Even specialized test types like accessibility testing and performance testing are largely performed in-house, indicating that companies are focused on building internal capabilities and talents while on the road to DevOps. 

Perhaps no technology has become essential as quickly as AI, which has sprouted from futuristic curiosity to must-have capability.

4-in-5 teams have started incorporating AI into their development processes, with 60% reporting that their development and QA teams have somewhat or fully embraced AI. Interestingly, that proportion rises to 76% for fully DevOps teams, indicating that DevOps maturity sets the stage for ongoing technological evolution and adaptation.

How AI is being applied to software testing varies by DevOps maturity and pipeline automation levels. When accounting for pipeline automation levels, the popularity of AI use cases shifts drastically. Organizations with fully automated pipelines are more likely to use AI to summarize test results. In contrast, fully DevOps teams are more inclined to harness AI to generate test cases.

The popularity of using AI to summarize test results is an interesting trend, considering that test analysis was ranked as the least time-consuming testing task.

In the same vein, fully DevOps teams using AI to generate test cases also aren’t using new technologies to tackle the most burdensome testing activities (test maintenance and test execution). This suggests that AI tools are still evolving to meet the most acute bottlenecks impacting testing in DevOps.

With a growing number of testing tools and quickly evolving set of AI capabilities available, a final question remains: when are DevOps teams testing in their development pipelines?

Both functional and end-to-end tests are carried out throughout development, though in notably different concentrations. Most teams (52% code stage and 41% pull request stage) are doing the bulk of their functional testing, most likely unit and integration testing, in the early stages of development pipelines. In contrast, end-to-end testing is more likely to be performed in the deployment stage of development. 

The number of issues discovered late in the development process is likely connected to the number of teams with low test coverage across their web applications and mobile applications. 67% of both web and mobile app development teams report having less than 60% test coverage, while 1-in-5 teams have less than 20% test coverage.

Test coverage challenges proved even more acute for non-functional testing: 73% of mobile app development teams have less than 60% non-functional test coverage, and approximately 1-in-4 mobile teams have less than 20%. On the web app development side, 71% of teams have less than 60% non-functional test coverage and 25% have less than 20% test coverage.

A lack in non-functional test coverage can introduce serious headaches - and therefore churn - to customers. A 1-second delay in response time can reduce web conversions by 7%, and 53% of consumers will abandon a mobile site if it takes more than 3 seconds to load. Likewise, 45% of people will leave a slow application with a bad impression of a business. A lack of accessibility testing can result in lawsuits and penalties, particularly for industries like government, consumer retail, and healthcare. 

Effective non-functional testing is particularly important for mobile applications since Google and Apple have accessibility and performance criteria for their respective app stores. Failing to meet these requirements can result in delays in app updates, downgrades in search results, and even removal from app stores. 

Identifying defects that late in the development process can cause headaches for deployment frequency and DevOps maturity, particularly most development teams (53%) are considered mid-tier performers who take at least one full work day to deploy fixes, with a median of 11.12 hours (1.5 days) necessary to address issues. Though this aligns with medium performance according to DORA, 11 hours can cause delays when deploying on a weekly or daily basis. 

A key part of being able to address bugs efficiently and mature DevOps processes is an organization’s ability to collaborate cross-functionally, which is reflected in the handoff process between QA and developers. According to the 2024 Testing in DevOps survey respondents, 43% of development teams need to improve the handoff process. 

Despite the fact that mobile teams are more likely to catch defects earlier in the development process (though 48% of bugs are caught in the deployment stage), issues collaborating between QA and developers is likely to undermine any efficiency gains made with early detection.