Software Testing Strategies for Improving DevOps Metrics

The software industry is staring down a tidal wave of disruption: AI, the shift to digital-first and hybrid customer experiences, and cloud adoption are all transforming how software is created, tested, and delivered to consumers. To manage these disruptions, companies are turning to digital transformation and DevOps to help their teams optimize and improve their delivery pipelines. 

Organizations of any size, industry, or maturity can harness DevOps best practices to improve their performance, particularly when they leverage the DORA research program. DORA, formally known as DevOps Research and Assessment, is a program run by Google Cloud that produces a plethora of resources for development organizations looking to build better products, improve efficiency, and improve team culture. 

Every year, DORA produces the Accelerate State of DevOps Report, a comprehensive look into the DevOps tools and practices that help software development teams perform better. This year’s report gathered insights from over 36,000 professionals from across the industry, giving the DORA team an in-depth understanding of team performance across the industry. Using this data, they identified the most important traits for high-performing teams:

• They build with the user in mind
• They embrace flexibility with cloud
• They use AI and automation for faster code reviews

Measuring DevOps Performance with DORA Metrics

Before exploring the role of software testing in DevOps, it’s essential to understand how DORA metrics measure DevOps performance:  

• Deployment Frequency: How often an organization successfully releases to production
• Lead Time for Changes: The amount of time it takes a commit to get into production
• Change Failure Rate: The percentage of deployments causing a failure in production
• Mean Time to Restore Service: How long it takes an organization to recover from a failure in production

Though DORA metrics are a short scorecard for DevOps performance, they capture a holistic picture of development team capabilities across velocity, quality, and efficiency.

How Test Automation Impacts DevOps Best Practices

Software testing directly impacts an organization’s ability to improve their DORA metrics and DevOps performance. If a team has a high deployment frequency and a high change failure rate, they’re more likely to have low test coverage. Conversely, a team with low change failure rates and a low deployment frequency may need to consider how to streamline quality through test automation, shift-left, and/or continuous testing. 

Looking at the trends identified in the 2023 Accelerate State of DevOps Report, quality teams have even greater opportunities to impact their organization’s DevOps performance moving into 2024. 

Build (and Test) With Users in Mind

The State of DevOps Report found teams focused on their user have 40% higher organizational performance, noting that “teams that focus on the needs of users build the right thing AND build the thing right.” The report defined user-centricity with a few key attributes: 

• A clear understanding of user goals
• A definition of success that considered the value provided to the end-user
• Continuously updated and reprioritized specifications and requirements that reflect current user behavior

The right software testing strategies, particularly when supported by the right test automation tool, play a critical role in building better customer experiences. Software testing strategies that contribute to better customer experiences include: 

• Refine software testing with real-time user data: Integrating automated testing with a customer data platform like Segment ensures that test coverage not only remains high, it remains accurate. Modern test automation solutions capitalize on the data offered by CDPs to flag popular, but untested, pages to quality teams and developers so that test coverage can be continuously tailored to user needs. 
• Embrace functional and non-functional automated testing: The user definition of quality can’t neatly be divided into ‘functional’ or ‘non-functional’ buckets. The more likely scenario is that consumers are simply annoyed by unstable or slow websites, illegible color schemes, or pages that are unusable with a screen reader. Adopting automated performance testing and automated accessibility checks ensures that all consumer preferences are considered in DevOps pipelines. 
• Test the entire customer journey: A simple customer journey can quickly become complicated with automated testing. An automated test likely needs to cover a marketing email, a coupon code, and an invoice email with a PDF attachment. But the story doesn’t end there: the same test also needs to test an API for a payment service like Square or Afterpay to ensure that customers can complete the checkout process. These comprehensive end-to-end tests, though traditionally difficult to do with traditional test automation tools, are becoming essential for building better user experiences in DevOps pipelines. 

Increase Infrastructure Flexibility with Cloud-Based Test Automation 

Cloud-based test automation tools contribute to the benefits of cloud infrastructure. The Accelerate State of DevOps Report discovered that using a public cloud leads to a 22% increase in infrastructure flexibility, which leads to 30% higher organizational performance. 

These benefits are clear when applied to automated testing, as development organizations

can seamlessly scale software testing as their DevOps practices, product, and quality engineering practice evolve. Cloud-native test automation that ensures the reliability and resiliency of tests gives software development organizations the flexibility and power to execute comprehensive end-to-end tests and regression tests without slowing velocity. 

Even when teams need to run cross-browser tests in parallel and end-to-end tests, cloud-native testing enables rapid results. Teams have the ability to run the same tests in parallel across multiple environments, including ephemeral preview environments, persistent staging environments, and production environments for consistent quality.

Use Automation and AI to Make Code Reviews Faster

The all-important code review is central to improving DORA metrics and overall DevOps performance. According to Google’s 2023 State of DevOps Report, teams with faster code reviews have 50% higher software delivery performance. 

Though software testing usually has a role in code reviews, that role has been primarily restricted to unit tests, static analysis, and other code-level checks. But that approach risks testing gaps that result in costly defects in the later stages of the DevOps pipeline. 

Balancing the value of more comprehensive testing during code reviews with the need for faster code reviews demands test automation and artificial intelligence. Modern test automation frameworks harness AI to overcome the challenges of automated testing by autohealing broken tests and adapting test execution time to match the pace of the application, so tests are less likely to require time-intensive maintenance. 

These more comprehensive tests make the code review process faster by providing reviewers with insight into the existence or the lack thereof code quality regressions, and help teams catch issues earlier on in the development process. Packing more software testing into DevOps pipelines gives the entire organization higher confidence in the product, even as they embrace greater pipeline automation and CI/CD. 

Harnessing DORA Metrics to Shape DevOps Best Practices 

The 2023 Accelerate State of DevOps Report is a valuable look into the DevOps best practices and DevOps performance metrics that shape high-performing software development teams. As engineering and quality leaders look to invest in tools and strategies to improve DORA metrics, test automation and AI are prime opportunities to support stronger software testing and adopt the best practices recommended by Google. 

Start your journey to better DevOps performance with mabl’s 14-day free trial.