As organizations continue to adopt DevOps best practices, they’ve sought out ways to track and improve their software delivery efforts. This has led many DevOps teams to utilize a data-driven approach with metrics to measure performance and drive improvements. And these same metrics can also be useful for quality assurance and quality engineering teams.

In this post, we’ll take a deep dive into the four DORA metrics and why they matter for quality assurance and engineering teams. We’ll also discuss how these metrics can be used to drive improvements in manual and automated software testing.

What Are DORA Metrics?

The DevOps Research and Assessment (DORA) team at Google analyzed DevOps practices across many organizations and identified four key metrics for measuring software development and delivery performance. Here’s a breakdown of these metrics.

Deployment frequency (DF)

Deployment frequency measures how often new code is successfully deployed to a production environment. This metric helps organizations create an overall benchmark for development velocity. If a company uses a lot of manual processes or has a slow error recovery time, they’ll likely have a low deployment frequency.

Lead time for changes (LT)

Lead time for changes is the time that it takes from an initial code commit to the code change being deployed. This is important for understanding whether there are any delays in development or throughout the continuous integration and continuous delivery (CI/CD) pipeline. For example, there could be a slow build process, time-consuming manual testing, and other inefficient processes that delay deployments.

Change failure rate (CFR)

Change failure rate is the percentage of code changes that break the system once deployed to production. This metric is the most obvious indicator of code quality, which helps organizations understand whether new releases are improving the user experience. Tracking the failure rate is important for balancing the need for quality with the previous two metrics that are focused on development speed.

Mean time to recovery (MTTR)

Mean time to recovery is the average amount of time it takes for a system to become available again after an outage or failure. Although unplanned downtime is inevitable, a low mean time to recovery indicates that the organization is adequately prepared to restore the system in a timely manner.

Why DORA Metrics Matter for Software Testing

The DORA metrics are important for quality assurance and engineering teams because they offer valuable insight into the entire software development life cycle, including manual or automated software testing processes. In many ways, putting a focus on the DORA metrics can help break down the barrier between development and testing teams, promoting collaboration, and leading to the delivery of high-quality software faster than before.

Now that we’ve covered why the DORA metrics are valuable to quality engineering teams, let’s take a closer look at how they can drive software testing improvements.

  • Deployment frequency: A low deployment frequency result could indicate that software testing or code validation processes are failing to discover broken builds early enough or simply slowing down the code release process. Strategically implementing automated test cases and minimizing manual testing could enable faster delivery with fewer failed deployments.
  • Lead time for changes: From a software testing perspective, a long lead time for changes could indicate that the testing process is causing bottlenecks in the CI/CD process. Any manual testing requirements or unreliable - or flaky - automated tests can easily impact lead times. Implementing a more modern solution in favor of reliable automated testing could help reduce lead times. Hear more in this video where the Xandr team talks about scaling automated testing in their CI/CD pipeline.
  • Change failure rate: A high change failure rate is likely one of the most obvious signs that there isn’t an effective software testing strategy in place. For example, this could mean there’s too few test cases or insufficient test coverage to adequately identify defects. By intelligently increasing test coverage with a more reliable end-to-end testing strategy, the change failure rate could be reduced and improve development velocity. Learn more about how ITS optimizes functional test coverage to build better customer experiences.
  • Mean time to recovery: Although a longer mean time to recovery rate could just mean there are limited development resources to remediate issues, by having a reliable software testing process recovery can include automated testing without slowing down recovery time. An effective quality engineering strategy should discover most systemic or high-impact defects before a build is pushed to production.

Improving DORA Metrics with Mabl

When many organizations adopt DevOps processes and begin tracking the DORA metrics, they quickly discover that the old approach to software testing with legacy solutions isn’t purpose built for today’s fast-paced development approach. They find that deployment frequency and the lead time for changes are being impacted by manual testing processes, or the change failure rate and mean time to recovery metrics reveal a lack in software quality.

A low-code test automation platform like mabl can help alleviate these delivery bottlenecks while also improving software quality. Mabl can be easily embedded in your CI/CD pipeline and as a native cloud solution it automatically scales to meet growing test coverage needs. Using mabl’s in-depth release test coverage reports, organizations can identify any gaps in their software testing strategy that could be impacting quality. And the low-code approach empowers anyone to create, run, and manage automated tests to improve quality without slowing development or straying from best practices.

The DORA metrics offer valuable insights for DevOps and quality engineering teams, but organizations still need to take action to improve their software quality and delivery speed. With mabl, organizations can lower the barrier to software testing and scale their quality engineering efforts to capture the full potential of DevOps.

Want to hear more? Sign up for your free demo of mabl here.