How DevOps Impacts Testing

Early on in my programming career I learned that it pays to make friends with the operations people and work closely with them. Back in the day, we depended on operators in the machine room to mount our tapes and keep our long batch jobs running smoothly. We needed the system administrators in the basement to keep our production environments healthy and make configuration changes for us. I often baked brownies and visited the machine room and the basement to share them.

Working with people of all specialties, whether or not they were actually on my development team, came naturally to me, I never felt siloed. In the 00s, we started getting better tools that let all of us participate in maintaining continuous integration pipelines, adding automated regression test suites, investigating build failures, checking logs to try to understand production failures. I collaborated every day with our sysadmin, who spent time every day testing and automating tests.

Our whole team took responsibility for quality, and for making sure all testing activities got done for each story and feature before we delivered. Testing is a team sport, not the responsibility of a test or QA team or individual.

In 2009, I read Continuous Delivery by Jez Humble and David Farley. It is pretty much all about testing! I realized my own team had been doing continuous delivery (CD) without using that term. The book pulls together all the practices and guides teams working towards CD.

2009 is also when we started hearing the term “DevOps”, after the first Devopsdays conference in Ghent. The term didn’t make a big impact on me since my teams had operations and development integrated already. But it helped me see that a lot of teams, even teams self-identifying as “agile”, were still throwing builds over the wall to operations teams to deploy and manage in production.

Whatever terminology you prefer, the culture, principles, and practices associated with DevOps continue to influence how delivery teams approach testing. Let’s look at a few examples.

A holistic, whole-team approach

A whole-team approach to testing and building quality in is a central foundation of agile development. Many teams fail to build the necessary relationships and break down the silos of specialties. They may do some of the rituals around agile, like short iterations and daily standups, but they succumb to “mini waterfall” or go off the path to agile in other ways.

Logan Miles recently wrote that teams that start with a DevOps transformation tend to find agile development along the way. Rather than trying to learn a set of practices, DevOps puts the focus on the desired outcome: delivering small changes to production safely and frequently. That focus helps teams address difficult challenges like building shared understanding of stories and features among all team members to avoid re-work and shorten cycle time. To get the confidence needed for CD and to shorten cycle time, teams learn to guide development with technology-facing and business-facing tests, automated regression test suites and deployments, and other time-saving practices. This frees team members up to do human-centric testing activities like exploratory testing and accessibility testing.

The whole-team approach to quality is backed up by data from several years of results from the State of DevOps Survey. The book Accelerate, by Dr. Nicole Forsgren, Jez Humble and Gene Kim, reports that having reliable automated tests is a predictor of IT performance. This reliability is a result of developers taking responsibility for creating and maintaining acceptance tests - in collaboration with testers.

DevOps is all about removing siloes. Separate test or QA teams are generally an anti-pattern. But we still need people with specific testing skills, knowledge and experience embedded in cross-functional teams, building a quality DevOps culture together. Team members with these specialized capabilities support the team’s ability to build testable code. They transfer testing skills across the team. They ask good questions, make risks visible, contribute to the team’s continuous quality improvement.

DevOps promotes team responsibility for the product not only during development but after release to production. We get more opportunities to learn operations-related skills. I often see testing specialists take their turn in on-call rotations. They also help test the on-call playbooks and processes that help teams respond to production failures. Win/win/win.

The scope of quality and testing is extending to include infrastructure as well as tooling to learn about production use of our applications. Many testers and QA professionals are evolving to take on more of a test consulting role on all these different aspects of testing, and encouraging a test-first approach to infrastructure as code.

Test infrastructure autonomy

As the teams I’ve worked on got control over our product’s infrastructure, and were empowered with new tools and technology such as cloud platforms and services, we were free to solve our own problems around test environments. Where we previously contended for too few test environments in VMs on physical servers, now we could spin up both persistent and temporary test environments in the cloud, and deploy whichever version of our product we wanted for testing.

Our pipelines automatically deploy builds from master/trunk or story branches to appropriate environments. We know when a new build is ready to test, where it is deployed, and what code changes it contains

The cloud capabilities associated with DevOps enable teams to refresh test databases at will, add test data as needed, change the configuration of the application, restart it when needed. We no longer have to wait around for sys admins to make changes for us, and task switch while we wait. We have more time for human-centric testing, and reliable test environments which we can control. This is a must for teams wanting to do continuous delivery.

Logging, monitoring, and observability

Cloud infrastructure and the ability to store huge amounts of data cheaply mean teams can instrument code to log every event in the application. We’ve had monitoring and alert tools for a long time now, but as data storage and infrastructure become more affordable, we have the ability to see exactly how customers are using our product.

The ability to safely test in production is expanding our testing approaches. No test environment can replicate production, and we can’t predict all the potential scenarios real people will do. Teams can use techniques like release feature toggles to hide new capabilities while they test them in production. A new range of products and tools are available to make it even easier to do dark launches, canary launches and A/B tests.

Chaos engineering is a scary-sounding term, but it’s a safe way to identify production issues before customers are affected. There are many prerequisites to doing these types of experiments in production, like the ability to deploy new builds reliably and quickly, good rollback strategies, adequate logging and analytics and the ability to trace across distributed systems, effective monitoring and alerting. This is a perfect example of how DevOps has opened up new possibilities for testing and improving product quality.

There are so many amazing analytics tools today. All of these capabilities mean we can track how individuals use our application and pinpoint production problems in a matter of seconds.

New risk-based approaches

Thanks to changes in team culture, infrastructure, and tools associated with DevOps, teams can mitigate risky areas of their applications in multiple ways. We can evaluate each risk and decide how to test it before and after deploying it to production. We can test a much bigger and more widely varying range of scenarios. We have more time for human-centric testing activities such as exploring, accessibility testing, security testing. It’s easier to determine the impact of different production loads and stresses.

How is your team applying DevOps principles and practices? How can you use those to improve how you do your testing, engage team members of all specialties in testing, and learn more ways to build quality into your product? These are great topics for conversations with your delivery team, including operations specialists.