Continuous testing

Recently someone pointed me to a Twitter thread which had a link to an article about “Continuous Testing”. I was surprised to read this definition of continuous testing in the article:

Continuous testing is the process of executing automated tests as part of the software delivery pipeline to obtain feedback on the business risks associated with a software release candidate as rapidly as possible.

Well, sure, that is part of continuous testing, and an important part. But there is so much more. 

After I read that, and looked around some more, I find there are entire books on continuous testing that use the same narrow definition of “only the automated regression tests in the deployment pipeline”. And whoever wrote the Wikipedia entry on it also agreed with that narrow view.

Not my definition!

I have a very different idea about what constitutes “continuous testing”. In my mind, “agile testing” is the same thing as “continuous testing”. Janet Gregory and I, along with others in the agile testing community, define agile testing this way:

Collaborative testing practices that occur continuously, from inception to delivery and beyond, supporting frequent delivery of value for our customers. Testing activities focus on building quality into the product, using fast feedback loops to validate our understanding. The practices strengthen and support the idea of whole team responsibility for quality.

For a more specific definition of continuous testing in DevOps, I find Dan Ashby’s definition, and especially his visual model (which many other authors, including myself, Janet and Katrina Clokie refer to in our books), reflects what my own high-performing teams have embraced over the past decades. He doesn’t have a short definition, but his model, based on the DevOps infinite loop, expresses it well:

Dan Ashby, “Continuous Testing in DevOps”, 2016

Dan’s post has another great model for continuous testing near the end of that post, showing that our testing starts with testing new feature ideas, and extends into the many other possible types of testing we might do on a software product. And he’s come up with another model too, you can see it in his tweet. 

More perspectives

I asked around on a couple of testing community Slack workspaces to see what other people think of when they hear “continuous testing”. I like what Benjamin Fellows described on the Ministry of Testing’s testersio.slack.com workspace:

I've taken it to mean no testing phase and instead taking elements from both shifting left and shifting right (testing throughout the development and operations process, from testing initial requirements to using customer metrics to further test after release)

Matt Lievertz posted his view on the same thread, and I find this one interesting too:

I’m not sure if there’s a normal definition, but if I heard it I would suspect it referred to either:

1) the testing component of continuous integration

2) continuous monitoring in prod — testing in prod continuously as a functionality-level monitor

The monitoring and observability aspects of continuous testing are definitely vital. 

Others in this conversation said they could see value in both definitions, but they tended to go with the Dan Ashby perspective. 

I blogged about my own views of continuous testing, based on both Dan’s models and on Jez Humble and David Farley’s Continuous Testing book. Peter Zimmerer’s comment and the links he includes are worth a read. He did some research on the “real essence of continuous testing”, and as a result, he uses two “fundamental objectives and characteristics”, which I like a lot:

Continuous testing means:

1. Testing over the whole lifecycle by shift left and shift right, i.e. testing continuously from begin to end

2. Strategic test automation, i.e. continuously reuse & adapt & execute (only the) needed tests in an intelligent manner

Other names for continuous testing?

When people use the term “continuous testing”, their meaning might be quite different from my own. Should we come up with some other term that’s not being used anywhere else yet? We’d love to hear your ideas on terminology, please use the Twitter button at the bottom of the page to let us know. 

Heather Reid started a discussion on the same topic on the Ministry of Testing Club. I liked Sergio Freire’s idea to use a different name for “continuous testing”, to make it more clear what we mean:

For me Continuous Testing is more Omni Testing (or Omnipresent Testing), in the sense that testing is always present during the whole development life cycle.

Some people may think about Continuous Testing as meaning tests are “continuously” running; these tests are assumed to be automated ones (i.e. automated checks).

In my opinion I would prefer Omni Testing as I see testing as something that is always present, no matter what kind of approach you’re following or at what phase of your development life cycle you are.

You use testing to guarantee and seek a bunch of quality attributes.

Janet Gregory and I write about continuous testing models in our upcoming new book, Agile Testing Condensed. Janet came up with a term I really like: “holistic testing”. 

We learn about a feature that our customers may want, we build and deliver it, and we learn how the customers actually use it. We use that feedback to decide what to build (or remove) next.

I continue to ask people about their own definition of “continuous testing”. I’m trying to keep an open mind, there is always more to learn. Whatever we call it, testing is not ever separate from software development. It’s integrated with coding and many other activities throughout the infinite loop of software delivery.