As we approach 2019, we can look around and see an amazing range of frameworks, libraries, drivers and other tools that help with automation testing. Yet there are many software delivery teams out there that still have little or no regression test automation. Desperate to release new features, they “save time” by putting off any effort to automate regression tests. As the saying goes, they’re too busy drowning to learn how to swim. Or, they fear they don’t have the skills to do automation testing, or lack money to invest in tooling. Let’s look at some ways to address these testing problems – which, as Jerry Weinberg said, are really people problems.
Getting over manual regression testing pain
When I first joined my previous team, it had thousands of automated regression tests at all levels, but still relied on lengthy manual release regression checklists to feel confident about deploying to production. If you’ve ever had to work through one of these checklists over and over, you know how soul-crushing that activity can be. It doesn’t take long before you’re tempted to skip checks for areas that haven’t regressed in a long time, or avoid scenarios that are difficult and take a long time. It’s easy to make an error doing a manual test and mistakenly think you found a bug, or miss a bug that was there.
Who feels this manual regression testing pain? Typically, only the testers (or QA professionals, if you prefer that term). Too often, the programmers (developers, coders, engineers, choose your preferred term) are being paid to write production code. Testing is seen as a separate phase. The programmers will feel pain eventually. When a bad bug is logged days or even weeks later, they have to stop what they’re doing, context switch, remember what that code does, and try to fix it without breaking something else.
Long cycle times and re-work aren’t conducive to a team’s ability to deliver new business value frequently, at a sustainable pace. If your team hopes to embrace continuous delivery, you’re not going to succeed if you have to stop and do manual regression testing. Even if you release only once a week or once a month, without automation testing, it will take longer and longer to deliver new features and bug fixes.
Make it a team problem
Here’s an idea that has worked like a charm for me on four different teams. Put your manual regression checklist somewhere that everyone can access, like a wiki or an online project tracking tool. Assign each team member a set of tasks to do. Before every deploy to production, everyone - programmers, product owners, designers, Scrum Masters, managers, BAs, database people, system administrators - stops their usual work and works through their assigned section of the manual regression checklist. With everyone contributing, you finish regression testing much faster. More importantly, now everyone feels the regression checking pain. Nothing motivates a team to start solving automation problems like having to do tedious manual steps over and over.
Yes, you need management support to do that. Explain the concept of technical debt to your managers. Translate that opportunity cost of long cycle times to money amounts for them. Ask them what nightmare posts they would NOT like to read on social media after the next big release.
Next, get the team - or if it’s a large team, a representative cross-section of 5-10 people - together to discuss how to tackle automation testing. There are good models to help you identify your riskiest areas and identify a strategy that will work for you. Start small, work on your biggest problem first. Design small, frugal experiments to make that problem a little smaller. Set SMART goals to track the outcomes. Revisit them frequently in your retrospectives, alter them or try new ones as appropriate.
Test automation is hard
In my experience, automating at the unit level is usually the place to start. We know that test-driven-development is hugely effective in many contexts. The problem is unit test automation - really, all automation testing, if you’re having to write the automation code from scratch - has a steep learning curve. Brian Marick called this the “hump of pain”. Until you have a substantial library of unit test pieces and parts, you’re doing everything from scratch and it takes a lot of extra time. Plus, programmers who’ve never done testing before aren’t sure how to specify the tests, what coverage is appropriate at the unit level, and how to write assertions that won’t get false positives or negatives.
Once you get enough experience and a big library of unit test components, it suddenly becomes easy. Everyone I have worked with who got over the hump would never go back to coding without unit tests.
Buying time to learn
"OK," you say, "automation testing sounds like a good investment to make." But it takes so much time to do manual regression checking, you don’t have time to learn this difficult art of automating unit tests. How do you stop the madness? You’ve made this a team problem, and normally, I recommend that programmers collaborate with testers to achieve automation testing. But if the programmers are busy learning how to automate unit tests, they might not be open to yet another new practice like pairing with a tester.
Here’s what has worked really well for me when I’ve been in this situation. First, I (and other willing and available team members, if possible) go talk to the business stakeholders - the operations folks, customer support, sales, marketing, and so on. Find out what parts of the application absolutely have to work. They’ll say “everything has to work”; remind them about how Twitter might light up after a high-impact regression failure.
Our product was a web-based application. My next step was to research tools that can make it easy to automate simple smoke tests through the UI. I found an XML-based tool that we could easily plug into our continuous integration to run regression tests. I quickly automated tests for the most critical capabilities of our app, and within a few months we no longer had to do any manual regression testing.
I have programming experience – I could have written automated test scripts. But the tool we used sped the process up considerably. It didn’t allow any logic in the tests, but simple, mostly happy-path smoke tests provided enough protection for our financial services app while the team ramped up on TDD and later, acceptance test-driven development at the services or API layer of the app. Later, we did need more sophisticated automated UI regression tests, and adopted a framework and test library so that we could write the scripts we needed.
Can your team come up for air?
Today there are lots of tools (such as mabl) that make it possible for non-coders to do automation testing with a minimum of help from coders. The technologies behind some of them, such as machine learning, provide benefits that appeal to coders as well. Reach out to testing forums and social media to find out what’s working for other teams. Make sure to get consensus from the whole delivery team on the automation testing tool you choose.
Look for tools that let you apply the patterns and practices that make tests easy to maintain while producing trustworthy results. For example, there should be some way to create reusable flows or modules for steps that are required in more than one test.
Invest time to learn some basic test automation patterns to produce the most effective automation. Start automating simple smoke tests for those highly critical areas of your product. Start small to see if the tests provide quick and reliable feedback before you invest a lot of time. Remember that these don’t have to be a permanent solution. The idea is that you are automating your release regression checklist, so you can use that time to build a long-term automation testing solution.
Your team will soon be able to make changes to the code more quickly, because now there’s starting to be a safety net of UI smoke tests along with some automated unit tests (hopefully at the same time). You may notice an even bigger benefit: everyone will be thinking more about testing, and designing the code to make automation testing easier.
Shorter cycle times ahead
After some months, the team can start thinking about automation at other levels such as API and services - there are useful automation testing tools available to help with that too. You may decide to add more UI automation, refactor your initial smoke tests, or find a better long-term solution if needed.
As your automated safety net grows, your team will get faster and better feedback from test suites run in your continuous integration. You can learn how to use executable tests, both technical- and business-facing to guide coding, as in test-driven development and behavior-driven development. The team will be able to deliver value to customers more frequently, while enjoying the delivery process a lot more!
I’ve talked a lot here about tools. I don’t want to imply that these tools will somehow do magic for you. Your team has to pull together and do the work, but modern automation testing tools can help you stop mushrooming technical debt and start making time to learn how to solve the problems.
With mabl, you can have unlimited users on your team in the same workspace. You can try mabl for free here.
