The continuous stream of new technologies and the demand for faster product development continue to drive quality standards higher and higher. New ways of testing, like automated and continuous testing, and movements such as DevOps are increasing development speed and adding flexibility in software product development pipelines. To remain competitive, development teams must always be pursuing pipeline and testing process optimizations.
Maximizing the trifecta of speed, cost, and quality is a goal that all development teams pursue—and it is often their greatest challenge. Quality assurance teams usually feel this pain the most, and often must make tough decisions to find a compromise between the three. The team might try to deliver the product faster, but there will be a higher risk of bugs being found in production. If management insists on maximum quality, the schedule might need to slip. By automating as much testing as possible—especially the easier, "boring" tests—your team is much more likely to get closer to reaching their optimization goals.
As your team considers adopting test automation to address the ever-increasing pressure for rapid delivery and higher-quality product releases, it’s very important to step back and consider if the investment of effort and capital will be worth it. Before you take any steps to develop a testing automation strategy, it’s important to first define your expectations—what you expect to result from the transition to test automation. Compare these gains with the necessary investments to determine the return on investment.
Avoid Automation Pitfalls
As you move forward in your test automation initiative, keep watch to ensure that you keep from making classic mistakes:
-
Remember that manual testing will remain important until the end of time. Though automation will have increasing importance for your team, there are many contexts that will continue to require manual test case executions.
-
Repetitive or redundant tests are in most cases quite suitable for automation, since running the same types of tests repeatedly is often tedious and eventually leads to error. By contrast, many less-straightforward tests may always require some level of human observation. It may be impracticable, for example, to automate the determination of whether a website is aesthetically pleasing. Another example: it may be challenging to automate a navigation menu test that accurately verifies if the menu remains user-friendly after changes have been made. Such tests are probably most effective if they remain manual.
-
It can be disastrous if you don’t make the effort to synchronize your automation tool stack together with the capabilities of your organization. To successfully implement your automation strategy, it’s necessary to maintain product knowledge while acquiring automation knowledge.
-
Plan for perpetual test maintenance. After developing and implementing much of an automation strategy, it will be necessary for your team to regularly update and maintain the created tests. As you build new functionality and add product improvements, your test suites will expand accordingly. Ensure that all of your tests remain relevant and reliable by maintaining them carefully. mabl eases the test maintenance inevitability by auto-healing your test scripts when your application UI changes under development.
Which tests should you automate?
Many tasks—those that entail the monitoring of system performance or behavior—are usually done much more efficiently by a computer or machine. Testing that involves interpretation, many forms of interactive participation, or uncertainty is best done by humans.
Another way to view it is that test automation is often applied to the performance of mundane and repetitive tasks. On most software projects, there are many such testing tasks, and automation of these tests can significantly reduce delivery time and costs. In many teams, this can save some members enough time for them to completely reallocate it to other productive activities.
Though it can save teams much effort and money, test automation cannot address all software quality issues. Too much automation in the wrong areas can actually decrease testing efficiency. Plan ahead and analyze so that you can determine which of your development and testing processes will actually benefit from automation. Here's further reading on keeping your tests simple and how to avoid test automation anti-patterns.
Spend sufficient time doing the research, evaluate existing tools to find the right ones for the job, plan your deployment carefully, and prioritize which automations should be done first. It will also be necessary to train developers on how to collaborate with and use the tools effectively. Provide standards and guidelines to ensure that all testers will construct relevant, effective, and high-quality tests.
When to stick with manual testing
Despite the industry-wide move to automation—especially for teams that seek to adopt Agile or DevOps—it’s important to remember that manual testing will continue to be important on every software project. For example, there are some types of UI bugs that are difficult to uncover and largely counterproductive to pursue, making the cost to attempt tracing and fixing them with an automation approach too high to be worth it.
As long as people maintain a few tests that are unsuitable for automation, manual testing will always be important. Anytime a tester or developer asks, “I wonder what will happen if…” is the moment at which manual testing shows its highest value.
Consider a team that for any given sprint, has 200 person-hours to devote to testing beyond the unit-test level. Recording and validating a segment of automation might require 10 hours, while manual testing might take one hour. The team must decide if they will create 20 automations or execute 200 human tests. The right balance is probably a combination of manual and automated testing.
Best Practices for Ensuring Proper Test Coverage
No matter the extent to which you automate your test suites, it’s vital that your team performs all types of testing across the entire product development pipeline.
-
Unit testing — Unless your software product is one of the rare exceptions, unit tests should be the most abundant type of tests across your entire testing suite. Many such tests are perfect candidates for automation. Over time, you can build a fully-automated, mostly parallel testing infrastructure investment, so that it’s feasible to execute thousands of unit tests within an hour.
-
Smoke testing — Setup your environment to get the minimum testing done in about 20 minutes the next time that you’ve got to push a hotfix. Maximize the amount of testing you can perform in the shortest amount of time by running automated smoke tests in parallel.
-
Cross-browser testing — Conventionally, one of the most time-consumptive manual testing activities is to validate an app across different browsers and devices. Much of this testing can be automated, and can be setup to run in parallel—giving your team the ability to execute thousands of tests among a wide range of browsers and configurations at the same time.
-
Regression testing — In many development teams, successive build releases can occur at a very rapid pace. An extensively automated regression testing suite can quickly check whether the functionality of a new build is comparable with the stability and user experience of the previous build. Choosing the right automation framework for your build type will help you automatically recognize and test the majority of your new functionality.
The testing pyramid is a popular model used to help teams organize their test automation strategy. Further reading on this model can be found here.
To maximize the effectiveness of your tests, your automated testing tools should give you record-and-replay capability coupled with automatic functionality change detection and easy test reconfiguration. Additionally, any tool with data-driven automated testing enables your team to quickly run any test involving the generation of random data sets. You’ll enjoy the ability to run more tests in the same amount of time, and have the option of expanding your test coverage, which should eliminate or minimize the impact of inaccurate tests.
Mapping out your pipelines to see where you can automate, where you'll need manual steps, and what you can run in parallel is a good activity to conduct across the entire team.
Moving from manual to automated testing
While manual testing is undeniably still an important practice, it’s hard to ignore the benefits and pressure of moving to automated testing. The industry is picking up pace and development cycles are getting aggressively shorter. To stay competitive and deliver quality software at the rate the market demands, many organizations and teams have started to – or already have – adopt automation.
Moving from manual to automated testing isn’t easy, and to do it right the first time is even harder. But if you're sure to outline goals, a structured roadmap, and well-built testing framework or test automation tool that can support you at every step of the way, you're sure to succeed.
