Okay, so you really want to learn more about automating your test processes. However, a quick search on automated testing seems to bring up more information than you can handle. For this reason, we will be breaking this down into digestible pieces to make it easier for you to get a handle on what this is all about.
As a tester, you may already have a testing framework that defines how user stories or functionalities will be tested before launching your website or application. In other words, at its most basic level, you want to make sure that if someone clicks link A, it will go to page B. Automating this process means that you can program an application to do all of this for you, saving you a ton of time, and makes sure you don't make any of those oh-so-human mistakes that most of us make.
There is, however, a large variation of tools that exist, for different skill sets, needs, and they behave differently. A test automation tool or framework can include everything from tools that blackbox the code and requires no (or only a little) bit of programming experience, to full fledged frameworks which are designed for testing large enterprise-level applications. What best for you and your team?
This article is a brief roadmap to the types of testing tools that are available in order to help you make a decision as to which direction to go.
For the purpose of clarification, the following are the main categories of testing tools. There is some overlap, and these are not hard and fast definitions, but in order to make it clearer, we have divided these into loose categories:
Linear Automation/Record and Playback tools
Frameworks
Modular frameworks
Library Architecture Frameworks
Data-driven Frameworks
Keyword-driven /Behavior Driven Frameworks (BDD)
Intelligent Tools
Examples:
Linear automation testing tools are the easiest form of testing. Sometimes called “record and playback” these are tools that will record each action a user takes on a website, and create a simple script which can be played back to mimic exactly the same behavior the user took.
The purpose of these is to identify if behavior taken by a user can be mimicked and be reproduced if performed again. They are useful during the prototyping process of developing a site. If you build something to work a specific way, it should be able to be reproduced in exactly the same way under the same conditions.
Advantages:
These are very easy to use, and are a good introduction to automated testing tools, and can be great for simple prototyping (testing a site while building it). They are great for beginners and useful for small applications, especially if pieces do not need to be reused.
Disadvantages:
Because everything is hard-coded, they are not dynamic or scalable. For each activity, you need to record a new test. If something on the site changes, the test will typically be needed to be re-recorded from scratch. Tests also tend to be brittle in that they fail even with miniscule changes to the application UI.
The vast majority of testing tools that exist fall into this category. They are not raw “tools” in themselves but instead a set of methods or functions that can help design testing plans.
Scripting Languages
To do any significant testing, particularly anything beyond record-and-playback, some programming becomes necessary.
Even if creating something using modular-based testing, tests still need to be tied together. In most cases it does not matter which language you use; if you know a little JavaScript, in many cases that will be enough.
The following languages (in no particular order) are popular for test scripting.
JavaScript
Perl
Python
Ruby
Tcl
For some of the more advanced frameworks, you may need to know more than one. You can, of course, write your own frameworks using your preferred scripting language, however many existing frameworks have already been designed to help you speed up the process.
Examples:
Unlike linear testing, modular frameworks break down individual activities into separate functions, which serve as building blocks for a master script which will run everything in a specified order. Or, instead of playing back of the entire test, each segment can be tested on its own.
Advantages:
Because pieces are separated, changes to an application will only affect the specific modules related to that change removing the need to recreate the entire script with each change. Modules can also be reused if the same functionality exists in multiple parts of the application. They work well for single applications which need a lot of changes
Disadvantages:
Like linear testing, each piece is still hard-coded, so they cannot be reused for different applications. If there is a different dataset, each piece will need to be recreated.
Library Architecture Frameworks
Examples:
Advantages:
Modular frameworks take it a step up, where you can use variables to further define the functions. This makes functions more reusable, as it only becomes necessary to change a few variables to be able to use them in different situations. The frameworks’ modules too, are made up of reusable libraries, and each of these can be drawn upon when needed.
Disadvantages:
While more scalable, the actual test data itself is still hard coded, and changes to data still require changes to the script. Working with libraries requires understanding some programming experience to create each of these functions. As a result, it can take a lot more time to develop tests for your projects.
Examples:
Advantages:
Data driven frameworks are high-level frameworks designed to be flexible enough to handle multiple testing environments and datasets. There are no hard-coded aspects of these; they are designed to handle data in multiple formats, ranging from websites, SQL tables, spreadsheets, and more. The data itself drives the test. By changing data, different scenarios can be tested quickly and as a result are highly scalable.
Disadvantages:
They have a huge advantage of flexibility. The drawback is that their complexity requires very highly skilled testers who can write in multiple programming languages. They also take a very large amount time to implement. The payoff may be great for very large ecosystems, however they are not particularly well-suited for smaller projects.
Examples:
“Keyword-driven” frameworks, which are sometimes referred to as “Behavior-Driven” are one of the most flexible types of testing systems. These are very high level frameworks, designed directly from business language. Simple commands can be used to perform some basic functions, by assigning a series of keywords to a lookup table with a specified set of commands.
Advantages:
Once they are set up, they are very easy for testers to use; they follow a natural language progression. Users do not need to know much coding and little scripting is needed. Keywords can be reused in tons of tests; each piece of the code is very easily reusable. Test scripts can be independent of a specific application.
Disadvantages:
The drawback is, as is common with many user-friendly environments, they require a considerable amount of back-end coding, including keyword definition, and the initial setting up of all object repositories. Setting up BDD frameworks requires a high level of proficiency in test automation. They also add some scalability issues as it can be difficult to maintain the repositories. As applications grow, the testing framework needs to keep being modified.
Examples:
Overwhelmed yet? What if you don't have a lot of programming experience, and you just need to be able to start testing? You want to be able to test your site, but would rather not get stuck with the limitations of linear record-and-playback tools? This is where the next generation of machine learning (ML) driven tools come into play.
Advantages:
mabl combines the idea of auto-healing linear record and play testing, with advanced capabilities to meet the needs fast-moving teams.
Auto-healing Tests
As modern websites have more dynamic user interfaces, such as AJAX applications, mabl collects multiple attributes to assist in recording specific data about an element. This enables mabl tests to navigate elements in a way that would normally require complicated scripting, and automatically update the underlying test scripts as apps evolve under development.
Test Output Processing
With built-in regression testing (checking to see if there has been a change to previously functioning parts of a site), mabl uses machine intelligence to compare against past tests to spot anomalies in behavior, performance, visual elements in the application UI, and learns more as you test.
Data-Driven Testing
You can define a set of parameters for a test, and run that test for each variation simultaneously, without re-recording each scenario separately.
Reusable Flows
Similar to modular frameworks, you can define a flow, or set of steps, that you can reuse in multiple tests, or connect together to build a master test. If you ever need to update a flow, you can do so without it affecting the other flows, or the master test.
Disadvantages:
They don’t have the same infinite flexibility as creating your own framework does, so if your application has complex edge cases, these types of tools might not be for your project. They’re also not open-source, so there is a subscription cost to consider.
As you can see, the automated testing landscape may be large, but once you get an understand of the different types of testing frameworks, you can get your bearings. We hope that this article has helped gain a better understanding of the different types of testing tools available, and pointed you in a direction that’s right for you.