Next Generation Codeless Automated Testing Tools - Alternatives to Selenium

Testing has long been one of the most thankless tasks in the world of development. Doing it manually can result in hours of repeating the same tasks over and over again trying to find the bugs that developers hope don't exist. With the 90-90 rule in full-effect, it seems like it takes as long to test software as it does to create it. The larger the site gets, the greater the burden of testing it becomes. Worse than that, it creates a bottleneck in the development process. No sites can be moved to production before this process is completed (unless, of course, you don’t care about quality). Test automation solutions came along as a response to this.

Selenium and Selenium IDE - why do we need an alternative?


Selenium is a powerful framework that enables developers to automate all the tests that they want. However, it's user hostile for someone without a coding background. One needs to be pretty much a full-fledged developer to use it, but many organizations tend to tread down the dangerous path of hiring developers or interns without testing experience to create test scripts. 

To be fair, there's Selenium IDE. With Selenium IDE, all one needs to do is fire up the Selenium plugin in the browser and start recording the user journey. You can play the test back at any time to see the test run live. You can also manually add steps, assert the existence of elements, and use a number of other nifty features.

However, reality being what it is, nothing is ever as we plan it (otherwise why would need to be testing in the first place?). The love-hate relationship with Selenium is a real thing. It has its merits, but Selenium tests, whether recorded or coded, fail a lot for all sorts of reasons; a slight slowdown in the internet connection or an errant mouse movement while recording the test cause strange test results. Tweaking the tests often break them, and you'll need to start from scratch.

They're also at risk of becoming quickly outdated in agile and DevOps software development teams where changes occur frequently to the app under test. Almost any change on a site requires re-recording or rewriting of the entire test. The larger the project, the worse it gets. Maintaining Selenium scripts quickly becomes no more efficient than manually running through tests on your own. If not done right (which is easily done!), Selenium can result in low-confidence tests and become a huge burden.

To summarize, Selenium has a lot of the following problems:

  • Tests are unstable
  • Tests are fragile to app changes
  • It is far too complex for beginner users
  • Being a tester becomes a full-time highly skilled job
  • The specialized skill set silos testers from the rest of the development team
  • It creates a new expense items for development firms
  • Qualified testers can be hard to find

Record and Playback Tools - Yes, we know, they suck.

So, of course, many companies saw an opportunity here. First generation record-and-playback tools came along, but suffered from some of the same problems as Selenium IDE. Record and playback tools were still fragile and broke with minor app changes. Record-and-play tests aren't modular, which means each test case with similar flows in it have to be modified across the board. This generation of record-and-play quickly got a bad reputation. They were still not scalable. Cue the next generation.  

Next Generation Automated Testing Tools - codeless automation that doesn't suck

So a new demand arises: a solution that can harness more of the power of Selenium that:

  • Doesn't require coding
  • Runs stably 
  • Adapts to code changes in the app

But this generation of test automation tools has gone beyond these 3 requirements, and incorporated many delighters like cross-browser testing, advanced features like email verification, and automated visual regression insights from your test runs.

Joe Colantonio has talked about this third wave of test automation tools that utilize up-and-coming technologies like machine learning. The point of this article is to look over some of these well-known codeless testing tools on the market and to evaluate them based on this list of features that the next generation of automated testing tools need by Angie Jones. Many of the items in this article have already been addressed by most of these major testing tools on the market. However there are a number of features that may not be as common, and each tool handles these a little differently.

It is important to state that this article is not a review of these tools, but instead a brief overview of some of their capabilities. Unless stated, all features discussed are available innately or in the tool's UI, accessible without needing to code.

The following tools will be examined:

A feature-by-feature assessment

Feel free to skip to sections you care about the most. We will be comparing the following features:

 


Smart Element Locators

As we already discussed, one of the flaws in first generation record and playback tools was tied to the fact that even minor modifications to code causes test scripts to break. One factor that would improve these tools is the ability to check for more than one element locator for every element interacted with in the test script. If element locators in the code change or no longer exists, the test can fall back on other locators, meaning scripts would be more stable and require less modification in the event of code changes.

mabl

mabl searches for closely matching instances of a specific element instead of looking for it at its last known XPath. CSS selectors and dozens of other identifiers are derived from the website automatically. mabl will run the test using the closest matching element, and rerun the test with all closely matching elements if there are many likely candidates.


TestIM

TestIM does not appear to have any built-in tools for this purpose. It can be done, but must be coded using Javascript to provide different parameters.


TestCraft

TestCraft uses what they refer to as “smart binding.” This enables it to assign tests to multiple elements at the same time. TestCraft provides a few ways to doing this. This first is technical, which captures multiple attributes at once, such as IDs, and contextual, which captures an element's position, text value, or size. It also allows elements to be rebound during a run, to match changes that have occurred in the code. This is handled manually.


EndTest

In EndTest, identifiers are bound to specific elements using “locators” to uniquely identify an element via CSS selectors and XPath. EndTest has a built-in “self-healing” function which should, by using machine learning, identify if changes have been made. It will also allow dynamic locators to assist if elements have different locators for each page rendering (this, however, is not codeless as it requires JavaScript to add to a variable).


GhostInspector

To handle this issue, GhostInspector supports CSS and XPath. In cases where items have dynamic IDs (as is common with some coding languages), it allows using partially matching attribute selectors, or will allow identifying by other methods. By using JavaScript it should be able to identify multiple conditions.


 


Conditional Waiting

A common issue with first generation record and playback tools is that any delays or unexpected problems with a browser or internet speeds could cause tests to immediately break. What was necessary was to be able to create conditional pauses, and to allow scripts to only proceed if a condition is returned as true, such as a page finishes loading or an API call has completed.


mabl

mabl handles this automatically with built-in heuristics that assign different wait times for different elements so that if a specific element usually takes longer to load, mabl will wait the extra few seconds for it. It is also possible to to add “wait” steps to specify an exact amount of time to pause before attempting the next action. 


TestIM

TestIM has a series of wait tools, such as waiting for an element to be visible/not visible. It can check for the existence of text, some allows some timed delays. Custom waits can be constructed using Javascript.


TestCraft

TestCraft has some built-in “if” statements. One can say that the test should wait for an entire page load before proceeding. It also has functionality to check to make sure all AJAX requests and CSS animations have completed as well. TestCraft also allows setting a configurable amount of time for waits.


EndTest

EndTest handles most of this process automatically. It does not require adding wait steps; the application identifies whether to wait or not. It should automatically detect whether or not, for instance, a page load has occurred. It allows setting max wait times, and will also allow entering a pause into a test.


GhostInspector

Ghost Inspector will wait automatically to identify if an AJAX API calls exists. Most other conditions require configuring within Javascript to identify whether a condition returns true, such as the existence of an element.


 


Control Structures

One of the advantages of coded automation frameworks was the ability to create control structures such as loops and other conditional clauses. Being able to loop through many different conditions is a very valuable factor for making tests scalable. This is one area where codeless tools have lagged behind coded frameworks. While most have advanced to the state where they can allow simple conditional checks, with a few exceptions, looping is not something that is typically available as of this point, without some additional coding.


mabl

mabl allows several if conditions, and has a data tables feature, where you can configure multiple data sets for which you need to run the same tests on, and will run those tests concurrently. mabl also has reusable "flows" which are a segment of test steps that can be modularized and re-added several times over into a single test to repeat any action. If all else fails, you can handle looping with some Javascript.


TestIM

TestIM has a built in graphical tool which will enable setting up to choose modules and run while loop through them. It is possible to loop through some of the basic built in functions as long as a condition returns true. This can also be done on custom conditions that testers can create using Javascript.


TestCraft

TestCraft has a number of built in codeless control structures, such as “if” statements, and has a reusable test flows structure. It also allows the running of concurrent tests with different datasets.


EndTest

EndTest has a built-in “if" tool, and they can be nested. It currently does not have looping but it is under development.

GhostInspector

Ghost inspect does not have any built-in codeless control structures, but will allow writing custom conditions using Javascript. 

 


Easy Assertions

While this seems like a relatively easy task to accomplish, many first-generation codeless tools did not allow one to easily assert the existence of an element. This is important because being sure that an element has actually loaded is an important part of any automated test. While it could be done by writing custom code, this could slow down automated testing considerably. For this reason, this is a fairly crucial need for codeless automated testing tools and it should be a required feature of next generation codeless tools.


mabl

mabl has tens of easy assertions built in for any element: validating the inner text, CSS properties, and advanced assertions such as cookie and URL validation. 


TestIM

While TestIM allows validation of elements, it does not appear to have any built-in functionality to add custom assertions. Any assertions will require writing some custom Javascript code.


TestCraft

TestCraft has a number of basic assertions like validation its presence as one of its core features.


EndTest

Adding assertions is part of the basic functionality of Endtest. One can assert an element is present, visible, or not visible, text matching, and more.

GhostInspector

Assertions are built-in for GhostInspector. It asserts whether an element is present, not present, text equal to, not equal to, contains, as well as whether or not a Javascript condition returns true

 


Reusable Steps

As anyone who has done any testing can testify, there are many functions that require repeating. Having to run through a segment of a test each time can often seem like a time-waster. There have been a number of ways that this has been handled with coded tools, ranging from the creation of libraries for a project, up to a fully object-oriented approach where individual functions common through multiple applications can be drawn upon to create much more efficient testing processes.

For codeless tools to be truly worthwhile, they should be able to provide the ability to reuse common functions, such as logins without having to record it each time. If these functions need to be updated, they should be updatable globally.


mabl

mabl allows recording sequences which can be bundled into “reusable flows.” These can be stored as a library and can be inserted into either tests within the same application, or can be used again in other tests.


TestIM

TestIM has an ability to create small modules which can be reused, called “shared steps” which can be inserted into tests. They can also be cloned so modifications can be made if necessary for individual tests.


TestCraft

TestCraft is a little different than some of the other testing tools described here. Typically tests are constructed manually in a flowchart instead of recording them (though there is a recording function). As a result, the modular approach is built into it, and individual nodes or modules can be reused for new flows.


EndTest

With EndTest, tests can be divided up into “test cases,” such as “logging in” or “searching” which can then be reused by importing a test case into a “suite” which can be used in multiple applications.


GhostInspector

GhostInspector uses “custom variables” to perform functions which are not already built-in to the application. These can be simple variables, but also can be full functions (or objects) which can be modularized and be reused.

 


Cross Browser Support

Many of the first generation testing tools allowed testing in only one browser, typically Chrome or Firefox. However, as we know, users use a wide variety of browsers, and unfortunately, not all of them behave the same way. A common headache in front-end web development involves dealing with variations between browsers.. Whether you are dealing with Chrome or Firefox or Edge, Or IE this is necessary. This also gets more complicated if we have to check against older versions. (Maybe some of your target market is using IE6? Anyone remember the old AOL browser? <shudder/>)


mabl

mabl tests creation is limited to Chrome, however the tests are platform-agnostic, so the recorded tests are created once but can run in Firefox, Internet Explorer, and Safari in parallel or in different test plans. You can configure the display size for responsive web testing.


TestIM

Running cross-browser tests in TestIM requires using the CLI, which requires custom setting up of node.js. However once this is set up, the CLI is available through their interface. The scheduler also appears to allow configuring for different browsers.


TestCraft

TestCraft is platform agnostic and allows running on multiple platforms simultaneously and will run tests in Chrome, Firefox, and IE.


EndTest

EndTest shows support for Chrome, Firefox, IE 11, and Edge, as well as various mobile devices.


GhostInspector

Ghost Inspector allows running tests in Chrome and Firefox if set up in a Linux environment.


 


Reporting

Of course when you are running many tests, it would be a lot easier to see a high-level view of whether or not test runs have succeeded. Ideally, when a test fails, it should be easy to see exactly where the failure occurred without having to dig through log files or to have run the test again manually. On top of this, sometimes failures occur under certain circumstances that can't always be easily reproducible with a manual test.


With a good reporting system, ideally we should be able to see all information regarding a failure so as to understand what circumstance caused the problem. Some tools provide reports on which tests failed and where, and some even go as far as to provide visual evidence of the failures in action, so as to not need to run these again manually.


mabl

mabl has a dashboard that shows high level statuses on passed and failing tests, Javascript errors, broken links, visual changes and other things. Root causes aren't pointed out, but the test logs include screenshots and a quick link to the last passing test for easy comparison.


TestIM

TestIM has some very high level managerial reporting tools, and it reports common failures. This requires digging into each test to see the failure results. This may be sufficient for most users.


TestCraft

TestCraft has some nice features, including detailed reports of every run, and it also records screenshots and and videos of moments when they failed, so it's possible to visualize the problem.


EndTest

EndTest only provides a record with the results of a test, and displays pass/fail indicators for each run.


GhostInspector

GhostInspector provides reports on every test run. It will provide a screenshot which compares the results the last passing test. As a result, identifying any differences is quite easy.


 


Custom code injection

Even with the best designed codeless interfaces, occasionally you will need to write some custom tests that don't fit within what has already been designed by the creators of these tests. While the point of codeless tools may be to reduce the need to write code to run tests, there are times when you actually want to do so. Almost all tools allow some modification of tests with some client-side Javascript. This makes sense as these tools tend to run in the browser, however even more useful would be the ability to run actions as services, much like can be done with many coded frameworks.

Frameworks such as Selenium are often language-agnostic. Granted, while this is something that might only benefit the most advanced users, it could be helpful if some of the codeless tools would allow other languages.

mabl

mabl allows inserting Javascript steps. It does not appear to have any functionality for allowing other code.

TestIM

TestIM uses Javascript, in fact most functions other than the most basic validations tend to require this. It only supports Javascript.

TestCraft

Code can be inserted if necessary, however TestCraft's philosophy is that they seem to discourage doing this, as they really want this entire process to be drag and drop. However, Javascript is supported. It accepts any web-based applications and is code-agnostic for any web-based scripting, but will not accept any server-based languages. 

EndTest

Allows simple one-line Javascript validations. They have consciously decided not to allow other languages due to security concerns.

GhostInspector

Ghost Inspector allows inserting Javascript and execution of custom Javascript. No other languages appear to be supported.


 


Integration in the development pipeline

As most web development teams work in an Agile and/or DevOps environment, testing has typically been integrated into the full development process. As a result, it is typically necessary to be able to connect to different tools which will enable tests to run throughout the entire workflow and by different teams.

With coded frameworks, typically custom coding is needed to integrate into existing processes. For automated tests to be truly part of the development lifecycle, it needs to have some integrations with other tools, or include a few basic triggers that can be called via API.

mabl

mabl has integrations with several CI/CD platforms, and for non-natively supported platforms, there's an API that you can use to trigger tests and send test results back from.

TestIM

With TestIM it is unclear. While it has internal tools for running tests on schedules, all information about APIs seems to suggest it is calling external APIs, and not providing triggers. That said, TestIM has an integration with another product, Testrail, which will provide most of the necessary integrations.

TestCraft

TestCraft provides full integrations into CI/CD workflow, with many integrations for the entire development cycle. Integrations include Jenkins, Slack, Zephyr, Jira, Gitlab, and more.

EndTest

EndTest integrates with various DevOps tools, such as Jenkins, Github, Bitbucket, and a few other pipeline tools.

GhostInspector

GhostInspector provides integrations with AWS Code Pipeline, Bitbucket, Docker, Gitlab, Jenkins, and many other tools.

Conclusion

While there is still no perfect replacement for the power of fully coded tests, codeless automated testing software has come a long way since the first record-and-playback generation of tools. Many of the complaints about the limitations of these original tools have been addressed, and new changes are occurring regularly.

As seen above in the selections we examined, almost all have addressed some of the basic needs, such as easy assertions, a few basic conditional operators and the ability to integrate with other development pipeline tools. The speed at which software companies have adopted many of the needs for testers gives us reason for optimism.