In a previous article, I did a comparison between a few automated testing tools from a beginner's perspective. I examined Selenium IDE, Test.im and mabl, walking through a basic process in order to become familiar with automated testing in general, and examined which tool seemed best suited for for those getting started.
I had intentionally chosen play-and-record tools for the purpose of gaining a familiarity of how prototyping a test plan would work, and how it would seem to an individual who had virtually no knowledge of these media, and how easily one could get started with no formal instruction.
The point was to see how easy it was to get started from a self-test perspective, from the perspective of one just beginning to get their feet wet.
Once setting up tests and scheduling them to run on their own was understood, what I wanted to try next was simulate heavy traffic on a site. Onwards I went, exploring Selenium vs. mabl.
What happens if multiple users are attempting to use a site simultaneously? Does it lag? Does it cause certain things to break or become unusable? Unless one possesses an army of eager interns willing to key in planned tests at the same time, this is essentially impossible to do manually.
What you need is an army of robots, which is exactly where automated simultaneous testing comes into play. If you can mimic many users coming from different devices at the same time, you can simulate a busy website running in a real-time environment.
Another reason for doing this is that if you have a large site with many different functions to test, while you can set up tests to run sequentially, each test and each function adds to the amount of time necessary to complete your testing stage. This can result in website delays, especially if troubles arise during the testing period. Even if tests have been divided into discrete units, good testing strategies involve retesting all dependent parts of a site. In today's Agile environments, these delays are unacceptable.
As a result, tools have been created to manage this. Typically, these involve setting up tons of virtual servers, and timing preset tests to run at the same time. The point of this article is to examine a couple of different approaches to this, again from the perspective of one trying this for the first time without professional guidance.
For these purposes, I pitted the open-source tool Selenium Grid against mabl, a closed source tool that also boasts the ability of running simultaneous tests. During the process of this test I took a modest approach and followed the following processes:
Installation
Configuration
Installing and Configuring an existing test plan
I will document these processes and then provide an assessment from the perspective of a first-time user.
Selenium Grid is the segment of the Selenium universe which enables setting up multiple testing servers or nodes to run many tests simultaneously. A brief examination of the Selenium website gave me a basic overview of what it does. However, like many open source projects, the documentation was a little sparse and incomplete.
As a Windows user, I quickly discovered that installing Selenium requires Java. This caused me some immediate concern, as Java has had some significant problems, with so many security vulnerabilities in the past that Firefox had blocked access to it. However, these issues have (theoretically) since been resolved, and not being a security expert, I trusted fate and went ahead and installed it, and added it to the Windows path environment variable.
I then found the download for Selenium server and immediately discovered that the official documentation provided information that appeared to be extremely out of date: it references version 2.44 but the current version is 3.141.59 (though it was unclear whether that references Grid or the whole application). Links to a user wiki appear to redirected to a Github channel, which had information about the same out-of-date version.
So I crossed my fingers and followed the directions, updating the version number, using this command:
java -jar selenium-server-standalone-3.141.59.jar -role hub
Thankfully it worked, and I was able to get it up and running on my machine.
In the documentation, it stated that the default port was 4444. If something was already running on that port, it is possible to set it to run on a different port, by adding the following argument (for example) -port 4441 to the above.
In my case, the default port was free, so I decided to test it out and I directed my browser to http://localhost:4444/ which immediately told me I was in the wrong place. At least it was definitely running. Thankfully there was a link on the page that sent me to http://localhost:4444/grid/console
It showed me this screen:
Now that I had the server running, I went back to the (loose) documentation, and realized that of course I needed to set up different nodes to be able to run multiple tests simultaneously. This was not too difficult.
In a separate cmd window (the server was running in the main one), I ran the following command:
java -jar selenium-server-standalone-3.141.59.jar -role node -hub
http://localhost:4444/grid/register
After doing this again multiple times (in multiple cmd windows) I was able to get two nodes running at once. Going back to the command window I was able to see something that looked like this, showing that both nodes were running:
More information could be seen about each node, by clicking Configuration
|
browserTimeout: 0 |
This showed me information on how to get to each node. So, I entered:
http://localhost:33580
and of course got to a page showing me I was in the wrong place, but followed a link to get to here:
http://localhost:33580/wd/hub/static/resource/hub.html
It showed me a screen that looked like this:
Creating a new session opened the browser window, and showed this in the browser window:
At this point it was time to load a script. This required entering a URL for a page that had the WebDriverJS client. (Incidentally, it is worth noting that the new browser session did not include any installed extensions such as Selenium IDE, so it was not possible to record a test in this session)
Creation of a script running this client of this was considered beyond the scope of this test, as it is somewhat involved. I did do some brief research to determine whether it was possible to import a Selenium IDE test into a page, and determined that the latest version of IDE did not have the functionality to export script results. However, a third-party tool, Katalon, which is essentially an IDE, allows this functionality, so it can be useful for prototyping tests in Selenium WebDriver, and as a result, Selenium Grid.
The setup process with mabl proved to be quite different and considerably easier than Selenium Grid.
Right away, I realized that mabl takes an entirely different approach to the process of simultaneous processing and required a different mindset. From the mabl perspective, the approach is to set up a test (or a Journey) through a “training” session, which creates a prototype test script.
From this point, it helped if I looked at at “Plans” as similar to to Selenium's “nodes.” Within each Plan, I could set up multiple “Journeys,” which seem loosely comparable to Selenium's “sessions.”
Before I get too deep into that, here is how I proceeded to test the same procedures above with mabl:
As mabl works quite a bit differently than Selenium, my procedure was also different. Setting up parallel instances of test runs in mabl seems to require having tests already in place. So, to get started, I ran a simple training session and created a journey, which looked like this.
I then cleaned it up a little so that it was in a state where it would pass a test (which is very easy to do in mabl; it's a matter of simply deleting, inserting, or moving steps to where you want them to run).
At this point I wanted to have it run several different instances at once. The documentation is extensive, and there are many training tutorials, but to create a fair comparison to Selenium Grid, I wanted to see how easy it was to handle this myself without assistance. I did, however, resort to the manuals occasionally.
I found that it is possible to create “Plans.” While it was difficult to find what a plan is in the documentation, it seems to be a set of test routines (or Journeys) that can be run together. It appears possible to run multiple Journeys at once, and Plans are set to run at specified times.
These appear to be loosely analogous to the “nodes” created on Selenium Grid, but with more features, such as settings for when they are set to run, and display a set of test run statistics in a dashboard:
Using the “run multiplier” for each journey within a plan allowed me to be able run many Journeys at the same time. After some digging through the documentation, there appeared to be a limit to the number of multipliers that can be run for a Journey; the level is 3 at a time for standard accounts, and up to 10 for enterprise level accounts.
If I wanted to run more of these, my assumption was that plans could be duplicated quickly so it seemed virtually effortless to have them behave the same way that nodes do on Selenium Grid. They also retain all of the aspects of other nodes without having to configure each separately, so if I was on a standard account, and I wanted to run 30 simultaneous duplicate tests, I could set up one plan to have 3 multipliers on a Journey, and then duplicate this plan 10 times.
A note about support: I wanted to confirm this so I began a chat session with the support area. I was able to get information confirming this, however the amount of time required to get this answer took about 30 minutes, which I attribute to the time of day (it was late in the evening). I got an immediate response from a member of the support staff, however they appeared to disappear and forwarded my question to another staff member (this was done without any notification). However, after I got the information I needed, I received a follow-up email from the original support person apologizing and offering to continue to help. This is, of course, better support than I could have expected from Selenium, which doesn't exist.
It is somewhat difficult to compare these two tools, as they behave very differently. Both had their strengths and disadvantages.
While documentation seemed sparse, after playing with it for a while, it became very clear that this is an extremely powerful tool, and ideal for those who wish to run multiple self-constructed tests. It, like Selenium WebDriver, is what it claims to be: a framework for further development. Tools for running and configuring tests exist, but everything needs to be done by the tester or developer at the outset. If an organization has a large amount of human capital, with many highly talented testers/developers and a large bank of servers, it is an ideal tool for managing large numbers of test. There is a reason why Selenium is a popular framework: its sheer power and flexibility.
The drawbacks are due to the complexity of running it. Getting it configured and running takes a lot of initial work (as this user discovered). Once up and running, it will handle a lot, but pretty much nothing will run “out-of-the-box.” The server and each of the nodes need to be configured, as well as each and every aspect of any tests. This also includes timing of tests, which will typically need to be configured via cronjob. Even in cases where tests can be imported from tools like Selenium IDE, configuring and running sites and servers is still a necessity. Support, where it exists, is in the form of online forums or Github issue pages. One is unlikely to get personalized service, and for a beginner, this could be a major drawback.
(It's worth noting that that there are a number of Selenium Grid cloud-providers, some of whom offer some free setup, such as https://www.gridlastic.com/pricing.html. However, the ease in getting tests to run here does not necessarily decrease)
mabl is very easy to use, right out of the box. The learning curve for setting up running multiple tests is very low. Configuring tests in mabl seems to be relatively easy; it's about getting the plans moving. The preliminary set up that was necessary with Selenium Grid servers, including configuration, seemed to be unnecessary with mabl; it's already taken care of, as everything is cloud-hosted. In other words, there's no need to set up a hub server or several nodes, either on one machine or on others; instances in the cloud are very easy to create, and mabl orchestrates it for you. Support is another area where mabl is very strong. They have a staff that appears eager to assist where they can.
The drawbacks are mainly in two areas: cost and scalability. For each instance added to a plan, the costs can increase. However, this cost should be compared against the amount of time and human capital, on top of the cost of servers and hosting that is necessary to run tools like Selenium Grid.
mabl is a codeless automation tool, but customization seems very possible. From quick observation, it appears that this is limited to javascript; Selenium allows a multitude of programming languages. However, in most cases this should not be too large of a drawback. If you are writing your own tests in other languages, chances are, you're already working with a framework such as Selenium.
In the experiences of this humble developer getting used to the idea of working with automated testing tools, whether to choose Selenium Grid or mabl depends on the size of one's enterprise, the level of complexity needed, knowledge level, and the amount of time you want to commit to your QA process.
From my perspective, while it was fun figuring out how to use Selenium Grid, the learning curve seemed to increase very quickly. Also, the need to make a large investment in additional independent resources in order to do any sort of parallel testing would make Selenium something that would not be worth the time-expense.
mabl provides a set of ready-to-use tools. The ease of getting started with some genuine testing, the speed at which it takes to set up parallel tests, and the ability to do all of this using the power of the cloud makes it a no-brainer. While I'd love to spend more time on Selenium, mabl to me feels like an obvious choice from a business perspective, all except for possibly the most complex of organizations.