One struggle we’ve constantly run into when trying to make mabl usable on all web applications is getting past reCAPTCHA. Few things stop mabl in its tracks like anti-bot security. After all, mabl is a bot, so with reCAPTCHA just doing its job, it constantly proves pretty much impossible to get mabl tests running on an app that uses it. When you train a test on a reCAPTCHA site, everything appears to work fine because you’re accessing the site on your own system, making reCAPTCHA see it as “real traffic”. Once you go to run the test in the mabl app through our execution engine, though, the traffic is instead seen as a bot trying to access that same site. This triggers the infamous square photos popup and messes up any tests that attempt to click through the reCAPTCHA or access the content on the site.

For a long time, reCAPTCHA seemed like an insurmountable barrier, leaving any site that used it incompatible with mabl. Then, a few weeks ago, one of our customers told us about a way their team had solved the issue using a solution in the Google Dev FAQs on running automated tests with reCAPTCHA. This response sounded simple, and the changes needed to get it working seemed easily surmountable to get it working, so we decided to test it out for ourselves and see if these changes could actually let mabl access reCAPTCHA protected sites.

Testing reCAPTCHA with mabl

To set up a mock “test environment”, Don McNamara, one of our software engineers, created a test page with two versions of reCAPTCHA v2 buttons on it, one that was set up normally and another whose key had been switched to one of the predefined keys defined by Google for test environments. He also added reCAPTCHA v3 behind the scenes along with a button that generated the reCAPTCHA v3 curl command you can use to see if the traffic detected from the current session is considered a bot.

3 versions of captchas: reCAPTCHA v2, reCAPTCHA v2 with test keys, and reCAPTCHA v3.

The test page for testing the different kinds of reCAPTCHA

Next, we trained a test using the mabl trainer where we clicked on both versions of reCAPTCHA v2 and then looked at the playback to see what the results looked like. While training the test, both clicks caused a green check mark to appear in the boxes, but when we looked at the screenshots in mabl for the first test run, we saw that clicking the non-modified reCAPTCHA v2 did not bring up the green checkmark, but the dreaded picture puzzle.

mabl can’t be trained to click the right pictures, which is understandable with reCAPTCHA being created to stop bots, so this would totally stop any test designed to go any further into the site from continuing. On the other hand, clicking the modified version of reCAPTCHA v2 did present the green checkmark during both the training and in the test run screenshots, showing that the modified tests keys did allow mabl to bypass v2.

A screenshot showing that green check marks appear in both boxes during training for a reCAPTCHA v2 test.

The green check marks appear in both boxes during training...

A screenshot showing that a green check mark appears on the modified reCAPTCHA box when running through the mabl app.

...but only on the modified reCAPTCHA box when running through the mabl app

To test v3, we set up an echo step while training the test that returns the curl command for that specific test execution in the mabl app.

A screenshot showing how to set up an echo step while training the test that returns the curl command for that specific test.

The test that gives us the curl command for this run of the test

We then put the result from the first test run into a local terminal window, which gave us a response showing us that for that specific, execution engine created test run, mabl was not considered bot traffic. However, we were about to run into a problem with v3.

reCAPTCHA v3: The Problem

Here’s a little info about how reCAPTCHA v3 categorizes incoming traffic: it assigns the traffic a score from 0 to 1 on how legit the traffic is, with 1 being absolutely legit and 0 being totally illegitimate, or bot traffic. Since it’s pretty much impossible for the software to be absolutely sure one way or the other, the most common results are 0.9 and 0.1, for almost certainly legit and almost certainly bot traffic. We set the v3 test to run on the hour every hour in 4 different browsers over the weekend so we could see what scores the test runs got. When Don set up the page, we had decided to not have reCAPTCHA v3 stop any traffic, no matter the score it got. Here’s the results from the weekend:

A screenshot showing the results from a test on reCAPTCHA v3 over a weekend.

As you can see, reCAPTCHA v3 initially saw the test runs as low risk on Thursday, Friday, and for around half of Saturday, and gave them a score of 0.9. However, it then started viewing them as high risk through the rest of the day Saturday and throughout Sunday, giving them a score of 0.1. While the change to v3 proved initially successful in getting mabl test executions viewed as legitimate, over time the test runs again started being regarded as bot traffic.

As of right now, we haven’t found a way to make this traffic be considered legitimate for an indefinite amount of time. I created a post on StackOverflow that could provide answers in the future. Until then, make sure to constantly check your v3 results page or set up your application to provide some feedback when bot traffic is detected and you may hopefully find more success than we did.

On the other hand, if your team is using reCAPTCHA v2, you should now have no trouble getting your test environment set up to work with mabl, meaning you can start running automated tests without worrying about reCAPTCHA. 

You can try mabl out for yourself here.

UPDATE: This is what our traffic report looks like after another week:

A screenshot showing the results of a test of 4 different browsers running every hour over the weekend.

You can see that the traffic has continued to be seen as high risk and given scores of 0.1, so once mabl starts being seen as a bot by v3, it continues to be seen as one.