Using Machine Intelligence for Clearer Regression Testing | mabl

Regression testing helps development teams validate that the existing functionality of their product still works when they fix bugs or add new features. Typically, these tests are kicked off as part of a continuous integration (CI) process so that if anything broke based on their new commits, the engineers know very quickly. The hope is that these tests will ensure quality of the user experience along with the new changes.   

Given new application architectures including microservices, where individual services are decoupled into small, individual components, regression testing has become even more important since unit testing rarely catches code dependencies for a new feature that runs across one or more microservice.

However, regression testing presents many challenges. Some of the biggest are:

• A pass/fail comparison is relatively simple, but comparing regressions for performance characteristics is difficult.
• Tracking, analyzing, and comparing historical results for tests that you want to run regressions against takes a lot of work.
• Once regressions are identified, finding the cause takes a lot of time.

These types of challenges, formed by the time and work limits of humans, are exactly what machine learning was built for and why mabl incorporates machine learning to improve its regression testing. As mabl is testing applications, lots of data is collected outside the basic test steps, including test run time, page load times, and screenshots showing visual changes in the app. This data in turn trains machine learning models which are used to notify teams when something changes - both for the bad or the good.

Let’s take a look.  

This screenshot is an example where mabl found a regression in test execution time for a specific journey. Specifically, you can see where mabl highlights that the run of this journey (which took almost half an hour) on January 20th was significantly slower than previous runs.

We're frequently adding new automatic regressions based on user feedback. One of the newest regressions is visual differences. Since mabl captures screenshots of every step of your journey, mabl can build a model of your app that automatically compares these screenshots to earlier ones to detect visual changes. You'll then get a notification about exactly where on the page the change was detected.  

These are two of the many examples of comprehensive details you’ll see when mabl detects regressions. For a higher level view, mabl offers an Insights feed on the right nav that keeps track of every new regression mabl detects. If you don't feel like hanging out in the mabl app, you can configure these insights to be sent to Slack as well.

With machine learning, mabl is making regression testing easier for software teams by not forcing them to comb through test history to find discrepancies in the visual state of their app or painstakingly compare test execution times.

Use mabl to test your app free for 21 days - sign up for an account: