I’m a serial entrepreneur who has spent time working at Google, VMware, my first startup Stackdriver, and now my second, mabl. We started mabl earlier this year to help developers with the painful challenge of testing their applications as development cycles are getting shorter with the adoption of CI/CD and automation testing.

How Can mabl Improve End to End Testing?

As we were starting the company, I did a lot of research, primarily by reaching out to developers, especially those that had embraced DevOps, to understand their take on software testing. We launched a survey to get quantitative feedback. One of my main sources were blogs written about the challenges of testing applications in a modern developer workflow. A blog post that I was drawn to was from Mike Wacker, a software engineer at Google and previously a developer at Microsoft. Mike wrote the post in 2015 about why end to end testing just doesn’t matter anymore and why developers should focus on unit tests and to a lesser extent integration tests as the basis for ensuring quality in their code. If you haven’t read it and are involved in end to end testing, you should. His premise was (I’m summarizing and probably not doing enough justice to Mike):

• End to end testing “should” deliver the most value for customers, since they’re replicating real user scenarios. Mike argues that all product stakeholders including developers, QA, and managers agree to this general premise.  
• However, end to end tests are not reliable (flaky), take too long to complete (run overnight), and when there are bugs, the cause is hard to diagnose. Mike argues that these issues with end to end tests should force us to consider a different testing approach for ensuring customers are happy.   
• Unit tests are the immediate solution because they find bugs and remedies faster. These, combined with integration tests which help test numerous components together and identify bugs before lengthy end to end tests are written, are the solution recommended by Mike to the broken promises of end to end tests.   

Mike was correct with his conclusion, for the time. Why waste time on end to end tests when you already know the challenges they present? Instead, compensate by adapting your testing process. 

Enter Machine Learning

As Sundar Pichai discussed at IO 2017, Google is revamping all of its products to be AI first – given the advancements in computer vision, voice recognition, and machine learning. We’ve already seen the smart technology Google has delivered from a consumer perspective for years, including self-learning thermostats, traffic navigation, and speech recognition. The next step for Google is applying AI for business use cases, which it’s already doing within Google Cloud. At mabl, we also believe ML can have a profound impact on improving the lives of developers, allowing them to spend more time building great products and less time on repetitive tasks. This is why we’re challenging ourselves to solve the specific problems that Mike pointed out with end to end testing.

Some of the things our team is focusing on include:

• Flaky tests:
  • As Mike mentioned in his blog, end to end tests often fail due to the test not being reliable. When this happens, developers lose confidence in their tests and just ignore them. We believe mabl should be able to automatically detect when steps in trained tests are failing because of a slight change to the UI. mabl should recommend a fix to the user and automatically update the test script if they approve it. Our goal is to eliminate flaky tests so developers can spend more time on coding and less time on maintaining their tests.  
• Isolating failures:
  • A developer searching for a needle in a haystack to solve a test failure isn’t the best use of that developers time. Given that mabl has different types of tests running on an application all the time, she’s learning lots about the application. She can use this data to provide evidence of what pieces of an application aren’t working and point the dev right to it. Helping developers figure out what is wrong and why is of high importance for mabl.  
• Speed of tests:
  • Mike specifically called out waiting for tests to run overnight, just to find out if a specific component is broken. We’ve also heard from several of our users that manual QA or even outsourced QA to manual testers still takes the time of a normal human to test. We believe end to end tests should be continuous, not overnight, and should run at the speed of machine time, not human time.  
• Self-learning tests:
  • One thing Mike didn’t mention is about learning from past test runs. We believe mabl should continuously learn from the tests she’s running as well as the feedback users are giving her, just like ML applied to consumer patterns. The more she learns about an app, the smarter mabl can be with the insights she’s providing users about the app's quality .  

mabl is solving hard problems

End to end testing is difficult today (as it was in 2015 when Mike wrote his blog post). However, instead of replacing end to end with unit and integration test coverage, we believe we should fix end to end testing. We believe advancements in machine learning allow us to both fix end to end testing and even go beyond what a human is able to do with test automation frameworks. We’ve been at this for 8 months and we’ve learned a lot along the way. We’ve solved some hard problems and still have many more to tackle. If you want to learn more, reserve your spot for early access as we’re launching the service very soon.