Is Selenium really that good?

50% of developers use Selenium today, yes 50%! Is Selenium really that good that so many of today’s developers use it, more than a decade after it was introduced? That’s debatable.

What is Selenium?

Selenium is an open-source project started by Jason Huggins in 2004. The framework allows developers to record and replay commands on web browsers to test functionality.  It has been adopted by millions of developers around the world to test the functionality of their web-applications running in modern browsers.  I’d think something invented 13 years ago within the technology realm wouldn’t be that useful today, but there are benefits of Selenium:

• It’s free
• It’s open-source
• It has a strong community, including 219 meetups across 40 countries around the world
• It can be integrated into developer other toolsets

That being said, I haven’t found a single developer who’s in love with Selenium.  Wherever I turn, the words flaky and brittle are synonymous as users describe the software.  More specifically, the biggest complaints I’ve heard are:

• It’s VERY time intensive to maintain
• Tests are often flaky, meaning tests fail for a reason un-related to what a test was actually written for
• Its out-dated
• Tests are brittle, meaning an innocuous change to a page element could cause a complete test to fail

For a web-app automated testing framework that was designed:

1. before the most popular web browser (Chrome) was released, and
2. before the most commonly used modern web-application deployment platform was launched (Cloud)

…is Selenium really so widely used?   

Numbers don’t lie

 As we started mabl, we developed a survey to get answers to specific questions as it relates to developer tools, with a focus on testing:

1. What are the most popular testing tools for modern developers?
2. How much do companies spend on these tools?
3. What is the makeup of the team that uses these tools?
4. What are the biggest challenges these teams have?  

Surprisingly, the data from this survey tells us that Selenium is the most popular, and there’s nothing that even comes close!  Selenium is higher by an order of magnitude vs. its nearest competitors including Sauce Labs and home grown tools.  Below are the top 6 tools which developers use today, based on our survey conducted in March of 2017.  

Is Selenium really that good?

Probably not. Based on such a significant difference between Selenium and all other options, I’d suggest that we as an industry haven’t innovated fast enough in developer testing tools as we have in other areas of a developer’s workflow like CI/CD, Collaboration, Bug Tracking, and Code Repo’s. Just look at what Slack has done for communication, or what Kubernetes for container orchestration, or Github for source code.  

In fact, as I dove deeper into the survey results, users themselves said this loud and clear –  

Developers just aren’t happy with the current choices for testing and therefore are continuing to use Selenium due to lack of alternatives.  

That being said, this may potentially change in the future. Just look at the acquisition and fundraising activity within the last 12 months. Testing clearly isn’t as hot a market as security software, but I think that’s because we’re still in the early stages of people becoming aware there’s an opportunity here:

• SmartBear Software is acquired for $410M
• Tricentis raises $165M
• Sauce Labs raises $70M
• QASymphony raises $40M

Perhaps given this lack of innovation, paired with high degree of users looking for new testing tools, along with new fundraising activity, developers will see test automation come back to be “state of the art” with the rest of their development toolset. I’d be interested if others share a similar view on whether Selenium is really that good or not?