FOX Broadcasting Corporation Scales Agile Testing With mabl

FOX Broadcasting Corporation is one of the most recognizable brands on the planet, reaching millions of viewers through numerous websites, live events, and applications. To deliver outstanding user experiences across all these platforms, their engineering organization has adopted agile practices that enable fast, frequent updates. But despite FOX Broadcasting Corporation’s sophisticated engineering practices, testing itself was disjointed, making it difficult to see quality from a high level. Though some teams had adopted automated testing internally, QA Manager Gregory Goldshteyn found limited value when attempting to implement automated testing on a larger scale:

“We’re living in an agile world with two week sprints, and trying to automate tests in such a short time frame is rather difficult, especially when dealing with complex scripts and flows. Even if the test works in the beginning and we have time to automate some user stories, quite often we don't have enough time to automate all that we need.” 

The Catch-22 of test automation proved to be painfully true: any time saved by automating tests would ultimately be spent on test maintenance. Rather than waste valuable time automating tests that would be a drain later, some FOX teams still rely on manual testing. 

Setting the Stage for Enterprise-Scale Software Quality 

With a testing culture focused on individual teams, there were few shared standards for software testing across the FOX organization. When Gregory joined the company in 2018 as the quality leader for the video engineering team, he realized that FOX’s extensive footprint demanded a standardized, scalable automated testing solution. Some teams had implemented their own testing tools, some relied purely on manual testing, and few quality professionals collaborated across different teams. Though the individual testing strategies worked for an isolated processes, Gregory saw the potential for creating a cohesive quality engineering practice that would enable FOX to nurture QA talent, improve collaboration between teams and disciplines, and support agile development practices for the long term. 

Minimizing Testing Costs at Scale With Low-Code and Autohealing 

“Even when we’ve automated all desired test cases, they would be brittle and break later on. And complex tests with lots of steps would require particularly extensive, time-consuming maintenance. That time and effort costs the company money.”  - Gregory Goldshteyn, QA Manager at FOX 

Due to the sheer size and complexity of the FOX product portfolio, managing the human costs of test automation was a priority from the onset. As the FOX organization had learned with other automated testing experiments, the time and effort demanded by high-code, high-maintenance testing tools had tangible costs to the company, especially at their speed and scale. 

Mabl’s autohealing capabilities proved to be the scalable solution to the test maintenance challenge. Using machine learning, the mabl platform automatically evolves tests as the product changes, significantly reducing the amount of maintenance required. The time saved by autohealing made it possible for some FOX teams to expand their testing strategies to better reflect the full user experience. 

Keeping pace with FOX’s two week sprints also meant identifying a test automation solution that accelerated test creation, further mitigating the human effort costs. To address this challenge, Gregory began investigating low-code test automation tools. After looking into mabl, Testim, and Sauce Labs’ AutonomIQ, he realized that mabl offered the most powerful combination of flexibility and simplicity. The mabl Trainer was the most intuitive test creation solution, while JavaScript snippets and an array of integrations enabled FOX to automate a wider range of tests.

In one case, a team testing backend APIs with Postman collaborated with a frontend team handling UI testing to migrate their Postman Collections into mabl. This allowed the frontend team to embed API tests into their existing end-to-end and UI tests, which improved the performance of those tests and enabled the frontend testers to better understand the user experience. Mabl’s integration with Postman and low-code test creation process reduced the time needed to start seeing value from integrated API testing, while autohealing ensures that the frontend team will be able to maintain the more complex end-to-end tests in the long term. 

Now that mabl has reduced the human cost of test automation with low-code test creation and autohealing, more FOX teams have been able to sustainably implement automated testing.Though many had zero automated test cases prior to implementation, most have 30 to 40 percent of their test cases automated, with some teams reaching 50 percent without slowing down their two-week sprints.   

Enterprise-Grade Test Automation Demands Flexibility  

Though low-code and autohealing tests ensure that automated testing is efficient enough to keep pace with the FOX organization’s agile practices, scaling an enterprise-grade automating testing strategy demands a testing solution that can handle the diverse needs of the FOX product portfolio. Every team has slightly different nuances and every product has numerous variations that demand powerful, adaptable testing. 

Most FOX teams run their pipelines on Jenkins, but how and when tests are run varies widely. Some teams prefer to run tests per environment, such as executing a more extensive test case when that runs when deploying to a staging environment. Other teams have pipelines with production deployments, where test scripts are tailored to run based on production data, in addition to a smoke test post-deployment. Mabl has proven capable of handling each team’s preferences, from end-to-end tests that assure the entire team of product quality to smoke tests that address specific QA concerns. Gregory shared that mabl’s enhanced branching capabilities are particularly useful for FOX QA teams, many of whom create dedicated branches for particular testing needs without impacting the main branch. Whatever each FOX team needs to understand quality, mabl’s extensive suite of features and functionalities is able to adapt.

Scaling is important at any organization, but at a company the size of FOX, scaling is an absolute necessity. Every product - whether it’s part of a website, application, or live event - must be tested for dozens of different scenarios to deliver high-quality experiences to every consumer.  With mabl, FOX teams have the ability to expand testing without slowing down: 

“Parallel runs and data-driven testing have helped our entire organization scale testing. DataTables in mabl make it easier to write a script once, then create all necessary iterations. For example, we might need to test the same content across multiple file types. With mabl, we can create and run those different tests in parallel, accelerating testing.” 

The ability to grow the FOX automated testing strategy to the level demanded by a major enterprise operating on agile principles is already improving confidence and product quality. Managing hundreds, if not thousands, of potential customer journeys is a tall order, but mabl is steadily making it easier for the FOX organization to improve quality with testing that covers each critical customer experience. 

“Our confidence levels have improved when we go to production. Historically, a few teams have had fires because a particular feature wasn’t fully tested, resulting in rollbacks or emergency fixes. Now we see less of those situations because quality is generally better. We can cover all critical paths before deploying into production.” 

Standardized Test Automation Elevates Testing 

Software testing is increasingly the make-or-break factor when it comes to improving development practices and the customer experience. For the FOX Broadcasting Corporation, testing is evolving from a disjointed effort to a scalable strategy that reduces friction across the organization. When testing is simplified through low-code and autohealing, it becomes easier to amplify the impact of quality. For Gregory, a key part of that effort is supporting QA professionals as they build their skills:

“I’d like our QA folks to be experts on quality, meaning they’re focused on the user story that the developer is trying to achieve. They need to connect testing to the customer, not be bogged down in the code. Mabl helps our people do that, and I love it.” 

Mabl’s centralized platform is helping the massive FOX organization unify automated testing practices for better quality practices. For the first time, test automation is scalable without costing the company money in valuable human hours. Low-code test automation is helping teams quickly create comprehensive testing strategies within their two-week sprints, while autohealing ensures that maintenance doesn’t slow them down in the long term.

Though Gregory dedicated much of his time to teaching mabl basics to other teams earlier in the FOX implementation, he finds mabl’s intuitive nature is enabling more people to start using the platform with minimal support. This means that developers, QA professionals, and even some marketing teams have started engaging in quality for better customer experiences across the extensive FOX footprint. Though it may sound simple, removing the barriers to expanded testing has elevated testing on the enterprise scale demanded by FOX. Gregory summarized the results succinctly: “With mabl, that maintenance is significantly reduced and it’s much easier to showcase our results with the centralized platform.”