Get a Free Trial
Creating, executing, and maintaining reliable tests has never been easier.
Wurl is a not-so-secret pillar of the Streaming and Smart TV world. Their massive network connects video producers, video services, and advertisers, enabling creators to reach a global audience and advertisers. Needless to say, managing quality for this large and growing ecosystem is a considerable challenge.
Using Low-Code Test Automation To Ensure Video Quality at Scale
Jagadeesh is a Lead QA Engineer at Rebaca Technologies, where he consults on Wurl’s quality strategy. Jagadeesh has years of experience in automated testing, including building his own custom software testing frameworks. For Wurl, custom automated testing frameworks were only useful to team members that knew how to code, limiting their ability to scale automated testing.
Wurl provides a feature-rich experience for media companies to schedule their media content. In order to ensure that the ad and promo placements for their video content play at the right location and that the transition is smooth during playback, Wurl has developed a video player that ensures ad placements are seamlessly placed in existing content
Video playback is difficult to test with most automated testing tools, since there’s no easy way to ensure a smooth transition on the player between standard video and the ad or promo as well as check that the captions associated with the video using existing functionality. With mabl’s reusable JavaScript snippets in mind, Jagadeesh designed an automation suite that can validate both the timing of ad placements and the accuracy of any associated captions. The automation suite relies on JS snippets to control video player actions, making it possible to test these nuanced scenarios that would otherwise be difficult to manage.
Validating Video Player Functionality with JS snippets
Most streamed television has approximately 24 frames per second, so the average 30-minute show has around 43,200 frames. Even a 30-second ad features over a thousand frames, making it difficult to assure quality across an entire program without automated testing. With mabl, Jagadeesh and his team are able to use automated smoke tests to validate text in different video frames.
Consider a 30 minute video: there may be 10 minutes of show content, followed by one or more ads, followed by more content. This happens multiple times throughout the day as the content lineup shifts, and each time, a smooth transition and consistent quality between a changing queue of ads and changing episodes is expected. Using manual visual and audio testing combined with automated testing with mabl, the team is able to validate the functionality of each sequence.
To test each scenario, the Wurl team designed their mabl test to pause the video at specific durations (5, 10, and 15 mins) for different types of asset playback like promos, ads, and episodes. At each given point, the video frame is expected to contain particular text. They validate the location of the text within the player by using custom JS Snippets, which utilize OCR technology. Each test can contain any number of steps that validate videos at multiple points to cover more complex scenarios, depending on the needs of the content. A library of reusable flows simplifies the test creation process as different team members look to test different assets.
Using a custom JavaScript snippet, the team is able to capture a base64 image of a video player as part of the test execution. That image is then converted to a JSON object via API, so team members can review the test steps, flows, parameters, and results, making it easier to triage failed tests. Being able to watch tests playback live also helps the Wurl team manually validate the video’s audio content, since testers can review the video knowing that the quality of the audio will match the quality of the already validated video. With this approach, 70-80% playback related test cases can be covered. The team then focuses on the remaining 20% of the playback validating the QoE and QoS of a playback manually.
The following screenshot demonstrates the playback sequence validation of Promo[1], Ad[2] and Asset[3] types. In each step, text within the video will be validated.
Before mabl, scaling testing to this extent was difficult since Wurl's QA team had different areas of expertise, in addition to the challenge of maintaining testing infrastructure. Low-code automated testing makes it possible for everyone to contribute to Wurl’s quality strategy, even for difficult test scenarios like validating video.
Measuring Quality and the Viewer Perspective
QoE (Quality of Experience) describes metrics that measure the performance of a service from the perspective of a user or viewer. Common streaming video related QoE metrics include rebuffering, playback failures, and video startup time. For example, when a viewer is streaming video, the network might experience a 2-second outage that would be captured by a QoS availability metric. However, if the video player had buffered enough video data ahead of time, the viewer won't experience any issues that would be measured by a QoE metric, such as rebuffering.
QoS (Quality of Service) refers to metrics that measure the overall performance of a service. In the context of streaming video, this often refers to networking and the ability of a network to deliver video bytes quickly and consistently. Common metrics include bandwidth, throughput, latency, and availability. With mabl, Wurl has been able to improve both QoE and QoS metrics, thereby ensuring that viewers have a seamless, enjoyable streaming experience.
Manual and Automated Testing Join Forces to Keep Wurl Going ‘Round
Managing quality at scale is no easy feat, but a combination of manual and automated testing helps Wurl’s QA team move quickly without missing a single frame. Since starting with mabl in March 2021, Wurl has been able to grow their automated testing strategy to 300+ test cases, and get more cross-functional team members engaged in test automation. Wurl’s network of creators, advertisers, and viewers is growing quickly in the streaming era and thanks to an effective testing strategy created by Wurl’s QA team, they’ll be able to ensure quality across all videos.
See how you can benefit from mabl's adaptable, powerful test automation platform with reusable JavaScript snippets by registering for a 14 day free trial.
