Am I Covered? How to Maintain Excellent Test Coverage

Looking for a way to improve customer outcomes and increase the efficiency and effectiveness of your DevOps teams? Strive for excellent - and maintainable - test coverage. But how specifically do you increase coverage and create better user experiences? Adeeb Valiulla, Director of QA at Sensormatic and mabl Engineering Manager Geoff Cooney explored what ‘excellent’ test coverage means, and how quality assurance teams can achieve it at their mabl Experience session. Based on Adeeb’s real experiences as quality leader, the session shared actionable tips, strategies, and techniques for software testing professionals looking to improve their own test coverage as part of their quality engineering efforts. 

Excellent Testing is a Requirement for Effective DevOps

Why should you care about excellent test coverage? As Geoff pointed out, it directly impacts your teams and your customer outcomes. Mabl’s 2021 State of Testing in DevOps Report highlighted the importance of testing and code coverage on teams and end customer outcomes. Survey results of more than 600 industry professionals showed:

• Only 4% of fully DevOps and automated teams say they have insufficient test coverage, while 23% say they have excellent test coverage.
• Nearly 80% of teams that self-described as having excellent test coverage reported a seamless issue handoff to engineering.
• Teams with excellent test coverage reported they could release production fixes in under eight hours — nearly twice as fast as all other levels of test coverage.
• 80% of teams with high test coverage reported high customer happiness, with only 3% reporting low customer happiness. Compare this to teams that said they had low test coverage, with a nearly 50/50 split between high and low happiness customers.

Like quality engineering itself, test coverage is one of the few metrics that connect code to the customer, helping developers, product managers, and testers understand the quality of the product and how well their software testing strategies reflect the user journey. 

Sensormatic’s Approach to Test Coverage

As the global leader in retail technology solutions, Sensormatic understands the critical role of software testing and has worked to develop a systematic approach to test coverage. Using data and test automation, Adeeb has created a strategy to ensure that test coverage is accurate and comprehensive. He shared the details of their approach: 

• Create a testing checklist: Creating a comprehensive testing strategy should consider the application’s requirements and the testing methods you will use. Create an inventory of all testing activities, developing a list of tasks that need to be completed and accounting for manual and automated tests, the size and experience of your team, and the type of application you’re developing.
• Prioritize efforts: Focus your testing resources on critical areas of the application with a high probability of having problems. 
• Understand your risk: Determine your risk coverage by performing a thorough risk analysis.
• Leverage test automation: To reduce overall testing time and free employees from doing repetitive, tedious, error-prone activities, it’s imperative to leverage test automation. By automating many tasks, you will expand test coverage for a larger portion of your application.

Adeeb explained that this automated testing strategy has brought Sensormatic significant results. “We saw a huge difference before and after. For product one, when we started, we were only at 40% test automation coverage. We couldn’t do any PDF testing, had no email testing, and API testing was the bare minimum. I'm proud to say that we’ve almost doubled our automation coverage to 80% and we’re able to cover emails, PDFs, and CSVs. For product two, we went from a staggering 10% to an exceptional 95% automation coverage. This has helped us achieve so many goals — not to mention happy customers!”

Improving Your Test Coverage

How can your quality engineering organization start improving test coverage? Adeeb recommended five specific strategies for companies that are tackling changes in their own test coverage. 

• Have a strong software testing strategy: This gives your quality engineering team a clear roadmap for effective testing that will keep them grounded, even in high-stress release cycles. 
• Actionable data: Quality engineering teams must embrace data, including key KPIs, internal and industry benchmarks, and customer sentiment. Having this data ensures your team is focused on testing the most critical part of your applications and provides a path for improving the overall customer experience. Review your data, learn from your mistakes, and improve every future release.
• Clear goals: Setting goals benefits the bottom line while enhancing team innovation and collaboration.
• Increase test coverage gradually: To be successful, it’s critical to gradually expand test coverage in stages. You cannot just “flip the switch” in a day or two — you must have proper planning with gradual milestones and timelines.
• Embrace automation: Automation can be a huge multiplier when it comes to coverage; it allows your QA team to use more of their time focusing on testing more complicated scenarios or participate in more exploratory testing. 

The Impact of Excellent Test Coverage

Excellent test coverage requires both thoroughness and accuracy. It’s unrealistic to expect that any feature or product will ever have 100% coverage, especially as more software development organizations embrace DevOps and deploy new code on a daily basis. Instead, quality engineering teams need to understand what customer experiences matter most to their users, as well as the riskiest areas of their application to effectively increase test coverage. Armed with the right data and a software testing strategy that balances manual and automated testing, any QA team can start improving their test coverage for optimal performance. 

How does mabl’s own engineering team think about test coverage? Check out the full recording for this mabl Experience session to find out.