How do we know we have sufficient test coverage? How much do we really need? In this session, Adeeb Valiulla and Geoff Cooney will share why test coverage is a critical metric for QE teams, and the strategies Adeeb employs to grow coverage, and create better user experiences.
Transcription
Katie Staveley
Hello everybody, and welcome to our second session of the day for Experience 2021. I'm happy to welcome you to this session called How to Maintain Excellent Test Coverage. We have with us today a couple of fantastic presenters. Before we jump into that though, I'm going to cover a couple of housekeeping items. So we'll do a 30-minute session, we'll save some room at the end for questions and answers. If you have any questions throughout today's event, feel free to submit those through the question tab on the event platform. I'll take a look at those and share those with our presenters as we get closer to the end of our scheduled time. If you have any other issues that come up throughout the session today, feel free to drop those into the chat as well. As a reminder, we are recording. So all the sessions today and tomorrow will be recorded and available afterward and we'll share those around for everybody who attends and even everybody who registers. So, jumping over to our next slide here, a little introduction of today's presenters.
I'm really pleased that we have a guest presenter from a company called Sensormatic, Adeeb Valiulla. So Adeeb, comes to the session today with a ton of experience in Quality Assurance, most recently at Sensormatic and he's going to talk a little bit about how they handle test coverage, some of the metrics that they look at, and some of the success that they've had so far. We also have Geoff Cooney here today. He's our engineering manager at mabl; he's been with the company for four and a half years, which is pretty amazing because the company is about four and a half years old. But also has tons of experience in Development and Engineering for tech companies and software in particular. So for today's agenda, we're going to cover a couple of different topics. Geoff and Adeeb are going to talk a little bit about why you should care about excellent test coverage, what that means. How do we at mabl think about test coverage for our own engineering efforts? Geoff has firsthand knowledge of how we handle things here for testing our own application. How does Sensormatic, in particular, and Adeeb, will talk about this, how do they approach test coverage? Examples about how that test coverage impacts the business and creates better quality software for their customers and users. Then last but not least a bit of advice for you all today on how to improve test coverage for your own organization. So without further ado, I'm going to turn it over to Geoff. Geoff, take it away.
Geoff Cooney
Thank you, Katie. I thought I'd start just by sharing some numbers from mabl’s 2020 State of Testing in DevOps report that highlight the importance of testing and code coverage on both teams and customer outcomes. For some of you who attended the keynote speech, some of these numbers may be a little familiar. So to start with, we surveyed a number of teams and engineers, and practitioners about how their teams operated, what test coverage they had. You'll see from this data here that of all the teams that self-described as fully DevOps and automated, only 4% of them describe themselves as having less than insufficient test coverage. And of teams that are fully DevOps and automated, 23% describe themselves as having excellent test coverage, as compared to 4 - 7% for all the other categories.
I think this really highlights how critical having effective test coverage is to be able to get fully automated in DevOps to be able to push things out with reliability and confidence. At the same time, we see high test coverage makes our teams more efficient. Teams that reported this had excellent test coverage, reported a good or seamless transition of each handoff of issues to engineering - nearly 80% of those teams reported that. We also see that the team's time to resolve issues is significantly faster. As you can see teams with excellent test coverage had nearly twice reported that they could release fixes to production in under eight hours for bugs and nearly twice as often as all other levels of test coverage. So that shows that good test coverage is critical to making our teams perform well, keeping up speed getting DevOps, but what about the customer outcomes?
So, at the same time, we see teams that reported having high test coverage, 80% of them reported high customer happiness, but only 3% of those same teams reported low customer happiness. Compare that to teams that said they had low coverage, which was nearly a 50-50 split, 30% of those teams reporting high happiness. But I also want to talk a little bit about how mabl thinks about driving towards effective test coverage. Presumably, we're all here because we believe that test coverage and testing is a critical part of the software development lifecycle. But how do we get to it for effective test coverage?
Now I want to highlight a few things that we think about here in the mabl engineering team and also ground that in some examples of what we're what we do internally in these different areas. The one thing I did want to highlight is when we talk about test coverage in this sense, if you Google test coverage, you'll often find the information about the metrics, right? How do we measure what's the line percent coverage? What's the branch coverage? When we talk about test coverage and when we think about test coverage, we're really thinking about how do we ensure that we're testing our application in such a way that it behaves as we expect and that users get the experience they expect with a slightly broader definition than just the measurements. The measurements are a piece of that, but not the whole thing.
The first thing we think about at mabl is, how do we make it easy? How do we make it dead simple for people to add additional test coverage for people to contribute to our testing story, and to make our app better tested? Maybe, one of the examples I like to think about there is, imagine we have a developer who's working on a bug fix. It takes him an hour to find the solution, maybe it's just a couple of lines, 10 minutes to get the change, and then now they'd like to add some test coverage to make sure that doesn't happen again. If it's 15 minutes, then add some new tests that are going to do it. If it's an hour, probably, you start looking at, if takes a day because the right pieces and infrastructure aren't there and they have to set up mocks and infrastructure or if they in work mabl, they have to define all the reusable blocks from scratch, they're much less likely to actually do that. You're also starting to push at, getting outside that eight-hour window delivering the fix. Now you're trading off. Do I want to deliver the fix quickly? Or do I want to make sure I have it tested so it doesn't occur again?
So maybe we think about this in two dimensions. One is how do we make it dead simple, easy to add new test coverage? At the same time, how do we make it easy for developers to get feedback and act on that test coverage? You saw Dan touched on both these dimensions a little bit in the keynote speech. In terms of making it easy to add tests, a lot of this is about the infrastructure and simplicity of the process.
If you're talking about unit tests, you know some examples of what mabl does there is that we have a unit test infrastructure that makes it easy for us to test simple HTML files. This allows us to validate that our test execution systems are running as expected and that if we add some new feature functionality there, it's relatively easy to add a quick example case and lift that up and test it. Similarly, when you're talking about API or other layers, you want to add mock services, and then when you get up to the end test level, you're looking at mabl building, building blocks in mabl - how do we have flows that can be reused in different tests? One example of this for mabl is when we find that we highly parallelize our test executions, the login flow we actually have some risk of getting blocked by our vendor because we have too many tests logging in at the same time by the same user. So we have a reusable flow we built that handles this case, and it makes it easy for not every engineer or product owner who builds a new mabl test has to rethink through that logic.
At the same time, running lots of tests is great, but we need to get that feedback to developers to act on it as quickly as possible. So some of the things we're doing there from a mabl perspective are, we have some infrastructure that basically automatically generates test environments for each pull request. You can see that in the image to the right. So when the pull request goes up, we automatically create a new environment link to it there. And then we also use the CI/CD integration for us. That's GitHub actions, you can see some of the actions we run on the right as well.
Those GitHub actions, they run code checks, they run unit tests, and then they also run our mabl tests, and that dynamic environment is critical to that because we're able to automatically kick off both headless tests and tests in the Cloud that run against that environment and give us real-time feedback on the end to end validation of our changes before we push them out. Then we also have a set of tests that don't necessarily run a PR but we do regular monitoring of our environments to ensure that they're healthy that the application behaving as expected and those actually pipe onto our on-call rotation and page people to take a look at them if they start failing and make sure that there's no problem with the application. So now we have this infrastructure that allows us to add tests without too much difficulty.
The next thing to think about is how do we focus our efforts to know where we don't have test coverage? Or where are we testing or is this a new feature? Or what are the things we don't have? So at mabl there are a number of test coverage tools we use for that. So we use jest, unit test, and branch line coverage, which tells us, which specific code has a test in which aren't for mabl, we actually use the mabl and mabl coverage metrics which also tell us which sections of the application are tested and which tests have we run most recently in the last release. And do we have gaps of things that aren't that should be scheduled to run their plan, but they're not in the right plan? That also layers in some non-functional information such as - we’ve demonstrated before, do we have pages that have gotten less performance or tested that much performance? Is that something we need to dig in on? Then at the same time, it's nice to know which paths we're testing which paths we aren’t testing? But you know, there's also the question of what is the quality of those tests that we're creating? What's the quality of the testing of those paths, and for that we use a lot of that code review, and feedback on our mabl test cases. So things we're looking for there - Do we have the right things covered that we thought would be covered? Do we have the right test cases? Do we have enough assertions? It's good to test the paths but you also want to have verification that the page looks like what it should and that various elements look like what they should.
One of the other things we think about is continual feedback. As we saw earlier, one of the goals of testing is to simplify the engineering process. But another goal is to make sure that users are getting the experience that they expect. Sometimes that means more than just what the team initially sets up for test coverage. So now that we have test coverage in place that allows us to go to the next level of quality engineering and we have what we call these Elmer Fudd, exploratory testing sessions. The Elmer Fudd, because we're hunting bugs, is kind of the paradigm there. What we do is we use those as a way to focus on a specific area of the app, often one that was either recently changed or a new feature we're rolling out and have, time for people who worked on it, and people who didn't work on it, who it's new to, to explore the application and find new cases that we can not only then ensure are working as expected from a user perspective, but then we can gather feedback into our test coverage and test cases that we had.
At the same time, we do the same thing with customer feedback sessions. Then the next step is once we release it to production, you want to know, are users getting the experience they expected? Are they hitting bugs? Or are there places where there are aspects of the feature release that are confusing and are not behaving nicely as people are trying to use them. We have a number of test coverage tools with their feedback from there and which are I think various types of application instrumentation telling us where people click, where they're getting stuck, where they're where they seem to be frustrated, and then also support requests are a great source of data for this as well. So we're getting a lot of questions. Even if the questions aren't leading to exact bugs, we are finding out that lots of people are confused and asking questions about a particular feature, and that's maybe a sign that is some practical coverage we need to add in that area and need to rethink.
Finally, test coverage isn't really a static thing. A lot of the metrics are static, terms of lines and branches covered. But when you think about it, am I testing the things that my users expect? That's something you're constantly learning about. This is constant learning about the product. With that, I'll pass off to Adeeb who's going to take us through the Sensormatic story around coverage.
Adeeb Valiulla
Thanks, Geoff. Thanks, Katie. Good afternoon. Good evening, everybody. Before we jump right into Sensormatic’s approach to test coverage, I wanted to take a couple of minutes to just let the audience understand what Sensormatic is all about. Sensormatic solutions help retailers drive frictionless, personalized experiences by powering accurate decision-making across the enterprise. We do this by leveraging our four main pillars which are inventory intelligence, loss prevention, traffic, and analytics. True views inventory intelligence harnesses the power of RFID for a single view of item-level inventory across your enterprise. Sensormatic empowers retailers to fight global shrink and retail climb with proven merchandise protection solutions. On the traffic side, big data in retail is being used to shape the customer experience by making shopping easier, more convenient, and more interesting. Sensormatic shopper track solution collects and delivers data that retailers can easily convert into meaningful and actionable insights. Last but not least on the analytic side, we help retailers harness that data across the entire retail enterprise to deliver exceptional shopper experiences and increase operational excellence. So you'll notice we have a ton of products, a ton of ways our customers use our data and use our services.
This brings us to quality to its core, we want to make sure we deliver optimum quality and exceptional customer service. This is why test coverage makes a huge difference to us. Let's jump right into what my approach is in terms of test coverage. Creating a comprehensive testing strategy, it should take into account the applications requirement as well as testing methods you will be employing. Moving on to creating a checklist of all those testing activities based on the strategy, the next step will be to create a list of actual tasks that need to be carried out. Now keep in mind you need to account for manual and automated, size and experience of your team and in the type of application, you are developing. You also need to prioritize areas of application failure areas of application that are critical and have a high probability of having problems. You then need to create a list of all those applications, perform a risk analysis, which will be helpful in your risk coverage. Last but not the least, you have to leverage your test automation. That way you will reduce overall testing time, free professionals from doing repetitive and tedious error-prone activities, and be able to cover more portions of that application.
Keeping this strategy in mind, at least in Sensormatic, we saw a huge difference before and after. I'll take you to a couple of examples for two of our products, both our web reporting products and for this webinar, I'm going to take you to these two applications. So for product one, when we started, we were only at 40 percent test automation coverage. We were not able to do any PDF testing, no email testing, API testing was barely minimum to zero. When we started our journey, I'm proud to say that, we have almost doubled our automation coverage, not only doubling our automation coverage to%. But we were able to add more and more comprehensive scenarios, data validations, we were able to cover emails and PDFs and CSVs and more and more test cases were migrated to the mabl platform.
For the other product, we ran from a staggering 10% to an exceptional 95% automation coverage. As in when more features come in, we try to be as close to 100% as possible. This has helped us tremendously achieve so many goals, not to mention happy customers. But you might ask how we got there, there were a couple of things we utilized together. We utilized the click and record functionality mabl gives us. We utilized the API testing functionality. We used the mabl extract JS snippet for more and more data validation. Then we parameterized flows to make our flows a lot more universal and reusable in most of our scenarios. We incorporated our automation first approach and our initiative to utilize the echo feature in the trainer to write our manual test scripts directly in mabl so we can automate.
Last but not least I do want to highlight the amazing work my team has done. So, the two products which I just showed you for the first one where we went from 10 percent to 80 percent. I just want to talk a little bit about the team dynamics - we had around 9 QAs, around 13 developers, and over 6000 test cases. The product where we went from 10-95%, it was around 3 QAs, 5 dedicated devs, and roughly around 1000 test cases. So you can understand the complexity and the number of test cases my team was able to deliver. Super happy about that.
In addition to that, mabl keeps on bringing out new features and I just want to highlight a few features I'm really excited about and my team can vouch for. First and foremost, mobile testing, I’m really excited about what mobile can bring to the table in terms of mobile testing. The use of configure find has improved our element reliability tremendously and we look forward to making our test stable and reducing maintenance time. We want to add more and more scenarios using JS snippets and use API tests. Using all of those features, obviously, our goal is to increase test automation coverage. All of this had a huge impact on our business.
There are some numbers I'm showing on the dashboard right now, because of this test automation coverage, we were able to reduce our production books to 35% year over year. In addition to that, our pass percentage increased to 95%. What that means is 95% of the time, whatever we deployed in production stayed in production. Then 90% control delivery of all of our initiatives, because we were able to run our regression cycles and our testing cycles and all utilize more and more of our automation.
Considering all of these metrics, obviously, we had great feedback from our customers, our customers are happy and in addition to our customers being happy, our team was also very bonded, we had great synergy amongst ourselves, all people were helping each other and a happy team. So happy customer, happy teams were the greatest takeaways I had from this exercise.
To conclude my advice on improving test automation coverage to whoever's watching today, I'll start by laying out five bullet points, starting with having a strong strategy, which allows the team to give a good roadmap for effective testing, that will keep them grounded, especially in high-stress release cycles. Data collection is a necessary practice for high-quality quality assurance, and quality engineering teams and knowing their key KPIs, internal and industry benchmarks. Most important customer happiness and sentiments can ensure your team is focused on testing the most important part of your applications and provide a path for improving the overall customer experience.
Moving on, setting goals assists teams and working towards a productive release that benefits not only the bottom line but also enhances innovation and collaboration within the team. To be genuinely successful, expanding test coverage must be done gradually and in stages, you cannot flip the switch in a day or two, you have to have proper planning and have gradual milestones and timelines. Last but not the least, test automation can be a huge multiplier when it comes to coverage. When it comes to enabling your test at runtime, you want to make sure you have less human intervention, allowing your quality assurance teams to use more and more of their time to be creative and focus their efforts on testing more complicated scenarios, or participate in more exploratory testing. So these are the high-level bullet points that I would lay down to whoever wants to start going towards the test automation coverage increase plan. Thank you guys for giving me this opportunity. We'll take questions now.
Katie Staveley
Terrific, thank you so much Adeeb, and thank you, Geoff, that’s some great information in all of those slides and throughout the session today. We did have a couple of questions that have come in. Adeeb, let's start with a question that came in. You just talked about actionable data. So could you share a little bit more detail around what you and your team consider actionable data and what that means?
Adeeb Valiulla
Yeah, absolutely. So we have created data-driven teams within Sensormatic and what we mean by a data-driven team is that we look at our data and learn from our mistakes and plan for our future and be better every single spring every single release. The data we capture is around production bugs, how many bugs are being missed in production, what's their severity? What areas are being missed out on? That's number one.
Number two is automation coverage and how well we are progressing towards automation? Are we adding more and more feature sets into our test automation framework? That's number two. Number three is obviously, production deployment versus rollback. So that's the change in failure rate. How many times you deploy certain things in production that stays in production. That's one of the metrics I mentioned in the presentation where 95% of the time whatever we deploy in production stays in production.
Other actionable data we track and act on is delivery timeliness, how punctual are we with our deliveries, if we are punctual, perfect if we are late, we also capture data on why we are late. Let's understand if there are problems in our requirement gatherings if there's a problem in our estimation, is there a problem in the testing and development. So a bunch of this gets clubbed into our tools which enables us to create these actionable insights and this gives my team the ability to also just reflect and see how well they're doing and where we need to improve on and rightfully so get the credit back to the team. Reducing production bugs year over year by 35%, improving our test automation coverage and having punctual deliveries of our initiatives or projects, it speaks volumes, it speaks that the team is going in the right direction, obviously, everyone has to improve and everyone can be better. But these are some of the data points we measure to make sure we're going in the right direction.
Katie Staveley
That's great. Adeeb, I have another question for you actually, this question is how are you creating and managing your testing tasks? Are you using a specific tool?
Adeeb Valiulla
Yes, we use Atlassian products. Yes, we use X-Ray. We have a common strategy. You know one of the points I mentioned even in test automation, which goes literally in any exercise you do is having a common strategy. Any project, any initiative, high, medium, low complex, simple projects, we have a set of testing activities we do. We have functional, non-functional, we have regression testing, and everything is automated. We have an automation-first approach. So the team is expected and we have alignment with the business that we will automate as we go and we have to cover all of these scenarios. It's basically a checklist: you do this, you do this, you do this and once everything is checked, we move on to the next level. So we utilize some tools but also a good strategy across what type of testing we have to do, what environments we have to use, and so on and so forth. Definitely helps my team stay on plan and follow that common objective.
