Get a Free Trial
Creating, executing, and maintaining reliable tests has never been easier.
Software development leader and mabl partner Atlassian found that half of DevOps teams are struggling to measure and improve their work processes. They discovered that while most software development organizations have a mountain of data at their disposal, few teams are successfully turning that data into actionable insights. When development teams can’t consistently improve, they’re unlikely to fully benefit from DevOps adoption.
As an enabler of DevOps, quality teams need processes and tools that allow them to unlock the full potential of data in software testing. At mabl Experience, mabl Software Engineer Lauren Clayberg explored how quality engineering teams can leverage data throughout the software development lifecycle for better software and more successful development teams.
How Quality Engineering Uses Data to Measure Change
Quality engineering, where continuous testing is practiced within a culture of quality, allows quality teams to expand the definition of application quality to better reflect the full customer experience. Actionable data makes this possible by helping quality teams make informed decisions about their application quality and quality strategies.
There are a few ways to become a data-driven quality engineering team. The first is using data to measure application changes over time. This is critical for understanding the impact of the quality team as well as how different aspects of the overall quality strategy affect the product. Data can reveal small changes that might be less noticeable with the full scope of quality, yet have the potential to make a big difference, such as subtle shifts in load time.
Lauren noted that she’s used mabl to identify declines in application performance well before they became noticeable to users. When performance testing an API endpoint, she used data to determine the endpoint’s speed was slowly worsening:
“300 milliseconds isn’t that noticeable to me, but performance testing made me realize that there was actually a lot I could be improving. Those kinds of changes can build up over time, and this was something I definitely wouldn’t have noticed without using data for that analysis.”
Tracking small changes results in actionable data that helps quality teams set concrete goals for their quality engineering strategy. Had the mabl team not tracked the performance of this specific API endpoint, they may not have noticed an issue until a user submitted a help request.
Using mabl for Data-Driven Testing
Ideally, quality engineering allows software testing to take place throughout the entire software development lifecycle. But quality teams still have an important role to play in the final stages of deciding if a project is ready for release. Once again, the right data allows quality teams to save the day.
Mabl’s release coverage feature allows quality teams to determine if their automated testing strategy is accurately evolving with the product. The dashboards enable anyone to quickly understand the state of the application under test as well as identify any gaps in testing.
Lauren explained how mabl’s test automation platform makes data actionable through the release coverage charts. One particularly useful dashboard measures the cumulative tests run on the application.
“I can make sure all of my test coverage is actually being run against my application. For instance, what if only 57 out of 60 tests were run? Before I would send that feature to production, I would want to make sure that I wasn’t missing any testing that I thought was happening.”
Other useful data can be found in the average app load time chart, which shows trends in application load speed. Like Lauren’s previous example with the API endpoint, data allows quality teams to spot defects earlier, making them easier to fix before customers experience any issues.
Page coverage helps teams prioritize where to add new tests for the feature or application. This feature lists every page explored by mabl in the app under tests over the last two weeks, filtered automatically by application. Mabl users can quickly understand how to adapt their automated testing strategy as the product evolves, ensuring smoother collaboration between quality engineering and developers.
Harness the power of data with mabl's BigQuery Integration
To fully unlock the potential of DevOps, quality engineering teams need to be able to connect the range of tools used throughout the software development process. This makes it easier to collaborate across the software development organization, create a culture of quality, and share data. Mabl offers a variety of integrations designed to make software testing collaboration seamless, including a BigQuery integration that allows test results to be exported to BigQuery and generate reports in DataStudio.
The BigQuery integration makes it easier to find test run information, which includes details like test status and the browser it was run on. Mabl users can find detailed information about plan runs, including application environment, deployment labels, and test run information.
Finally, mabl users are able to categorize every test failure, which can also be displayed in BigQuery. This data is particularly valuable when understanding overall application quality since it captures test trends at a glance.
Lauren noted: “This is a fantastic way to improve communication across your team. It helps you understand what is going on in terms of quality within your app and see trends.”
Quality teams can also use whatever dashboard software they prefer to create a custom dashboard to better fit the needs of their software development organization.
Identifying the Right Data for Each Team Member
An essential part of ensuring data is useful: sharing the right data with each team member. In a mature quality engineering environment, every person in software development has a role to play in software testing, but that doesn’t mean each team member needs the same information to participate in the overall quality strategy. Lauren’s mabl Experience presentation highlighted the most useful data points for each quality stakeholder:
Quality Engineers and Analysts
These team members are most likely focused on creating and updating tests, monitoring test performance and failures, and ensuring test dependability. Metrics they may be interested in include:
- Tests with the lowest passing rate
- Lowest passing rates for specific browsers
- Counts of new tests added for each feature
- Labels to determine distinctive features
Quality Engineering Managers
QE Managers typically focus on the quality of a specific product area and prioritize quality engineering efforts within their teams. Metrics they may be interested in include the following:
- Unique test runs
- Unique tests created per feature
- Failing runs in development and production
- Pass rates per browser
Executive Teams
Executive teams are focused on understanding how the overall organization functions, as well as broader trends in software testing and product quality. Metrics they may be interested in include:
- Number of automated tests by product areas
- Number of tests run
- Time spent testing
Software Developers
Developers are focused on creating high quality software, while minimizing regressions and tech debt. Metrics they may be interested in include:
- Pass rates for their branch or by browser feature
- Counting failed runs due to regressions over a period of time, specifically ones that made it to production.
Data Is Key to Smart Quality Engineering Decisions
Software testing produces vast amounts of data that provides value to the entire software development organization, if quality engineering teams have the tools and processes that allow them to share that information with the right stakeholders. With mabl, quality teams have the ability to track and share crucial information that makes it easier to identify quality trends, assess their testing strategy, and improve overall product quality.
As Lauren noted during her mabl Experience presentation on harnessing data: “You want to be able to use data to track those changes and set concrete goals for your team.”
See how mabl can help your team embrace data-driven testing with our two week free trial. You’ll have access to our BigQuery integration, release coverage features, and more for smarter, simpler automated testing.
