As experts in test automation, our team gets a lot of questions about how we use mabl to test our SaaS application:
- How does mabl approach test automation?
- What testing strategies does the mabl team use?
- What role does end-to-end test automation play in enabling the engineering team to continuously ship code to production?
We decided to take you on a behind-the-scenes journey of how we do testing at mabl. We’ll go into detail on how we use mabl to test the mabl application. In the end, you’ll have some tips on key strategies to help your team make the most of end-to-end testing. Let’s get started!
Testing strategies at mabl
At mabl, we take a holistic approach to quality assurance, which means that everyone on the product team participates in testing activities. This translates to covering every level of the typical testing pyramid from unit testing to exploratory testing. We also use our own product wherever possible, and take full advantage of mabl’s automation capabilities for integration and end-to-end testing across our web apps and sites.
Applying mabl to the testing pyramid
Here’s how we think about the different types of testing:
Unit testing
The primary role of unit tests is to validate that individual units of code, like a code branch (e.g. IF/ELSE statement) or function, work as expected. We consider writing unit tests to be best practice for developing software in general because they help us test code with code. Therefore, we have thousands of unit tests across our codebases and, in some cases, we have an ambitious target of 90% minimum coverage. The tools we use for unit testing include: React Testing Library, Jest, JUnit, Mockito, chai, mocha, and sinon.
Integration testing
Integration tests help ensure that several pieces of code (e.g. functions, components) can work together without unintended consequences. These tests are very valuable in places where it’s challenging for our team to have good end-to-end test coverage, such as the mabl Trainer Chrome extension. Unfortunately, browser security prevents us from being able to use the mabl Trainer to create tests against itself since it’s not possible to have extensions access each other. So we use CodeceptJS to create and run integration tests for the mabl Trainer extension. For the mabl web app, we rely solely on mabl itself. We also use Java for integration testing the mabl APIs, but since we are expanding the API testing capabilities we’ll soon be moving those over to mabl as well.
End-to-end testing
End-to-end tests are the most valuable tests in our arsenal because they are the closest representation of the user, and validate that everything functions, looks, and performs as expected. While having unit tests is considered a best practice for testing code, we consider end-to-end testing critical for testing the holistic user experience with the application or service.
This is where mabl shines. We have easily automated all end-to-end tests for our app. With mabl, we don’t need to write and maintain code and we don’t have to rely on manual testing either. Everyone on our team - developers, QAs, product managers - can build and run reliable tests, unlike when we need to test the Trainer with CodeceptJS. Mabl is currently set up to run end-to-end tests across our CI/CD pipeline environments, including production.
Exploratory testing
Even mabl does some manual testing, most often in the form of exploratory test that can't be automated. Mabl typically does team-based exploratory testing, where the team dedicates set time together to explore various areas of the application. The goal is to identify issues that automated tests may have missed. All findings are shared in a common Slack channel, which the meeting organizer reviews. Any issues that need to be resolved are logged in Jira so they can be addressed. We also use the insights from those sessions to identify areas where better end-to-end test coverage is needed to prevent future regressions.
Integrations
When it comes to testing, we are strong advocates of automation (naturally) and team collaboration. We are always looking for ways to better incorporate testing across our software development and delivery workflow. Some of the most valuable integrations we have are:
- GitHub Actions to help run tests on every deployment, and show test results in the context of GitHub Pull Requests.
- Slack to notify our team if a test plan for a given deployment fails. Slack also gives us added visibility into mabl insights, such as auto-heals and visual changes.
- Jira for issue tracking: when a mabl test identifies an issue, we can easily log that issue directly in Jira from the test results page in mabl.
- PagerDuty automatically notifies our on-call engineer when the mabl test that monitors our production environment fails.
- Segment helps us identify test coverage gaps in mabl by mapping real usage data to test activity across the application.
- BigQuery stores comprehensive app and user data layered with DataStudio for in-depth reports and analysis.
- Email for additional notifications, including a mabl workspace summary report, that team members subscribe to.
Continuous testing workflow
We typically organize the core capabilities of mabl into three main buckets: Chrome Extension/CLI, React web app, and Google Cloud services. While we test every part of the application, this next section will focus specifically on the web app.
The engineering development cycle at mabl is heavily reliant on a robust testing strategy. Our team is constantly developing new features and enhancing existing ones; some weeks there are up to 100 individual changes to the app. Without a reliable and thorough set of tests, we wouldn’t be able to deliver at our current velocity while also maintaining quality. Our development team incorporates end-to-end testing early and often in order to catch and fix issues as quickly as possible.
Testing is important at every stage of our development process
After determining which feature to create or improve, a mabl engineer will start development by creating a code branch for the project. Testing at this stage will consist primarily of unit tests and mabl tests. Since our web app is based on React, we use the React Testing Library to write unit tests for it. Engineers are required to add unit test coverage for every new component and feature. In addition, the engineer will use the CLI to run the mabl regression tests that are relevant to the feature to identify and fix any issues with our common user workflows.
PRO TIP An easy way to run focused tests is to label your mabl tests for a given feature and then run tests with just that label.
Before merging the feature into the product, the engineer will open a pull request. In the pull request, a fully functional preview environment with the new feature will be generated. This is a great time for other team members to give feedback and perform exploratory testing. All unit tests and our full set of mabl regression tests will also run at this step. If a new mabl test is required, it will be added to the regression plan at this time.
An example of a pull request with a preview environment and mabl tests
After feedback, the engineer will merge the change into the main branch. Now the feature is a part of the development mabl environment. This merge will trigger a deployment and run the full set of tests again, including the full set of regression and smoke tests for the app. Once the feature is working as expected in the development environment, it’s pushed into production.
Synthetic monitoring in production
While testing in production is sometimes frowned upon, we believe this is an important step in our continuous engineering development cycle. Today’s cloud-based applications like mabl are often heavily reliant on third party services, so it’s important that we run our full set of regression and smoke tests when tagging a feature for production to ensure that there are no outages from any parts of our infrastructure or third party services. Additionally, a set of these tests runs every four hours, in both cases enabling us to proactively look for issues and fix them with minimal impact on our customers.
When a monitoring test for the production environment fails, the on-call engineer is notified - any time day or night - to resolve the issue. In order for these tests to be effective when the mabl production environment is having a problem, we actually test our production app using our development app. This enables the two environments to function independently while also giving the mabl team an external method of determining if our production app is healthy.
We monitor our production app for issues with third party services and infrastructure
We also use Rollbar and Google Cloud Operations (formerly Stackdriver) to monitor our production environment for errors and capture logs.
Feedback flywheel
Our team is constantly looking for ways to improve our testing strategies. We take a customer-centered approach to improving our testing. Beyond adding tests for new features, we also add tests for any customer-reported issues to ensure that no similar issues appear in the future. Feedback is a core part of our culture, and gathering customer feedback is particularly important. Team members across the organization aim to connect with customers on a regular basis to help inform product decisions and improve overall quality.
Another way we look for possible improvements to our testing strategy is to use the coverage feature. This mabl feature shows the pages of the application based on the link crawler and our segment integration. We use this feature to identify which pages are visited the most compared to pages that are tested the most, allowing us to adjust our test coverage for maximum customer impact.
Key takeaways for your team
We covered a lot of ground in this post about how testing is ingrained at mabl. We also provided an in-depth look at how we use mabl to test mabl. Here are a few takeaways that may be helpful to you and your team:
- Testing is a team sport and there is no one-size-fits all approach.
- You can use mabl to do reliable end-to-end testing at every step of your development process to catch and resolve issues early.
- Integrating mabl with your workflow tools helps keep your team informed, facilitates collaboration, and increases productivity.
- CI/CD tools make it simple to establish and run mabl tests during deployment, a critical and often overlooked opportunity for testing.
End-to-end testing insights and coverage reporting help identify important quality gaps. - Applying synthetic testing to production can help your team proactively detect regressions, especially in third-party services which are not necessarily in your control.
To learn more about our story, check our recent session from Experience 2020.
New to mabl? See for yourself how to embed automated tests into your entire development lifecycle - sign up for a free trial today.
