Testing for DevOps Pipelines: Pull Request Testing

So, you’ve completed your work. You’ve developed your new feature or created a patch, and you’re ready to create a pull request and submit your work to the main branch.

But wait, are you?

As we mentioned in our previous post on testing for the DevOps pipeline, only those at the apex of the testing maturity model tend to conduct tests during the coding stage. Instead, most organizations will begin testing at the pull request stage. This has some disadvantages—there’s more testing to do and less time to do it in.

Given these constraints, what should your testing model look like? Let’s dive back into our infographic on testing for DevOps pipelines and explore.

Click the Diagram to Expand

Understanding the Pull Request Phase of the development lifecycle

For illustrative purposes, we’ve split what we’re calling the Pull Request Phase of the SDLC into two parts. The first part is the commit stage, in which developers add their work into an experimental branch of the source code. There’s also the PR approval stage, where the experimental branch is merged with the main branch.

In an ideal world, developers would be able to run one series of tests every time they commit, and then a second set of tests before the pull request is approved. In the first stage of testing, you might perform:

• Linting to eliminate obvious coding errors
• API smoke testing to ensure that basic functionality still works
• Unit testing to validate the functionality of individual commits

Looking at the infographic, you’ll see that many of these tests could also be performed during the coding process. We’re assuming those aren’t completed yet. You’ll also notice that the second stage—before the PR approval—is where we typically see code review for the first time. Traditionally, the pull request phase is where the organization reviews code, suggests additional changes, and asks for further commits. The pull request is then approved and merged once this process is complete.

More mature users may have a “build” step where they add a testing feature that automatically attempts to build the experimental branch before it gets merged with the source code. This confirms that the new features won’t break the build once they’re approved and acts as a foundation for unit-level testing. This more advanced level of automation makes it less likely that developers will need to immediately roll back a pull request due to code-level failures.

Primarily focusing on the code at this stage, however, often leads to blind spots. Missing these blind spots can increase the risk that bad code can sneak into production, potentially creating serious problems down the line. Adding more tests at this stage can greatly reduce that risk, saving your team time, effort, and stress.

Expanding and streamlining pull request testing

In an ideal world, developers would be able to perform three extra steps before the pull request phase is complete:

• First, in the commit phase, developers should be able to perform UI smoke testing to ensure that their new code hasn’t broken critical functionality
• Second, developers should be able to validate that they have not broken existing tests
• Third, developers should confirm that they have the right test coverage for their new feature

Beyond just reviewing the code and making sure that it builds, developers should be able to make sure that the application works as intended—for example, by ensuring that the end-to-end tests pass. You should also be able to ensure that you haven’t broken your tests before you commit your changes to main.

If a test breaks at this stage, you have an important decision to make—do you rewrite the test so that it passes with your new code, or do you rewrite the code so that it passes the test?

The moment when you propose merging your changes into the main branch is not the time to make this decision. Instead, you should be able to measure the coverage of your tests during the commit stage. Whether tests pass or fail is only part of the equation - you should instead be asking whether the tests still provide coverage, whether you still provide the right assertions, and whether the tests are still ensuring high quality. In this manner, you can apply the same standards and engineering discipline to testing as you do to coding.

Add quality to the development and testing lifecycle

Here at mabl, we make it easy to eliminate the seams between testing and development. Our solution provides a highly intuitive way for any team members - even those who might not necessarily be skilled at testing specifically - to incorporate automated testing into their workflows.

This blog is the second entry in our series on testing for DevOps pipelines. To learn more, check out our posts on understanding the code stage, deployment stage testing, and testing in production. 

Sign up for your free trial of mabl to see for yourself how easy it is to build reusable end-to-end test flows that can be integrated into any phase of your development lifecycle.