Testing isn’t technically supposed to have stages. But it helps us imagine an ideal test execution scenario across our software development lifecycle. In this session, Joe will share how to approach testing across four distinct phases of development. He’ll share who should execute which tests, where and when those tests should be executed, why it’s important, and how you can leverage mabl at each development phase.
Transcript
Joe Lust
In this talk about the 5 W's of Continuous Testing, we're going to focus on integrating testing throughout your pipelines with mabl. But first, a little about myself. I'm Joe, Lust a Cloud Engineer here at mabl, and I live and breathe testing, which has made it very apparent to me that there is a great need for continuous testing. That's what we're going to take a deep dive into today.
As we might have alluded to in the title, there are 5 W's and actually an H, we're going to talk about why you should be using continuous testing, what and where exactly, you would use Continuous Testing, when you should be doing that testing, who is going to participate as an actor in that testing process and finally, drop into the nitty-gritty, concrete details of how you can integrate mabl in your CI/CD pipelines using our rich tool belt.
First, the why? Why is Continuous Testing important for you? In short, we really want high-velocity high throughput pipelines for delivering code to production, for creating and deploying changes constantly. As you are probably aware, there's an accelerating rate of innovation throughout the software industry, where change is becoming faster than ever and we have to find a way to make all those changes.
For example, as shown here in this diagram, configuration changes, or code changes, or upgrades you need to make to your apps or even the many APIs and dependencies that you pull into your applications, which could be in the 1000s. We need to take all of those changes, and rapidly convert them into user impact, user value by deploying them. The challenge, though, is CI/CD or continuous integration and continuous delivery is easier said than done.
I wonder on our session right now, is everybody comfortable with shipping all code and code changes to production automatically? Probably not. Why? What's missing from this picture? I think most folks in our session probably know what it is. I'm going to guess for you. That's right. It's quality. We need quality throughout our pipeline to ensure that we rapidly deliver high-quality changes.
As I said, we need continuous quality testing in our pipelines to make sure that we can transfer all these changes into user-facing changes without shipping defects quickly. Now, don't just take my opinion and mabl’s opinion here. Also consider what major research groups in the industry like Forrester have said in the reports like the recent Forrester Wave report, where they identified that, while we see software tools, engineering tools, evolving rapidly to accelerate delivery. Currently, the tools for quality are not keeping up. The current state of quality testing and engineering is not able to cope with modern software development. We need a better way and that's where continuous software testing comes in.
So what exactly does this look like and where are we going to use it? Let's talk about continuous software testing in detail and how we're going to integrate that watchful eye into our system. So we can deliver with a high velocity. Typically, when we think of a software delivery pipeline, we think of something similar to this cartoon diagram here, where there are approximately four stages.
We have a coding stage, where we're going to make changes that need to get shipped, the pull request or review stage, we have to make sure that these changes are of high quality. Then we need to deploy this code somewhere. So it can be further evaluated and tested. Maybe it's your integration or your UAT or QA environments. Finally, we're going to need to send this code to run and its production environment. We'll take a deep dive into these four different stages and how you can integrate continuous testing in each using mabl.
So to start with, there's the coding phase where changes are made and I have a little secret to share with you a dirty little secret of engineers like myself, we might like to think that we're adding a lot of value, we're making features. But we are also the source of the majority of the bugs, we break a lot of things that we don't mean to. So we need a better way to help engineers isolate these bugs early at the first stage in our pipeline so they can't get any further. Two goals we generally have in mabl, when we integrated this stage is how do we make sure that their code changes are working and how do we have a rapid dev-test loop, that is the time between when you can make a code change, and then see that change reflected as quality or not. The components that we offer, that are in your tool belt here are mostly the mabl CLI and the desktop app. Those are our core continuous software testing services. If you want to get even more deeply integrated tools like our jest, library, and our GitHub application, and these are going to let us run headless tests, and smoke tests, various means of finding those bugs very early.
I'll touch real quick on the desktop app, the simple video here, give this one moment to load. The key takeaway here is that in the desktop app, we can simply run a local test run and all we did is just click one button here and now we see this panel open and it's testing my blog website, it's actually doing a rich end to end test, it's actually using a Chrome browser running headless in the background. I'll note that that's running in the background. So we're not interfering with your flow state, we're not getting in the middle of your workflow. You've probably seen this before Selenium, or some other tools where Windows start popping up and closing all over your computer, you can't do much.
But with tools like the mabl desktop app, an engineer can easily run those automated tests and then get back into their flow state. We saw there that that test ran in about, I think 10 seconds. What I just did here is I checked another box that said run headless and that's where I mentioned that we're not going to pop up anything in the way. As we see it only took five seconds to run. That is we can even run faster than that headless mode. The final thing I'm going to do right here is I'm going to show a neat feature, which is that we can put in localhost, we're overriding this test in the desktop app and I want to start a server over here in the background and now I'm testing my local code, I'm not even going out to a server. As an engineer, now I can make a change, run it locally and see it. Here my test is going to run end to end locally, a few seconds and now I'm really sure that everything works, right. There we go nine seconds and I have high confidence in that code change. Only 10 seconds to make sure my code is quality. It's much easier than some other tools. Let me just close this window. There we go.
Now the important thing to note here is we want the engineer to be in their flow state, we don't want to interrupt their context where they're most productive. So if that state for you, if that productive place is your integrated development environment, we have ways I mentioned earlier, or just tests and you can get all these green checkmarks to tell you if your code is working well written as JavaScript unit tests. In this case, just hit a button that turns green. Now as the engineer was sitting in her IDE like IntelliJ, here, there's no need to switch context, you automatically know if that change is quality or not using rich end-to-end headless tests.
Finally, for the most brazen of individuals, maybe like myself, if you really want the command line, you can do all the same directly from the command line if that's where you're productive, and you get the same quick, rapid feedback. So that is the first stage where you're integrating continuous testing into your software delivery pipeline.
The next stage is the pull request stage. A coder has made a change, but we really need to be confident that it's high quality before we send it further down the pipeline. Now, we can bring more tools to bear. We can bring tools like headless testing, which we've already seen, but also more smoke tests, API tests, visual tests, cross-browser tests are many different tests we can bring to bear here as we run these tests in the Cloud and give us feedback directly into our review context to understand how good or bad is this change. We need to make a correction before it gets any further downstream. That really sets us up for long-term success because as we all know the earlier we catch a bug closer, we can shift it left to a source, the less time it takes to fix it.
The tools in your tool belt here, again, are the mabl GitHub integration. We'll touch on that in a moment. With GitHub Actions, you can get these tests done automatically. Then our many other rich CI/CD integrations that work with most CI/CD vendors. So let me give you one example of how you can run end-to-end tests as a PR quality gate. Let’s make sure that we don't merge buggy code.
Here we can see this line down here. It says mabl deployment tests, simply by installing and configuring properly, the mabl GitHub app, you get this extra check placed into your pull request. Now if we clicked on this button for details, we'd see all the information if anything had been wrong, but luckily for this engineer, we see our changes are good. We have a green checkmark, everything turns green, this is ready to merge and this is a great way to use those rich end-to-end tests and everything you've already built into mabl automatically in your pull requests.
When you're making that decision is this code, something that we want to ship, you can have all that information at your fingertips. So we did that pull request and we decided we wanted to merge it. So now we're going to deploy it. When we deploy it, it's going to go somewhere for further testing. For example, it might be your UAT, or your QA your integration environments. The biggest thing we need to make sure we avoid is getting defects into production.
That's why now we're going to bring the full power of your testing suite to bear for comprehensive, end-to-end test coverage. We can do additional tests such as regression testing, cross-browser testing, maybe you used Chrome in your local tests and chrome in your pull review, but now we can use Chrome and Firefox, IE Safari, we can use emulated mobile browsers, we can try every possible confirmation of your application, we can also look at performance information, we can be more subjective, not just red or green and we also can bring in API testing, basically, we can throw the kitchen sink at this application change and make sure that we have very high confidence and also a very broad clear picture of the quality of the change.
How would you do this in your pipeline? It's the same tools as we saw before our GitHub app, our many CI/CD integrations, and additionally, our mobile CLI, which lets you, you know, really use all the power of mabl. So let's give you an example of what that might look like. In mabl, if you're on the desktop app, or the web UI, you've probably seen a deployment that really all of the quality information about the change that you're considering moving to production. It gives you a broad picture quality. So you can see all the rich insights mabl has collected, and really make your decision if you want to promote this change.
Here, for example, we can see on the UI that all these plans were run, they were triggered, because you told mabl when you made a deployment, and mabl went and tested that new deployment wherever you deployed it and we've got all the information about it. As we'll see later, there are additional subjective measures that are brought to bear here as well to again, give you that complete picture of the quality. Now, we can approach the final stage in your pipeline, which is getting that change to production so it has a user impact.
Now, it might seem like that's the end of our journey. But quality, it doesn't end, when you deploy to production. For example, maybe at 2pm on a Friday, you ship code to production, but it still has to be quality code at 3am on Sunday. That's where we really think that continuous testing moves into production as well. For example, we can continue it. So you run all of those rich tests that you created earlier, the regression tests, the cross-browser testing, and we can harness your QA investment to become a monitoring tool to continuously validate production. So that we know that, again, that user workflow that was very important to your company, for example, being able to check out with the checkout cart or being able to create a new user account that still works every hour every day.
The how you would do this in mabl is very simple. You just use your scheduled plan runs. If you ever go in to create a plan run, you'll see that there is an option, a trigger you can use and you can tell it to run for example, every day at 2am or just run every hour. Then you simply attach this to whatever your monitoring tool is, and now you have a very powerful monitoring tool, which, as I said, harnesses all the testing, end to end testing that you've already created.
Just to reinforce that, you might ask again, when is it important to be using continuous testing? That's where the answer is, we want to test continuously, basically, all the time the DevOps symbol you see here; it looks like an infinity symbol and that's no coincidence, quality, it never takes a break. So we need to test whatever defects could occur, which for many applications that are deployed in the wild, new SaaS applications, that's anytime things could break at any time. So again, don't just want to test we make a code change, or do a build release, we want to test continuously, we want to monitor. Doing so is, again, very simple to do, because of luck while you're using mabl.
Once you built all these rich end-to-end tests, you can simply integrate with your pipelines, as I mentioned earlier with these schedules and these different events, and you get that continuous testing at all these stages of the pipeline. Now, you might ask, who is going to do all this? Who are the audience and the actors in our continuous software testing play? The answer is really simple. Everybody, everybody is a participant in quality with mabl, we really see this as a way to democratize testing, that everybody in your organization can participate in quality and it starts on the left here with our quality engineers, and they're going to craft our tests, maintain these tests and they're also principally going to be reviewing all of the rich results, reports that come out of mabl.
But they're not the only people that can help our coders. People like myself very happily contribute all the bugs. We also, as we saw earlier, in the PR stage, we can see all the results in our pull requests, the quality that was created by the quality engineering group, we can see that in our commits when we make a change. A really important place we can see it is in the debugging ability of mabl. We've heard from many customers, and we've done it ourselves here mabl as engineers, that an engineer gets a ticket, someone's reported a bug is a ticket, and rather than try to go through the complex process of reproducing a bug and setting up their development environment, and trying to get all their new debuggers ready and try to find that bug. They don't need any of those things, they simply get the link from the Jira ticket for the bug and they go and they look at the result of the failing test and with very rich information made available by mabl, for example, the DOM snapshots of the complete HTML of the page, or all the network traffic and HAR files, or the complete chrome trace, which shows you everything like even processor level things happening in Chrome, you've got the screenshots, you've got the console walks in the browser, with all the information made available. It's literally as if, as an engineer or a coder, I'm right there at that computer when the user experiences the issue. You can actually just look at that and make the fix right there. But I really think that's a really powerful place that coders can get to be a participant in quality just by using mabl.
Finally, I want to touch on the third group of people, business stakeholders, and disclaimer of the patterns I'm talking about here, quality engineers, and coders and stakeholders, this is a way that we've seen this be done at mabl by smart customers, but everybody does it differently. So this is just a suggestion. But the way we've seen business stakeholders integrate, are that new business analysts and product managers, and other management groups, they can also see mabl, they can just walk in and they can easily you know, see the results of tests, they can see the coverage reports, and they also become participants in our quality play.
Even they can make simple no-code test updates, for example, a string changed on a button, in mabl, you can go in change that one thing hit save in the middle UI, and now even a non-technical contributor has been able to help bring quality to bear, they've been empowered and that actually frees our quality engineering group to focus on higher levels, objective quality tasks, rather than changing strings on buttons, they can actually focus on making even more complex tests or going and doing exploratory testing. The takeaway here though, remember is that everybody can contribute when you use a tool like mabl, and you got the continuous testing set up throughout your pipelines.
Let me just give you an example of what that can look like. Here we have a preview, and I'm sorry, it's a spoiler for Lauren, she's in the next talk of a release coverage report. This is an example I wanted to bring up getting that complete picture of quality, it's not just red, green, it's, for example, we have a percentage here, it's a continuous value, how well is the test coverage of my user paths and the application.
here is another example, here from the release coverage, again, you can see that on the left, we have that binary state. Yes, no, passed, fail. But we also get a continuous curve here, right, we get these percentages, how well are things trending? The takeaway I have here, as far as the actors, and the who goes is now anybody can come and see these reports in your organization, and help make those decisions about what do we need to fix as engineers? Or where do we have to shift our quality efforts, everybody can see if that big release is ready or not and it really is empowered all actors to contribute.
Just to give you another example of how it also empowers folks, is it lets everybody see quality ubiquitously through tools, like our notifications enable, so it's something you really should turn on so that the who have continuous testing is everybody and in this case, I just stole something from some of my own testing apps, we can see that on my blog, some broken links were found, wow, it's a bunch of them, I really need to go and take a look what's going on there. This was sent to me in Slack automatically, these insights are dropped directly into Slack for me.
There are other ways to get these notifications as well. For example, an email. If you turn on reports, for a given workspace or given plan, you can get your rich email reports. In this case, it's showing me my demo workspace, and how many tests passed, and how many failed. Unfortunately, it looks like the Flappy test is a little flappy right now, so I need to take a look at that. But again, the takeaway is that with these notifications, you can further bring everybody into the fold, so everybody can see what quality looks like in your application.
Finally, as I mentioned, at the beginning, we're kind of cheating, there's five W's, and an H, the how. The how, is the rich suite of tools we give you in the middle tool belt. I'm just going to take a moment to kind of touch on each one of these because if you're not using these, I really hope that you start, there are a lot of resources available at help.mabl.com and mabl University that will get you set up in no time to really leverage these resources.
So on the left hand, we've got local testing. We talked on this a little bit earlier. But the mabl CLI is an amazing resource, if you want to integrate it any way you can conceive of basically, you can hook up the mabl CLI, whether you're an engineer typing in your laptop, or if you're putting a deep into your CI/CD workflows, we've seen many customers use mabl in ways we never conceived of, by using the CLI. Of course, the desktop app is an amazing tool, as you saw earlier when I show new running headless. Those tests, pointing them at a web app, pointing them at my localhost server. All I had to do is make one quick, if you ever used other tools, you have to install tons of components, get them all configured just right. Get everything running just right and then you can test, here with the desktop app, everything is really mabl easy point and quick. Definitely, something you should be using if you're not already.
The Jenkins integration. If you're somebody that likes to write unit tests for react or any node js or JavaScript applications, this again is a way that you can just run your tests as unit tests using mabl’s headless engine on the background is basically all the CLI goodness. Right there in your unit tests. It's really an awesome tool. Getting into that CI/CD stage of the pipeline again, Jenkins, I'm sure everybody here has heard of Jenkins. Probably one of the most used CI/CD tools. They kind of invented CI/CD servers. mabl has a first-class Jenkins integration. If you've got Jenkins, go to the Jenkins store. Click Install and you've got the mabl build step and you can directly put it into any pipeline you want there. This is going to let you trigger those tests and create those deployments we spoke about earlier that nexus of quality where you're going to be able to see everything that happened about a deployment, track the coverage, all sorts of things like that. Just go into the Jenkins store point quickly.
As for DevOps, we've also got an integration there somewhere point quick go and integrate mabl as your DevOps pipelines. Octopus Deploy. Big shout out to one of our employees who loves Octopus Deploy and brought this to life. If you use Octopus Deploy, it’s a piece of cake. You just incorporate that orb and you're off to the races. Bamboo, another popular enterprise tool, we've got a mechanism to connect directly to Bamboo. Similar to CircleCI, all these first-class entities. I'm just going to note down here, the REST API for mabl, if you feel really comfortable with you know, making API calls, you can go straight to the source from the horse's mouth and you can call the REST API. If you just go into the mabl app, actually, and you go to Settings an API, there's a rich tool in there, that'll show you all the commands you need to use the REST API directly.
Source Control, we integrate with all these, all the major players, GitHub, GitLab, BitBucket and if you just go and search mabl, any of these major brand names here, you'll see our integrations with all of them. The richest of which is probably GitHub, we showed you here in the talk today, seeing those rich results directly under commits directly in your pull requests. So this is your tool belt. Most of these tools are point and click. So it's really easy to start dropping these pieces in. I know that CI/CD is bespoke, so nobody on this call has an identical setup. But with a strong set of tools in your tool belt, everybody should be able to easily integrate mabl and get that continuous testing harnessed in their pipelines.
That in a nutshell is setting up continuous testing in your CI/CD pipelines with mabl most of our customers after not being customers too long, or asking us all the details of how to jump into this. I hope that you guys found this talk helpful. As always, you can find us on friends of mabl and you know, ask us any questions you might have. You'll find me there @Joe. Feel free to email me as well if you've got any questions, Joe at mabl. With that, I would love to take your questions about continuous testing and your CI/CD pipelines.
Zane Halstead
Awesome. Thank you so much, Joe. That was great. First one that came in is: do I have to be a mabl admin to configure CI or integration such as CircleCI?
Joe Lust
Great question, so you obviously, you're going to have whatever permissions on your pipelines that your organization has set up that your operations group have, with mabl, most integrations use our API keys, you'll see those if you go into the settings in API panel. So I believe its editors can see these keys. If there is a key, you can use it. But in some cases, if you don't have any keys and administrator in your mabl account, one of the people in your workspace would have to create that key. But again, it really depends on what your operational group is like, and so forth.
Zane Halstead
Great. Do you find a stakeholder sometimes needs convincing to participate in continuous testing and a follow-up to that? What if the stakeholder does not feel comfortable looking at the test? How would you entice them to look at it?
Joe Lust
Good question. I guess we've generally found that folks really like to be able to see brass tacks, show me the money. So I think that with mabl, it's so simple to see this information historical, I use things like, many years ago, Jenkins, and sure, if I took a stakeholder and told him to go look at a million wines and log output on a Jenkins UI, it's probably really daunting. It's not really user friendly to somebody that isn't, you know, a Jenkins, Maven, but I think with mabl, when you can just give somebody a link and say, here's the output, you can see that test, if that's what you want to see. Now, if you're a product manager, and you're like, why are we allocating resources to fixing this bug, it becomes really simple for them to just click a link and see what's happening. So I don't really think there's a lot of trouble getting like stakeholders to look at this information. I think there is additional high-level reporting, showing high-level values, not just red or green, but those percentages, those coverage reports, the performance reports, I think that also can help managers and other stakeholders in their budgeting and, you know, resource allocation decisions. So I think that is oftentimes the kind of rich information those individuals like to take you straight from the horse's mouth.
Zane Halstead
Wonderful. Next one is: we are only testing pre-prod right now. But we want to expand where we're testing. What's the next best step for my team?
Joe Lust
Sure, great question. So, we mentioned kind of four stages in that pipeline, we mentioned, the code changes the pull request, review stage, that deployment to pre-prod, and then production, we've found that when you can kind of again, move left shift to left the discovery to that review stage, or that local stage, it really is going to help you, flush out those bugs sooner. So, there are tools available now, you know, for example, like preview environments, or review apps, there are some words for it, but when you deploy some code, just to a simple environment, so you can test it at that earlier stage, that that really helps folks, we do that here in mabl. When you deploy something, a UI change, a little bot pops up in the pull request and says go here to watch it. If you've ever had to go through and look at CSS changes and try to figure out what they mean, well, now you can just click a link and go and see it at an earlier stage than pre-prod. So I'd really encourage folks to think about preview environments. We've actually given a number of talks and mabl have some documentation around that. But even before that, that's too daunting, running locally is even easier. That's where I think things like the desktop app, lets you, go all the way left and shift as close to the source of the bug as possible. So preview environments, and that local testing, I think are a great thing to investigate.
Zane Halstead
Awesome. Do you leverage maple capabilities for production monitoring?
Joe Lust
That's a great question. Yes, it's a little secret that mabl tests mabl, we have many environments and we run tons of tests against mabl. So we're very confident in mabl. I'm actually starting 10 minutes ago, a production on-call engineer. So I know that there's a bunch of tests watching mabl in production and if anything doesn't look exactly right, I'll get paged. So we use mabl on mabl in our monitoring, and we feel pretty confident about it.
Zane Halstead
Great, another one came in. How can I get access to the beta runner?
Joe Lust
So we've got our CI/CD runner, we kind of mentioned that earlier there. I believe you can just reach out to your CSM. If you don't know who that is, you can also just kind of reach out to support. That little intercom pop up in the desktop app for the web app, you can ask folks there, and I believe if you just Google for it, there is a signup form as well. So there are many ways you can get access to start using that feature, which lets you run mabl headless tests directly inside of your CI/CD pipelines.
Zane Halstead
How do SREs fit into continuous testing?
Joe Lust
Sure, SRE so we mentioned for folks that are familiar with that term, Site Reliability Engineer. There are some folks who are really focused on keeping things up and looking great all the time for your SaaS applications. I think a great place for SREs is in part that, continuous testing and production aspect, now that you can kind of use mabl as your monitoring tool, so an SRE can set up various tests to monitor for that performance or that coverage and make sure that those real-life end to end flows that a user sees continue to function correctly in prod. Oops, I'm getting into the details here. If you do some simple synthetic tests, they might just make sure your website returns anything. But that's not going to tell you that the page is down showing 404. So that's where those rich end-to-end tests, make sure a user can get to your shopping cart can open it, put an item in and check out, give you that confidence that everything is really working correctly. That's the sort of confidence that an SRE is looking for.
