We’re living through an unprecedented transformation in the way software is built, delivered, and consumed. The shift to continuous delivery enables industry leaders to innovate and react to evolving market conditions almost instantaneously. However, achieving delightful and differentiated digital customer experiences while innovating faster requires us to embrace a holistic approach to quality. In this keynote, we’ll share our vision for elevating the impact of testing within this new reality, laying out a roadmap with real-world examples to illustrate how teams are improving quality and velocity with quality engineering, and we’ll give you a glimpse into the future with mabl.
Mary Kate Morrison
Hello, everyone and welcome to our third mabl Experience conference. Many of you may know that Experience is our annual event that brings together the Friends of mabl, and the broader quality community all to discuss the adoption of quality engineering and user-centric software development. I'm Mary Kate Morrison, I head up our Customer Success team at mabl, and we're thrilled to have so many of you with us for this year's event. We'd love to hear where you're coming from and what you're looking forward to most. So feel free to say hello and share where you're joining from today in the session chat.
Our theme this year is all about elevating testing. As you'll hear throughout our keynote today, we're seeing an extraordinary transformation in the way software is built, tested and released continuously. And organizations are really innovating rapidly to meet these demands. We believe quality is at the center of these trends, and to fully adopt DevOps or make progress on digital transformation, testing and quality need to be your priority. You'll have a chance to hear from a number of mabl customers about how they're elevating quality, and implementing quality engineering best practices within their organizations over the course of the next couple of days.
It's going to be a really fun and jam-packed event. We have over 1000 folks attending, 20 different content sessions, and 26 total speakers. I do want to take a moment to thank everyone at mabl who has worked so hard to put this event together, as well as our speakers for all the time and effort that went into preparing for their content sessions.
Here's a quick snapshot of a few of our various customers and partners who you'll hear from over the next two days: From growing test coverage, integrating testing into automated pipelines, building a culture of quality, to implementing non-functional test strategies, there's a session for everyone across both days of Experience.
Another thing we want to mention as we're kicking things off today is that we have a very diverse audience, which is wonderful to see. We've got about 70% of attendees in quality assurance or quality engineering roles, about 10% in developer or engineering roles, and the remaining 20% in other roles such as DevOps, Enterprise Architecture, Product Management, etc. We also have a great mix of about 70% of attendees currently working in individual contributor or practitioner roles, and about 30% in leadership or management positions. So we're so happy to have folks coming from different backgrounds and perspectives.
For a full rundown of our Day One agenda after our keynote this morning, we'll have a fireside chat with Vince Esquilin at JetBlue. Next we'll take a quick break where everyone can enjoy our virtual piano bar. After that, you'll have a chance to hear from Stacy Kirk from QualityWorks, Janet Bracewell from SmugMug, and Madhura Kamat from iCIMS for an engaging panel on leading connected quality teams. Following that, we have our very own Darrel Farris presenting on the journey to quality engineering adoption. We'll have one more short break this afternoon and wrap up today with a final keynote from Debbie Levitt at Delta CX discussing customer’s definitions of "quality" and "done."
We'll be back tomorrow with two ways to participate: virtually there's a number of great customer spotlights sessions, panels, and more. Additionally, we look forward to seeing a number of you in-person tomorrow at the MGM Music Hall with a great day of customer sessions, discussions, and networking as well.
We're so delighted that you've chosen to join us at Experience and whether you're looking to elevate your career, your QA strategy, or the role of QA in your organization, we hope that everyone comes away with new knowledge, perspectives, and connections. So without further ado, I'm delighted to introduce our first session, today's keynote and welcome Dan and Gev to the stage.
Dan Belcher
Thank you very much, Mary Kate. I'm very excited to be here. Thank you everyone for joining. I'm Dan Belcher, I'm one of the founders here at mabl. I've been telling everyone that this is one of my favorite holidays now because the very first mabl Experience two years ago was just as we were emerging from COVID lock downs, and it was the first time that many of us within mabl had seen one another. And it was really the first time ever that we had gotten so many of the passionate people in the quality engineering community together. So I'm excited to be here, I think it'd be a great event. And I'm also really looking forward to meeting many of you in person who have made the trip to Boston for Day Two tomorrow. I'm also joined by my colleague Gev.
Gev Hovsepyan
Hi everyone, super, super excited to be here. It's really a special day for me. I joined this community exactly a year ago, right after Experience 2021. And today, I couldn't be more humbled to have the opportunity to be part of this opening keynote. We have a lot to learn, to share, to celebrate during this event. And I can't wait to engage with you all, and continue transforming software equality around the world together.
Dan Belcher
Thank you Gev. In this opening session, our goal is to introduce the themes that we'll be exploring in greater detail throughout today and tomorrow. The first theme is really that quality needs to be at the center of these incredible transformation initiatives that we're undergoing across many, many industries, right, that's DevOps transformation, digital transformation, and technology transformation. And we'll talk about how you can elevate quality to help your organizations meet their goals in each of those areas. We'll also hear from other teams and other companies who have been successful in rolling out quality engineering as a way of improving quality and customer experience. And in this session, we'll be excited to hear from Stack Overflow about their quality journey. Next, we'll talk about the investments that we've made over the past year, and continue to make to support you in your quality transformation.
But first, I want to acknowledge the incredible momentum that we've had within this community over the past year. It was just after mabl Experience last year that we announced that mabl had secured $40 million in additional funding. Now, of course, that's important to us as a company. But I also think it's significant to this community for two reasons. First, it gives us more resources to invest in supporting you, whether that's with technology, or people, or content, or training - and you'll hear a lot more about that today. Secondly, I think it's really important validation that a broader group of stakeholders and investors in the industry are recognising the central role that quality plays in the future of software engineering. And so it's a great validation of that.
From a momentum perspective, it's also been a great year in terms of hearing more of you tell your stories within the community and share the incredible results that you're delivering for your teams. And with that momentum comes scale, right, so there are 1000s of you using mabl on a monthly basis now. We're executing millions of tests in our cloud for you. We're collecting billions of test artifacts, processing those and generating insights around quality that help you support your teams. Finally, we're doing this globally with users from this community in more than 60 countries. And for our part at mabl, we've also more than doubled our team since last year's mabl Experience. We now have over 100 mablers all around the world supporting and engaging with you.
And around the world is very important because increasingly, our user base is distributed globally. So we have fewer than 50% of our users here in the United States. We have growing communities in Europe, India, Japan, and beyond. And we actually expect the proportion of users outside the United States to increase over time.
One of the great highlights for me this year has actually been getting back on the road to meet with many of you all over the world, to hear your stories about how you're integrating quality engineering into your work. This photo is from an event that my co-founder Izzy and I attended in Japan, where many people were speaking about the incredible results that they're delivering by elevating testing within their companies and in their teams. So we look forward to doing much more of that in the years to come.
The other key thing from a momentum perspective is seeing the ecosystem that's expanding around quality engineering. QE lives in a broader software delivery pipeline, and we have a set of great partners who are providing capabilities as part of that pipeline, whether that's GitHub for source and CI, or Segment to help us match test coverage with usage, or Atlassian supporting issue management with JIRA. We also have incredible resellers around the world who are helping more organizations evaluate, purchase, and deploy our unified platform for quality engineering. This year we were also invited to participate in the Google Cloud Marketplace. So now Google Cloud customers can purchase mabl as part of their Google Cloud contract, which is very significant because Google and its partners are leading many application modernization initiatives. And so this gives us more validation that quality engineering needs to be part of application modernization.
So we're seeing tremendous momentum with this community, and we're very thankful to be here today. So where is this momentum coming from? Of course, it's coming from results, but there are also external forces that are placing more and more emphasis on software quality. And here, we'll talk about the role that software quality needs to play in digital transformation, in DevOps transformation, and in technology transformation.
With digital, I'd ask you to think about, just since the start of the pandemic, how many new digital experiences you have. How many new things you can do online, that you had to go into a branch or an office or fill out paperwork, or pick up the phone and talk to a representative for only three years ago. Whether that's opening new bank accounts, or purchasing new products, or you know, opening new policies, or the like, there are so many new capabilities that digital transformation has brought forward at an accelerated pace since the start of the pandemic. And then I'd ask you to think about now how important it is for the companies that are offering those new capabilities that they're delivered with high quality and that they're reliable. Those digital experiences have become core to those businesses! So I hope it's evident how important it is that we manage the quality of those digital experiences.
And when we think about quality, the bar for quality is also going up. Right? So we have these new digital experiences. This year, the Department of Justice in the United States reminded us of how important it is that we account for people with disabilities and accessibility, when we introduce and manage those digital experiences. And they backed that up with lawsuits and settlements with companies who are not meeting the standards for digital accessibility. And so when we think about quality, we need to think not only about the functional quality of these digital experiences, but also ensuring that they're accessible, both from a business risk perspective and also because it's the right thing to do.
Beyond risk, we're also looking at revenue, right? So this year, for the first time, while Google has been telling us for years, that it's critically important that we manage page load time as a key aspect of customer experience. This year, for the first time, they incorporated page load time metrics into their search ranking algorithms. And what that means is that if you rely on Google search results for your customers to find your company, or your customers to find your product, that you'll have lower results if you have slower speed and higher results if you have faster speed. And so managing performance becomes vital for anyone who relies on search results for their customers to find them in their products.
So that's a digital experience and I hope it's evident that quality plays such a critical role in enabling these digital transformation initiatives. The second area where we see quality playing just as an important role is in DevOps transformation. Today, we see that the majority of organizations are on the DevOps journey. It's clear that in order to embrace DevOps practices where you have continuous integration and continuous delivery, you have to have a continuous view on quality. And that requires elevated automated testing.
It was exciting this year to see more concrete results published that helps solidify the idea that higher performing software teams embrace DevOps practices. They're much more likely than low performing teams to use version control, to use continuous integration, to use continuous delivery, and so forth. It was also exciting this year, from an enterprise perspective, to see us coalescing as an industry around a standard set of metrics that we can use to measure our software delivery performance. That's lead time for changes, mean time to resolution, deployment frequency, and change failure rate. Quality plays a vital role in having as low as possible lead time for changes (that's the time from when you have an idea to when the product is in customers hands), mean time to resolution of issues (so we have confidence that we can deploy fixes without doing manual testing), deployment frequency (so we need to have continuous testing to make sure that we can deploy continuously without failures), and then change failure rates (so reducing the number of times that we have to roll back a change because we've introduced some defect). And so I think this is the type of thing that helps enterprises gain confidence in their deployment of DevOps, and we can see many of our customers are on that journey where they're building intelligent pipelines to deliver software to their users continuously.
The third pillar is technology transformation. We see accelerated adoption of cloud computing, whether that's for new applications and digital experiences, or teams that are migrating from on premise to the cloud. This is evidenced, of course, by the incredible growth in revenue for Google Cloud, for Amazon Web Services, for Microsoft Azure, combined doing well over $100 billion in cloud revenue on an annual basis. Now, the critical thing here is how do we manage quality in that dynamic cloud world. So on one hand, when we're migrating we have to have control of quality before, during, and after the migration. And then on an ongoing basis, we have to be sure that now that we have much more dynamic infrastructure, that we have a level of test coverage to ensure that there aren't unexpected issues as a result of that dynamic nature.
Beyond the cloud, another technology transformation that's happening is we're becoming much more programmatically connected. On one hand, many of you are offering new capabilities to your customers and partners that are based on APIs. And so if you think about the number of other vendors or partners that are integrated with you via API, that's increasing exponentially. We're also integrating API-based services into our applications: so if you think about how you are sending email, how you are billing your customers, how you are authenticating your users, so much of that happens now via API. And again, the importance of quality is how do we ensure that those third-party APIs that we're integrating into our product that we don't control, that are updated on an unpredictable basis as far as we're concerned, how do we ensure that we know whether there are any changes that affect the quality of our applications? And so again, it becomes much more important to place quality front and center for those integrated services.
And so we view quality and quality engineering in particular, as a key enabler in ensuring success for digital transformation initiatives, for DevOps transformation initiatives, and for technology transformation initiatives. So the question is, how can we embrace quality engineering practices to see that level of success?
And really, what we're focused on is driving better business outcomes. And so that transition we see as really fundamentally changing from a quality assurance mindset to a quality engineering mindset. There are four pillars that we think about there. The first is: how can we build quality throughout the entire software development lifecycle or software delivery pipeline - that means that we're thinking about quality from when the first line of code is written on the developers desktop, to when the code is built to continuous integration environments to when it's deployed to a staging environments to when it's in production and customers hands, how do we ensure that we have a continuous view of quality throughout that entire lifecycle? In order to do that, we also talk a lot about democratizing testing, because there are many stakeholders involved in that end to end process, and we need to be sure that all of those stakeholders can participate in quality and participate in automation. The second pillar is ensuring quality across that entire digital surface area that we talked about earlier, whether it's your user accessing your web application from a browser, or that user accessing that web application from a mobile device, or any browser, or accessing the APIs that you're offering, or downloading a statement from the application in PDF and ensuring quality there, to the emails that your application uses to communicate with your users, we need to ensure quality across the entire experience. Next, we need to move beyond strictly functional verification, to validating quality for non-functional attributes like performance, accessibility, user experience, and so forth. And then finally, because we're talking about quality engineering, we have to measure and use data to improve our work. This doesn't happen overnight, so the key is having refined processes, measuring your efficacy, and using the data to affect how you improve your testing and your processes moving forward. So there's this idea of data-driven, continuous improvement within the team.
Now the teams that can do this, they really see unprecedented performance. First, when they embrace quality engineering practices, they can accelerate time to market and improve their agility. That means faster regression cycles, they can deploy more, they can have shorter lead time for changes and lower failure rates in production that can slow down the pipeline. So speed is the first benefit. The second benefit, and just as important, is customer experience and customer satisfaction. So by employing quality engineering practices, they can have fewer defects that their users find, they can have faster page load, faster API response times, fewer accessibility issues, and ultimately deliver a better user experience to their customers.
And the value of this is very clear. In the mabl Testing in DevOps Survey for 2022, we saw very clearly that there was a major difference in how you view quality assurance and how satisfied your customers are. You're twice as likely to have satisfied customers if you view quality assurance as a critical asset for your company. And really, I can think of very few organizations that value customer experience quite as highly as Stack Overflow. And with that, it is my pleasure to welcome Dale Cooke, Head of Product Engineering from Stack Overflow to talk about their quality journey.
Dale Cook
Thank you, Dan. Great to be here. And thanks to the whole mabl team for letting me share our experience with you today. As Dan said, I am Dale Cook. I'm the VP of Product Engineering at Stack Overflow. And I'm sure pretty much everyone here knows what Stack Overflow is, but perhaps less well known is our SaaS offering, which is called stack of Stack Overflow for Teams. So this version of Stack allows you to bring the power of Stack Overflow inside your own organization and access a rich set of tools for content creation, discovery, moderation, curation, etc.
When I first joined stack overflow in February of 2021, we were making Stack Overflow for Teams, one of our big priorities for growth. But we had a couple of challenges. At the time, we were relying very heavily on manual blackbox testing for Stack Overflow for Teams. And while this testing was comprehensive, it was somewhat slow and cumbersome - and I think we all know sort of the challenges with using manual testing to do these kinds of things. So we recognized fairly quickly that what we needed was an automated, comprehensive testing suite to put in place to replace the manual blackbox efforts.
So, working with our head of QA, we put together this sort of laundry list of things that we wanted not necessarily to solve immediate problems, but also thinking in the future as well what we wanted, you know, how we want things to work going forward. I've done this kind of process before these transformations, and I thought I knew the marketplace reasonably well. And we looked at about a dozen offerings from different companies, but there was really only one that hit every one of those items on the list, and that was mabl. I was surprised, I didn't think we'd get everything, but mabl offered them. The thing that really sold me was the amount of effort that the mabl team put into making sure that we got what we needed, not just sort of selling us things, but listening to what we wanted to do and helping us understand what was on their roadmap - it was really a phenomenal experience to get on board in this. So that was sort of, you know, an awesome thing and really, really critical because we weren't just looking for software, we were looking for a partner that we could grow with over time, we had other aspirations as well, we just didn't want a piece of software, we wanted a partner and we definitely got that with mabl.
So that was about 12 months ago. And over the last 12 months, mabl has really become a cornerstone of our QA experience. You can see some of the stats here, but it's really transformed how we think about QA and what we're thinking about going forward. One of the things that's really important to call out here - and often doesn't get included in these things - is just team satisfaction. The teams love working with mabl, it's a joy to use, and it makes things a lot simpler. And that's really key because ultimately, if the teams aren't happy with it, then none of the other stuff really makes that much difference. So with our partnership with mabl and with the mabl team helping us understand things and helping us solve complex problems, it's been an absolute joy to work with them. And it has fundamentally changed the way we look at quality. So where do we go from here?
So immediately, one of the things we're just about to roll out now is - originally the idea was using mabl just for Stack Overflow for Teams. But it's been so successful there that we're actually rolling it out into our other offerings: StackOverflow.com and our Stack Exchange sites as well. We had planned to do that at some point in the future, but we've moved that up significantly just because it's such a good tool to work with. We would be wasting the opportunity if we didn't move ahead with that straightaway. And then even thinking beyond that, one of the things that we're very interested in this democratization of testing and getting away from the idea that testing and QA is purely an engineering function, and thinking about how we can push this further out into other parts of the organization with the product or sales or marketing or whatever. So that's actually something that we're really interested in and that's one of the things we're going to be tackling over the next sort of 12 months. You can imagine that, instead of product writing up an acceptance test in a ticket somewhere, they can write it in mabl and then it just gets run automatically and no one has to come back and manually check these things which just won’t happen. So we're very excited about that, and more so more than anything we're excited to continue this relationship - there's a real partnership we have here ongoing. I love talking with the team and saying, “Hey can we do this?” and the answer is always “yes, we can”. Right? That's really amazing and I really love the relationship that we have. So again, thank you for this opportunity to share that with you. And Dan, I'll hand it back to you.
Dan Belcher
Well, thank you again for sharing. And it's really been such an honor to support you and your team over the past couple of years. We're looking forward to continued partnership in the years ahead. And if, as the community here, if it's not clear how the idea of support is very core to mabl, it's a core value for us - we talk about it day in and day out internally and in our work with our clients and the community. And it's in that spirit that I'm also really excited to invite Gev to talk more about the investments that we've made and continue to make to support the quality engineering community.
Gev Hovsepyan
Thank you so much, Dan! As I mentioned at the beginning I’ve been part of mabl for a year, and let me tell you: I wouldn't be where I am now without the support of my fellow mablers and this fantastic community. So let's continue the thread of support, shall we?
So, today, we all operate in a world of constant and unprecedented change, from evolving customer expectations to various transformational efforts that enable us to meet those expectations. As quality leaders, our role is to elevate quality as a cornerstone of customer experience, and ensure that we all have a high-quality digital world at the end of these transformations. And we are here to support you in that endeavor.
As Dan mentioned at the beginning, we all are elevating quality at scale, with 1000s of monthly active users, millions of monthly test runs, and billions of artifacts. And all these require massive investment to support your growth. Now you have more than 100 people supporting you day-in and day-out with 98% resolution satisfaction, and under three minute response time, 24 hours a day.
But we didn't stop there. We brought to you new technical account managers to help you accelerate your quality engineering progression. We also made significant investments to support our customers globally, from language support to a big team over in Japan.
And today, we're excited to expand our portfolio of tools supporting you with two major announcements: as an addition to our existing world class university, we are super excited to announce a new mabl certification program with availability of both foundational and advanced skilsl assessment. This newly created program enables you to expand your knowledge of quality engineering, and demonstrate proficiency in using mabl through free self-paced learning.
But why stop there? Why not push this beyond the expected? Let me introduce you to Careers in QA, an industry specific job board that connects software quality professionals with employers actively hiring. The demand for skilled testing professionals is growing rapidly, and with Careers in QA you will have the stage to explore opportunities outside of your organization where you can continue to elevate your impact.
But we didn't stop there either. We delivered over 100 new product releases with innovations that enabled you to continue progressing in your quality engineering maturity journey from achieving high functional test coverage all the way to continuous improvements to quality metrics. So now let's take a look at a few product highlights from 2022 to support you in achieving high functional test coverage and ensuring high quality regardless of what browser your customers use to access your service.
- We delivered support for Microsoft Edge.
- Similarly, support for Shadow DOM enabled us to expand our platform to solutions where component encapsulation is a must. Great examples of these are simple things like chatbots, or complex solutions such as Salesforce.
- But achieving high coverage at scale requires a solution that is maintainable at scale. To continue to reduce the test maintenance burden, we brought to you more intelligence when it comes to building effective tests with our intelligent waits as well as dozens of API enhancements.
- From more Postman compatibility to more effective debugging with enhanced console logs from static IPs to file upload support, we brought to you innovation that enables you to apply the same engineering rigor to test automation with branching conflict resolution and test review.
- And last but certainly not least, to meet customer's expectation of quality, functional testing is not enough: you need a solution that brings functional and non-functional testing together as a unified experience. This year we brought to you more non-functional testing focused on two key domains: Accessibility checks and reporting that provide you the right tools to ensure your services are accessible to anyone. And enhancements to performance reporting with trend reports for your application and APIs,to provide you with visibility on how your releases impact your customer experience.
And let me tell you, we did all of this together! I'm sure you see that there isn't an hour that goes by without ideas and opportunities you share with us. From regular product ideation sessions to friends of mabl, you name it. And I can't thank you enough for your support and drive to make our product, company, and community better every day.
But we're not done yet. I truly believe that the great days are ahead of us full of opportunities for more innovation. As many of you know, this year, we launched a new product portal where you share new ideas and contribute to ideas that others share in the community. And the energy there is infectious. And your voices are being heard there: from database access to message flow, from native mobile to performance testing, the opportunities are endless.
And actually talking about performance testing, I heard that one of our fearless leaders from the technology team is about to execute a crazy API load test with 100,000 requests. Hey, Joe, should we all buckle up now?
Joe Lust
Thanks a lot. Hi, I'm Joe Lust, an engineer here at mabl who focuses on our infrastructure, and a watchword of what you’ve heard a lot today: Scale! That's deploying infrastructure and applications at scale, and of course testing at scale. Just like we heard Dan mentioning earlier, the millions of tests and the billions of test results we routinely are parsing.
But speaking of innovation, let me talk about how we are helping our large enterprise customers elevate their quality at scale. We are working to help everybody test at scale, which means many things, but to us, it means looking at many use cases, not just testing one or two paths to an application, but testing the hundreds or 1000s of paths through a large enterprise application.
And not just running those tests with one or two users, but running with hundreds of users to represent the varied and diverse composition of individuals that makes up the actual user base of a large enterprise app. And all the while, we need to make sure that we are seeing a positive and quality user experience while under such load.
One tool in your toolbox when testing at scale is load testing. Load testing helps us find those limits to the application, push it until we hear it creak and squeak. We do so using real world workloads, not just running one test, but running that myriad test case bundle we mentioned earlier with the numerous and varied users so we can replicate what real users pulling an application looks like. And all the while, we need to be collecting rich metrics so that we can validate your compliance with service level agreements for things like latency and timeouts. But more importantly, we need to validate that the user experience is quality for the end user.
For example, you might have seen before some screens and apps you use like Reddit, and their cartoons that try to paper over the fact that they keep typing out, or the many pop ups we see in our applications day to day about unreachable servers or web pages that won't load. or in one case, as you've probably seen before, the Twitter Fail Whale: the image they used to put up whenever their site crashed, because it crashed so often.
With load testing, we want to ensure that your worst case scenario is still a positive user experience so that your most prominent UI element is not actually your most infamous UI element. So let me show you an example of what this would look like.
When testing at scale, in our example, we're going to launch many mabl API tests from the mabl cloud against an API under test that I've set up also a Google Cloud. And we'll see what happens when we try to pull it with traffic. In this case, we're going to try to send 100,000 requests in a little over a minute. So let me show you that example.
Will jump over here and look at the mabl CLI. So here we see the mabl CLI, one of my favorite tools, we can create and launch very large scale tests. In this case, I've just asked it to create a large deployment. And this deployment is going to run against an application. And we can even use the CLI to see what that application is. So let's do a quick look up here: that application happens to be named API under test, very creative! And on the right hand side, we're going to start looking at the server logs to see all the traffic from the application under test as the load comes in.
So what tests are we running here? Well, we're running all the tests with the label “pretty big plan”. So let's go look those up, something we can also do in the CLI, and we can see here: we've got a number of plans, we're going to run the whole lot of tests, bunch of tests, a lot of Wowza and stifling request tests, you get the picture: we're sending a bunch of requests to this app. On the right hand side, we can see the traffic is already pummeling the application, each one of those lines is a request going through from the app side. And now it's really picking up, we're gonna step on the gas for a second and just push this example a little faster so we can skip to the end, just about 90 seconds in the future. And there we are, there are the results. We ran a whole bunch of examples of mega API tests, and every one of them completed successfully. That's what it looks like from the mabl CLI standpoint.
Let's jump into Google Cloud and look at the application itself and see what it saw. Here we're looking at the logs from the application under test. And we can see there was a big spike in requests here. So I want to zoom in on that and look at the shape of the load. Load shape is important: it needs to replicate a ramp-up and ramp-down like real users hitting your application would. And we can see we got just that: a nice ramp-up and a nice ramp-down, and almost all those requests happened in one minute. That's a bunch of load.
And on the left hand side, we can see 100,000 requests came flying in that short time. And they actually all executed pretty quickly. And we see on the right 200: that means success. Okay. This is what it looked like on the application side.
Let's jump into mabl and see what you would see. Well, here we see our deployment: and we hit the application under test with 1000 tests. That's a lot of tests. Let’s drill in and see what we got in the results. Well, we see all our plans ran, every one was successful. Okay, things are looking quality. But we can go deeper, we can double click go into each one of these tests we ran. And from there, we can actually see every single request we sent. That's why we have 100,000 records here for each test. And we have tons of detailed results for each one, from the response to the headers to the body. Of course, we have the basic stats, you know, we have the status, the 200 success, we have the timing, how long did it take. And then we also have more detailed cross test analysis. So here, for example, we're loading the complete history of this test. And we can see the purple line at the bottom showing that we generally have pretty good average execution, but there are some outliers may we might want to click on and dig into. Overall though, this is 1 million results that came out of this test the mabl cloud in just about a minute. That's a lot of scale!
Alright, so that concludes our example here. Let me explain what we just saw. So to summarize, what we saw was 100,000 requests made to the application under test in a little over a minute. And mabl saved back to the mabl cloud a million results. That is scale!
Now, testing at scale that capability, it doesn't happen by accident. Trust me, I've worked really hard on this. It happens for mabl because mabl was designed from the ground up as a cloud native, highly scalable testing platform capable of elevating quality for everybody. Now, I don't know when Gev or Dan might plan to ship load testing as a product feature, but from my perspective, it looks like our architecture and our infrastructure are more than ready. Back to you, Dan.
Dan Belcher
Thank you very much, Joe. You know, it's good to know that you have our back and the rest of the tech team has our back from a scalability perspective. You know, I think it's also worth noting you can see in the stories between Joe and Dale and Gev and some of the examples that we provided earlier, you can see this continuum playing out.
We heard Dale talk about being able to dramatically expand test coverage because he was able to democratize testing, use a low code approach to get more people involved, and take advantage of the intelligence in mabl to get lower maintenance for that initial test coverage. And it's once you can get past that, that you can start to get into validating a lot of these non-functional requirements, that frankly, we've always wanted to do, I think, as an industry but have struggled to get there given the burden of using a legacy approaches to validating functional quality.
So thank you very much, Dale. And thank you all for joining this opening session. Just to recap, first, it was really a pleasure to reflect on what a great year it's been for this community, this community as you scale, as you demonstrate much greater results, as we see a broader ecosystem growing up around the community, and really looking forward to spending more time with this community over the course of 2023.
You also heard about the role of Quality Engineering and I hope it's clear now how, by embracing quality engineering practices, we can meet the growing demands set upon us by digital transformation, and DevOps transformation, and technology transformation.
I hope you can also see that we at mabl are here to support you, we have your back, and we'll continue to invest in people, in technology, and in content, really excited about the certification program to support you and elevating testing within your organizations.
What we'd ask in return is that you continue to be great and vocal partners and community members. And so please be open and sharing your feedback whether that's with us privately or publicly and our friends of mabl Slack community. We really value the benefit of this community kind of sharing openly and trading ideas and most importantly, sharing your results from transforming quality within your organization.
And personally, I'm very excited for the next session where Vince Esquilin will talk about how they've transformed quality at JetBlue. So thank you very much.
