We all know that continuous testing is key to shipping high-quality software. Between strategies such as shift-left and shift-right, how do teams prioritize the right strategies to help you achieve continuous testing? In this fireside chat with Mike Schober and Izzy Azeri, Mike will share the strategies to shift your testing, how to drive change across your development team, and how to overcome challenges faced along the way.
Transcription
Izzy Azeri
Thanks a lot for having me here and I'm joined by Mike Schober. Just as background. My name is Izzy Azeri one of the co-founders of mabl. I spent a lot of time with our enterprise clients, folks like Mike over at Schwab, in terms of how they increase efficiency and get better test coverage from a quality engineering perspective. I was actually amazed before diving into this call doing some research on Schwab, that they're the third-largest asset manager in the world. So congrats on that, Mike, and maybe you can just provide some background on yourself.
Mike Schober
Well, thanks, Izzy. I'm happy to be here. I actually lead a team called enterprise engineering at Schwab, it's a horizontal organization. One of the key initiatives that we're currently leading it's a multi-year effort to transform Schwab's software delivery pipeline, which includes all quality aspects as software moves from code checking to production, deployment, and activation. Quality will be an automated yet gating factor across the continuous delivery pipeline. I think that's a key thing that we're going to be automating. So teams are doing it in various fashions today. But this is intended to be an effort that will standardize for the most part, how teams function.
Izzy Azeri
That's great. Appreciate that background, Mike. I know lots of large enterprises are initiating projects like yours, that are driven by cloud migrations or SaaS migration has that been an impact on your organization, Schwab as well.
Mike Schober
It is actually, Schwab, as a technology organization is on a large-scale transformational journey, with the target goal of moving to the cloud for applications as well as SaaS-based tooling and vendor capabilities. So that's a big move for us. It's been in the works for a while, but I think there's a much bigger push now to make that change and especially in my area. The team that I'm on has a large on-prem footprint and we have a goal of moving out of the data center to a SaaS-based model over the next couple of years, we don't want to do it too fast because transformation can be disruption if you go too fast. But we are on that journey right now. That is a large part of the continuous delivery pipeline effort that I'm on to provide SaaS-based solutions for just about everything that we're providing to the enterprise.
Izzy Azeri
That's great. Integrated delivery pipelines are kind of a buzzword these days. What does that mean specifically for Schwab?
Mike Schober
Well, the strategy that we're working under it's simple, but it's to provide infrastructure, software capabilities to the development teams, but also have a standard set of tools that provide the capability to observe and measure the continuous delivery pipeline in an objective way, but also to integrate and automate and do it in a standardized way, we're not foolish enough to think that we'll get 100 percent standardization, but get it much more standardized than we are today. That's the journey that we're on, it's going to be a multi-year journey. We're currently working with pilot teams at this phase of the effort.
Izzy Azeri
That's great. I know oftentimes, when people are redefining their pipelines, they're thinking about what stages do I integrate automated testing into those pipelines? Is that something you're considering also? Have you thought at all about production as an area or a focus from a software test automation perspective too?
Mike Schober
Yes, absolutely. Automated testing should start at the point of code check-in and go through the continuous delivery pipeline into production. Some teams are doing it in different manners, but we want to enable through the continuous delivery pipeline, all teams to do it in a more consistent way. The production part is an interesting mindset change. I think teams, in general, get shift-left but can be challenged sometimes when you talk about shift-right and testing in production. You actually need to do both to create higher-quality deployments.
I've even seen organizational barriers. For example, teams think that they aren't allowed to test in production with automated testing tools. That's not the fact. You can and we do and there are actually things that can only be encountered in a production environment. In a company like Schwab, I like to say in the wild, but the key is to be able to fail fast and limit the blast radius of any changes to your clients. That's the key.
Izzy Azeri
That's a great point in terms of failing fast. I've seen some customers I talked to, there are the development teams and the automation teams doing testing in their workflow up until it gets to pre-production and then the DevOps team takes it over when it comes to production. What I've seen happen is as the developers start talking about this automated testing tool they're using, they start showing it off with their DevOps folks and the DevOps source like, wow, I want to try that as well. Because it's so much easier, perhaps, or, more kind of synonymous with what you guys are using the development side. So I want to use it on the operation side as well. Have you seen that pull it all going one way or another?
Mike Schober
Yeah, we have with the ops folks, but also, we've started to see it with the product owners - the democratization of testing with low-code software test automation tools. That's something that when I started on this journey, I wouldn't have thought about until we started using mabl and I realized how simple it is for any persona to use that capability. So we actually have product owners that are doing software test automation in production, building out full UAT test suites and then rerunning them and current releases, and then future releases. So that's been a big change, too.
But yes, I do think that the various personas around the production deployments are starting to share and leverage automation in production. We actually had a use case where one team that was able to work to test dark in production, which is a new concept that we're starting to have added to our use case for the continuous delivery pipeline, they actually found an issue in production, testing dark, fixed it a couple of days before the day that they were turning the features on, and we're able to have a quiet release with no issues. So that's another area that we're starting to expand the type of testing that we do in production.
Izzy Azeri
I'm sure the product owners and executives were happy. It was a quiet release and not a loud release when it comes to customer feedback there as well. Right?
Mike Schober
Yeah, that's a good point. I mean, the product owners and operations teams, when they see stuff like that they start to have more trust in deployments and trust in the team's automation, when they can see it run, but also, more importantly, when they can see an issue caught like that before the feature was turned on to the clients.
Izzy Azeri
How does an organization as large as Schwab - 32,000 people globally - how does an organization like that adapt to change where now you have POS kind of participating in testing, you have developers obviously involved you have price, very skilled automation engineer somewhere in the middle somewhere? How does that shift happen at a company as large as Schwab?
Mike Schober
It doesn't happen overnight, I can tell you that for sure. It takes time. Any change at a company like Schwab that has the scale, for example, the assets that you mentioned, the scale of assets, we have to be careful when we do production deployments and be as safe as possible. So that type of change does take time. That's part of the continuous delivery pipeline transformation that we're working through. That's going to be the change over time. It's a multi-year effort, that we'll basically roll out through the enterprise.
Izzy Azeri
For those of us listening, perhaps to this, and being at a similar size, organization, any tactics on how you democratized testing, or how to get other people involved that perhaps don't have the experience or the skill set?
Mike Schober
Yeah. I mean, I think there's, there's always going to be doubters that'll never change. It'll always evolve. So you can't go forward and say, hey, there's this target state that we're going to get to, and then that's the end state, because as we know, for those that have been in technology for even a short while, but then others that have been in for a while, like me, things continue to change, and you have to change with it. So that's part of our strategy is that we're going to do something, we're going to roll it up. But we're going to continue to take feedback and evolve over time.
But to get to how do you get people to make the change? I think the simplest way is to use the tell-show-do technique. So you tell them what you're going to do. Then you show them what you're going to do and then you do it with them. That's been the most effective way, especially when we first started working with software test automation. I told him what we were going to do and they were excited, but of course, they're like yeah, right. Then we showed them and then we got them hands-on to do it themselves and doing it themselves. So I think that's probably the most sound technique that I've seen.
Izzy Azeri
That makes a lot of sense. You know, I was on a webinar actually last week with CRO for Atlassian and I think he said they hired 700 People last quarter and reorg every 18 months. He said, the way that you're successful at Atlassian, is you have to just embrace change, embrace what's coming at you and, you know, get on board and be successful that way. So it feels like it's something similar at Schwab, where teams are embracing that change and the benefits that could bring them.
Mike Schober
Absolutely. They're seeing the benefits, or they're gaining the trust in the automation, there was another scenario where one of the teams that are using a low-code software test automation solution was testing in production and I was shadowing the deployment. They ran the automation with mabl, the product owner ran the automation in production, and then they were going to run some automation. They said no, we trust what we just saw, we're good, we can move to the next phase. So just to see that level of trust in the team, and the automation that was built was pretty amazing.
Izzy Azeri
So as you're moving testing into these integrated pipelines, and as you're shifting-left and shifting-right, what have been the impacts across either the business or the team that you've seen?
Mike Schober
There's an interesting use case that we had on my previous team when I was leading a delivery team. Traditionally, a lot of teams at Schwab do Selenium development for automation, that's just a kind of a traditional automation tool. But this one team brought forward mabl as a low code solution and I guess they did a pilot, but it was a live pilot in real testing and compared the two testing paradigms, I'll put it that way. What they found from a benefits impact perspective is they reduced the time to create a test by 85% and the developers, by the way, were the testers on this team, they were doing their own automation. So they were thrilled by that and so were the product owners.
Another positive impact is the meantime to resolution (MTTR) was reduced by 80 percent, using a low-code test automation solution. Then they were using a separate visual verification tool, and they were able to eliminate that separate integration of another tool in their toolchain. Then this was really the kicker as the experience for the employees’ product plus technology teams were improved by 140 percent. So that's a pretty, pretty large impact on a team in terms of efficiency and engagement.
Izzy Azeri
Wow. I've never thought about the impact on employee happiness themselves. But it sounds like also is a huge factor and probably helps with retention, especially in these days where hiring is so difficult, and we're all trying to retain the great talent that we have. So perhaps I guess, for the audience here serving the team members, if they implement a low code solution or a new product and understanding how they feel about it is probably a great suggestion. Sounds like that came in well for you.
Mike Schober
Yeah, I mean, it did, I think the experience or two as we started our journey, two years before that, focusing on Selenium automation, and in the beginning, is fine. But over time, it's a codebase. So it grows like your application codebase, and it gets more and more complex. And that's where the trust of the POS starts to drop. So when they did this, their automation codebase had grown pretty big. So I think that their engagement started to drop because they actually had to maintain that like an application and their developers. So going to a low-code solution really streamlined their testing abilities and that's what brought up their engagement.
Izzy Azeri
That's fantastic. We do have a question here from the audience and maybe this is rolling back to that pilot project you were talking about. The question is: when you do a project like that, do you pick one pilot team or several pilot teams? How do you go and share those results once you have them broadly across the organization to help others understand the impact?
Mike Schober
Yes, you do start with one or two teams. That's the best way to do it for the continuous delivery pipeline. For that, we actually have 10. But we're starting to focus more on a couple because that's just too many. Then share it as through depending on your communication channels. If you're on a delivery team it's a little more challenging than being in a horizontal organization like me. As we have wins across this transformation, where we literally have organizational change management people and comms people on our team. So that's going to be our method. They're the experts. We aren't the experts. We're doing the continuous delivery pipeline facilitating the automation. But being horizontal, we have the ability to leverage and tap into those types of resources and have them as part of our extended team.
Izzy Azeri
Oh, that's really interesting. So these are folks that help drive more broadly across the organization in the impact, who are, I guess, trusted professionals and doing that, you know, on a day-to-day basis?
Mike Schober
It's their full-time job organizational change management, and then we have some comms people as well. So they're the experts. We tell them that some of the concepts they come up with is just fascinating listening to them talk because I wouldn't be able to do that. Because that's not what I specialize in, but having them on the team is fantastic.
Izzy Azeri
That's really great. What a great way to get the impact out there through professionals doing this. That's awesome.
Mike Schober
Yeah, absolutely.
Izzy Azeri
Well, we talked about pipelines, we talked about automated testing. Outside of mabl or other testing technologies, what are the technologies that are important or capabilities that are top of mind?
Mike Schober
Well, I mean, I mentioned for us that, I call it the legacy automation footprint, but it has a big footprint that we have here at Schwab. Over time, it does start to show cracks. Because as I mentioned, it's a codebase, it becomes an application over time. API testing is a very big deal for developers and testers alike. But that's an area where product owners do not venture into because that's not something that they specialize in. They can do the UI testing a lot easier though, and API testing is something that they haven't ventured really deeply into. But API testing is another major area, visuals is a big one too.
In client-facing applications. I had a scenario in my last team where for those that have gone to financial institution site, you have the webpage at the bottom, you have all this fine print, which is longer than the web page. Those are the terms and conditions. In regards to visual, there was a release where we had some CMS come over because that's who supplies the T's and C's for us. The visual picked up a truncation in that. That's a legal issue for Schwab. So we picked that up and we went over to our PO and we said, hey, is this what is supposed to look like? The PO said, how did you find that? Because nobody reads those things? Because you just assume that they're right. So that's another big area from an automation perspective that's it's starting to become more widely used.
I think a big area that we're focusing on moving forward, is you hear a lot about CI/CD. So that's more kind of tail end, right-hand side, we're looking more closely and as part of the continuous delivery pipeline, we're looking into release orchestration, to be able to provide our ability to seamlessly move through the delivery pipeline. So that's a big focus for us as well.
Izzy Azeri
That's great. Then are the product owners involved there too, or is that a different audience?
Mike Schober
They will be because it'll get a view into that. Another area that we're looking into, as we start looking at continuous delivery pipeline is, you can go all the way to production and then you find out how things are working with you in production. What we want to be able to do, and there are some capabilities that we've been piloting, is predict the risk of a change before it goes to production. So there are some AI-based capabilities that we're looking at, that will allow us to predict the risk. Also actually, teams probably won't like this, but I think they will over time, assign a credit score to a team based on data that's in Jira and remedy. So it models that data and then creates a risk score and a credit score so that change management can then say, yes, certain teams can go all the way to production. Other ones, we need to do a little bit more work on others, where your risk is too high, we need to figure out why before you go to production. So that's some of the capabilities that we've been we've been looking into with our continuous delivery pipeline.
Izzy Azeri
That's, fascinating that so it's allowing people allowing the organization really be prepared for the potential risk of certain releases happening based on prior data or things that they can predict that potential.
Mike Schober
Yes. If something is a higher risk, you can say, okay, development team, go back and let's do some more testing on this to make sure that all the bases are covered. When you get to production, let's make sure we monitor this closely and run those tests in production as well. So you can have some heightened visibility on higher-risk releases.
Izzy Azeri
That makes a lot of sense. We do have another question from the audience. Mike, you talked about this, but maybe just diving one level deeper for those of us who aren't familiar with it, but what is release orchestration when it comes to Schwab.
Mike Schober
There are capabilities out there that that can be leveraged for that. But you're orchestrating the full release through from CI through testing, automation testing. Though you're automating release management, basically, at Schwab, we have some manual tasks around what was defined historically run definition of “done”. So it's automating all of those. So the delivery teams don't have to do that anymore. In the tool that we're looking at, specifically, allows you to set your orchestration workflow, and then automate that through some pretty simple interfaces. So it's really taking the manual release management out of release management and automating it.
Izzy Azeri
That's great. It sounds like the benefit, there would be savings from a human interaction perspective to have to do this manually versus having to rely on a system to automate.
Mike Schober
Yes. It's really simple. I mean, we can call out to software test automation capabilities to run automation tests and get based on that, you can automate approvals through your change management, your ITSM system. For low-risk teams, you can even stop a release orchestration, if the change risk is too high and require additional approvals. So it does take the manual out of release management. Another thing we have some monitors and the first and second line of defense on our team as well, for risk. Today, most of those types of things are done manually. It's up to the development team or their release managers to basically backtrack and find all that information for the auditor's well, with the capability that we're looking at, you can just go download the otter important give it to the auditor, and it has everything that's been done. So it simplifies things as easy as or as complex as on it can be for those that have been audited before. They're useful.
Izzy Azeri
So Mike, we talked about a lot of the advanced capabilities Schwab either doing today or planning on doing, it does seem like Schwab is ahead of the curve when it comes to new technologies and new ways of doing things automation. Has that always been the case? Or was that kind of a shift that had to happen in the organization over time? Can you just talk to what that's been like at a company? I think I read somewhere that, Schwab was founded over 50 years ago. Right. So there's a lot of history there. Can you talk through that kind of technology evolution over time?
Mike Schober
Yeah, I mean, I wasn't there in the beginning, or even in the early 2000s. But Schwab has always been an innovative company from the start. So there's, I think it was, might have been the first discount brokerage way back in the day and it's far from that now. But even back in the early 2000s, from what I understand there was in the tech bubble, which I was part of, but just not as not a part of Schwab. But that's why it was one of the first companies that built out a website. So I think all along, it's been an innovative company, I was leading Intelligent Portfolios before this current role. So we were one of the first large-company robo advisors or automated investing as what's called on the market. We were competing with startups. So Schwab has typically been at the forefront of innovation in technology.
Izzy Azeri
Do you see the role of QA changing as teams integrate testing further into the development pipeline?
Mike Schober
I do. I think traditionally some teams still do this. QA has been more of the right side of the SDLC. I think, shifting left and shifting right. QA is part of all of that. So I do think that the rule changes. I think QAs become more consultants to the development team. So it's a consultative role in terms of quality. So yes, I do. I think that changes over time.
Izzy Azeri
And that's actually what we've seen in other companies as well, where QA gets embedded into the development teams as opposed to sitting separately as a coach as opposed to solely responsible for quality within an organization.
Mike Schober
Yes, exactly.
Izzy Azeri
Great. Well, Mike, it's been great having you on. Really appreciate you being here and sharing more about what you're doing at Schwab from a test and integrating testing earlier in development. I look forward to seeing you next time.
Mike Schober
Sounds great. My pleasure.
Izzy Azeri
Thanks.
