Quality Engineering is a team sport. Without the right collaboration strategies and proper processes in place, it can become difficult to maintain quality and product velocity. Join Janet Bracewell, Andrea Wood, and Jessica Mosley as they unpack what a culture of quality means to them, and how their teams, and customers benefit.
Transcription
Katie Staveley
Hello, everyone. Welcome back to the next Experience session of the day. My name is Katie Staveley. I'm going to be the moderator for today's panel. The topic we'll be discussing for this particular session is Managing People and Processes in Quality Engineering. I'm really pleased to have a couple of fantastic panelists here today that have joined us.
We have Janet Bracewell. She's the Senior Engineering Manager for SmugMug. Janet's been with SmugMug for nearly nine years and has worked her way up from being a QA tester all the way through an engineering manager. I think she's going to add a ton of great insights and we'll have some great experiences to share with you all. We also have Andrea Wood, Andrea is the VP of Engineering for Barracuda and similarly, Andrea has been with Barracuda for some time for about four years and prior to that she was with a company called Synovia that was acquired by Barracuda. So Andrea has, again a wealth of experience as a senior tech person for a number of different tech companies and I look forward to hearing what she has to share. Last but not least, we have Jessica Mosley, who's the senior quality analyst for Enbridge. She's been with Enbridge for a few years and prior to that was at Worldwide group, but has nearly two decades of experience in tech as well. So welcome to each of you. Thank you for being here today.
So without further ado, I want to get right into some of the questions that we have prepared for this great panel. So first up, why don't we talk a little bit about inhibitors to DevOps, adoption, and testing in DevOps. So in our recent survey, mabl put out a survey on testing in DevOps we had uncovered that most people shared that one of the biggest inhibitors is not necessarily technology, but slow processes and reluctance to change. So for our panelists, it would be great if you could share a little bit about what you have experienced in your organization? Are you seeing that, and some of the ways that your team has overcome this as a challenge in terms of adoption and digital transformation? Somebody want to go first, maybe Andrea, maybe you can jump in here and share your experience.
Andrea Wood
Sure. So Barracuda is a cybersecurity company and what comes with that is a desire to deliver quickly, new features, new changes that are coming as a result of us becoming aware of new threats and things. So it's always a constant battle of balancing quality with speed of delivery. I think, at Barracuda, we haven't felt as much friction with getting mabl into the CI/CD pipeline, because it's really seen from the top down, I think that software test automation is necessary to drive that kind of speed and delivery without sacrificing quality. So for us, I think all the teams as they have adopted, mabl and you know how it's growing and expanding within the company. Whatever that particular team CI/CD pipeline is, at that moment, most of the teams are incorporating mabl into their CI/CD pipeline effectively.
So for us, we haven't seen that friction so much. But I think earlier on the friction would have been more around continuous delivery and what that means. So to have continuous delivery, you have to have all these elements in place. In large companies like Barracuda the autonomy of teams to move in that direction, or friction that might come with centralized teams not having a CI/CD pipeline ready that can support that can get in the way. So I think those were some of the challenges that we've overcome. To get to that point where it's past it's technically viable now.
Katie Staveley
Perfect. Janet, do you want to jump in here as well and share your experience?
Janet Bracewell
Sure. We came from a totally manual process when I joined the company. We're a photography platform. We provide a way for folks to store and share and sell their images and so the primary driving force for us is that the images look fabulous on our platform. They are safe. If you upload you get it back just the way you sent the originals to us. So it is a very visually driven platform. So I think early on the idea of automation, where you wouldn't see what was happening was difficult.
The other part of that also was the idea that we were doing waterfall, we were always the last stop before the customer. Often, our developers weren't really thoroughly testing their code, they'd hand it to us and say, find everything. So there was a fair amount to use the term friction between developers and QA and there was nothing especially speedy in that process at all.
Katie Staveley
Terrific, Jess, do you want to offer your perspective as well.
Jessica Mosley
Sure. What I've noticed over time, well, images are a little bit different. I'm a part of the technology and innovation lab within Enbridge, which is essentially a very unique opportunity, where we're kind of like a startup within an already established company to be able to innovate processes and procedures that are currently within the company for applications or other aspects of whatever is actually being done for the company, to make sure that it's of quality and it's using the best type of technology and innovating on the ideas currently in place.
But what we did find, and it was kind of the status quo at previous positions I was in was that a lot of the times it was more of what Janet was saying, where it's just like us being seen as some critique on whatever the previous process was necessarily, and it was never anything on them, it was just mainly to improve what was already there and that was the idea. As she said, with automation, a lot of times people think that, oh, it's going to be something that's going to take away from me or make my job harder, in some way. But that's never the case, it's just to make what is already been created a lot more efficient and optimized for the most part. But so I understand the kind of like hesitation from some companies to not want to necessarily jump on the digital bandwagon was like, well, we spent all this money on this, or we've done all this. So now you're coming up with something new, it was just basically innovating on it. It's not taking away from it anyway.
Katie Staveley
Yeah, that makes a lot of sense. I think, regardless of what the technology is, or what it is that's changing, change is hard in any organization. So getting folks on board can be a challenge both for individuals, as well as in addition to the processes in the technology, so I'm sure that a lot of folks can relate to that on the call.
So let's jump to the next question here. Before I do, I just wanted to remind attendees that we are taking questions from the audience. So feel free to submit those through the Q&A tab on the platform at any time. I'll take a look at those and add ask them to our panelists as they come in, but, and also a reminder that we are recording today's session, so you'll have access to this entire session afterward as well. So let's jump into the next question here.
With osftware test automation tools like mabl, manual testers, product owners, and other folks within the organization, oftentimes have the opportunity to participate more in automated testing. Can you share some examples of where you've had success getting more people involved in testing and what that impact has been? Janet if you don't mind why don’t you take this one first. I know that have some good insights to share on this topic.
Janet Bracewell
We adopted mabl just a year ago, October and their team itself was excited about the opportunity. Just to be really honest. There were some of the thoughts in the back of their heads, just like you mentioned am I going to be replaced? The answer's no, not ever because humans are smarter than any machine learning we have to tell it what to do.
So we have several test engineers now, we did not have when we started with the exception of one guy that we hired, because he already had a background with software test automation, and we thought, okay, he is going to be the one who's going to help us and up-level the team apart of the problem with being a very busy platform is that you have to literally carve time out to learn. Not that mabl is a difficult platform to learn; it has intricacies. But we are also a fairly unique platform that our customers and users are able to customize very highly. So as users and attempting to automate, there were so many permutations and opportunities for users to make changes to get themselves in really interesting states, we have some really fun bugs that have come through. But you can't automate everything. It's just not possible.
We had a two-hour block on Wednesdays that we call QA learning, we put mabl learning time practice and what have you in there. We also have QA who are embedded and this might come up in further conversation. But originally, we were that group over there and several years ago, embedded QA on each of our product teams. So there's a voice that keeps reminding them about mabl and automation and we add it to Jira tickets.
One of the things that we did with our QA learning time is we set up mabl office hours. So on the company calendar, there are 30-minute blocks within there that anybody can sign up for. Most often it's been QA who are working. But one of the other things that we did relate to those office hours was that the bugs that came in either during the week before or even the days leading up to that particular Wednesday, the team of our test engineers would grab the developer who would end up usually doing the fix on it and we would park him with the QA who was on call at that point and they would walk through having the test writing process for mabl. So what it did was the link to the test in mabl went with the bug ticket onto the product team backlog so that when it got fixed, there was an easy way to test it to see if it actually was fixed and QA had hands off at that point.
Katie Staveley
Terrific. Jess, or Andrea? Do either of you want to jump in on this question as well on how you're helping others get onboarded and use automated software testing just to do raise your hand?
Jessica Mosley
So I started out with mabl and this was back in 2018 at a previous company, and completely loved it and even then my thing with the developers, the scrum master or product owner was always trying to get them in and showing them what it is, how it works and all the secret sauce of mabl, what makes it great.
When it came to Enbridge that was the first thing that I sat down with our leadership and was like, we got to have this, this is something that's going to take us to the next level and scaling up. Basically, wanting to be something showing in oil and gas and innovation. So one of the things that I initially started off doing was doing things, little sessions called Dangerous with mabl. The first Grandmaster that I got on was one of our flagship products in the lab. Patrick Nazmi, he took it and ran with it. After that he was doing little sessions training other people on mabl and we continued with that in just wanting to demystify what it was and not basically having what we were doing or creating was something that was just being tested.
We were quality engineering, making sure that this was a quality culture. It doesn't have to just be where the testers or the quality assurance engineers are just doing this. We all have a part in this to play. So why not know how it goes from beginning to end, even down to how the tests are being created. You always have a great perspective when it comes from someone, such as the scrum master, or even the product owners in doing something as simple as a user flow that I never would have thought of, where they have more knowledge on some of the back end business stuff for it. But it essentially just created a great atmosphere for everyone to be able to learn about mabl and that's been very, very fun.
Katie Staveley
That's awesome. I love the concept of Dangerous with Mabl and how that's become a part of your culture. That's terrific. Andrea, did you want to add to this topic as well?
Andrea Wood
I think everything that Jess just said pretty much covers this topic. But one thing I would say I would add is that mabl gives us the opportunity to continue to have that quality influence upstream and because it's not purely a tool that only quality assurance engineers or automation engineers should be in or could be in. It gives that exposure to developers and product managers. So during like grooming, in an agile process, you can have those conversations now about what the test coverage should be for this new thing. Or, it could be a mabl test and having that be thought of as a first-class citizen further upstream than it normally would have been, or further exposed to the wider group than it normally would have been if it was just isolated to a downstream activity. So I echo everything that they were saying.
Katie Staveley
Terrific. Andrea, why don't we stay with you for this next question, if I remember, you had shared some really terrific insights again on this question. So we all know that in any transformation project, not just quality engineering, that it's a marathon, not a sprint. So if we take this approach, crawl, walk, run, and apply and make those changes within the organization. How are you today? How was your team building resiliency into your process for creating and maintaining tests and in what ways is your team evolving?
Andrea Wood
So the evolution is coming across the company, we care about the mission of providing a more unified customer experience across our products. So it always starts, starting with the customer. But in providing that unification of experience for the customer, it necessitates a lot more consistency around how we develop products, how we test them, how we deploy them, how we get things out the door. So mabl is part of this more strategic initiative that we have to bring some consistency to our CI/CD pipeline and even to the UI layers of these products.
We're using React now and we have a Barracuda design system that is influencing the look and feel of these products. What comes with that is this opportunity to have automated software testing in place consistently across the products. Rather than it growing and transforming from the top down, you will use this new technology. What we found with mabl and I think with any good engineering tool, is that things that are good resonate and mabl came into Barracuda with a few teams doing a trial and it expanded very rapidly. I think now is across 10 or 11 teams in various stages. But because it is such a powerful tool and will be something that really moves the needle towards that end state that we want to get to. It's growing within these teams very quickly.
So many teams are finding that it is transforming how they think about QA it is moving a quality influence further upstream in their discussions and it is making it possible. Well, I would say at Barracuda probably at this point it's still quality assurance engineers or qulity assurance automation engineers who are creating the mabl test. There's a lot more involvement from the team as a whole, I think in what that test coverage should be and what the test plans should look like and having those conversations.
But I think the selection of mabl in this bigger picture transformation has been something that's been proving out to be pretty successful, I think through experience at Barracuda.
Katie Staveley
It does. I have to admit, I don't hate hearing that.
Andrea Wood
Yeah. I think what is great about mabl is that it has some depth to it, but it's not hard to get started. So the teams pretty quickly can get up and running with, you know, one team is running. On every merge, they run tests, their 10 tests are very high-value tests. So they've already proven themselves to be worthwhile and catching bugs. But then you start to realize there are greater possibilities, and you can build more and more, but you just can see some immediate value with it, which was really important for gaining adoption and buy-in for the tool across the larger organization.
It wasn't hard for mabl to build a reputation within Barracuda a little bit. I think that's helping with adoption and growth. One thing I would say is, there's a lot of teams using it now and they are in various stages. I think our next stage will be how do we share information across people using mabl? How can we have everybody visible to everyone else's workspaces and mabl so they could see other tests and say oh, they did this cool thing, and they added some JavaScript in here and we can use that too? So that'll be the next stage I think is leveraging. Building and leveraging mabl expertise across the organization and how we share that out.
Katie Staveley
That’s great.
Jessica Mosley
Once one person hears about mabl within a company, it's almost like it goes viral. It is like social media influence or something I don't know. It just takes on a life of its own.
Katie Staveley
So great. Janet, I know that you had shared in our previous prep call a little bit about how your team's evolving. Your team has been using mabl for a little over a year now. So do you want to share some of the things that your team is doing as well?
Janet Bracewell
Yes, as I mentioned, we all started as manual testers and with the introduction of mabl, there was the opportunity for a brand new career path. One of the things that I have also done and been a part of as we have grown with our people processes, we spent months this year developing competencies for our ICS as well as our managers. Because of where the competencies started in engineering, they were very developer-centric and I said I'm sorry, can't apply these to my group. It's not going to work and it's patently unfair. So I wrote competencies that followed the pattern of the new company ones for our QA, as well as a new block for our test engineers.
What it has meant is that our QA now has a new career path that they didn't have before. So it's not just QA and growth along that path in IC competencies. There is now a test engineer career path that gives them a career path. I can't think of another term for that. But another way to grow and mabl has provided that.
Katie Staveley
I love that. That's amazing. I think it's always hard as a manager of people to figure out how you set those right paths for employees to keep them engaged and we're all talking about this great, I know, there's the Great Resignation, the Great Reshuffle, whatever term you're using, I think that's an amazing thing to offer your team to give them a path to learn and continue to grow as individuals and professionals.
We had a question come in from the audience that I think it would be a good time to add that here and maybe just since you are an advocate for mabl at Enbridge, after having used it at a different organization, I think maybe it would be great if you could start on this one. So the question was, could you share your experience convincing the leadership team about asking for more money, resources, time, etc. for getting automation into the software development lifecycle?
Jessica Mosley
Oh yeah, this one was a doozy to do. One of the things that was the real turning point, I believe was being able to assess it in man-hours acquainted to money, especially when it came to maintenance and set up of the automation. It was, this is something we need, we need to be able to scale it and be able to then maintain it. The problem that a lot of leadership well, in the roadblocks that I've run into previously was okay, well, how much is this going to cost me? How much is it going to cost you not to do this, basically? That's essentially how I presented it to a majority of the leadership in trying to convert them to the mabl enthusiast, basically.
So also being able to showcase its features, one of the biggest things is as Andrea mentioned, is being able to get those tests up and running pretty quickly. If you've done any type of software test automation, or even coding, for any application, it usually takes some people, depending on their skill set weeks to get things going. Whereas with mabl you can get a nice little test suite up and running in a couple of days, if that. So the time that it would take and the money that they'd be losing in just waiting on that was how I presented it and made sure that this is an investment and you're going to see that ROI very, very quickly.
Andrea Wood
I echo that because, for one, I mentioned earlier, speed of delivery is important. Well, there's a return on investment to that speed of delivery as well that the business usually can understand from the top down that, if you can quantify that, if we get software test automation in place, we will be able to deliver features faster, then that's a clear win. But the other thing is one of the QA managers here for one of the products has a spreadsheet going and it gives visibility to what that ROI is because on there you can see previously when we did it manually, or we did it another way, it took five hours now with mabl it takes 45 minutes. So you know that you can start to see the value, not just in the test creation, but in when the tests are running, things like that.
So I think being able to, it's hard to get that initial buy-in for the trial. But easy after that, I think to show where the value is going to come in and why it's a worthwhile investment to get mabl. At a larger company, I will say what's been trickier for us is not tricky. It's just it was an unallocated path. When you bring a tool in like that, and a few teams start using it, and then all of a sudden you want everyone to use it, being able to expand that contract so that everyone is able to use mabl is some is a process that I think now we've gotten placed pretty streamlined. So the company now knows how valuable it is, when a team is able to consume it and get moving with it, then there's no friction now to adding more workspaces, which is good.
Janet Bracewell
I remember the moment in a quarterly planning meeting where we had engineering leaders and product leaders and then part of my slide deck was a graphic that I had put together about, these were best guesses on cost. Here's what it costs when you find a bug in development, and here's what it costs when you find it in integration, and here's what it costs when you find it on production and here's what it could cost if you have to roll it back and redo it.
I remember the look on the face of our VP of Product when he finally got it, because he is a very visual person. But it was like, oh my gosh, I get it. So to say it was easy to say whatever you need with mabl after that point was kind of when he got it, it was like, oh, yeah, this is very effective.
Katie Staveley
It is funny that the second you start to propose tangible costs to the business it really kind of changes the conversation mabl or not, certainly with mabl but in any sort of technology discussion that once you start to tie it back to real costs to the company and to the team or to the function, the conversation, for sure changes for everyone involved. We have a bunch more questions coming in from the audience. So if it's okay with the three of you, I think I'll jump in with another one of those. The question is how do you make sure that each team member involved in QA delivers the same quality of testing as a more experienced automation engineer?
Janet Bracewell
Oh, I'll start to tackle that, and I can tell you what we do. That we have another tool that we use is called ClickUp and it's really designed for programming project management. But what we have done is my queue, our test engineers put this together. So it looks like a Kanban board with statuses and what have you. So as they write a test, it goes through a PR review process, just like code does. So the more experienced test engineers get a notification pumped out into a Slack channel, they go through it, they check it, they comment on it, the writer of the test actually will review it, revise it, and then it's ready to be added to a suite or it's packed and ready to go into a suite until the code gets to whichever development environment it needs to be in to run. But that's been super helpful.
As people have been learning to write tests that includes the developers who are learning to write, the QA learning to write, so our more experienced folks are doing actual reviews.
Katie Staveley
Love that. I'm sure that's great for both those more senior folks, as well as the folks that are new giving them that exposure and experience in both coaching and receiving feedback. Jess, or Andrea, do you have any additional thoughts on this particular question?
Andrea Wood
No, I had been about to jump in with the same sentiment that really, there's always review from seniors to juniors, it's just like a software developer putting a PR request up. It's also there's certain if someone's more junior, during the test definition phase, they're getting feedback from their more senior peers on the scope and the definition of the test that they're going to create. So I think the normal checks and balances on a development team come into play.
Jessica Mosley
I wholeheartedly agree. I mean, you just have to have that fostered relationship with the senior test reviewers or QA manager depending on who's actually reviewing the test will be able to have where the junior person can come to them with questions, and even helping in showing different skill sets necessarily, of how I got to this conclusion, how did you get to that conclusion, sometimes it can be a little bit more granular and breaking it down, but being able to speak with them, and connect with them on that level, just to have that understanding, I think is key to it, such as Janet was saying.
Katie Staveley
So I'm going to jump into another audience question, Andrea, this one is specifically for you. So how would you describe the end state that you're using mabl to help you work towards achieving?
Andrea Wood
So how I would describe it is that the short version is that things can go out when they're ready. So we want to get to a model where we're able to develop consistently across the product portfolio, and that there's automated software testing in place that allows these changes to go out and we can feel confident in the quality and trust of the change can go out.
So that's the tactical piece, the philosophical pieces, that there's a benefit to consistency in the tool usage and there are in a large company, especially one like Barracuda that's grown a lot through acquisition. There are lots of automation frameworks. There are lots of languages in play, there are lots of things in play, and in this particular case when it comes to test automation or CI/CD pipeline tools that we're using, there's a lot of benefit to the consistency. So in the end state, we will have teams and products where we can start to leverage resources across the portfolio based on what's important, not just these resources within this soloed product team, these resources within this solid product team, and oh, if these guys don't have the bandwidth, then their product manager just has to wait for these things in the backlog versus if people are familiar with a consistent tool, we might be able to ask someone on another team, can you help out these guys for a bit, they want to get these high priority features out. So there's business value to that.
But the general buy-in to an end state where there's are modern CI/CD pipeline tools that have automation, as a first-class citizen to ensure the quality of things going out the door, I think, is something that is bought in from the very top of the company now down. So we have many strategic initiatives towards this end, of which mabl is now a piece and a very fruitful piece. It's been pretty successful. So I had something else I was going to say about that end state.
Yeah, but I think mostly, that's it, we have an end state in mind, on our customer experience, and that's driving really the rest of it, and one of a very unified customer experience. That necessitates a unified UI look and feel, and that when I'm interacting with this type of product, it's the same calendar picker, it's the same thing and when I'm interacting over here, it's a familiar experience. So it would only make sense that at that next layer down of actually building those pages, that there'll be consistency in the testing.
Katie Staveley
That's terrific. Yeah, I think that it goes without saying, but I'll say it anyway, customer experience, really, or user experiences is the defining separation between brands today, whether you're a B2B brand, or B2C brand. I love hearing your story about aligning both the technology, the people the processes toward that ultimate business goal, and making automation truly at the forefront of what you're doing, I think it's inspiring, but it's also really interesting from a business model for sure.
Jess, how about this question for you? Do you have a balance of dedicated QA resources as part of your QA strategy? Or did you maintain existing resources, learning and maintaining mabl on top of their historical roles and responsibilities?
Jessica Mosley
Yes, so that we have the dedicated resources and basically just wanted to kind of upgrade their skill set with mabl and continue to allow them to grow within their current jobs and duties and things like that. But I want to say that the biggest benefit, I guess, the secret sauce to mabl is it will take you to different places. So if you start out knowing about automation, you know, that's great. How are you going to be able to utilize that with mabl? You can start by knowing a little bit more about the DevOps piece of it. Then that grows into something more, and you're able to learn a little bit more about the Cloud and things like that. I think mabl is a really good jump-off point for that and just allowing for that little bit more of growth for not only just as an employee, but as a person when utilizing it. That's just what I see.
Katie Staveley
That's terrific. Anybody else have any thoughts on that? Are your team structured differently? Do you have dedicated teams? Or is this in addition to current roles and responsibilities for your QA team?
Janet Bracewell
Many teams are different. Teams are different. But what I would say is that, within any quality assurance pool, this idea that everyone can develop software test automation is definitely believed. It is a growth path for even formerly QA engineers, who are just manual testers are now moving in that direction of your new role is primarily going to be working in mabl and developing automated software tests. All of us know you can't always get 100 percent and there's always going to be that element of manual testing.
On most of the teams, there are still dedicated QA engineers who do some manual testing work, but the lines are really blurred towards that agile ideal of team members and so QA engineers who are moving in the direction of mabl development, you know, the opportunity is there for them to also move more into development if they want to and learn how to fix the bugs that mabl is finding and layout. So there's real career growth opportunity in moving to this model. Especially with a tool that doesn't require that you are, you know, a normal type that we think software developer in order to use, while still being exposed to, to that. So we still see a mix. But I would say the people who are doing the most development in mabl, within the company right now have had a QA role or continue to have some type of QA role.
Katie Staveley
Great. Janet, a few folks have asked a very specific question for you. So we have about two or so minutes left in the session. So I'm going to hand this next question your way. So my company is using a product that's very visual, how did you and your team overcome heavy visual testing?
Janet Bracewell
A large part of that is still going to be done manually. I will say that, mabl if you have a historical run of tests, you can see the visuals, and that it will note any changes, especially if in a page, but if you're looking at for us the quality of images that's very much manual, if you're checking EXIF data is all of it there if you upload it, and save it, and then you download it from our platform, compare very, very large images, videos, all of that typically is. We can automate then did it get there, and did it come back again. But if you are looking for quality information, that piece of it is still very much manual.
Katie Staveley
That's not terribly uncommon. I think a lot of organizations, whether it's visual testing or other types of testing, they are mixing. It's not all automation or all manual testing anymore. We do see that across customers that it's a mix of both of those together. So we are at our final minute here. I do want to take a second to thank Jess, Andrea, Janet, thank you so much for being here with us today. I have been excited about this session since we spoke last week. I've been putting it in Slack probably daily. I'm sure my team is sick of hearing about how excited I am about this session and I really think it's delivered. I think you all offered some really valuable insights.
