Stack Overflow is a mature product with a rich history, and legacy, among the software development community. As Stack has grown over the last 13 years, some of the benefits that have arisen are that users can arrive at an answer to their questions in a number of ways, and the community members have a variety of ways in which they can participate with each other. While this is very desirable, it does lead to complex pathways that users can navigate, and even small changes can have a ripple effect. The Stack Overflow Team had been relying on manual testing for the Teams product for some time but realized they could do a much more effective job by bringing in test automation. In this session, Dale and Meredith will talk about how they automated testing as much as possible, the importance of democratization of testing, how they applied DRY principles to test creation, and how they trigger tests within the release pipeline.
Transcription
Izzy Azeri
I'm super excited for this session to talk to Dale and Meredith from Stack Overflow. If you're familiar with Stack Overflow like I am, I've used it a lot to answer those technical questions that I'm curious about but what you may not know is Stack Overflow has a private product as well for private businesses. So we're going to learn a lot more about how Stack Overflow approaches quality engineering with their product. So, as background, I'm one of the co-founders, spending a lot of time with our enterprise customers here at mabl and I'll hand it over to Dale and Meredith to introduce themselves and we'll dive right in.
Dale Cook
My name is Dale Cook. I am the VP of Product Engineering at Stack Overflow. I'm pretty new to Stack. I joined in February. I'm really excited to be here. Stack Overflow obviously is a cornerstone of the software development technology industry. I've been using Stacks since there was a Stack. As I said, I'm fairly new here joining February. But I've spent the last 25 years doing technology primarily and startups in Silicon Valley.
Meredith Hatzenbuehler
I'm Meredith Hatzenbuehler, I'm based out of San Antonio, Texas, and working as a software engineer focused on automated software testing at Stack Overflow. So I'm super excited to be here and talk all things testing.
Izzy Azeri
Great. Well, thanks, Meredith. Thanks, Dale. Why don't we jump right in? So when it comes to quality and testing before you ran into mabl, can you talk a little bit about what that looked like at Stack and perhaps some of the challenges that you were having that made you explore something new?
Meredith Hatzenbuehler
Yeah, so we have a wonderful 13-year-old codebase, that was in need of reevaluating how we approach things and things like our Team's product didn't exist in the beginning and now they do. So one of the things that mabl and looking for a new testing automated testing tool helped us realize is: rather than looking at things on a product-level, looking at things in a component-level, because we have a lot of reused components throughout different products. So finding a tool that could help us use and reuse that type of thing, and be a little more time-effective was definitely something we were looking for.
Dale Cook
Yeah. I think taking a holistic view of this, we had relied on some sort of older testing methodologies previously, and so when I came in, Meredith and I talked a lot about well, how do we move that forward? How do we take this very complex product that we have, regardless of it's now a public platform that everyone probably knows, and loves? Or whether it's our Team's product, which is assessed, offering private Stack Overflow? How do we build a testing framework that can test these very mature and complex products? How can we do it in a way that doesn't require us to spend a lot of like, engineering resources? Engineering resources are really expensive. We want engineers to be building new things and doing new things they are interested in that kind of thing. How do we build a really robust testing platform for a complex product without having to take a lot of engineering resources?
So we started to look at, well, how can we do that? How can we automate that? How can we do it so that we can do it in a fairly simple way? I've done this before in a couple of other companies but with different tools. We ran a process where emeritus did the bulk of the heavy lifting here, went out and surveyed the market and we came up with a set of criteria that we went out and surveyed the market and looked at a whole bunch of tools.
What we came back with was mabl was a clear winner. There were some other software testing tools that allow that ease of testing creation that mabl does, but they weren't performing. I won't mention them, but they were slow in the way that they work, we can build tests really fast, the tests themselves were not particularly fast. There were other products that were fast on their tests, but they didn't offer the flexibility or the ease of tests construction. Then the other thing that really sold us on mabl was its integration with the engineering workflow. So not just with building tests, and getting mabl integrated into our CI/CD pipelines and that kind of thing. But integration with quality engineering workflows, so that engineers can be running mabl tests, not just the unit test, not just their integration tests, but actual mabl tests as part of their workflow. That was a really powerful thing. So all those things added up to make mabl a clear winner.
Izzy Azeri
That's great. I appreciate that insight for the people on and so I think you've been using the product now for close to a year. How's it going so far? How has automated software testing impacted either product delivery or the happiness of the development team?
Meredith Hatzenbuehler
Yeah, so I think one of the things that have stood out is, when I was looking through a number of different products, part of the goal was like, let me see how far I can get, can I read a test? Can I run the test? Can I see the results? Are they helpful? So mabl obviously, came to the top of the list very easily.
But one of the things that I've even been surprised about is how much value we've gotten out of things like screenshots and every step, network information, the DOM, et cetera. That has so helped that loop with other teams, even if it's SRE to be able to come in and say, I can say, hey, here's some maple credentials, go log in, go look at this test. It's on step 43, or whatever and they're like, great, I have exactly what I need. Whereas before, maybe you have to have some back and forth discussions, or maybe you don't have that detailed of a record with other types of automated testing. So you're like, okay, well, let's wait till this happens again. We'll try and get more information and that loop of solving problems has definitely become much more efficient.
Izzy Azeri
That's great. Dale, you mentioned earlier, one of the things you saw was how mabl integrates into the different areas from a development process perspective. Can you talk a little bit about what are those areas you've integrated mabl into that have been beneficial for you?
Dale Cook
Yeah, absolutely. So at the moment, we're really concentrating on the Team product. So again, Stack Overflow for teams is our SaaS product. So we're really concentrating there so we can do a better job servicing our customers. But going forward, we have a much grander vision for how we'll implement mabl, we definitely want to bring it into the other parts of the engineering organization so as to serve the public side and all of the other parts that we have like we've got an advertising platform, it'd be great to bring it in and service that as well, and really spread it around and have it as an integrated part of our overall quality and testing strategy.
But more than that, one of the value propositions that we see from mabl is that we can push it into other areas of the organization that may not traditionally look at testing as a part of their realm or part of the value that they don’t want to enjoy. So for example, we want the product organization, to be much more involved in the test creation process. So instead of product having to write up and write something up and then send it off to testing, testing implement it and come back and go, is this right and they're like, No. This is a very convoluted cycle that takes a long time and resources, we want a tool, that product could go, this is my acceptance criteria for this particular story, or whatever, I'm just going to write it, I'm going to go and build it in mabl, it's going to take me less time to build that test, than it is to write up a document, just to send it to testing or to work with testing to try and build something for me. So this tool allows the front line to have more contact with the thing that they want to achieve.
We're also thinking about, well, how do you implement this in other areas that might even be further away from the engineering? Some things that have come up it's like, could you use it in the sales cycle. So if you're doing targeted sales. So I'm going into a company, and before I go into that company to do my sales pitch, I'm rolling up an instance that's branded for that company already, it looks like their website, like has their logos, and maybe I've even taken some content and put it in. Well, that's great. But as a salesperson, the last thing I want to have happened as I do that, I've usually got an automated tool that will do that across bits and pieces. But the last thing I want to happen is that I go into that sales cycle and something has broken in there.
So you can imagine I put my materials in an automated system, I create that specific instance for this sale. Then we run mabl against it for the path that sales don’t do and then sales gets a report that so they know before time, that that is correct. They don't have to spend their time going and checking that that is the case, they can do more of what they're trying to do, which is sales. So this is an incredibly powerful way of thinking about how to utilize this technology and how to push it outside of just engineering and production into other parts of the organization. Yeah, so hopefully that gives some insight on how we're thinking about in the future.
Izzy Azeri
Yeah, and it really sounds consistent. We heard a message yesterday where there was a similar kind of use case at Schwab, where product owners were getting involved in acceptance testing, and really what it did for them is it drove inclusion, they felt connected now to the development team and the testing team, but more importantly, confidence that what the feature was that they actually developed is exactly working the way they expected it to because they're the ones that wrote the test board. So that's a great use case.
Meredith, since you've probably been hands-on doing a lot of this from a test and development side, can you talk a little bit about the best practices you've instituted as you've ramped up your rollout of mabl internally?
Meredith Hatzenbuehler
Flows, flows, flows. Just like I said, we have so much potential where we're kind of focusing on our MVP with the team's product, but so many things that we're working on for this product also correspond to public and other areas of Stack Overflow. So we're trying to think of that upfront, which is like, how can I do this in actionable components that I can just plug into a different environment, but I know that this feature is part of that and now it will work. Silly things to like credentials, how we're organizing that, how we're naming those across different environments, just to minimize the amount of rework we have to do to make mabl run easily. I would say those are probably the two big ones.
Izzy Azeri
That's great. Do other people collaborate with you on those flows? Is there a way that other people can use those as well?
Meredith Hatzenbuehler
Yeah, we have a small group that's dedicated to mabl right now. So they're doing a ton of work iterating on that, and then we'll bring in like, a different tier, a different version. How did we do anticipating needs for this type of thing and slowly grow out and adapt as we go?
Izzy Azeri
That's fantastic. Thank you. Well, why don't we switch gears a little bit? I know part of the challenge right now across the world is around hiring, and I'm sure Stack Overflow is hiring right now as well. Can you talk a little bit about, what are the types of skill sets you look for in folks? What types of people are most successful at Stack Overflow and feel free to talk about some of the open roles you have as well.
Dale Cook
Thank you, Izzy. Absolutely, yes, we are most definitely hiring. We've had some growth plans for Stack for some time and as you probably know, the recent acquisition of Stack Overflow has accelerated there. We need to hire more people we need to hire across the board. If anyone out there wants to work at Stack, please come and work at Stack, we'd love to have you. You can go and check out our careers page and we have open positions.
As for what makes someone successful at Stack, it's an incredibly collaborative environment. We're not very big, and we get a lot done, where we get about 100 million uniques per month. That's pretty good traffic. I think we're on the list of the top 50 most trafficked websites in the world. So we get a lot of that done with a small engineering team. We're about 60 at the moment, I think, although, again, we have lots of opportunities to expand it.
Obviously, we want smart people and problem solvers, and good engineers, one of the things that's really stunning about Stack Overflow is just how collaborative it is. It struck me coming into Stack fairly recently and having spent such a long time in Silicon Valley, about how anyone can go and talk to anyone. The most junior engineer, we have can go and say, hey, VP of Engineering, I'm having a problem with this, can you help me? The answer is yes. We love to come together and share ideas. I know everyone says that, but it really does happen at Stack. It's really quite amazing to watch and have people who have 15-20 years of engineering experience, listening to and taking in and working with people who are very early on their career and incorporating their ideas into solutions and that's really an amazing place to be.
I think one of the things that we have planned over the next couple of years is really revitalizing our technology. As you know, as Meredith said, it's an older codebase, and we're going to put a lot of effort and resources into bringing that more up to date. One of the other things that I think is really interesting is things that we're looking for is people who are up for a pretty good technical challenge, like there's a lot of things that we need to do, and we're not going to shy away from that. We don't try and hide that. But I think trying to make these fundamental changes while keeping stack up and running and running in the way that it has for 13 years and highly efficient product. That's going to present a number of technical challenges. I'm certainly interested and I know, one of the teams that make something on that too. Meredith, do you have anything to add there?
Meredith Hatzenbuehler
I definitely agree with everything you mentioned. I've never been a “rah, rah”, so happy to work at a company before but it's really hard to not feel that way here. For me, it's been an incredible opportunity to work with developers from all over the world. The Netherlands, Germany, Brazil, Canada, Israel, the US, this is just on my team that I talk to every day. So that's really awesome. From a people connection standpoint to an ideas sharing standpoint, we're all coming from geographically and so many other points, and it’s fun to see the solutions that we can come up with in that kind of team. So, that's huge.
The other thing that's huge is remote work. We've been doing remote work well before I arrived. Our transition over the last couple of years was pretty seamless because we already operated in a remote-first standpoint, which is if you're having a meeting, for example, and let's say four out of the five people are in the New York office, but one is remote. Everyone will do the meeting remote from their desk so that you don't have that situation where there are four people hovered around a table and you're a floating head off to the side. So iterating on that and working on that has made it where, yeah, we're all working remote. We're all distributed on time zones. But you don't really feel that as a disconnection, it actually adds to like how we can work together. It's been amazing.
Izzy Azeri
That's fantastic. Dale, just so you know, we're almost at 100 million uniques as well. So we'll be there.
Dale Cook
All right, then the race is on. Challenge accepted.
Izzy Azeri
For those who aren't familiar with Prosus or what's changed since the Prosus acquisition can you just share if anything's changed internally? Or how you're thinking about things? Maybe there's a bigger opportunity now you're chasing as well?
Dale Cook
Yeah, absolutely. I've been through acquisitions many times before, like, just I've had a long career and so I've been through these things. I got to say that the Prosus acquisition has been really good for Stack. So there's very little change in the way that we work, quite frankly, basically, Prosus was a little unknown to me before this, and they have majority holdings, or significant investment in a very large number of internet properties, many of which the audience probably knows. Their process is to really come in and provide more resources, but not really to change anything.
One of the ways that they provide resources, of course, there's money investment, but the other way they do is because they have such a large portfolio, they're able to get companies that would have a lot of, I'm going to use the word synergy. I don't like it but it’s a word for this. Working together. So for us, we look at things like say, EdTech and we say, well, is there an opportunity to add a training component in here more than just answering questions? Could they be in Stack or in relationship to Stack, could there be some kind of organized training as well? When we look at that, we might look at a bunch of companies out that are doing that we think about partnerships, and that's a very big deal. There's a lot of friction in that it's actually harder to do partnerships than a lot of people expect and so it's a challenge. So we have to say, well, do we have the resources to do that?
One of the things Prosus does since it has a lot of these properties, and can facilitate these conversations and remove a lot of that friction, it makes things a lot easier. So we're not sure exactly where this will lead, one of the things that we are looking at is well, how do we fit in that process portfolio? What are other companies that we're aligned with and how can we all work together to do a better job of servicing our clients and provide more value to them and of course, increase the value of stack and the other companies as well. So that's one of the things that they offer, that I think is actually very hard to get in other places. So that's been really great.
Izzy Azeri
That's fantastic. Yeah, it makes a lot of sense. Don't interrupt you, but give you more value as you continue to grow. That's great. We did get a question from the audience here. As you started to use maple and get going with it, how long did it take for your team and the rest of the organization to really feel like they understand how to use it, and also get value out of the product?
Meredith Hatzenbuehler
Yeah, so our rollout that we had in mind initially isn't the rollout that we ended up finding effective. So one thing that, like a pivot that we made, that I think was really effective was rather than try and bring in all these different teams from day one, and try to get an advocate in these groups, and let's all get together and spend some time you know, getting familiar with mabl, that's hard when everybody's rocking and rolling with their normal work.
But, what helped was, okay, let's get a smaller subset of people to dedicate some time to this on a part of our product where continuous testing can really be a bigger burden, and let's just iterate on that over a couple of release cycles. We took the low-hanging fruit from our test plan that we know is going to be some of the easier stuff to learn to write and maintain and mabl and then let's start building that to show the value. Once we've been able to do that now, it's like people are more coming to us, like, hey, I heard you have this thing and everything's easier it's getting easier, like how do I use it? Do I need to make an account? Where do I do that? You're like, okay, cool, I can help you. So that approach has been most effective.
So I would just say whatever your MVP is that you can come up with, and whatever the subset is of your test plan that you can start to plug into a mabl and start to demonstrate those results. Like, when a step is working, everything's green, it's cool. When a step is broken, here's the data you get, here's how it might be helpful and just really slicking it down to those most important parts. Then, after, after doing that, and kind of getting the word out, but also, having like a broader test coverage base that we're slowly growing, and now, that part of the process, we have a much higher level of confidence on certain things that's hard to resist.
Izzy Azeri
We had a similar kind of conversation yesterday with Andrea Wood, who's the VP of Engineering at Barracuda. She said something similar where, as the product started to evolve in rollout across the organization, at one point, it just went viral, like, people just wanted to come in and try this new shiny thing, because they heard about all the benefits it was providing. I'm curious when it comes to your team, Meredith, the people using mabl, did they have experience with automation previously or was this brand new to them?
Meredith Hatzenbuehler
I think we have a mix, where some people have worked on some automation, and some haven't. And, those that haven't found it easy to get up to speed with mabl and those that had Selenium or something like that where there was that little bit of in-between time, well, what's this equate to over here and that sort of thing. But honestly, we had great support from the mabl support team too, to kind of get us through that first round of questions of like, okay, how do I do this and why isn't this working? Or what did I forget to configure or whatever, the turnaround time was so quick, that we were able to get up to speed pretty quickly?
Izzy Azeri
That's fantastic. One more question here about just maybe broader for Stack Overflow, what are some of the big projects you're tackling over the next 12-18 months that you're excited about or are going to have a big impact on the business?
Dale Cook
Yeah, definitely, this product release, there'll be product improvement, product release that's coming out. But one of the things that I think I mentioned before, one of the things we're really excited about is, we're doing a lot of tech investment. So some of that's dealing with technical debt, third in your codebase because we're going to have some technical debt. So that's dealing with bringing testing more across the platform. Automated software testing is where mabl comes in, mabl and other tools. That's a big investment we want to make. We continue to think about how we can make a better a better product for us internally. So that it's easier for engineers to build things and more automation, better codebase, like that kind of stuff allows us to move faster in our product releases. So over the next year, we're going to spend a lot of resources doing that we're very excited about that. It's a bit overdue. We think it's going to have an incredibly important impact on the future growth of Stack Overflow.
Izzy Azeri
That's fantastic. Appreciate that insight. Dale. One last question here is: how do you think about the future of quality at Stack? You know, a lot of people think, automated software testing is the end, but there are other areas of quality engineering that people think about whether that's around accessibility testing, performance testing. Are there other areas of quality engineering, you're thinking about, over the next couple of quarters, as you continue evolving the platform?
Dale Cook
Absolutely. You know, accessibility is definitely one of them. In order to meet some of the ISO standards that we want to want to reach, that becomes much more important. There are security aspects that are really important. There's a whole suite of things that we really need to do and we're about to be able to automate that as much as possible. So that's going to be something that we're looking forward to utilizing because I think it's time to provide a huge amount of value.
Izzy Azeri
That's great. We love to support you in that journey, Dale. As you re-architect and re-platform Stack we will be here for you. So I do want to thank you, Meredith and Dale, it's been my pleasure speaking to you both today and excited to see what's next for Stack and if anyone's interested, Stack Overflow is hiring all around the world. Thank you.
Dale Cook
Yes. Thank you.
