This post discusses how we organize our exploratory testing sessions at mabl.
You’re welcome to download our template for our “Elmer Fudd” sessions:
Quality is like Mom and apple pie - everybody thinks it’s a good thing!
When your product is an automated testing tool like mabl's, the bar for quality is high. Ensuring the quality of our software is a top priority for the team, especially since at the end of the day it helps our customers deliver better products to their customers. It’s core to our culture and we're constantly working on ways to improve.
No matter what your domain is, many delivery teams struggle with the seemingly competing goals of frequently delivering new features that solve new customer problems, and maintaining a high level of product quality so customers don’t feel the pain from bugs or other issues. Like many of you, we have to work on balancing the needs of both as well. We strive to deliver new features and enhancements as quickly as possible without losing focus on the quality of our output. It's an ongoing process for our team that we are always looking for new ways to improve upon.
Our product is evolving at a rapid pace. The team is working very quickly, releasing new features and improving existing ones. For example, there were over 70 new enhancements released in the last 90 days. To keep up with the pace of development while also meeting the high standards for quality, we initiated a program called Elmer Fudd - a nod to that legendary hunter of Bugs Bunny. We've added more structure around involving more team members in exploratory testing of the product from a user experience point of view. Though we're ardent dogfooders of mabl, applying a good exploratory testing session before a build goes live, to find the edge cases not covered by existing automated tests, helps us give our customers a better overall experience.
A small experiment
Our quality catalyst, Bertold Kolics, suggested an experiment: have volunteers from various teams, including field engineers from sales support, customer support and success, UX designers, product managers, and marketing team members, participate in a one-hour “bug hunt” before releasing a new feature to beta or production. This was easy to try. No special testing knowledge was needed. We got together, with a couple of us participating remotely. We had engineers, product managers, support team members, and more. Through ad hoc and exploratory testing, we found many opportunities to improve the product or fix an issue before the feature went live.
Everyone agreed the Elmer Fudd sessions were productive, and decided to continue them as needed. The sessions were informal. People could ask each other questions and discuss how features should behave. Issues found were noted in a dedicated Slack channel. Engineers went back through the channel later to note and prioritize bugs to be fixed or new stories to write for any missing functionality noticed.
The Elmer Fudd sessions continued, generally at least once per week. Some focused on testing new features, while others focused on making sure there were no regression failures due to a new change.
Reflecting and iterating with new experiments
After a few weeks of holding Elmer Fudd sessions, we did a quick retrospective on what was going well in the sessions and what could be improved. We had observed that multiple people were often testing the same piece of functionality, while other aspects of a feature might not get tested at all during a session. Despite using the Slack channel, sometimes details discovered during the sessions were getting lost.
We came up with some new ideas to try as experiments. One was to do more focused exploratory testing, using charters based on Elisabeth Hendrickson’s template:
Explore <target - a feature, requirement, configuration...>
with <resources, such as tools, data, a technique, another feature>
to discover <information - about system behavior, performance, security, accessibility...>
(from ExploreIt! Reduce Risk and Increase Confidence with Exploratory Testing by Elisabeth Hendrickson)
Another way to be more organized about who was testing what is have people pair up to help get multiple perspectives on each exploratory charter.
Before our next Elmer Fudd session, Bertold shared information on how to write exploratory testing charters, referring to Elisabeth Hendrickson’s Explore It! book (which we obtained for our library).
He created a document for planning the session. Engineers added links to the stories to be tested and notes about what needed testing. They created exploratory testing charters in a separate spreadsheet. Personally, I find writing charters can be challenging, which isn’t a bad thing. The thought process, such as thinking of resources to use for testing, brings up fresh testing ideas. I paired with a couple of engineers to help them get started writing charters.
When we started the next Elmer Fudd session, participants paired up and assigned themselves a charter. We still shared issues found in Slack, as well as recording notes in the charter spreadsheet. One of the engineers volunteered to remote pair with me. At the end of the session, we had a quick debrief. We felt good about the amount of testing accomplished. The engineer who paired with me said he thought we had a lot of good testing ideas that he wouldn’t have had by himself. Others agreed they experienced similar benefits.
Anyone can organize an effective bug hunt
Thanks to Bertold, we now have a pretty smooth process so that anyone can organize an Elmer Fudd session. He created a template we can use to create a doc for each session. It includes a checklist of activities to be done:
The Elmer Fudd session document has a section to specify the test environment and mabl trainer version (since we may be testing a branch) to be used in the session. The schedule for the session is also laid out in the doc.
The next section is “What’s in this release candidate?” People can add the tickets and PRs along with other notes about what to test. This is followed by a section about the exploratory testing charters, using the format shown above, and a link to a spreadsheet for exploratory testing charters. We have a template for creating this spreadsheet in Google Sheets:
The document also has guidelines for the session. These support good conversations during the session and help ensure all valuable information discovered during testing is captured.
Report all potential issues in Slack on #elmerfudd:
-
Over-communicate so that information captured on the channel can be used in the post-session effectively
-
Post screenshots, videos, logs as needed
-
Use threads for discussing a problem to avoid overloading the main channel
The resulting session document is pinned to our #elmerfudd Slack channel. The meeting organizer schedules the session and sends an invitation to the team requesting the Elmer Fudd, and to managers both in and outside of Engineering. A general announcement also goes out on Slack, and anyone can request an invitation.
Outcomes
Our team tracks many different metrics on production usage and customer-reported bugs. And there are many different efforts going on to improve our product’s quality and user experience. As with many modern complex, distributed systems, it can be hard to pinpoint trends and their cause.
Overall, there’s been a positive trend. The rate of bugs in production that require intervention by engineers is down. Field engineers report fewer roadblocks as they work with trial customers.
With all the information gathered during the Elmer Fudd sessions, everyone is more clear about what testing was done. We feel more confident about making a decision to merge changes and get them deployed or released in production.
We're all getting better at writing exploratory testing charters. Our engineers are getting a chance to hone their own testing skills even more, so they can do more testing as they are coding new changes.
An important benefit to the Elmer Fudd sessions is that sales, customer support, and marketing can participate, help with the testing, and get a glimpse of what’s about to be released and see how the product is evolving. We have several feature teams, and each one is continually releasing changes to different parts of the product. Participating in the Elmer Fudd helps everyone stay informed, and get hands-on with the latest features.
As we test a new feature in the Elmer Fudd, the engineers and designers who developed that feature are in the room to answer questions. This has ensured that people across all teams share the same understanding of the goals of each new feature and how that feature works.
The Elmer Fudd sessions keep quality at the forefront of everyone’s mind. It’s easy to get heads down coding new features, and this has given us a chance to step back, look at the big picture, and think deliberately about how customers might use the new changes. No doubt, as we reflect and learn from our testing activities, we will continue to improve them and do an even better job of building quality into our product.
You’re welcome to download our template for our “Elmer Fudd” session document.
| A group of mabl employees working around a conference table. |
