SmugMug Builds a Culture of Quality with Mabl

Describe mabl in 3 words: marvelous, magnificent, magical

SmugMug is the safe, beautiful home for photos, providing passionate photographers at any level of experience a place to store, share, and sell high-quality photos. As the leading platform for photographers worldwide, SmugMug prides itself on building beautiful experiences for all users. The SmugMug QA team plays a critical role in ensuring that every user has a delightful experience on their platform, from casual smartphone shutterbugs to professional portrait, wedding, event, and landscape artists. 

Managing A Complex User Interface for Creative Expression 

SmugMug’s photography platform is highly dynamic due to its variety of user experiences. Customers range from hobbyist photographers sharing their work with friends and family to professionals who manage fully virtual galleries that require payment processing, sales histories, and advanced account management features. With several different tiers of paid accounts as well as a plethora of customization options, possible user journeys are almost limitless. 

To meet the needs of every SmugMug user, Senior Engineering Manager for QA Janet Bracewell and her team maintain a thorough - and evolving - software testing strategy. This strategy includes a variety of local and cloud-based tests with comprehensive end-to-end testing with accessibility checks, regression testing, and API testing. Further complicating execution is that tests are performed in several different environments: a developer’s sandbox, an integrated development environment, and the production environment. 

Letting Testers Focus on the User Experience, Not Test Management

Prior to mabl, SmugMug QA primarily relied on manual testing to ensure a delightful user experience across thousands of possible user journeys. Though Senior Test Engineers Chris Skopec and Dan Laabs automated some tests with Nightwatch.js, scaling automated testing proved to be a challenge. But even when automated testing worked as intended - without flaky or broken tests - Chris and Dan found themselves focused on maintaining test infrastructure or test maintenance. When combined with their exploratory testing and other manual testing responsibilities, automated testing remained limited to approximately 70 test cases.  

The small scale of automated testing meant that the overall value of that testing was limited. Even when tests failed, diagnosing the cause was time-consuming. Dan and Chris spent too much time triaging failed tests, only to realize the problem was the test itself, not a bug. The lack of test review tools proved to be just as limiting as code-heavy test creation. 

The high learning curve for code-heavy testing frameworks also hindered the impact of automated testing at SmugMug. Their QA team consisted of nine members, most of whom were focused on manual testing. Balancing the need to maintain their existing testing strategy with learning test automation was difficult, especially considering the scale and complexity of SmugMug’s user interface. The user experience was always the QA team’s number one priority, so automated tests remained only a small part of their quality practice. 

Mabl’s low-code test automation platform made it possible for more SmugMug people to learn automated testing. Since starting with mabl in 2020, Chris and Dan have expedited the automated test creation process significantly, onboarding QA people and developers in an effort to democratize testing across SmugMug. One QA team member, Mike Hurst, has been able to master automated testing in addition to his manual testing role, and several more team members contribute regularly to test writing. Even as more people become regular mabl users, Chris and Dan have more time to support them since mabl handles all infrastructure requirements. The result is more routine testing that builds team confidence in product quality. 

Routine Testing Builds a Culture of Quality

Code pushes used to take place on Thursday nights, and there was always some stress as new code became customer-facing. Routine automated tests created with mabl have made it possible for the SmugMug team to deploy “any time--all the time,” without the team working overnight hours. The simplicity of low-code testing has enabled more people to contribute to testing, shifting how the entire organization views testing.

A few developers have become much more comfortable executing local tests while they code, making it easier to manage quality across the development pipeline. Chris, Dan, and the rest of the QA team have implemented automated end-to-end testing, regression tests, and API tests as part of a comprehensive testing strategy that covers the entire user experience. With less maintenance and diagnostic work needed to manage existing tests, the QA team has been able to increase their overall test count to 425 tests. Though the volume of automated tests has dramatically increased, time spent on maintenance hasn’t seen a corresponding increase. Dan and Mike have instituted a “Rolling Re-Review,” which ensures that all automated tests are routinely reviewed for effectiveness, flakiness and adherence to current best practices. This not only supports better testing, but a better overall quality strategy. 

In one case, the ability to easily scale test plans and report trends made it possible for the SmugMug team to track down a ghost bug that only appeared sparingly when web traffic was exceptionally high. Though their organization had plenty of “anecdata” that proved the bug existed, the inconsistency made it difficult to identify what caused the issue. Mabl’s test automation platform enabled SmugMug QA to multiply test plans runs at a scale that allowed patterns to emerge, ultimately leading to the resolution of a long-term (though occasional) issue. Testing at scale with mabl means that SmugMug can deploy at any time with confidence, knowing that any feature that has been through their test environment functions beautifully. A culture of quality has made everyone more confident in their product - even as change happens more frequently. 

Faster, more consistent test execution across local runs, cloud runs, and CI/CD headless pipeline runs has been instrumental in showing the value of testing across the development organization. More frequent testing acts as a barometer of site health, flagging issues faster than previously possible. The entire SmugMug team sees the value of testing as more bugs are caught in pre-production and fewer tickets are created by SmugMug’s Heroes (customer support). 

A Quality Partnership 

Though the mabl test automation platform has enabled SmugMug to transform how their organization thinks about testing, it’s the partnership behind the platforms that really makes a difference for Janet’s team. Since starting with mabl in 2020, the SmugMug QA team has worked with mabl folks from across the organization, and every one has supported SmugMug as they evolve their testing strategy. In one recent case, the SmugMug QA team was able to add automated accessibility checks to their existing end-to-end tests in mere minutes as part of mabl’s Early Access Program for beta features. The entire organization could quickly realize the value of these new testing capabilities, and another facet of the user experience is now factored into the SmugMug automated testing strategy. 

The wholehearted support from mabl has made the SmugMug team even more confident in their ability to continue delighting photographers around the world with beautiful, secure photo sharing experiences. For Janet and her team, that’s picture-perfect quality.