Budgeting for Test Automation in 2021

Despite months of feeling like it would never end, 2020 is coming to a close. For most of us, that means a sigh of relief and a lot of Zoom calls with our loved ones. But for those in the QA and software development fields, it also means working out a budget for next year, which is a challenging task in the best of times. To get the inside scoop on how to make a compelling case for test automation, we spoke to Andrea Wood, Senior Director of Software Engineering at Barracuda Networks about: 

• How to know when it’s time to explore new tools
• Ways automation can help your team be more productive
• How building a business case can justify the investment in the right solution

As an enterprise security provider, Barracuda Networks offers a wide variety of products that protect email, cloud and application security, as well as corporate networks. This means that their engineering team is typically developing and testing multiple projects at once. As Senior Director, Andrea is responsible for managing several teams and optimizing their development process - no easy feat as demand for security products surged in 2020. 

On a macro level, poor software quality is the elephant in the room, costing American enterprises an estimated $3 trillion in 2018 alone. Nearly 20% of that - $480 billion - was caused by finding and fixing defects. For companies providing critical security services like Barracuda Networks, the costs of faulty software could exponentially increase if customers were put at risk. 

In addition to the financial risks of poor software quality, customer satisfaction is a key metric that has a strong connection to software testing. In our annual "State of DevTestOps” survey, we asked an audience of CEOs, managers, developers, and software testers how they think about their user satisfaction as well as their test coverage: 

The results clearly show that the better test coverage an organization has, the happier their customers are with the product. Particularly in security, where poor UI has resulted in serious privacy issues and the market is extremely crowded, supporting a positive customer experience is crucial to success. 

Ch-Ch-Changes

Given the stakes of software quality, it’s no surprise that convincing your team to make a major change like the shift from manual to automated testing is a major challenge. But as Andrea stated:

 “The walls are breaking down between QA and [development] and CI/CD and DevOps. And for us, automation is now this first class citizen in being able to accelerate our deployment pipeline.” 

She notes that in their previous testing system, Barracuda Networks relied more heavily on manual testing and there was a disconnect between QA and product teams. That increased the handoff time between the two and resulted in a longer calendar cycle for development. As mabl is deployed among more teams, automation is becoming a given and testing is implemented much earlier in the pipeline. This allows them to do more continuous deployments and even deploy based on the risk of a change. 

The key to convincing your organization to make the change from manual to automated testing is metrics: 

• Speed: historically tests have taken a long time to create, usually due to a difficult UI or required scripting that makes the process clunky and difficult to learn. 
• Arch Insurance achieved 98% faster test creation with mabl’s automated testing. 
• With mabl, the low-code UI helps users to quickly create flows. These flows can later be reused across all tests, to save time and reduce maintenance. 
• Cost Savings: with manual testing, a large proportion of users’ time is invested in maintaining or repairing tests. To effectively communicate the need for automation, break down the cost of maintenance for manual or even open-source testing frameworks. 
• ITS, a corporate travel solution provider, reduced costs with mabl by 80% over Selenium. 
• The reduced time spent on test maintenance enabled the ITS team to spend more time on exploratory testing. 
• Test Coverage: As explained above, customer satisfaction has a strong correlation with test coverage. More coverage in less time equals happier customers and a healthier bottom line. 
• MaestroQA discovered that higher test coverage increased their daily deployments by 50 percent. 
• As a third-party call center quality assurance software vendor, MaestroQA had a minimum test coverage quota that needed to be hit before shipping any code, which greatly increased deployment time with manual testing. 

These metrics were the fundamental building blocks for Andrea as her team built the case to move to automated testing. During Barracuda Networks’ proof of concept with mabl, developers reported creating tests 3-4x faster compared to their existing testing tools. Test coverage also proved critical as Andrea says, “you can’t improve what you don’t measure.” With test coverage closely monitored on each sprint, management above Andrea was able to use test coverage as a key metric in understanding how test automation impacts product quality. 

Particularly with budget restrictions tightening in 2021, presenting a business case backed by solid numbers can be the make-or-break factor to having budget approved for automated testing. As Andrea mentioned, a PoC period or mabl free trial can give your team time to establish concrete metrics to convince management to make the investment in 2021. 

Engineering quality across the organization

The benefits of the mabl free trial also extend to quantifying “soft” metrics like collaboration and employee satisfaction. With integrations in common tools like Jira and Slack, mabl supports increased collaboration across the entire DevOps pipeline, particularly between product and QA teams. This “quality engineering” mindset, where QA and development work together throughout the development cycle, shortens the time to identifying failure results and resolving issues, giving back time to your employees that’s usually spent on frustrating manual tasks and improving their quality of work life.  

In Andrea’s case, test automation with mabl has transformed how her teams work. Compared to the traditional model with siloed development, QA, and Ops people, the lines are blurred and testing is happening upfront. Functional testing is now a common occurrence and engineers have greater contextual knowledge of how customers actually interact with the product. Genuine test-driven development requires all teams to work together to build better products. With automation, testing truly becomes a team sport. 

The selection process

Buying software is something we all do in our regular lives, especially in the time of COVID-19 and remote everything. But at the enterprise level, the process can be confusing and even outright convoluted. 

Some key steps to have in mind:

• Understand the people involved: most organizations will have a process in place for new purchases. If you can understand the perspective of each stakeholder involved at the onset, you’ll be able to make a better, more complete business case to each one. 
• Baseline criteria and success: have a mission and quantifiable requirements and goals in mind from the very beginning. This will not only help you evaluate different vendors and products, but also convince non-QA or development stakeholders why the purchase is needed and how the solution will impact the business.
• Have an implementation plan: Rome wasn’t built in a day, and neither will a new workflow. Be prepared with the basics of a plan to adopt a new tool and manage use questions in advance.
• Security first: Ensuring that a new product can fit into your security protocols is a major step. As a large security vendor, Andrea notes that Barracuda Networks has a security team, compliance team, and legal team that all play a role in managing cybersecurity. Consider the security posture as an early part of the process in evaluating vendors so it doesn’t become an obstacle at the conclusion of the acquisition process. 

• You have to walk before you can run: it’s easy to zero in on your team’s use of a new tool, but as Andrea discusses in the presentation, testing is ubiquitous. As her teams began using mabl, other teams came out of the woodwork to experiment with its automated testing capabilities. Teams often start with an application group and then spread adoption across the organization as new projects or new teams emerge. 

• Timing: Understanding the purchasing calendar for your organization helps ensure that your proposal is accepted by each stakeholder in the acquisition process. As Andrea mentioned, her team needed to work with legal, compliance, and security to adopt mabl and had she timed the purchase at an inconvenient time, her request may have been delayed or denied. 

• Understand the internal budget cycles and approval timeline as much as you can and plan accordingly. Asking co-workers that have been through the process previously can help provide tips for managing the process and reveal otherwise unforeseen time barriers that could potentially hold up your purchase. 

Key Takeaways

Data will be one of the keys to helping you drive decisions and adoption. Poor software quality is reported to cost enterprises billions. Understanding how it can or has impacted your organization’s customer happiness and retention can help you build a strong case to improve testing with automation.

Establish the baseline metrics that demonstrate business impact for your organization. Speed, cost savings, and test coverage can be valuable data points worth measuring and communicating to management. Look beyond hard metrics to understand the full impact of automation - collaboration between teams, employee happiness, and customer satisfaction - to paint a bigger picture of test automation’s value. 

We heard from many of our customers this year that purchases of all sizes have been scrutinized. That doesn’t mean that purchases weren’t happening, just that it’s more important than ever to make sure you do your due diligence and manage the process. 

Assuming this trend (scrutinized budgets) will continue at least for a little while longer, you’ll benefit from taking the time to understand how the purchases are made at your organization - know and proactively communicate with key stakeholders, assemble the most important metrics for success, and try to align with internal purchasing cycles where it makes sense. Taking the time to get organized in these areas will help you put your purchase on the path to approval. 

With these factors in mind, you’ll be prepared to work with your organization to invest in a test automation program that can adapt and scale from the onset. To learn more, watch our recent webinar about building your business case for test automation:

If you’re looking for an automated alternative to manual testing, or just want to improve test coverage and reliability in 2021, sign up for a free trial today to see for yourself how mabl compares and let us know how we can help with your evaluation!