Migrating to Low-Code Test Automation

Community Brands empowers associations, non-profits, and schools to manage communities, fundraising, events, and more. Product development moves quickly at Community Brands with UI transformation, adding new features in the product and existing code updates to their apps, which also means frequent, fast-paced testing. Maintenance of test scripts was one of their biggest challenges preventing them from keeping up with development. In this session, Aniket will share how his team identified challenges in their previous approach to testing, their migration strategy to low-code test automation, and tips for navigating through this journey.

Transcript

00:27

Hello, everyone, we are very happy to have you attending Experience. And I'm excited to introduce this session migrating to low code test automation. Before we get started just a few housekeeping items. If you have any questions throughout the entirety of the presentation, please feel free to leave those in the Q&A panel on the right side of your screen. For any comments or discussion, you can also use the chat feature you'll find both of those on the right side of the session page. And when the video is minimized, we will leave questions for the end at the presentation for Q&A. And so with that, I'll go ahead and hand it over and take it away.

01:04

Thanks Zane, Good morning and good afternoon everyone. Okay, first of all, I will start with my introduction. My name is Aniket, and welcome. I'm working from our India office. And I'm having overall experience of around 16 years in quality automation.

02:09

Today, I will be sharing the story of why we adopted mabl as an automation tool for one of the interface products which we call NetForum Enterprise. And we have actually made good progress there and I think I would like to share the story around it for today's session.

Okay, so I think first of all, I will go with the NetForum. What is NetForum? So NetForum is enterprise association management solutions. We are a top rated member management software for the larger associations. We are a part of Community Brands. And we are a leading provider for cloud based software associations, nonprofits, and K-12 schools. For any organization who wants to adopt the Community Brand solution, I think it's very simple. By adopting this, I think they can just manage the memberships, career centers, learnings, accounting related stuff, the fundraising donations, admissions enrollments, and events. So, we have a full package of around 30+ models in our solution. Recently, we are in the process of integrating a nucleus which is the data analytics tool which is adding the analytical capability for our NetForums solutions. And we have also migrated our on devices NetForum product on cloud native Microsoft Azure. For those who are interested I think this is the link, you guys can just go to this link and see the additional information about the NetForum product and in the Community Brands organization. Okay, well so, thank you this is all about my organization and the product. Now we will move to the next slide.

The agenda for today's session is this story about our product automation. So as I mentioned that NetForum enterprise solution is our product and I will just give you the quick overview. How overall we have done the product automation for the NetForum enterprise. At the start of this year we have adopted mabl, and so far, what is the progress we have made and what are the challenges we are facing.

Okay, so, first of all, I would like to introduce the NetForum Enterprise automation technology. So before adopting mabl or before making mabl as our primary automation tool, I think till last year 2021, we were using our in house selenium automation framework, which we had developed. And we were working on this framework for four to five years, around 1000+ test cases we have automated under this framework. And these are a few technologies, which we have used. So, the primary technologies in our in house selenium based automation framework is the programming language that is C#, we use Visual Studio for ID for writing the automation script. Of course, the automation Library is a Selenium library, then we have the test data driven test cases, initially we have developed, then after that we have more of a behavioral driven development approach in the automation. And for that, we are using specflow MDM unit. And I think Jenkins is our CI tool. And then Octopus is our deployment tool, GitHub were using for our versioning. And I think all these are a bunch of technologies. And then of course, we use the virtual machine managers, we have our virtual machines, which we use as a node and we execute the automation test on those virtual machines. So I think this is a heavy in-house automation framework, which we developed. We had been working around four years on this framework. And there are a lot of limitations we found and then we decided that we need to move for something which is really easy to shift upon.

And I think the challenges which we face with our open source automation framework is like access and setup. In the earlier slide you have seen that there are a lot of technologies, we are using a lot of configurations we are required to do on machines before kicking off the executions or before enabling the project automation project on any engineer’s machine. Then with C# as our primary language. And we are using BDD and spec flow. So of course we require some coding skills. And I think that increases our learning curve for this automation framework. We use all the open source related technologies. So we are not getting any technical support available from anybody. As I mentioned that it's a long learning curve, because it’s a bunch of technologies we have wrapped together and built our in-house framework. So engineers need to know everything about that. Then, at its core, the coding skills required for us to write or develop the automation script. And we really don't have any record and playback option there. So, of course, there is more script development time. And test script maintenance since it requires them to write code for developing the script. So if any changes happen on the front end UI side then of course, we need to maintain those scripts and to maintain we need to write again the code to fix the existing code. And of course, there is the first test creation time and then the maintenance time is also more. So these are few challenges we were facing with our in-house automation framework and we are looking for some options which can speed up our automation across the product. And that’s when we came up with mabl.

And I think we decided to adopt the mabl. So I will just go through how we have started our life with mabl. So first thing last year at the end of 2021. We just completed the POC, and in the POC we have downloaded the mabl tool on our machine. And since our product is an enterprise product, we have simple test cases, we have medium and then complex test cases also. Sometimes the test cases are more than 200 or 300 steps and a lot of navigations are involved. So we really need to check that all these things will really work with mabl support; then we require some parameterizations; we required a data driven approach in the framework, running the test cases with multiple data sets; and we really just wanted to check the feasibility before adopting the mabl for our enterprise application. And that feasibility study we completed around one and half month, we were on a trial basis, we automated simple medium, and complex test cases in mabl. And surprisingly, we found that we are able to automate it quicker than our in-house framework. And overall, the stability of the test cases were also good. So I think the feasibility we completed and then we adopted mabl as our primary automation tool in this year, January 2022. Of course, after that, we have got a couple of training sessions for our automation engineers from mabl training team. And in just one and half months, I have seen that all my engineers in my team, they really pick up the mabl tool very well. And they have just started automating the new test cases or features in our product. So for my in house framework, that learning curve was around six months. And here I see I have observed that it's a drastic change and the learning curve is reduced to just one and half one for this tool. So to date, 190 manual test cases have been automated. I have some statistics related to these 190 test cases that will come in my next slides. But then this is a combination of simple medium and complex test cases. And our medium and complex test cases, if any engineer is required to execute it manually, then it takes around one and half hour of execution or two hours of manual execution to complete the test case execution. And we have automated those test cases in our automation repository, currently 190 Manual Test Cases. As I mentioned, this is a good list of low code, and the capability of this mabl tool really help us to speed up our new automation test creation velocity. So there are some charts. For reference I have put them in my presentation and will come in the next slide. The biggest advantage I can see is with cross browser testing. We have been developing our product for the last 15 years. So a lot of revamping, refactoring and then the UI transform transformation happened in our product. And earlier, we have seen that with the help of our in house automation framework, we can just execute those test cases with one primary browser. And to just make those test cases available for cross browser, I think a lot of coding efforts are required. And I think we are really not getting any automation advantage in our UI transformation project. But then with the help of mabl, it has inbuilt cross browser testing support. And I think in just the last 8 months I've observed that these 190 test cases I was able to execute with the other browsers also, apart from our primary browser, Chrome. So I think that is the biggest advantage I can see. Then easy to kick off, and start the automation test execution. So that is also the one biggest advantage I have seen. So as I mentioned in my earlier in house automation framework, we have Jenkins. And we use Octopus as a deployment tool for deploying the latest version of automation code on our VMs or agents. And from there, we kick off our executions. So anything goes wrong there. I think we require, we spend a lot of time troubleshooting those problems. And I think make that VM up, resolve the Octopus and deployment related issues, and then kick off the execution. So I think sometimes we spent a complete day just kicking off the automation test execution. And I think that that is where I'm seeing that in mabl, it's very easy. We can just like on a single click, we can kick off the execution. And it has that capability so we can show you our executions. So we really don't require any manual intervention to start the execution. So I think this is the good stuff. And under the integration of Microsoft Teams, so we recently started using Microsoft Teams as our primary communication channel in our organization. And mabl has a wide range of integration support. And I think we are, for this Microsoft team, I found it really helpful, because after every execution for just the automation team members, they're getting notification in their team's chat that this execution has completed, this many tests have passed, this many tests have failed. And we even get the link from there, we can actually navigate to the mabl site and see the reasons of the failed test cases and any deviation in the product. So I think this early notification is really helpful. And not every time we need to go physically on the mabl site and check the result. So we will just come to know in our teams channels. So I think this is a good integration, which we have used . We are also using the Azure build pipelines integration with mabl. So that is in progress, we have not completed that so far.

Okay, so this is about the snapshot I was talking about, the Microsoft Teams notification. So if I see that this is my plan, we follow the product increment and scale agile framework. So this is the last PI, which we completed in October, and my automation team has automated around 65 test cases in one PI that is two and a half months of time. And we are daily executing those automation test cases. And if you guys see here that of the 64 test cases we have executed, that just four failed, we are getting the exact number of those 4 test cases. And then there are the engineers who have last updated or made the code changes. And yeah, and then if you want to go to see the more details of the output, then there are buttons that are available here that we can just go and view the output. Or we can go and view the plan and these notifications are coming on our automation team of Community Brands. So this is a very cool feature. And I'm seeing a lot of proactiveness on the team. And they're just going in and seeing and fixing the test cases, and making sure that everything is passing in the execution. If anything is failing, then from the product division, I think everything they will fix immediately.

And these are some velocity charts I put. I mentioned that we follow the PI. So this PI is like four, five sprints. And in this five sprint with the help of my four engineers, our team was able to automate the 69 new test cases. And I think this is the velocity of every engineer how many they have activated. And I think with our in-house frameworks, it's a lot of coding involved there. So we could hardly activate 25 to 30 test cases in one PI. So here I'm saying that the output or the velocity of the team has almost doubled. And yeah, and I think in every PI we write around 100 new test cases. And out of that now 70% of the test cases we are attributing in the same PI. So the backlog of 30% test cases, of course, piling up, but then as we were just started a couple of PI's back, so once the team is more familiar with the mabl tool, I think I'm sure that we will activate all the test cases which we are developing in the same PI. And I think we will almost eliminate the manual testing direct manual testing of those test cases. I think this is the PI objective for us as an automation team.

And these are another angle for the same data. So if you guys see that, the first engineer he has like spend of around 21 story points and in the 21 story points he has automated the eight hours and I think the 45 minutes, eight hours and 45 minutes of test case automation. So total around 94.5 story points. And 52 hours close to 53 hours of automation of manual test cases we are completed in this PI.

So I think everything is good so far, as I mentioned that we have adopted mabl at the start of this year, and we have just now around eight to nine months, we are using mabl. And we observed that some challenges with the mabl tool as well. There are a few important challenges we have mentioned here that I think the record and playback, that is the mabl trainer capability, it's good. But then sometimes we observe that if the test case is long, like more than 200 manual steps, or 300 manual steps, and if the recording has started. So in between we observed that sometimes it gets bought, and then we have to start recording, again from scratch. So we already lost the support ticket for this, I think the mabl the support team, they have responded very well, they have given us a couple of workarounds. And we have almost now got rid of this problem. But then I can still see some scope that this particular feature, it's a backbone of the mabl tool. And it has to be stabilized more. Just in the last version of mabl, we got the support of Edge browser, it's in an initial phase. But then I think the execution of the Chrome browser, the Safari browser, and Firefox browser, it's very stable for each browser. We have observed a couple of teething issues. But then apart from that, I think the Edge browser is also a very stable browser. And I think we're getting full cross browser support for automation testing with mabl. I think the last thing, as I mentioned, the need for an enterprise application. So the complex test cases are very long. Sometimes we have a single test case, we automate, I think its execution time is like 40 minutes, sometimes it's more than 45 minutes as well. And when we assume of such a test cases, we observe that the exhibition is going more than 10 hours. And then we observe that sometimes after 10 hours, the rest of the test cases from that suite, it has started aborting and they are getting terminated from the execution. So we again logged the ticket for this. And the solution for this is to start the execution in a staging manner in a test plan. So if we create a test plan in a mabl, and if I have 100 test cases, which takes around a twelve hours of execution, then I need to split those test cases in stage one, stage two and stage three, then I need to start the execution. So will not face this test case termination problem after some six hours of execution. So I think Thanks for waiting for the solution for these challenges. So I think these are the major challenges we can observe. And that's that's all in the challenges. And that's all from my presentation as well. Thank you.