DevTestOps According to the Experts

DevTestOps According to the Experts

Talking about DevTestOps in the abstract helps us understand the concepts behind the practice. I asked the following industry experts about various aspects of DevTestOps. They talked about topics such as creating confidence in DevTestOps practices, risk mitigation, challenges in test automation, addressing testing gaps, and building relationships among team members. The results of my interviews appear below.

Ashley shares her experience in testing and engineering productivity in her keynotes and workshops at international conferences. As a member of the Selenium Project Steering Committee and co-chair of the Selenium Conference, she promotes diversity and inclusion in the industry, and supports open source.

Federico Toledo, PhD., CPIO & Director, Abstracta

Federico is well known Quality Engineer and co-founded of Abstracta. Abstracta is a world leader in quality assurance and testing focused on improving the performance of software applications. Federico and Abstracta help companies implement DevTestOps practices in the modern enterprise.

Abby Bangser, senior DevOps practitioner and international speaker

Abby is a hands-on DevTestOps practitioner working on quality challenges across a diversity of business domains and tech stacks. She is presently focused on applying exploratory testing skills to system and platform challenges.

Elisabeth (Lisi) Hocke, Principal Agile Tester, FlixMobility Tech GmbH @lisihocke

Lisi is a senior Agile tester who's passionate about the whole-team approach to testing and quality as well as the continuous learning mindset behind it. Lisi has years of experience applying modern, Agile based testing principles to building great products which deliver extraordinary value. Building great products together with working great people is what motivates Lisi and keeps her going.

 
Michael Hüttermann, Principal DevOps Consultant, huettermann.net

Michael is an independent DevOps consultant with an international practice. Michael’s work focuses on DevOps engineering services in the areas of Continuous Delivery, DevOps, SCM/ALM.

On Creating Confidence in Product Quality

Lisa: What is the most effective thing that you recommend to give teams confidence in the software product they are deploying?

Ashley Hunsberger:
The most effective thing I've seen is faster feedback, and more layers of tests that were answering different business questions! We're used to deliver code where changes may not have been tested for weeks (sometimes months).

As we moved to faster feedback on every single pull request (or earlier if a developer so choses), and different suites that gave us different feedback - we saw confidence in our products increase. It was no longer a massive regression suite run every few weeks, or nightly.

Federico Toledo:  
For me, it would be having feedback loops at different levels. For example:

(1) Continuous Integration engine with different verifications (test scripts, performance tests, code analysis, etc.)

(2) Testing which is done by humans.

(3) Monitoring production, getting a variety of information (JS errors, response times, resources consumption, adoption and effectiveness of the changes, new features and so on)

Abby Bangser:
The thing I focus on is moving away from a fail safe focus and into a safe to fail mentality. What I mean by that is the confidence in a consistent, repeatable, reliable change process which means if anything does go wrong we can have confidence we can fix/improve it.

Elisabeth (Lisi) Hocke:
This depends a lot on the context of the product, but the first thing that comes into my mind is an effective and well-tested rollback strategy, including data.

The second thing would be alerting and monitoring so the team has the chance to quickly notice that something critical happened which requires a rollback. Other teams out there showed already that some checks can be automatically executed on a pre-production system which can automatically trigger a rollback before actual users are affected, which I see as a great approach to catch certain issues, but I would not assume we can ever think of all scenarios so sometimes critical issues will slip through to production. Therefore we need a good strategy what to do in these cases so we can still confidently try out things.

Michael Hüttermann:
The most effective attribute is internal quality. Good quality improves confidence dramatically. This means a team should strive for actions such as increase test coverage, find a good mix of complementary test categories, and build up a high degree of automation.

Following the t-shaped model, the whole team should have strong awareness for quality, however, you most often need dedicated testing professionals, as part of the team, with their experiences and skills. Some DevOps guys think everybody can do anything, or you can completely skip important roles. This may work perfectly, depending on requirements and basic conditions, however, often a better way is to appreciate Adam Smith with his idea of division of work.

On the Biggest Challenges in Test Automation

Lisa: What do you see as the biggest test automation challenge for continuous delivery?

Ashley Hunsberger:
I still hear 'but I need to test everything.' This is a huge challenge to overcome because our comfort zone, as humans, is in what we can see. So, naturally, people tend to think they need more UI testing. Educating teams on risk-based testing, and more to a 'automate the right things, not all the things' approach is a challenge, but worth pursuing.

Federico Toledo:
My first response would be the maintainability of scripts. But on second thought, I'd say that it is defining a good test automation strategy, which includes the former. This is also related to the design of the pipelines in your CI engine. Which test scripts you will run, at which level (unit, API, UI), where (environment) and when (critical tests early, full regression daily, whatever works better for the team, but define it accordingly).

Abby Bangser:
In many ways I see it as the same as any other delivery mechanism. Knowing what to test and where to test it. I think that this is that much more magnified when it comes to continuous anything because time is of the essence and you have to have high confidence in the automation as human eyes won't be around to question things.

Elisabeth (Lisi) Hocke:
So far I've never worked in a team that actually does continuous integration and delivery according to their definition. The biggest challenge might be to decide which automated tests to run in which step of the build pipeline to get quick and actionable feedback as early as possible, and which to run regularly or even continuously on production. Furthermore, these tests might not only be checking the product but also functionality or quality attributes of the infrastructure the product is running on.

Michael Hüttermann:
There are a lot of challenges impacting the team on a daily basis, e.g. a sudden collapse of initial process requirements or team commitments. It is a long way to build up trust and high degree of quality, but it is a quick action to destroy any trust if bypassing quality gates is more the rule than the exception.

Challenges also include the pitfalls I've identified in my DevOps book some years ago particularly the law of marginal costs, verb/noun mistake as well as paradox and irony of automation.

On Filling in the Testing Gaps

Lisa: What other gaps in testing do you see affecting teams’ ability to successfully build a DevOps culture?

Ashley Hunsberger:
There are a few things that I think affect the team's ability to build a devops culture:

- Testing in production. I often hear, "you want to do what?" This is a new realm as we move to Continuous Delivery, and it makes people squirm (even more than NOT testing everything we can imagine). Design experiments, think through the work to safely execute those experiments, and put that sales hat on to get there.

- Ability to fail and learn from that. Part of learning is not getting things right all the time. While we say we want a culture of learning, are we really fostering that without punishing when things don't turn out quite as we planned?

Federico Toledo:
Communication and human relationships. Confidence is crucial, not only in the software and tools you have but also with your team members and colleagues. To build this trust, it is important to work on human relationships and how team members communicate with each other, in order to have teams emotionally solid, and capable of solving any kind of problems they face.

Abby Bangser:
I think ideas about testing ownership is a huge potential barrier. If people do not see quality (and in turn the tests which help validate quality) as a team owned activity then DevOps can not be achieved.

To extend this further, if people define quality as delivering stories to sign off (rather than through and including end user impact) then the team may be under achieving in DevOps as it is cutting the feedback loops short.

Elisabeth (Lisi) Hocke:
This is not so much about testing, but I see knowledge silos as the biggest gap. I saw dedicated "DevOps" people getting hired who only worked on infrastructure topics without sharing their knowledge. They quickly became bottlenecks and when they were not available or even left the company, nobody knew our infrastructure and how to solve our problems. In my current team this issue got only remedied by using the mob approach, working as whole team together on the same topic, same time, same place, on the same computer. Sharing the existing implicit knowledge and building up new knowledge together, instantly sharing it across the team. This way we managed to heavily increase our lottery factor in this area!

On Building Relationships

Lisa: How can testing/QA professionals build relationships with Dev and Ops?

Ashley Hunsberger:  
Pair! Mob! Give everyone some insight into how we think and test, coach them to learn our skills. Learn about their skill sets! Show interest in each other as human beings, not just cogs in a wheel. Build empathy for each other! Ask, "is there anything I can do to help you today?"

Federico Toledo:
In the same way that they would with any other person. By asking "how are you?" and really taking an interest in the person and getting to know about their families, their likes, hobbies, etc as much as which skills they learned in the last training session.

Something that happens in Latin America is that we kiss each other more frequently, it is easier for us to be closer as a result of our culture. We like to welcome people, to make teams, we enjoy working together, sharing a mate or a coffee, and we like to talk. This really helps us to improve the way we interact with different teams. If you improve your personal relationship, for sure you will improve your professional relationship with these people as well.

Abby Bangser:
Show value and build communication channels. Basically helping each side better understand the other as well as better articulate their pain points into business cases worth investing in. (e.g. investing in better CI tools etc)

Elisabeth (Lisi) Hocke:
In my experience you can only build real relationships when you're actually working together on a topic. Through collaboration you get to know each other better and can develop a sort of empathy for each other's role, perspective and everyday life. I guess you can have only a sort of customer-client relationship, however, I don't think that's providing the most benefits as it can hinder knowledge sharing a lot.

Michael Hüttermann:
Testing should be an essential part of both Dev and Ops. Testing professionals often have a strong background with Dev, and I'd like to implicitly include testing professionals to the team, to make up the Dev team. In other setups, testing pros did cover Ops parts already, and they applied DevOps concepts for many years already, even before the term DevOps was coined.

With customers, I often emphasize that quality should be an inherent part of the process and the product, for many reasons. From one perspective, DevOps is just an evolutionary step to broaden good Agile Dev practices to Ops, thus testing professionals can contribute to the whole team in many ways, on many levels, including helping to define and to achieve shared goals, shared concepts, and shared tools - the DevOps triangle.

Putting It All Together

DevTestOps empowers the tester to be a campaigner who advocates for every teams' goals, from product, to business, to customer, to development. Testers must be in every conversation; in the early planning stages to understand the product requirements, in the ops stage in production to understand the customers, and the business side to understand the intents behind the software. DevOps unites the silos in software delivery, but DevTestOps unites the silos in the entire business.