In this article series, I’ll show you a test strategy I’ve used with several different companies to achieve high quality and productivity.
Test automation saves money, improves time to market and raises quality, but it’s also expensive. This test pyramid gives a guideline on how to allocate your efforts.
I’ll give you the basic definitions, roles, success measures, best practices and some possible maturity levels — enough info to put a functional test strategy in place for your project to achieve high quality at a reasonable cost.
What does this automation strategy do?
This is a general-purpose test automation strategy that is designed to:
Improve and maintain developer productivity
- Minimize duplicate testing
- Making it fast and easy to validate code changes as soon as possible
- Help developers know when they have “enough” test automation
The pyramid shape comes from the goals of minimizing test duplication and putting tests as close to the thing they’re testing as possible. For example, once you’ve tested all the variations of the login functionality through unit and component testing, you need only a couple additional variations as integration tests (positive and negative), and the end-to-end testing only has the basic positive login as part of larger tests. You don’t need to test all the login variations through the UI.
What does it NOT do?
This strategy only covers functional automation. For a complete test strategy, you still need:
- Non-functional automation like performance and scale testing
- Manual and exploratory testing
- Tracking of code and quality metrics
- Tracing customer value to test result
and for services:
- Passive and active monitoring of the production site
- Experimentation (A / B testing, betas, feature flags, etc.)
- Scale and resilience testing in production
Finally, the test pyramid strategy is very efficient, but not necessarily the most effective strategy for what you may need right now. For example, it focuses a lot of effort on unit tests, which help a lot with productivity, but help a lot less with quality unless you have complex logic. If you have quality problems, do some root cause analysis. Unit tests may not be the first thing to fix.
The blog series …
In these posts, we’ll go over each test type. We’ll detail their purpose, who is responsible for them and how to measure them.
- Structure of the test automation pyramid
- Unit testing
- API / Component testing
- Integration testing
- End-to-end testing
- In what order should functional test automation be implemented?
Summary
As companies ship software faster, test automation is needed more than ever to maintain quality. Good automation can reduce your time to release from months to hours. It can document your code and improve quality and productivity for the team.
However, test automation is also a big investment, so before you drop a few $100K on an offshore UI automation vendor, consider how to use this pyramid test strategy to create the right mix of functional testing for your product development roadmap.
Copyright © 2019, All Rights Reserved by Bill Hodghead, shared under creative commons license 4.0
