Technical Article, Testing

The Test Pyramid Part 2: Structure of the test automation pyramid

November 5, 2019
Bill Hodghead

This post is part of a series on functional testing using the test pyramid.

I’ve been engaged by five different companies to help them create a quality strategy.

For each company I’ve worked with, I used variations on the same strategy. To create this strategy, I took ideas from all over, but the core came from Martin Fowler.

Martin Fowler uses a simple test pyramid:

Martin Fowler, simple test pyramid — *Figure 1 – Fowler’s simple test pyramid*

This is a nice, simple idea. You’ll do more unit testing and less UI testing. The best testing is the testing closest to the code. The problem with this is that it confuses “UI testing” as both a type of testing and a thing to test. You can often test a UI using unit tests and integration tests. It makes more sense to have end-to-end testing on top (as some others have done).

Martin Fowler and Toby Clemson have another test pyramid that looks like this:

Martin Fowler and Toby Clemson, test pyramid — *Figure 2 – Fowler’s microservices test pyramid*

For this test strategy, I only want to talk about automated testing, so I skip the exploratory part at the top of their pyramid since it’s a manual process. I’m not going to try to quantify how much exploratory testing you should do compared to other techniques. Their pyramid suggests it’s small — and I’m not sure I agree. I strongly recommend doing exploratory testing. For that, I refer you to the books and videos of James Whittaker.

I also disagree with the order of integration and component. In my thinking a component is typically a group of units and components need to be integrated.

Here’s my functional test automation pyramid:

For each company I’ve talked to, the ideal pyramid has been really flat! The general pattern fell out that each layer down has about 10 times the number of tests as the one above. Don’t get too hung up on the number of tests, but if someone says, “We’re going to add hundreds of end-to-end tests,” ask a few questions about why those couldn’t be done with smaller faster, more reliable, component and unit tests.

In general, tests higher on the pyramid find more bugs but run more slowly and take more effort to maintain.

Tests lower on the pyramid are fast and reliable and change less often. They find fewer bugs but do more to keep the code maintainable and documented.

In the next few posts, we’ll dig into the specific layers.