Lately, I have seen some confusion at the level of testing that should be done when a daily build is generated. How do we know that the daily build is ready for general consumption? What’s the difference between BVT and smoke testing?
Although general definitions vary throughout the industry, below, I will provide our definitions of smoke tests and build verification tests, which is based on how Microsoft does this testing. The exact definitions and terminology are not important – what is important is that your team shares a common vocabulary for referring to the different types of tests.
Smoke tests
A “smoke test” is a cursory test that ensures the basic functionality of the application works. The primary outcome is validating that the build can be considered for further testing, and ultimately, early problem detection. If a smoke test fails, there are serious blocking issues that need to be addressed quickly. Early detection is important so that a larger group of people does not become blocked by installing a bad build.
Smoke tests are:
- Extremely fast to run
- Ideally, automated
- Run on every build, including interim builds generated throughout the day
- Focus only on critical functions (e.g., a user can log in)
- Shallow and wide (touch as many parts of the app as possible)
- Small in number (i.e., 1 or a few smoke tests per build)
- Typically built into other, more thorough, test cases
- Executable in minutes (not hours)
Build verification tests
BVTs, build verification tests (also called build acceptance tests or sanity tests), are a superset of smoke tests, but in some cases, the terms are used interchangeably. They are the top priority test cases that exercise basic functionality in the build, slightly more thoroughly than smoke tests. BVTs ensure that the daily build is usable for testing.
BVTs are:
- Still run as part of the overall test pass (your highest priority tests)
- Typically automated
- Run immediately after a daily build is produced
- Executable in minutes
- Executable in minutes (not hours)
The term “smoke test” comes from the hardware industry. Once a circuit is created, the very first test that a hardware engineer performs is turning on the power. If it catches fire and / or smokes, you know you have a problem. No functionality is tested at this point, other than powering on. No smoke means that further functional testing can be performed (or that you have no power at all!).
One could argue that the term “smoke test” should be replaced for software, but it is a very widely used term in the software industry.
