The term “smoke test” is usually used to describe a suite of basic tests that verify that all the major features of an application are working. Some use the smoke test to determine whether a build is stable and ready for further testing. I usually use a smoke test as the final check in a deploy to production. In today’s post, I’ll share a cautionary tale about what can happen if you don’t have a smoke test. Then I’ll continue that tale and talk about how smoke tests can go wrong.
Early in my testing career, I worked for a company that had a large suite of manual regression tests, but no smoke test. Each software release was difficult, because it was impossible to run all the regression tests in a timely fashion. With each release, we picked which tests we thought would be most relevant to the software changes and executed those tests.
One day, in between releases, we heard that there had been a customer complaint that our Global Search feature wasn’t working. We investigated and found that the customer was correct. We investigated further and discovered that the feature hadn’t worked in weeks, and none of us had noticed. This was quite embarrassing for our QA team!
To make sure that this kind of embarrassment never happened again, one of our senior QA engineers created a smoke test to run whenever there was a release to production. It included all the major features, and could be run fairly quickly. We felt a lot better about our releases after that.
However, the tester who created the test kept adding test steps to the Smoke Test. Every time a new feature was created, a step was added to the smoke test. If we found a new bug in a feature, even it was a small one, a step checking for the bug was added to the smoke test. As the months went on, the smoke test took longer and longer to execute and became more and more complicated. Eventually the smoke test itself took so much time that we didn’t have time to run our other regression tests.
Clearly there needs to be a happy medium between having no smoke test at all, and having one that takes so long to run that it’s no longer a smoke test. In order to decide what goes in a smoke test, I suggest asking these three questions:
1. What would absolutely embarrass us if it were broken in this application?
Let’s use an example of an e-commerce website to consider this question. For this type of website, it would be embarrassing or even catastrophic if a customer couldn’t:
- search for an item they were looking for
- add an item to their cart
- log in to their account
- edit their information
- wish list functionality
- product reviews
- recommendations for the user