The Hasty Generalization Fallacy is a common one in software testing. But before we look at its impact on testing, let’s learn what it is. This fallacy occurs when someone draws a conclusion based on just one example, or a few examples.
You may have fallen for the Hasty Generalization Fallacy as a child when you met someone from another country for the first time. If they were very nice, you may have concluded that everyone from that country is nice. If they were cold and unfeeling to you, you may have concluded that everyone from that country is cold and unfeeling. This is silly, because countries have millions of people and it’s unreasonable to assume that everyone in an entire country will have the same personality!
This fallacy is very dangerous in software testing, because it results in us not testing enough. If you run one or two tests on a feature, and from those tests determine that the feature is working just fine, and you stop testing, you may miss important bugs. Here are some examples where this might happen:
• Running a passing test in the QA environment, and assuming that it must work in the Production environment without actually checking
• Running a passing test with one Admin user, and assuming that it must work with other types of users, even those that have different permission levels
• Running a passing test on an Android device, and concluding that it must work on an iOS device as well
Recently, I got this question from a team manager: “Why are we expected to test in the Stage environment when we’ve already tested in the QA environment?” There are so many reasons!
• Data could be different in the Stage environment, which might expose a missed bug
• APIs that the application is dependent upon could be at a different state of release, exposing a bug that might then go on to Production
• Connections, such as those that go to a database or call an email service, could have the wrong connection strings. If the connections are wrong in Stage, they could also be wrong in Production.
Before you stop testing a feature, ask yourself whether you might be committing the Hasty Generalization Fallacy. Think about the other tests you could run to make sure that the feature is really working as expected. Always ask yourself, “What else could I test?”