Have you heard the ancient fable, “The Boy Who Cried Wolf”? In the tale, a shepherd boy repeatedly tricks the people of his village by crying out that a wolf is about to eat his sheep. The villagers run to his aid, only to find that it was a prank. One day, the boy really does see a wolf. He cries for help, but none of the villagers come because they are convinced that it must be another trick.
Similarly, we do not want to be “The Tester Who Cried Bug”. As software testers, if we repeatedly report bugs that are really user error, our developers won’t believe us when we really find a bug. To keep this from happening, let’s take a look at the things we should check before we report a bug. These tasks fall into two categories: checking for user error and gathering information for developers.
Check for User Error
- The first thing you should always check is to verify that the code you should be testing has actually been deployed. When I was a new tester, I constantly made this mistake. It’s such a waste of time to keep investigating an issue with code when the problem is actually that the code isn’t there!
- Are you testing in the right environment? When you have multiple environments to test in, and they all look similar, it’s easy to think you are testing in one environment when you are actually in another. Take a quick peek at your URL if you are testing a web app, or at your build number if you are testing a mobile app.
- Do you understand the feature? In a perfect world, we would all have great documentation and really well-written acceptance criteria. In the real world, this often isn’t the case! Check with your developer and product owner to make sure that you understand exactly how the feature is supposed to behave. Maybe you misunderstood something when you started to test.
- Have you configured the test correctly? Maybe the feature only works when certain settings are enabled. Think about what those settings are and go back and check them.
- Are you testing with the right user? Maybe this feature is only available to admin users or paid users. Verify the criteria of the feature and check your user.
- Does the back-end data support the test? Let’s say you are testing that a customer’s information is displayed. You are expecting to see the customer’s email address on the page, but the email is not there. Maybe the problem is actually that the email address is null, and that is why it is not displaying.
If you have checked all of the above, and you are still seeing an issue, then it’s time to think about reporting the bug. But before you do, consider the questions the developer might ask you when he or she begins to investigate the issue. It will save time for both of you if you have all of those questions answered ahead of time.
Information for the Developer
- Are you able to reproduce the issue? You should be able to reproduce the issue at least once before logging the bug. This doesn’t mean that you shouldn’t log intermittent issues, as they are important as well; but it does mean that you should have as much information as possible about when the issue occurs and when it doesn’t.
- Do you have clear, reproducible steps to demonstrate the issue? It is incredibly frustrating to a developer to hear that something is wrong with their software, but have only vague instructions available to use for investigation. For best results, give the developer a specific user, complete with login credentials, and clear steps that they can use to reproduce the problem.
- Is this issue happening in Production? Maybe this isn’t a new bug; maybe the issue was happening already. This is especially possible when you are testing old code that no one has looked at or used in a while. See last week’s post, The Power of Pretesting, for ideas on testing legacy software.
- Does the issue happen on every browser? This information can be very helpful in narrowing down the possible cause of an issue.
- Does the issue happen with more than one user? It’s possible that the user you are testing with has some kind of weird edge case in their configuration or their data. This doesn’t mean that the issue you are seeing isn’t a bug; but if you can show that there are some users where the issue is not happening, it will help narrow the scope of the problem.
- Does the issue happen if the data is different? Try varying the data and see if the issue goes away. Maybe the problem is caused by a data point that is larger than the UI is expecting, or a field that is missing a value. The more narrowly you can pinpoint the problem, the faster the developer can fix it.
The ideal relationship between a tester and a developer is one of mutual trust. If you make sure to investigate each issue carefully before reporting it, and if you are able to report issues with lots of helpful details, your developer will trust that when you cry “Bug”, it’s something worth investigating!
Great post 👍 I just want to add that under certain circumstances (in Android apps, for example) bugs remain as floating ones. Nevertheless you spent a lot of time to locate it. Usually I comment in the bug report how often bug reproduces (3 times from 10, for example).
Hi Ilya- I'm glad you like my post! That's a great point about intermittent bugs. Letting the developer know how often you see the issue, like 3 times out of 10, is very helpful.