Think Like a Tester

I often talk with testers who are feeling frustration with their manager. Some of their complaints include:
• My manager doesn’t give me enough time to automate
• My manager expects me to test all the sprint items at the last minute
• My manager signs our team up for too much work
• My manager doesn’t appreciate how much work I do

Have you ever struggled with any of these issues? Then it’s time to learn how to manage your manager! Read on for six ideas on how to do this.

1. Think about what your manager wants
   The best way to get someone to behave the way you want is to figure out what they want, and then show them how what you want and what they want align. What does your manager want? Your manager probably reports to a manager themselves, and your manager is probably accountable to their manager for things like releasing software on time and reducing the number of customer complaints that occur after a software release. You want good software to be released on time as well! So when you talk with your manager, point out the ways your ideas can achieve this.
2. Explain how your strategy will help your manager
   Once you know what your manager wants, you can tailor your suggestions to show how your manager will be helped by them. For example, you could say: “I know that we’ve had some issues with defects escaping to Production that have resulted in customer complaints, and I know that your manager isn’t happy about that. I think if we schedule a one-hour Bug Bash before each release, we could catch most of those bugs.”
3. Be a team player
   Are you a team player? Do you show up to work each day with a positive attitude? Do you help and encourage other people on your team? Do you go the extra mile without being asked to do so?
   The kind of attitude you bring to work has a huge effect on whether or not your manager respects and listens to you. Nobody likes working with a complainer. When you are pleasant to work with, your manager will be more likely to want to have conversations with you about how you can work together to improve your team’s processes.
4. Approach your manager with data
   It’s always easier to convince someone of something if you have cold, hard facts backing up your assertions. If you feel that your workload has increased over the last six months, you can show your manager the average number of stories you tested six months to a year ago, and the average number of stories you tested in the latest six months. If you think your manager should give you more time to write test automation, you can use metrics to show how much time would be saved if you had an automated regression suite that could be run with each release.
5. Enlist the help of other team members
   You are not the only person on your team! It’s likely that there are other testers or developers on your team that feel the same way you do. Why not talk to them about the situation? You could share your ideas and listen to their ideas as well. It could be that the Dev Lead on your team has a great idea for limiting the number of sprint items that the team is taking on. If you work together, you can make a convincing case to your manager.
6. Suggest an experiment
   Sometimes you might have an idea that you are sure will improve things on your team, but the team doesn’t agree. They might be resistant to change, or they might think the change represents too much work. In cases like this, you can suggest an experiment: “Let’s try doing Dev-QA handoffs for two sprints. If it doesn’t save any time for us after those two sprints, we can stop the handoffs and go back to the way things were.” It’s been my experience that most of the times I suggested an experiment, the rest of the team realized I was right! But even if that doesn’t happen, you show your manager and the team that you are an innovator and someone willing to try new things.

Managers want what you want: a happy and successful team! By following these six suggestions, you’ll be able to work more effectively with your manager and team to build and deliver software your customers will love.

It’s a new year once again, and time to think about what improvements you and your team can make to increase the quality of your products! One complaint I often hear from testers is that they have become a bottleneck on their team. They feel constant pressure to get their testing done, and they feel that they don’t have time to do good exploratory testing or write quality automation.

In my experience, there are nine main reasons why testing becomes a bottleneck. Read on to see if any of them apply to your team!

Reason One: The team has too much tech debt
When a product has crushing tech debt, testing and fixing bugs both become more difficult. The software often requires careful configuration before running any tests, and any test automation will likely be flaky. When developers fix bugs, it’s likely that they’ll break something in the process, resulting in additional bugs that then need to be fixed and validated. The remedy for this is to prioritize fixing tech debt, which will pay off in faster development and testing processes.

Reason Two: The team is given work without adequate acceptance criteria
Have you ever been assigned to a project that doesn’t have clear user stories and acceptance criteria? It results in the whole team flying blind. No one knows when a last-minute request for a forgotten feature will come in. When you don’t know the scope of a project, you can’t correctly estimate how long the work will take. When the development work takes longer than expected, it’s often the testers who are expected to work miracles and test twice as many features as time allows. The best solution for this problem is for the team to refuse to take on any work that does not have clear user stories and acceptance criteria. If the Product Owner wants to add a last-minute feature, the project deadlines should be renegotiated to allow for extra time to do the development and testing.

Reason Three: The developers aren’t completely coding their stories
Developers can sometimes get so mired in solving coding problems that they forget to go back and check the acceptance criteria of a story to make sure they’ve met all the requirements. They pass the story off to the tester, who quickly discovers the missing requirements and sends the story back to the developer. This kind of story ping-pong can slow things down for everyone. The developer has probably started work on their next story, and now has to switch their context back to the first story. The tester will now have to find something else to work on while the story is completed, and will then have to switch contexts back to the story once it’s completely ready. If this is happening on your team, calling it out in a team retro meeting might be enough to get the team to understand the problem. They can then develop the habit of checking the acceptance criteria before handing off the story.

Reason Four: The developers aren’t scheduling a handoff meeting with the testers
In a handoff meeting, the developer who worked on a story meets with the tester who will be testing the story. They demonstrate the work they did, show how the work can be tested, and outline which adjoining areas should be regression tested. Then the tester can ask any questions they have about the feature and how to test it. When a team does not use handoff meetings, it’s more likely that the tester will discover that the feature hasn’t been deployed, or that they don’t understand how to test the feature. This results in more back-and-forth questions that slow down the testing process.

Reason Five: There are not enough unit tests
Unit tests, typically created by developers, are an awesome way to catch problems quickly. When the developer pushes their code to a branch, the branch is built and unit tests are run against it. Any functionality that the developer broke with their code is detected in a matter of seconds. They can then fix the problem before handing the feature off for testing. When there aren’t enough unit tests to catch problems, the problems will go directly to the tester, who will then have to report the bugs. The time spent reporting bugs is time that the tester could have been doing exploratory testing and finding more challenging bugs.

Reason Six: The team has not tied API automation to builds and deployments
Almost as useful as unit tests, API automated tests catch any changes in the code that may have introduced bugs into how the API behaves. For example, a code change might have inadvertently changed a response code from a 401 to a 403, which would then result in an error in the UI. If API automation is run with every build and every deployment, problems are detected before the feature goes to the tester, once again saving them time and energy.

Reason Seven: There is no good process for reusing regression test plans

Exploratory testing is a great way to find bugs, but when you have an existing product or feature, it’s important to have some record of what should be tested during a regression. Without a record, it’s easy to forget to test some less-used part of a feature. For example, when testing an e-commerce application, the tester could forget to test removing an item from a cart, and instead focus only on purchasing the cart items. Without a good system for creating and saving regression test plans, the plans will need to be recreated with every release. This takes up valuable time that could be used for testing. Having a set of regression test plans for reuse can save enough time that the tester can do exploratory testing before the release.

Reason Eight: Developers are not contributing to test automation
Yes, good software testers know how to write good test automation, but developers are great at writing code, and they are a valuable resource for writing clean code, creating reusable methods, and setting up mocks for testing. When a team has only one tester who is expected to do all the automation on their own, the process of automating the tests will be slow. It may be worth pointing out to your team that when there are good, reliable, automated tests, it saves time for everyone, not just the tester. Developers will get fast feedback, which prevents time-consuming context switching.

Reason Nine: The team is relying too much on UI automation
There is definitely a place for UI automation, because this is where the visibility and functionality of UI elements are tested. But most of the business logic of an application can be tested with unit and API tests. We all know that UI automation tends to be slow and flaky, even with well-written tests. With unit and API tests, teams can get accurate feedback more quickly. And these tests are easier to maintain because there is less flake.

Do any of these reasons apply to your team? Do most of them apply? In this New Year, I challenge you to pick one of these reasons and discuss it with your team. See whether you can come up with a plan to address it, and watch the testing bottleneck start to speed up!

As you know, this blog has focused for the entire year on logical fallacies. We’ve learned about all kinds of fallacies, from the Red Herring Fallacy to the Appeal to Ignorance Fallacy! It’s time now for the last blog post of the year: the Slippery Slope Fallacy.

The Slippery Slope Fallacy occurs when someone assumes that one negative event will lead to a chain of negative events, causing disaster, when there’s no proof that each event will be the cause of the next.

This is a common fallacy used by parents when they don’t want to let their teenagers do something. Imagine this scenario between a father and his daughter. “If I let you go to the rock concert and stay out until 2 AM on a school night, soon you’ll be staying out until 2 AM every night. Then you’ll be too tired to get up and go to school on time, which means that your grades will suffer, and then you won’t get into a good college.”

The fallacy is obvious to teenagers: staying out until 2 AM one night will not lead to staying out until 2 AM every night, because the parent won’t actually let that happen. The father in this example is using the Slippery Slope Fallacy as an excuse for why he doesn’t want his daughter to attend the concert.

The Slippery Slope Fallacy happens in software testing as well! You may have encountered a well-meaning tester who has found a small UI bug in the team’s application. They log the bug, but rather than letting it go to the backlog, they insist that the bug be fixed NOW. The logic they use goes something like this: “If we don’t make the developers fix this bug right now, it will mean that they will ignore bigger bugs in the future. Then we’ll wind up with a ton of tech debt that we will never be able to get out of, and our application will be filled with bugs. Our customers will desert us and then we will go out of business.”

The fallacy here might be a bit harder to see for testers who feel strongly that their application should be as close to perfect as possible. But here’s the error: putting one small bug on the backlog will not necessarily result in the team ignoring big bugs. A well-functioning team will have a triage process in place where the whole team can determine the user impact of a bug, how important the fix is compared to other tasks the team is working on, and the potential cost of waiting to fix the bug.

Yes, ignoring too many bugs can result in too much tech debt, but a small UI bug that doesn’t impact the functioning of the application is not going to significantly contribute to that debt. It’s important that a tester choose their battles and let some small bugs slide, because if they protest loudly about every bug, the team will stop taking them seriously.

I hope that you have enjoyed my series on logical fallacies! If you would like to learn about more fallacies, I have great news for you! In early 2024 I will be publishing an mini-book called “Logical Fallacies for Testers”, which will include the twelve fallacies I wrote about this year, plus three additional fallacies!

The Appeal to Ignorance Fallacy is an interesting one: it states that something must be true because it hasn’t been proven false.

This fallacy is often used by people who believe in entities like Bigfoot, the Yeti, or the Loch Ness Monster: they will say that no one has proven that Bigfoot doesn’t exist, therefore he must exist! With an example like this, it’s very easy to see the false logic.

The same kind of fallacy is common in software testing as well. Consider this statement: “We know our software is secure because we’ve never had a security breach.” Having no security breaches does NOT mean there are no vulnerabilities in the software. It is possible that there are dozens of security holes in the software, but the company hasn’t grown enough for a malicious actor to decide they are worth exploiting. Some companies might also say “We’ve never found a security vulnerability in our software.” That might be true, but it could be that the reason it is true is because they’ve never looked for vulnerabilities. It’s bad logic, and bad practice, to say something doesn’t exist because you’ve never looked for it.

Another example of the fallacy happens when someone announces that their company’s app is “bug-free”. This is an impossibility. Lack of found bugs doesn’t mean that an application is bug-free. It means that the testers haven’t found any bugs recently, nothing more. A simple application with just two buttons has at least two different testing paths. Add a third button and you have at least six different testing paths. So imagine how many testing paths an application with a dozen different features could have! There is no possible way to test them all, so there is no possible way to prove that the app is bug-free.

Watch for this fallacy when your team is discussing software. When someone makes a bold claim, ask yourself if it has actually been proven, or if it is merely wishful thinking.

Equivocation is a technique used to mislead others through the use of imprecise language. There are many words in the English language that have more than one meaning, such as the word “light”, which could mean “bright”, or it could mean “not heavy”. It’s also possible to use equivocation by being deliberately ambiguous about time or quantity. Children are excellent at equivocation, as you will see in the example below.

When I was a much younger woman, I taught piano lessons, mostly to young students. I gave each student an assignment book in which I would assign their lesson for the week. I expected each student to practice for fifteen minutes a day, six days a week, and the assignment book included a little chart where they could enter their practice time.

I discovered all kinds of ways that children equivocated about their piano practice! When a child’s mother asked her, “Did you practice piano?”, she might answer “Yes”. But upon further examination, it became clear that what she really did was practice yesterday. Other students would mark down the time they spent sitting at their piano bench looking out the window as “practice”. Still others would play the piano, but not play their assigned music, and call that “practice”. And one creative young man recorded one fifteen-minute practice session and replayed it on a tape player every day so his mother would hear him “practicing”.

The same thing happens in software testing. Many terms are used in an equivocating fashion to convey that extensive testing has been done, when in fact it hasn’t. Consider the following examples:

• “Code coverage”: a team could boast that they have 95% code coverage, when many of their unit tests are simply set to return true regardless of their state
• “Automation coverage”: saying that a team has 100% automation coverage could mean that they only ever run ten manual tests and they’ve automated all ten
• “Test plan”: this could refer to anything from a pages-long document to an idea for a few tests to run that the tester thought up while in the shower
• “Test results dashboard”: is this a tool that shows successes and failures over time, highlighting flaky tests, or a colorful page that doesn’t convey any meaningful data?
• “Continuous Deployment”: for some teams, this could mean “when I commit code, it is automatically deployed to production”, or it could mean “after I submit a change control request and it is evaluated, approved, and scheduled, then it goes to production”

I could go on and on. Even the word “testing” has been highly debated. Are automated scripts that exercise an application’s functionality “tests”, or merely “checks”? My point here is NOT to arrive at common definitions for everyone. My point is that it is very easy to equivocate to give an impression of software quality that is simply not true.

As software testers, we owe it to our end users to be honest about our testing practices. This means reporting our activities to our team with clear definitions and metrics.

You may have heard of the term “red herring” if you have ever read a mystery story. When a mystery author wants to keep their readers guessing about who the murderer is, they may throw in clues that point to another suspect. These clues are called red herrings.

The Red Herring Fallacy is similar; rather than addressing an important issue, the speaker diverts attention from the issue by introducing information that might seem to be related, but is in fact irrelevant.

Here’s a real-world example: in the debate about green energy, some environmentally-minded people point out that the use of solar panels puts stress on the planet because solar farms often reduce the number of trees in a community, and because discarded solar panels can fill landfills. Another proponent of green energy might counter this argument with data about how the solar industry is booming and providing many people with good jobs. While this fact may be true, it doesn’t address the actual issue being discussed, which is whether solar panel usage puts more stress on the planet than it relieves.

The Red Herring Fallacy is present in software testing as well! Here are a couple of examples. A company is getting ready for an important release, and a tester finds a bug one day before the release is scheduled. The team begins discussing whether the fix should go in the next day’s release or wait for the following release, but soon the product owner and the engineering manager get in a heated discussion about how the bug was missed. While it might be important to determine how the bug was missed, it’s not relevant to the current issue, which is to decide when the bug fix should be released.

Another example is a situation where customers are complaining about a feature that an engineering team created. The team is asked to explain why their feature was released with so many bugs. The team responds with the fact that that they have a large number of automated tests that run with every release. This fact is not relevant, because the bugs went undetected. The number of automated tests is a red herring; what should be discussed instead is why the team didn’t discover the bugs.

Good software testers and teams know how to stick to the issue at hand when discussing product quality. The next time you are confronted with a problem, remember to focus on the problem and its solution rather than getting distracted by other details.

This month we continue our journey into logical fallacies with Circular Reasoning. Circular Reasoning can be explained in these two statements:

• X is true because Y is true
• Y is true because X is true

A quick examination of these assertions shows that they aren’t proving anything. It’s possible that neither X nor Y are true, but the person asserting that X is true will go around and around with these two statements as if they prove their assertion.

Here’s an example: your neighbor insists that driving over 55 miles per hour is dangerous. When you ask her to prove that it is dangerous, she says that driving that fast is illegal. Consider those two statements:

• Driving over 55 miles per hour is illegal because it’s dangerous
• It’s dangerous to drive over 55 miles per hour because it’s illegal

That’s Circular Reasoning at work!

We also encounter Circular Reasoning in software testing. Consider these two statements:

• All of our automated tests passed because our feature is working correctly
• We know that our feature is working correctly because all of our automated tests passed

At first glance, this seems to make sense. If our tests are passing, it must be because the feature is working, right? But there is something else to consider here. It’s possible that the tests are passing because they aren’t actually testing the feature.

I learned this lesson several years ago when I first started writing JavaScript tests. I was really proud of my tests and the fact that they were passing, until a developer asked me to create a condition where the value being asserted on was incorrect. I was surprised to see that my test passed anyway!

I wasn’t aware of how promises work in JavaScript. When I thought I was asserting that a value was present on the page, I was actually asserting that the promise of the value was present on the page. I needed to add async/await logic to see my test fail when it was supposed to fail.

To avoid circular logic, make sure to challenge your assumptions. Ask yourself, “How do I really know that this is working?” Test your automated tests: each one should fail if there is a condition present that should cause a failure. Additionally, don’t blindly trust metrics. Dig into the data and make sure that the metrics are measuring what everyone assumes they are measuring.

When we are sure that our tests fail when they are supposed to, and when we are sure that our metrics are measuring what they are claiming to measure, we can then have more confidence that our passing test reports and positive metrics are indicating product quality.

The Hasty Generalization Fallacy is a common one in software testing. But before we look at its impact on testing, let’s learn what it is. This fallacy occurs when someone draws a conclusion based on just one example, or a few examples.

You may have fallen for the Hasty Generalization Fallacy as a child when you met someone from another country for the first time. If they were very nice, you may have concluded that everyone from that country is nice. If they were cold and unfeeling to you, you may have concluded that everyone from that country is cold and unfeeling. This is silly, because countries have millions of people and it’s unreasonable to assume that everyone in an entire country will have the same personality!

This fallacy is very dangerous in software testing, because it results in us not testing enough. If you run one or two tests on a feature, and from those tests determine that the feature is working just fine, and you stop testing, you may miss important bugs. Here are some examples where this might happen:

• Running a passing test in the QA environment, and assuming that it must work in the Production environment without actually checking
• Running a passing test with one Admin user, and assuming that it must work with other types of users, even those that have different permission levels
• Running a passing test on an Android device, and concluding that it must work on an iOS device as well

Recently, I got this question from a team manager: “Why are we expected to test in the Stage environment when we’ve already tested in the QA environment?” There are so many reasons!

• Data could be different in the Stage environment, which might expose a missed bug
• APIs that the application is dependent upon could be at a different state of release, exposing a bug that might then go on to Production
• Connections, such as those that go to a database or call an email service, could have the wrong connection strings. If the connections are wrong in Stage, they could also be wrong in Production.

Before you stop testing a feature, ask yourself whether you might be committing the Hasty Generalization Fallacy. Think about the other tests you could run to make sure that the feature is really working as expected. Always ask yourself, “What else could I test?”

This month, I’m taking a look at the Bandwagon Fallacy. This fallacy happens when someone makes a choice because “everyone else is doing it”. When you were a child, you may have tried to convince your mother that you should be allowed to do something because all of your friends were allowed to do it. This is the Bandwagon Fallacy at work!

The Bandwagon Fallacy is prevalent in many areas of society. One area where this is very obvious is with diets. In the 1990’s, low-fat, high-carb diets were popular. Then in the 2000s, people switched over to the Atkins Diet, which was a high-fat, low-carb diet. Other recent diet trends include the Whole 30 Diet, the Keto Diet, and the Paleo Diet, all of which have different demands.

Just because a diet is very popular and you know people who feel great and lose weight on it does not mean that it is right for YOU. Everyone is different, and it’s important to run a small test on a diet and see how you feel before jumping on the bandwagon with everyone else.

The Bandwagon Fallacy is also frequently seen in the testing world! Think about how many articles you’ve read recently about AI. It seems that everyone is using it to think of new test cases, write test automation, create self-healing tests, and so on. But as with a diet, just because some teams or testers are finding success with it doesn’t mean it’s right for YOUR project. And dropping your current automation solution just because something new comes along results in wasted time.

Another example of a trending tool is Cypress. Cypress is very popular for both API and UI automation because it’s so easy to set up. Cypress comes with good documentation and examples, and it has a vibrant community. But there are some software projects for which Cypress would not be helpful. Cypress can’t test native mobile code, for instance, and it also doesn’t support Safari. And it only supports JavaScript, so if your team doesn’t know JavaScript, it might be better to use a different tool.

It’s fun to try out new tools and techniques. And it is helpful to learn new skills to stay in-demand by employers. But be sure when you are adopting a tool that you are adopting it because it meets your team’s needs, not just because it’s what everyone else is doing.

In this installment of my Logical Fallacies series, I’m taking a look at the False Dichotomy fallacy. The False Dichotomy fallacy is used when someone presents two opposing options as if they are the only possibilities; that no middle way exists. This is detrimental to progress because it limits people’s thinking; they feel that they must choose one side or the other. In more extreme cases, this can make people afraid to speak their mind for fear of being associated with the “wrong” side of the debate. And it can make small-minded people unable to look at both sides of an issue objectively.

There are many examples of the False Dichotomy fallacy in society today, but let’s examine one from a few decades ago. Back in the 1990’s, there was a phenomenon known as “The Mommy Wars”. This was a debate about whether mothers should stay at home during the day with their children, or whether they should go to work and put their children in day care. The sides were extremely polarized: the stay-at-home moms cited studies that showed that children thrived when they were at home wth their mothers, implying that working moms didn’t want what was best for their children, and the working moms group cited studies that showed that women who weren’t working outside the home were unfulfilled, implying that staying at home was hurting the movement for women’s equality.

Of course, with the wisdom of thirty years behind us, we can see that this was a False Dichotomy. It’s possible for moms who stay at home with their children to have thriving at-home businesses, and it’s possible for moms who work to choose flexible hours so they can be with their kids when they come home from school. And we can see that fathers were clearly ignored in this False Dichotomy; today I work with many dads who step away from their desks to pick up their kids from school or drop them off at day care, and dads who work longer hours on some days so they can take one day off every two weeks to spend time with their kids.

In the area of software testing, there are two obvious False Dichotomies. The first is the Manual vs. Automation debate. I’ve written about this before, but I’ll summarize why this is a ridiculous debate:
• “Manual” and “automated” are arbitrary designations. Things can be automated as part of a manual test (such as using a script to create users), and things can be manual as part of an automated test (such as doing a visual check after a script runs).
• There are some things that are best tested through running a script, such as performing a load test, and some things that are best tested through a manual test, such as driving up the road with a cell phone to check that the GPS location services in an app are working correctly.
In order to ensure that our software is of the highest quality, we should use all the tools at our disposal, including our hands and eyes, and think of ways to use those tools as efficiently as possible.

The second common False Dichotomy in software testing is the debate about whether we need software testers at all. Some software teams believe that all their testing can be done by software developers and that testers are irrelevant, while other software teams believe that testing should be left to the testers and that it’s not the job of developers to test their code. I believe that both of these positions are misguided and potentially harmful. When developers try to do all of their own testing, they may miss important bugs that are caused by the interaction between two different feature areas of the software. And when developers don’t test at all, they may create buggy software that slows down the team’s progress as testers log more and more bugs to be fixed.

Software testing is most effective when the whole team focuses on quality. What does this look like? It can vary by team, but here are some examples:
• Developers and testers work together to do manual exploratory testing just before a release. Each engineer uses their own expertise to think of ways to test the application.
• Developers create test harnesses for things that are difficult to test. For example, to test file uploads, a developer could create a web page connected to the API that would allow the tester to easily upload files.
• Testers attend meetings where features are architected to provide important insight about the current behavior of the product, raising concerns if the new feature might impact existing functionality.
• Developers and testers work together on test automation. The tester provides insight about what should be tested, and the developer reviews the code for clean coding practices.

Is your company suffering from a False Dichotomy fallacy? If so, see if you can work with people from the opposing side to brainstorm new and innovative solutions.