Think Like a Tester

One of the best developments in software testing over the last two decades is that software testers have learned to code and write test automation. But the downside to this is that sometimes testers can get a little TOO excited about writing code! This can result in tests that are over-engineered. Below are four signs that you may be over-engineering your tests:

Sign One: You Automate EVERYTHING
Not everything should be automated. Some tests are very difficult to automate, and those tests are the ones that are the most likely to be flaky and require constant maintenance. You should always be doing some manual exploratory testing anyway, so why not just keep that tricky test as a manual test and save yourself time and aggravation?

Sign Two: Asserting On Too Many Things in Your Tests
As the great Nikolay Advolodkin states, “For UI automation, every single step is a chance for something to go wrong.” This is why he recommends atomic tests: tests that check only one thing, and that run very quickly. It’s better to have a hundred small tests that each take less than thirty seconds to run than to have ten tests that each take five minutes to run. With the small tests, you will be much less likely to have false failures.

Sign Three: Abstracting Out Too Much of Your Test Code
It’s a great idea to observe the “Don’t Repeat Yourself” rule of clean code, but it can be taken to the extreme. If you have a test that ends with something like this:
assertErrorMessage()
consider that whoever maintains this test is going to have to search for this method to find out what the test actually does. Something like this could be much more maintainable:
cy.get('#error').invoke('text').should('contain', 'card is expired')
This tells the reader exactly what the expected error should be and where it should be found.

Sign Four: Constantly Rewriting Your Tests with the Latest Automation Tool
It’s great to update your test code periodically, but if your tests are running and passing, there’s no real reason to switch to a new testing tool. It just makes more work for you and for everyone who needs to maintain your tests. There’s an American expression: “If it ain’t broke, don’t fix it!” This means that if something is working, there’s no need to change it or replace it with something else. If you find that your tests have become flaky or difficult to maintain, then by all means look for a new tool. Otherwise, be thankful for the tool you have.

The primary goal of test automation is that it should provide a way for you to do automated checks on your product’s code reliably and efficiently. Your code should also be readable and maintainable. Anything that doesn’t meet those goals is likely over-engineering!

After many years of working in software testing, I think it’s safe to say that almost no one enjoys writing documentation. Even people who enjoy writing (like me) can find it a chore, compared to other, more interesting, activities such as creating test plans or writing test automation. But documentation is extremely important! In this post, I’ll outline five reasons why, and I’ll also share five strategies for ensuring your team or company has good documentation.

Reason One: It is helpful to YOU

Think of documentation as a letter from your past self to your present self. Often in software testing we’re required to set up complex application configurations. Do you remember the details of every single setup? I know I don’t! Whenever I need to set up something complicated, I take notes about what I did. I keep those notes organized so they are easy to find the next time I need to configure something. When I find my documentation I need for something I have forgotten, I usually say “Thanks, Past Me!” out loud.

Reason Two: It is helpful for your team
When you have been working on a team for a while, you often internalize details that might not be obvious to new team members, and those details are probably not written down anywhere. Something as simple as the login information for your test users will remain unknown to other team members if it’s not written down. Would you like to stop what you are doing several times a sprint to give your teammates login information when they need it, OR would you prefer documenting this information and placing it in a location that is easy for your teammates to find and bookmark? I know which I’d prefer!

Reason Three: It is helpful for people outside of your team
Your team probably has specialized knowledge about how your feature or product works that people outside your team don’t possess. The more you can document how to set up your feature, how to configure users for the feature, how to test the feature, answers to common questions, and how to troubleshoot the feature, the less you will have to stop what you are doing to help others with their feature problems.

Reason Four: It helps prevent missed requirements
I was once tasked with testing a rewrite of a very important feature. The feature was very complicated and had dozens of configuration settings, resulting in hundreds of testing combinations. None of the features were documented, leaving me to guess, experiment, and ask the product manager questions to get the information I needed. Shortly before the rewrite release, the product manager discovered that we hadn’t coded one very important configuration setting. We didn’t code it because we didn’t know about it! Having written documentation about every single configuration option would have prevented this problem. (And yes, I was documenting everything I learned, so the next group of people to test the feature would have the information they needed!)

Reason Five: It helps prevent missed test cases
Think about the most complicated product you have ever tested. When you think about all the different ways it can be tested, do you remember them all? Are you sure? Documenting how a product works and all the different settings and user journeys is extremely helpful when determining what should be regression tested. With proper documentation, you will be much less likely to forget about testing a feature. And yes, documenting your regression test plan for reuse is a great idea as well!

Now that we’ve examined the five reasons documentation is so important, let’s look at some helpful documentation strategies:

Strategy One: Know that not everyone is good at documentation
Because most people don’t enjoy writing documentation, many teams who want to add documentation decide that everyone on the team should take turns writing it. Unfortunately, this will result in some poor documentation from the people who aren’t good at it. Not everyone is good at taking a complex topic, breaking it down into smaller parts, and describing those parts in a clear, organized fashion. If you make everybody write documentation, expect to spend a lot of time fixing the poorly written items.

Strategy Two: The people who are good at documentation usually hate it less than the people who aren’t good at documentation
There are people in the world who actually enjoy figuring out the clearest way to explain things. Some of those people are technical writers, and if you have access to them, consider yourself lucky! Most teams will need to settle for finding the developers, testers, or product managers on their team who are the best explainers, and asking them to create the documentation.

Strategy Three: Make sure the documentation doesn’t fall to just one person
If you choose just one person to write the documentation for your team, chances are they will wind up resenting it, because they will need to take so much time away from their coding and testing to do it. That’s not fair to them. Choose at least two people to share the load.

Strategy Four: Build in times for documentation
For testers, It’s already hard to balance activities like creating test plans, executing test plans, writing test automation, and fixing flaky tests. Developers have to deal with meeting product deadlines, conducting code reviews, and fixing tech debt. Rather than expecting people to prioritize documentation on their own, a really useful strategy is to build in the time to do it. This can be done in a few different ways: documentation tasks could be added to the sprint board and estimated, so the time is included along with the other tasks; a team could choose one day a month to focus on documentation; or a team could set aside one sprint every quarter to focus solely on documentation. This reduces the anxiety that testers and developers feel when they are trying to balance too many priorities.

Strategy Five: Know that maintaining documentation is much easier than starting documentation
When you first start documenting your team’s information, it can feel like it will take forever to get it done, and you may imagine that it will always be that way. But once the documentation is written, it will take much less time to maintain. The team will simply need to remember to update the documentation whenever something changes. Building in a regular time to review the documentation will help keep the documentation at the front of everyone’s mind. Additionally, your documentation will help your teammates and others find the information they need on their own, reducing interruptions and context switching. Some of that saved time can be used to maintain the documentation.

With strategic planning and focus, teams can create useful documentation that will serve themselves and others in the company, helping them release quality products on time.

I’ve always been interested in health and fitness because I want to live a long, healthy life. But as I progressed farther into “middle age”, I discovered that the methods I used to stay healthy were not working for me any more. I gradually gained 30 pounds, and I realized I needed to make some changes. I made those changes and am now back to my original weight (for the curious, this is the program I used).

I’ve been involved in many conversations lately about software quality and the reasons why it’s difficult for teams to adopt good quality practices, and this morning it dawned on me that there are many similarities between staying healthy when one matures and keeping software healthy when a company grows. Let’s take a look at some common excuses that keep teams from getting or staying healthy.

“Updating our tests takes too much time”:
Application software is always changing and improving, which means that testing methods used five or ten years ago might not be as effective as they once were. But to completely rewrite a suite of automated tests takes a significant amount of time; time which teams often don’t feel they have. However, NOT updating the tests will result in even more time spent releasing software: the test automation will have many false failures, which will take a lot of time to investigate. This will mean that the team can’t trust the tests, which will result in the need for more manual regression testing. Spending more time on manual regression testing means that the team has even less time available to update the automated tests, and the cycle continues downward.

This is very similar to the excuses given by many people that they don’t have time to exercise. Eventually that lack of exercise will result in being achy and unfit, which means taking extra minutes to climb a flight of stairs or spending a month recovering after a surgery that could have been prevented through fitness.

Make the time to get your tests “fit” now! The investment that you make will compound as your updated tests save you time; time which you can then use to update more tests.

“It’s too hard to learn a new system”:

When you are working for a small company, it’s relatively easy to keep track of all the tests you need to run. But as a company grows, the complexity of the software grows. Where there were once 1000 tests to run, 100,000 tests may now be needed. This is more than one or two people can store in their heads or on spreadsheets. But learning new systems to keep your tests organized will take time and patience. Yes, it would be easier to continue to manage your tests as you always have, but as time goes on it will become more likely that you will miss something and release buggy software.

This is similar to the excuse that it’s too difficult to learn how to prepare healthy food. People become accustomed to getting takeout several nights a week or buying frozen meals at the supermarket. To eat more healthfully, it becomes necessary to learn new skills, like how to grill chicken or toss a salad. It is certainly easier to rely on restaurant food for your meals, but this also means that you have very little control over the quality of food you are consuming.

Learning new systems will enable you to test more accurately and efficiently. It also keeps your mind sharp and uncluttered, which you can put to use in more creative exploratory testing!

“It’s more important to release this feature”:
Software companies need to stay competitive by releasing features that customers want, but that can’t come at the expense of quality. Growing tech debt will eventually result in catastrophic failures, and what customers want even more than new features is software that works. But it can be difficult for teams to say no to aggressive timetables pushed by product owners and management.

This is similar to the pressure that dieters feel when people around them are encouraging them to have a beer or eat a slice of birthday cake. There’s nothing wrong with these things- they taste good and are fun- but over time, that consumption adds up in the form of extra pounds or declining health. Dieters need to remember their “why” and be able to articulate it to others. For example, “I’m working on losing weight, so I’m not drinking alcohol at the moment”.

Similarly, teams need to be able to clearly articulate why a shiny new feature needs to wait an extra month or two before release. Tech debt needs to be catalogued and its impact needs to be quantified so that everyone is clear that addressing the debt will be better for the company in the long run.

This month, talk with your team about things you can do for the health of your software. Do you need a “diet” of delaying a feature release? Or do you need to “exercise” your code by shaping up your test automation? The efforts that you make will help improve your product and ultimately your end user’s experience.

Most software testers know how to conduct boundary testing: for example, if the tester knows that a text field should only accept 20 characters, they will test with 19 characters, 20 characters, and 21 characters, and perhaps even 100 characters. But I have often encountered situations where a tester does not think to do boundary testing if no boundary is specified in the acceptance criteria. This is dangerous!

There are many reasons to know the limits of your input fields and of your application. Not knowing can mean that your application is vulnerable to attack, or that the user experience will be degraded. In this post, I’ll discuss three types of limits you’ll want to know and test for.

Size Limits:
Whenever you encounter an input field in your application, you should test to discover what the limits are. What is the lowest input you can enter? What is the highest? For example, if the field is expecting a numeric value, find out what the highest value is that the field will accept. Then find out what the lowest field is. Will it be zero? Will the field accept negative numbers? For text fields, discover whether the field will accept a single character, and find out how many characters you can add before the submission fails.

There are two reasons why you should always do this: first, you want to make sure that the user receives an appropriate error message if they exceed (or go lower than) the limit. Imagine a scenario where someone with a last name with more than twenty characters tries to enter their name into a field that will only accept twenty characters. If they don’t get an error message telling them why their submission failed, they will be very frustrated and have a poor user experience.

Secondly, a field without appropriate limits is vulnerable to a buffer overflow attack. This is where a malicious user floods the text field with hundreds or thousands of characters in the hope that it will trigger a server error that will provide them with information that they can use to reach the database. You’ll want to make sure that there are input limits in place to prevent this kind of vulnerability.

Size limits are also important with file uploads. What is the maximum file size that your application will accommodate? Not knowing this can result in users seeing their upload fail without knowing why, or malicious users crippling the application by uploading files your software can’t handle.

Type Limits:
Whenever a file can be uploaded to an application, you’ll want to discover exactly which file types are allowed and which are prohibited. Again, one reason for this is that you’ll want to make sure that the user receives an appropriate error message if they try to upload a file type that is not allowed. You’ll also want to validate that all allowed file types can be processed properly. It won’t matter if your application says that it allows .png files if the application doesn’t actually process them! Finally, it’s important to make sure that the application is not vulnerable to malicious file uploads. If a malicious user can upload a file with embedded code, you are opening up your application to attack.

When you are testing file types, don’t believe the file extension. A hacker could disguise an .exe file as a .txt file to try to bypass the file upload checks. You’ll want to make sure that there is file-checking software in place that validates that the file is actually what it says it is.

Performance Limits:
A third kind of limit you should know about is the performance limit. You should know how many requests your application can handle per second before performance is degraded, and you should also know what the user will experience when the performance is degraded. To gain this information, you’ll want to conduct load and stress testing. Find out how many users your application is expected to support; then run load tests with that number of simulated users to confirm that the application works as expected. Next, increase the number of simulated users gradually until the response time slows or the requests start backing up. Make note of what your real users will experience if this happens: will they get an error message? Will the application freeze or crash? Or will they be staring at a progress wheel for several minutes? Once you have this information, you can bring it back to your team and discuss any needed changes.

The great benefit of software testers is that they are always thinking about what could possibly go wrong with an application. Be sure to know your limits in every area of your software!

Usually when I learn new things, like a programming language or a new automation tool, I like to take a complete course. Courses are structured for logical, incremental learning. But sometimes when I want to learn something quickly, I use what I call “The Ripping-Out-Code Method”. I don’t know if anyone else uses this method, but I’d like to think that my name for it is original!

The Ripping-Out-Code Method involves taking a code project written by someone else and working to understand it by gradually removing lines of code until you are left with the basics. Here’s how it works:

Step 1: Find some code you want to understand
This could be some test automation code in a new framework, some production code that you will be testing soon, or some kind of script that you’d like to use.

Step 2: Pull down the code
Pull down the code from the repository where it is located.

Step 3: Save a copy of the code

When using the Ripping-Out-Code Method, you might occasionally find that you’ve ripped out too much code, and then can’t remember how to put it all back again! If this happens, it’s nice to have a fresh copy of the code to switch to so you can start over.

Step 4: Read through the code
Before you make any changes at all, take a look at the code to see if you can figure out what’s going on. See whether you can locate functions that are being called, whether you can find the values of variables that are being used, and so on.

Step 5: Run the code

Once you have a sense of what the code is doing, it’s time to run the code. Watch what happens when the code runs, either through the log statements or through the UI. If the code consists of automated tests, watch the tests run and pass.

Step 6: Make a small change to the code

Now that you have confirmed that the code is working, make a very small change. For example, if there is an automated test that enters a value, change the value and run the code again.

Step 7: Start removing lines of code

At this point, you should probably have a rough idea of how the code works. Now it’s time to start ripping out the code! Look for things you don’t need. For example, maybe the code contains some UI tests, and you’d just like to focus on API tests. Start ripping out the UI tests. Don’t rip out too much, though! Make a note of what you are doing, perhaps even copying the code into a separate document, so that you can put the code back in if you need to.

Step 8: Run the code again

Once you’ve ripped out some lines of code, run the code again to make sure it still works. If the code doesn’t work any more, you know you’ve ripped out too much! Add the removed code back in, and take a look at the code to determine why it was needed. However, if the code does work, it’s time to rip out more code.

Step 9: Repeat steps 7 and 8 until you have removed as much code as you can

In the example above, where you wanted to focus only on API tests, you might look through the code to find classes called by the UI tests but not the API tests. Rip out those classes one at a time, running the API tests each time to make sure they still pass.

Step 10: Use what you’ve learned to write some new code
Once you’ve ripped out as much code as you can, you should be left with the basics of what you need. Moreover, you should have a pretty good idea of why the leftover code is still needed. Using this knowledge, try creating a new project and writing the code on your own.

The Ripping-Out-Code Method does not work for everything. If you are learning a new programming language, for example, it’s probably best to start with a beginner’s course. But if you already know a bit about the code or tool you are working with, this method can be a quick way to learn. If you try it out and find success with it, please let me know in the comments!

There’s nothing like a shiny new year to make people think about how they can improve their careers! Here are five ideas for software testers in 2025:

I. Learn Test Automation

Will this be the year that you finally- FINALLY- learn to create automated tests? Coding skills make you more employable and provide better-paying roles. I’ve made learning test automation easy through my completely FREE YouTube course, Monday Morning Automation! The lessons start with the command line, then move into Git commands, then teach JavaScript basics, and finally get you started on automating tests with Cypress. Each lesson takes about ten minutes, and when you have completed the lessons, you will know enough to create automated tests for your own company’s product.

II. Read a Book

Books are a great way to dive deeply into understanding a topic. Here are four books I recommend:

• The Complete Software Tester: I wrote this book to help beginning and middle-level testers improve their testing in all areas, including API testing, security testing, and performance testing. It’s available in Kindle and in paperback, and is also available in Russian!

• Logical Fallacies for Testers: This is a super-short book that you could read in one weekend. I wrote it to help testers think critically when evaluating their test projects.

• Explore It! by Elisabeth Hendrickson: I recommend this book to every tester I know, and to everyone who is considering a career in testing. Exploratory testing is so important, because this is where the tricky bugs are found.

• Perfect Software and Other Illusions About Testing by Gerald Weinberg: This book is a classic. It’s filled with amusing anecdotes from the author’s career and helps testers think about why they are testing what they are testing.

III. Take a Course

Whenever I’m struggling to understand something, I know it’s time for me to take a hands-on course. Following along with the course and actually doing the activities is a great way to solidify your understanding of concepts. Here are four courses I recommend:

• Postman Essential Training: I created this LinkedIn Learning course to help people understand how to create and execute API tests. API tests are faster and more reliable than UI automation and also provide a great way to test business logic and find potential security issues.
• Playwright: Web Automation Testing From Zero to Hero by Artem Bondar: This Udemy course is a great introduction to Playwright. Playwright is a powerful test automation tool, but it’s a little tricker than Cypress. I found this course extremely helpful for understanding Playwright concepts.
• The Complete Node.js Developer Course by Andrew Mead: I took this Udemy course when I needed to learn how to create a test application. It is super clear and provides exercises that force you to think about what you’re doing. Understanding Node and JavaScript is very useful for Web testing.
• Android 14 App Development Bootcamp 2024 by Vin Norman: This is another Udemy course that I took when I was doing mobile testing, and wanted to understand more about how mobile apps are developed. When I took the course, I was surprised to find that the course was incomplete, but it still provided me with a good understanding about how Kotlin development works.

IV. Attend a Meetup or Conference

Meetups and conferences can be really helpful in introducing you to new concepts. It’s so great that modern technology allows us to attend virtual meetups and conferences anywhere in the world! Here are a few meetup and conference ideas:

• Next Wednesday (January 8, 2025), I’ll be presenting at the Testomat.io Test Automation Meetup. My topic will be “Configuring Playwright for Test Automation Success”. If you are just getting started with Playwright, or are thinking about using it, this presentation should be very informative!
• Automation Guild ’25: This online conference is coming up in February and is a great way to improve your test automation skills. Once you have registered, the training sessions are available for you to attend in real-time, or to view anytime via recordings and transcripts.
• Testflix: I love this conference because each presentation lasts less than 15 minutes- great for my short attention span! The conference features presenters from around the world on a wide variety of themes, such as API Testing, Career Development, and AI and ML in Testing. The conference usually happens in October, so be sure to watch for it later this year!

V. Learn a New Skill

Is there an area of testing that you have always wanted to know more about, but you either haven’t had the time or the will to learn it? For me in 2024, that was API Contract Testing. I had always wanted to learn it, but reading about it in blog posts wasn’t providing me with enough understanding. Fortunately, Marie Cruz and Lewis Prescott came out with a great book in 2024 called Contract Testing in Action! Between the examples in that book, and the hands-on learning project at pact.io, I was able to finally figure it out.

Learning new skills can be daunting, and sometimes frustrating! But it’s worth the journey. You will become a better thinker and a better tester in the process and the more you learn, the more your skills will be in demand! Let’s make 2025 our best year ever!

We live in a time where it is easy to measure things. Websites measure visits from users all over the world; YouTube videos measure views and likes; mobile apps measure crash statistics. So it makes sense that software managers would want to measure quality activities.

Unfortunately, we don’t always have clear language to describe what is being measured. You may have heard a manager talk about getting to “100% Test Coverage”. But what do they mean by this statement? Here are a few things this could mean, and one thing that this cannot mean.

Test Coverage can actually mean Code Coverage

Sometimes when people refer to test coverage, what they really mean is code coverage. Code coverage is the number of lines of code that are executed when automated tests run against the product or feature. Areas that are not touched by automated tests could indicate gaps in testing. Unit tests are a great way to increase code coverage, because they are designed to test the code directly. And there are number of tools available to measure code coverage, such as DotCover, Coverlet, Cobertura, and SonarQube.

Test Coverage does not mean the percent of every possible test

A common mistake made by those who have never tested software is thinking that there are a finite number of things that can be tested in an application, so it is possible to have 100% test coverage. This is simply impossible. There is no limit to the number of tests that could be created even for the simplest applications. Consider a simple web form with five fields, none of which are required. A user could fill in just one field (five different possibilities), just two fields (ten different possibilities), three fields (ten different possibilities), four fields (five different possibilities), or all five fields. That’s 31 test cases, and that isn’t even considering negative testing. What if one field has an error and two fields are correct? What if three fields have an error and one field is correct? It’s easy to see how the number of test scenarios can quickly move to the millions of test cases. So no team will ever reach 100% test coverage as defined here.

Test Coverage can refer to Automation Coverage

It’s possible to use the term test coverage to refer to the percent of manual test cases that have been automated. This is useful for showing the status of an automation project. If you have 500 documented manual test cases, and you have automated 200 of them, your automation coverage is 40%. However, this meaning is less useful if you don’t already have a documented suite of manual tests. If only a handful of manual tests have been documented, a team could claim to have 100% automation coverage after automating those tests. Meanwhile, hundreds of other test cases could be neglected.

Test Coverage can refer to Feature Coverage

Another possible meaning for test coverage could be the measurement of which application features have existing tests. If a product has ten features, and the tester executes tests for five of those features, it would be possible to say that 50% of the features have been tested. In this usage, it’s also important to distinguish what is meant by “tested”. Tested could mean that exploratory testing has been conducted and documented, but no regression test suite has been created. Or it could mean that manual regression tests have been created, or that test automation has been written. Also keep in mind that it’s not possible to say that a feature has 100% coverage, for the same reason listed above.

What can Test Coverage tell us?

Test coverage can tell us how many lines of code are executed by tests, how much testing has been done on a feature, how many regression tests have been created, or how many automated tests have been written. But it is only useful if all members of the team or organization have agreed upon what the term means. If your team is asked to start measuring “Test Coverage”, share this blog post with the requester and ask them to clarify what information they are looking for.

Working for a startup is fun, because a small group of people get to make all the decisions. You know the whole team well, and decisions can be made with a few simple conversations.

As a company grows, you don’t know all your colleagues as well, but it’s still fairly easy to think of them as part of one big family. Each team can have its own autonomy to decide things, and teams can operate in spite of cross-team differences.

But when a company grows to have 50 or 100 software teams, team differences can start to take their toll. It can become very cumbersome to determine who is responsible for which product, and which team’s deployment broke your team’s product.

At this point it’s time to make some quality changes. Here are five you should consider:

Naming
When you’re at a small company, it can be fun to give your teams whimsical names: The Jetsons! The Hobbits! It’s easy to know who’s who because there are so few teams. But the more teams you have, the harder it’s going to be to figure out who does what. Which team should you talk to about authentication? Is it Team Magneto, or The Borg? Which team pushed a change to their API which broke your entire backend: Team Fluffy Bunnies, or Team Doge?

At a large company, team names should reflect either the product the team works on or the team’s function. Yes, having serious names is less fun, but trying to remember whether you should page Team Black Widow or Team Hawkeye when your site goes down is also not fun.

Documenting
At a small company, knowledge can be passed down tribally. If a new hire has questions, they can just ask you, and you can spend a few minutes with them telling them what they need to know. You can even keep this going for a while at a medium-sized company, if you have some motivated people who are happy to spend time answering others’ questions. But at a large company, trying to spread tribal knowledge becomes an incredible time suck.

Documenting information not only helps with cross-team communication, it can also lessen your workload. If you are the only person who knows how to do something, who’s going to get called when that something needs to be done? You, that’s who! Providing training to a few others can be helpful, but that just means the workload has been spread to a handful of people, who will all be overwhelmed with inquiries. But if you can create clear documentation so that anyone can read exactly what to do and do it without asking any questions, you’ve just freed up a ton of time for yourself and for your fellow experts.

Monitoring

At a small company, everyone knows what’s going on. A deployment is a common knowledge, because most of the company has been working on it. Even at a medium-sized company, it’s fairly easy to keep track of what other teams are doing. But at a large company, a team that you don’t even know about can do a deployment that you don’t know is coming, and that deployment can have ripple effects on your team’s product!

This is why monitoring is so important. It’s not just the changes that your team is making that can result in failures of your product. It’s not even just the teams adjacent to you that can affect your product. Having monitoring in place can alert you the second things go wrong with your application. You may even be able to pinpoint the issue, identify the team that caused the failure, and have them fix the problem before any of your end users notice.

Standardizing
Can you imagine a scenario where each developer on a team decides to code in their favorite language? One might choose Python, and another C#. Of course it’s easy to see that this makes no sense. How could the team share their code? But for some reason, teams don’t realize that this is also a problem with test automation code. In large companies, the different systems are dependent upon one another. One team creating a chat feature will be dependent upon another team’s authentication software, and yet another team’s notification software. Having automation code that is in the same language and follows the same standards from team to team makes it easy to understand all the automated tests. Moreover, it makes it easy to set up systems where a build in one team’s product can trigger tests in another team’s product. It also makes it easier on the team setting up the CI/CD environment, because they don’t have to set up systems to accommodate many different test frameworks.

Measuring

In a small or medium-sized company, it’s pretty easy to notice when there is a quality problem. When a product is unreachable or when a customer complains about a critical bug, everyone hears about it. But the larger the company gets, the more likely it is that some problems won’t be common knowledge. For this reason, it is a great idea to set up metrics to measure team success. Metrics to measure could include bugs found by the team, bugs found by customers, average response time, and average error rate.

As I mentioned in a previous post, you often can’t compare the metrics of one team to the metrics of another team. But you can compare a team’s metrics to their previous metrics. What was the team-to-customer bug ratio six months or a year ago? Was it better or worse than it is today? And if one team’s metrics seem wildly different from all the other teams’ metrics, that might be a data point worth looking into.

Change is not often easy, especially when it comes to companies that have become used to their own way of doing things. But as your company grows, it’s important to grow in your quality maturity as well, and this can mean adopting new standards for naming, documentation, monitoring, tool standardization, and success metrics.

Recently, I was talking with some technology leaders at my company about how we needed to encourage testers to do more exploratory testing. We were getting ready to hold our second annual Hackathon, and one of our group suggested “Why not have a competition for testing as well?” And thus, Testathon was born!

To prepare for Testathon, we invited people to sign up in teams of three to five people. Testathon was open to any participants, not just software testers. We had 48 participants sign up in ten teams; most of the participants were testers, but we had a few software engineers, managers, and product owners as well.

Once I knew how many teams we had, I chose ten products for the teams to test. I made sure that no person was testing their own product. I worked with the product teams to create documentation and user logins that the test teams could use during Testathon.

Testathon lasted two days; during those days the teams tested their assigned products in as many ways as they could think of. They logged the bugs they found in a special Jira board, and marked them with their team name.

Once Testathon was over, five judges- all Principal Software Test Engineers- divided up the bugs and worked for days to attempt to reproduce them. This showed the importance of being able to write up good repro steps in a bug; if a judge wasn’t able to reproduce the bug with the instruction steps provided, they moved the bug to Closed.

Bugs that were reproduced were assigned a severity and labeled by bug type, such as API, Performance, Security, Usability, and so on.

Next, each of the judges nominated which bugs they thought should win for each category. Our categories included: Most Unique Bug, Best Bug Report, Most Critical Bug, Best Security Bug, Best API Bug, Best Usability Bug, and Best Performance Bug.

The judges convened and voted on a winner for each category, and then announced the winners to the whole Engineering group. The winners received cash prizes.

We received lots of positive feedback from the Testathon participants. They really enjoyed working with testers from other teams, testing new products, and learning test strategies from each other. Moreover, some of the bugs logged were great finds! A serious security bug was identified and fixed in two days. Another tricky bug that a product team had known about, but couldn’t consistently reproduce, was finally given good repro steps so the team could fix the bug. Many API bugs were found where the response code was incorrect. And there were several small UI bugs found that when fixed will help improve the user experience.

After Testathon, all of the bugs were presented to the product teams for analysis. They sorted through the bugs for duplicates, and moved any new bugs to their Jira board.

Testathon was a great way for our company to learn the value of exploratory testing! We are already looking forward to our next Testathon. Perhaps you might find a Testathon valuable for your company!

As software companies grow, they often find it necessary to start measuring things. When a company has just a couple of dozen people, it’s pretty easy to see whether a software tester is performing well, or whether they are struggling. But when a company grows to several hundred or several thousand people, it becomes more difficult to keep track of how everyone is doing.

At this point, the management of the growing company will conclude that they need to look at metrics to gauge how a software team is performing. And often, it will be suggested that teams start counting their bugs. But what does a bug count mean? This post will discuss what a bug count will and will not tell you.

A Bug Count By Itself Tells You: Absolutely Nothing
Saying that a team found fifty bugs this month means nothing. It makes as much sense as trying to determine whether a book is good by counting how many pages it has. “Was it a good book?” “Well, I haven’t read it, but it has five hundred pages!!!”

Comparing Bug Counts Between Teams Tells You: Absolutely Nothing
The number of bugs a team finds is dependent on many things. Each product tested is different. One team’s product might be very simple or very mature, while another team’s product might be quite complex or brand new. Further, each team might have a different way of logging bugs. One team might have a practice of verbally reporting bugs to the developer if the developer is still working on the feature. Another team might have a practice of logging every single bug they find, down to a single misplaced pixel. Finally, one person’s single bug might be three bugs to someone else. Consider two teams who test their product on iOS and Android. One team might log a single bug for a flaw they’ve found on both operating systems, while another team might log two bugs for the flaw, one for each operating system.

Comparing Bug Counts by the Same Team, Over Time Tells You: Maybe Something
If you are tracking the number of bugs a team finds each month, and there is a big change, that might mean something. For example, an increase in the number of bugs found could mean:
* The testers are getting better at finding bugs
* The developers are creating more bugs
* There’s a new complicated feature that’s just been added
But it could also mean:
* There’s been a change in the procedures for logging bugs
* Last month, half the team went on vacation so there was less work done, and now everyone is back

Analyzing Who Found The Bug, the Customer or the Tester, Tells You: Likely Something
When bugs are logged, it’s important to know who found them and when. Obviously, the best case scenario is for the tester to find a bug in the test environment, long before the feature goes to production. The worst case scenario is for the customer to find a bug in production. So if you pay attention to what percentage of bugs logged are found by testers and what percentage are found by customers, this could tell you something, especially if you look at the metric over time.

If your team started to track this metric, and the first month you tracked it, 75% of the bugs were found by the testers and 25% of the bugs were found by the customers, you’d have a baseline to compare to. Then in the second month, if 85% of the bugs were found by the testers and 15% of the bugs were found by the customers, you could surmise that your team is getting better at finding the bugs before the customers find them.

Metrics are a double-edged sword: on the one hand, they can be used to illustrate a point such as how well a team is performing, whether more hiring is needed, or whether there’s a problem in your software release processes. But on the other hand, metrics can be weaponized, manipulated, and gamified. By considering all the possibilities of what the number of bugs could mean in a specific context, you’ll be more likely to find metrics that will be helpful and instructive.