Think Like a Tester

It’s been a while since I’ve written a post, mostly because I felt that between my three books and my blog posts, I had said everything I wanted to say about software testing. But things have changed! In the last year, AI has drastically changed the software development landscape. After working with various AI tools for a while, I now have a series of posts about AI queued up! In this post, we’ll take a look at the basics. When using AI, it’s important to continue using your brain. Here are some things to remember:

AI is not intelligent

In spite of its name, artificial intelligence is not actually intelligent. The AI agents we use are based on a Large Language Model (LLM), which is a database of trillions of words. When we ask an AI agent a question, it is answering based on the information in that LLM. It makes predictions based on training and previous interactions about what you are asking and what you might want next. Do not make the mistake of believing that the AI knows how to “think”. It knows how to fetch data, organize information, and make predictions.

AI can be wrong

An AI is only as good as the data it was trained upon, or the data it knows how to fetch. If that data is wrong, its answer will be wrong. To use a simple example, if there are 10 websites that say that Beethoven was born in 1775 and the AI reads from only those 10 websites, it will tell you that is Beethoven’s birth year. (It’s actually 1770.) When using AI, remember to verify its answers.

You get what you pay for with AI

Free versions of AI do a smaller amount of research than the paid versions to answer your questions, and they do less confirmation on their findings. The more computing power you pay for, the more sources the AI will search to have a greater level of certainty of the correct answer. When working with AI, remember to consider the level of service you are getting.

AI will lie to you

The legendary James Bach once did an exercise where he asked an AI to provide him with information about a page on his website. It confidently returned a response with great detail about the page. The page did not exist. This is called “hallucination”, and an AI will do this because it is programmed to give you an answer even if there isn’t one. It will find data to make guesses about what you are looking for. I’ve experienced AI hallucination when I’ve asked it to solve a UI automation problem and it has recommended that I use methods that don’t exist.

AI will agree with you

The opposite of the previous problem is that AI will agree with you whenever it can. It might stick to its opinion when you are obviously wrong, such as when you tell it that the sky is green, but if you ask it to do something in code that is not efficient or sensible, it might do it anyway because that’s what you requested. Keep this in mind when you are giving AI coding tasks, and periodically ask it if there is a better way to do what you are trying to accomplish.

AI will use too many words

AI has the tendency to provide wordy explanations for everything it does. This can quickly fill up your code and documentation with unnecessary text, which can make it hard for humans to read the code and interpret what is happening. And when AI generates documents, it tends to return institutional double-speak like “focusing engineering capacity on foundational platform investments that will unlock greater leverage and compound velocity over time” instead of using the simpler “prioritizing tech debt”. AI creations that are too wordy and obtuse are referred to as “workslop”, and workslop should be avoided so that software projects are clear, organized, and maintainable.

AI will do things you didn’t ask it to do

More than once, I have caught an AI agent I created doing something I didn’t ask it to do, such as create a Python script to make its work more efficient, or find a way to get an API key that I didn’t provide it with. Others have told me they have experienced this as well. This is not only annoying, it can also be dangerous, since the agent could get access to systems that it shouldn’t have access to. When working with AI, it’s important to set guardrails on what it is not allowed to do.

Use your brain when using AI

I am not saying to avoid AI. On the contrary, AI can be extremely useful and can save you massive amounts of time. But it’s important to be aware of AI’s tendencies, limitations, and dangers so you can use it wisely. In the coming weeks, I’ll be diving deeper into various AI applications with advice of how you can test it and use it.

My new book, The Professional Software Tester, is now available! This book focuses on career guidance for testers, including how to determine what kind of tester you want to be, how to sharpen your skills, how to get the job you want, and how to be successful when working with others.

To get you excited about the book, I’m sharing a free chapter below: “Know What You Don’t Know”. If you enjoy it, please be sure to pick up your copy of The Professional Software Tester in Kindle or paperback on Amazon today!

Chapter 16:  Know What You Don’t Know

In 2012, I got a job as a QA Engineer at a small software company that had never had a tester.  They wanted me to write test automation in addition to performing manual exploratory and regression testing.  I had never written automated tests, but the company was willing to take a chance on me because I had taken a few programming classes in college. 

Learning test automation proved to be quite a challenge!  One of the things I discovered during this process was that it’s important to know what you don’t know.  In other words, you need to have enough familiarity with a topic to be able to ask the right questions to fill in your skill gaps. 

One area where I struggled was JSON formatting.  I didn’t understand why sometimes the data was in curly brackets and why sometimes it was in square brackets.  My automated test was failing to build, and I couldn’t see where the error was.  After I asked for help, a developer on my team realized that I needed to learn the difference between an object and an array.  Once I knew that curly brackets were used around an object and square brackets were used around an array, I was able to ask more questions about formatting that gave me the information I needed. 

So, the first step in taking charge of your education is to discover what you don’t know!  Perhaps you have dabbled in a programming language while writing test automation, but you are pretty sure you are missing some key concepts.  One way to identify your skill gaps is by taking free quizzes online.  I am a huge fan of the website W3Schools (www.w3schools.com).  It offers completely free courses in languages like JavaScript, Python, Java, and React, as well as database query languages such as SQL, MySQL, and MongoDB.  Each of these courses has a quiz associated with it, which you can find by looking at the course directory or by searching on “Quizzes”.  Once you’ve taken a quiz, you can look at which questions you answered incorrectly, and that can provide guidance to the gaps in your knowledge.

Another useful way to discover what you don’t know is by using AI.  You can enter a prompt such as “What are the most important concepts to understand when using TypeScript?”, and the AI will return a list of concepts.  Reading through them, you can discover if there are any you don’t understand, and focus your learning in those areas.  Finally, AI is useful for when you need information about a specific area of programming, but you can’t remember what it’s called.  If you ask a question like “What is the name of the process in JavaScript where you iterate through all the items in an array?”, the AI will return a list of common ways to iterate, including a for loop, a for…of loop, the forEach method, and higher order methods like map, filter, and reduce.  Chances are that one of those methods was the one that you were looking for!

Recently I watched the AppleTV series, Silo. The show focuses on the lives of 10,000 people living in an underground silo. They know that their ancestors have lived in the silo for hundreds of years, but they don’t know why, and they don’t know why it’s unsafe to leave the silo.

The silo is powered by a generator and maintained by a team called Mechanical. In the third episode of the show, it’s made clear that the generator that powers the silo hasn’t worked correctly in 30 years, and it’s rapidly moving toward failure. Of course, this made me immediately think of software tech debt! In this post, I’ll examine the eight steps a team should take to address tech debt, with examples from Silo and from a project I worked on several years ago.

Step One: Be aware of the problem
In the silo, Mechanical was well aware of the problem with the generator, but no one else was aware, not even the mayor. The first step in fixing the problem was letting the mayor know that there were serious issues with the generator.

In the real world, I worked at a company that had an important payment module. The module was being built in an already overloaded server. Only a few developers were aware of the problem until they shared the issue with management.

It’s easy to stick one’s head in the sand when there is a tech debt problem, but ignoring the problem is not going to make it go away. Viewing the problem clearly is the first step in determining how it can be fixed.

Step Two: Make a case for why the tech debt should be fixed
The generator in the silo was the only thing keeping the power running and the lights on. If the generator failed, there would be no way to fix it and the residents of the silo would be plunged into permanent darkness. Without power, there would be no way to grow food and the entire population would collapse.

At my company, if the load on the server became too great, all software on the server would begin to fail, and there wouldn’t be any good way to update it. Response times for regular user activities would slow to a crawl.

In both scenarios, it was important for people with knowledge of the problem to communicate to management the consequences of not fixing the issue. These decision-makers can then give their permission to fix the problem and allocate the resources needed for the fix.

Step Three: Understand the risks involved with fixing the tech debt
Tech debt is not just a risk in itself; fixing the debt also can pose a significant risk. In the case of the silo, the generator would need to be turned off in order to be repaired. There was a chance that the generator might not turn on again after it was stopped. Additionally, when the generator was stopped for too long, the reactor would begin heating up and would eventually explode.

In the case of the payment module, the module would need to be moved to its own server to address the tech debt. If this move wasn’t successful, millions of people wouldn’t be paid.

It’s important to communicate to management exactly what the risks are with fixing the debt so they can make an informed decision. In some rare cases, the risk of fixing the debt can be greater than the risk of not fixing it.

Step Four: Formulate a plan with all the stakeholders
In Silo, the Mechanical team needed to discuss the plan to fix the generator with the mayor and the deputy sheriff, because the entire populace would be without power during the fix. The mayor and the deputy needed to put together a plan that would keep the people from panicking during the process.

At the company where I worked, we needed to discuss the movement of the payment module to another server with both the team that maintained the payment software and the team that oversaw the weekly payments.

All stakeholders should be aware of how the problem will be fixed and what the potential impact will be while the tech debt is fixed.

Step Five: Limit the scope of the project
The silo’s Mechanical team only had thirty minutes to fix the generator before the reactor exploded, so they knew that they had to focus on the most important fix. The large rotor that was out of alignment was what needed to focus on, and while it was being fixed they just did some light hammering on other dented areas.

When I was testing the payment module, I discovered two bugs. We were told we couldn’t fix them because the fixes would have required additional testing that would have complicated the project. Both bugs had workarounds, so we left them alone.

When you’re dealing with tech debt, it’s important to focus on the most important problem, rather than getting sidetracked into other issues.

Step Six: Have a rollback plan
When fixing tech debt is risky, be sure to have a rollback plan. In the silo, the only rollback plan was to turn the generator back on and leave the rotor unfixed. This wasn’t a great solution, but it would at least mean that they could keep the lights on for a few more months.

With the payment module, we had a way to return the code back to the original server if the migration to the new server didn’t work.

Step Seven: Communicate to everyone affected
In Silo, the mayor knew that it was extremely important to alert the population that the lights would be going out. Anyone unaware would panic. So she broadcast a message to the entire silo, and sent out deputies to make sure everyone was either in their living quarters or in a group shelter while the lights were out.

For the payment module, the stakeholders were already aware of the project and they were alerted to what time the move to the new server would take place.

When it’s likely that service could be disrupted or that users will see a big change, it’s very important to communicate that information.

Step Eight: Don’t go it alone
While Juliette, the main character in Silo, was a loner who often insisted on doing maintenance herself, she understood that fixing the generator would require the effort of the entire mechanical team. Some team members worked on removing the outer plates of the generator, some worked on fixing the rotor, and some monitored the pressure of the reactor.

Similarly, when we moved the payment module from one server to another, we had a whole team of people watching for error messages and checking that the payments went through.

Fixing tech debt is best done when the whole team collaborates, ensuring that changes made will not negatively impact end users.

Conclusion
Most likely your tech debt is not so dangerous that it will kill off an entire community, as it was in Silo! But the principles outlined here can make fixing tech debt a rewarding and low-stress exercise that will benefit your customers.

One of the best developments in software testing over the last two decades is that software testers have learned to code and write test automation. But the downside to this is that sometimes testers can get a little TOO excited about writing code! This can result in tests that are over-engineered. Below are four signs that you may be over-engineering your tests:

Sign One: You Automate EVERYTHING
Not everything should be automated. Some tests are very difficult to automate, and those tests are the ones that are the most likely to be flaky and require constant maintenance. You should always be doing some manual exploratory testing anyway, so why not just keep that tricky test as a manual test and save yourself time and aggravation?

Sign Two: Asserting On Too Many Things in Your Tests
As the great Nikolay Advolodkin states, “For UI automation, every single step is a chance for something to go wrong.” This is why he recommends atomic tests: tests that check only one thing, and that run very quickly. It’s better to have a hundred small tests that each take less than thirty seconds to run than to have ten tests that each take five minutes to run. With the small tests, you will be much less likely to have false failures.

Sign Three: Abstracting Out Too Much of Your Test Code
It’s a great idea to observe the “Don’t Repeat Yourself” rule of clean code, but it can be taken to the extreme. If you have a test that ends with something like this:
assertErrorMessage()
consider that whoever maintains this test is going to have to search for this method to find out what the test actually does. Something like this could be much more maintainable:
cy.get('#error').invoke('text').should('contain', 'card is expired')
This tells the reader exactly what the expected error should be and where it should be found.

Sign Four: Constantly Rewriting Your Tests with the Latest Automation Tool
It’s great to update your test code periodically, but if your tests are running and passing, there’s no real reason to switch to a new testing tool. It just makes more work for you and for everyone who needs to maintain your tests. There’s an American expression: “If it ain’t broke, don’t fix it!” This means that if something is working, there’s no need to change it or replace it with something else. If you find that your tests have become flaky or difficult to maintain, then by all means look for a new tool. Otherwise, be thankful for the tool you have.

The primary goal of test automation is that it should provide a way for you to do automated checks on your product’s code reliably and efficiently. Your code should also be readable and maintainable. Anything that doesn’t meet those goals is likely over-engineering!

After many years of working in software testing, I think it’s safe to say that almost no one enjoys writing documentation. Even people who enjoy writing (like me) can find it a chore, compared to other, more interesting, activities such as creating test plans or writing test automation. But documentation is extremely important! In this post, I’ll outline five reasons why, and I’ll also share five strategies for ensuring your team or company has good documentation.

Reason One: It is helpful to YOU

Think of documentation as a letter from your past self to your present self. Often in software testing we’re required to set up complex application configurations. Do you remember the details of every single setup? I know I don’t! Whenever I need to set up something complicated, I take notes about what I did. I keep those notes organized so they are easy to find the next time I need to configure something. When I find my documentation I need for something I have forgotten, I usually say “Thanks, Past Me!” out loud.

Reason Two: It is helpful for your team
When you have been working on a team for a while, you often internalize details that might not be obvious to new team members, and those details are probably not written down anywhere. Something as simple as the login information for your test users will remain unknown to other team members if it’s not written down. Would you like to stop what you are doing several times a sprint to give your teammates login information when they need it, OR would you prefer documenting this information and placing it in a location that is easy for your teammates to find and bookmark? I know which I’d prefer!

Reason Three: It is helpful for people outside of your team
Your team probably has specialized knowledge about how your feature or product works that people outside your team don’t possess. The more you can document how to set up your feature, how to configure users for the feature, how to test the feature, answers to common questions, and how to troubleshoot the feature, the less you will have to stop what you are doing to help others with their feature problems.

Reason Four: It helps prevent missed requirements
I was once tasked with testing a rewrite of a very important feature. The feature was very complicated and had dozens of configuration settings, resulting in hundreds of testing combinations. None of the features were documented, leaving me to guess, experiment, and ask the product manager questions to get the information I needed. Shortly before the rewrite release, the product manager discovered that we hadn’t coded one very important configuration setting. We didn’t code it because we didn’t know about it! Having written documentation about every single configuration option would have prevented this problem. (And yes, I was documenting everything I learned, so the next group of people to test the feature would have the information they needed!)

Reason Five: It helps prevent missed test cases
Think about the most complicated product you have ever tested. When you think about all the different ways it can be tested, do you remember them all? Are you sure? Documenting how a product works and all the different settings and user journeys is extremely helpful when determining what should be regression tested. With proper documentation, you will be much less likely to forget about testing a feature. And yes, documenting your regression test plan for reuse is a great idea as well!

Now that we’ve examined the five reasons documentation is so important, let’s look at some helpful documentation strategies:

Strategy One: Know that not everyone is good at documentation
Because most people don’t enjoy writing documentation, many teams who want to add documentation decide that everyone on the team should take turns writing it. Unfortunately, this will result in some poor documentation from the people who aren’t good at it. Not everyone is good at taking a complex topic, breaking it down into smaller parts, and describing those parts in a clear, organized fashion. If you make everybody write documentation, expect to spend a lot of time fixing the poorly written items.

Strategy Two: The people who are good at documentation usually hate it less than the people who aren’t good at documentation
There are people in the world who actually enjoy figuring out the clearest way to explain things. Some of those people are technical writers, and if you have access to them, consider yourself lucky! Most teams will need to settle for finding the developers, testers, or product managers on their team who are the best explainers, and asking them to create the documentation.

Strategy Three: Make sure the documentation doesn’t fall to just one person
If you choose just one person to write the documentation for your team, chances are they will wind up resenting it, because they will need to take so much time away from their coding and testing to do it. That’s not fair to them. Choose at least two people to share the load.

Strategy Four: Build in times for documentation
For testers, It’s already hard to balance activities like creating test plans, executing test plans, writing test automation, and fixing flaky tests. Developers have to deal with meeting product deadlines, conducting code reviews, and fixing tech debt. Rather than expecting people to prioritize documentation on their own, a really useful strategy is to build in the time to do it. This can be done in a few different ways: documentation tasks could be added to the sprint board and estimated, so the time is included along with the other tasks; a team could choose one day a month to focus on documentation; or a team could set aside one sprint every quarter to focus solely on documentation. This reduces the anxiety that testers and developers feel when they are trying to balance too many priorities.

Strategy Five: Know that maintaining documentation is much easier than starting documentation
When you first start documenting your team’s information, it can feel like it will take forever to get it done, and you may imagine that it will always be that way. But once the documentation is written, it will take much less time to maintain. The team will simply need to remember to update the documentation whenever something changes. Building in a regular time to review the documentation will help keep the documentation at the front of everyone’s mind. Additionally, your documentation will help your teammates and others find the information they need on their own, reducing interruptions and context switching. Some of that saved time can be used to maintain the documentation.

With strategic planning and focus, teams can create useful documentation that will serve themselves and others in the company, helping them release quality products on time.

I’ve always been interested in health and fitness because I want to live a long, healthy life. But as I progressed farther into “middle age”, I discovered that the methods I used to stay healthy were not working for me any more. I gradually gained 30 pounds, and I realized I needed to make some changes. I made those changes and am now back to my original weight (for the curious, this is the program I used).

I’ve been involved in many conversations lately about software quality and the reasons why it’s difficult for teams to adopt good quality practices, and this morning it dawned on me that there are many similarities between staying healthy when one matures and keeping software healthy when a company grows. Let’s take a look at some common excuses that keep teams from getting or staying healthy.

“Updating our tests takes too much time”:
Application software is always changing and improving, which means that testing methods used five or ten years ago might not be as effective as they once were. But to completely rewrite a suite of automated tests takes a significant amount of time; time which teams often don’t feel they have. However, NOT updating the tests will result in even more time spent releasing software: the test automation will have many false failures, which will take a lot of time to investigate. This will mean that the team can’t trust the tests, which will result in the need for more manual regression testing. Spending more time on manual regression testing means that the team has even less time available to update the automated tests, and the cycle continues downward.

This is very similar to the excuses given by many people that they don’t have time to exercise. Eventually that lack of exercise will result in being achy and unfit, which means taking extra minutes to climb a flight of stairs or spending a month recovering after a surgery that could have been prevented through fitness.

Make the time to get your tests “fit” now! The investment that you make will compound as your updated tests save you time; time which you can then use to update more tests.

“It’s too hard to learn a new system”:

When you are working for a small company, it’s relatively easy to keep track of all the tests you need to run. But as a company grows, the complexity of the software grows. Where there were once 1000 tests to run, 100,000 tests may now be needed. This is more than one or two people can store in their heads or on spreadsheets. But learning new systems to keep your tests organized will take time and patience. Yes, it would be easier to continue to manage your tests as you always have, but as time goes on it will become more likely that you will miss something and release buggy software.

This is similar to the excuse that it’s too difficult to learn how to prepare healthy food. People become accustomed to getting takeout several nights a week or buying frozen meals at the supermarket. To eat more healthfully, it becomes necessary to learn new skills, like how to grill chicken or toss a salad. It is certainly easier to rely on restaurant food for your meals, but this also means that you have very little control over the quality of food you are consuming.

Learning new systems will enable you to test more accurately and efficiently. It also keeps your mind sharp and uncluttered, which you can put to use in more creative exploratory testing!

“It’s more important to release this feature”:
Software companies need to stay competitive by releasing features that customers want, but that can’t come at the expense of quality. Growing tech debt will eventually result in catastrophic failures, and what customers want even more than new features is software that works. But it can be difficult for teams to say no to aggressive timetables pushed by product owners and management.

This is similar to the pressure that dieters feel when people around them are encouraging them to have a beer or eat a slice of birthday cake. There’s nothing wrong with these things- they taste good and are fun- but over time, that consumption adds up in the form of extra pounds or declining health. Dieters need to remember their “why” and be able to articulate it to others. For example, “I’m working on losing weight, so I’m not drinking alcohol at the moment”.

Similarly, teams need to be able to clearly articulate why a shiny new feature needs to wait an extra month or two before release. Tech debt needs to be catalogued and its impact needs to be quantified so that everyone is clear that addressing the debt will be better for the company in the long run.

This month, talk with your team about things you can do for the health of your software. Do you need a “diet” of delaying a feature release? Or do you need to “exercise” your code by shaping up your test automation? The efforts that you make will help improve your product and ultimately your end user’s experience.

Most software testers know how to conduct boundary testing: for example, if the tester knows that a text field should only accept 20 characters, they will test with 19 characters, 20 characters, and 21 characters, and perhaps even 100 characters. But I have often encountered situations where a tester does not think to do boundary testing if no boundary is specified in the acceptance criteria. This is dangerous!

There are many reasons to know the limits of your input fields and of your application. Not knowing can mean that your application is vulnerable to attack, or that the user experience will be degraded. In this post, I’ll discuss three types of limits you’ll want to know and test for.

Size Limits:
Whenever you encounter an input field in your application, you should test to discover what the limits are. What is the lowest input you can enter? What is the highest? For example, if the field is expecting a numeric value, find out what the highest value is that the field will accept. Then find out what the lowest field is. Will it be zero? Will the field accept negative numbers? For text fields, discover whether the field will accept a single character, and find out how many characters you can add before the submission fails.

There are two reasons why you should always do this: first, you want to make sure that the user receives an appropriate error message if they exceed (or go lower than) the limit. Imagine a scenario where someone with a last name with more than twenty characters tries to enter their name into a field that will only accept twenty characters. If they don’t get an error message telling them why their submission failed, they will be very frustrated and have a poor user experience.

Secondly, a field without appropriate limits is vulnerable to a buffer overflow attack. This is where a malicious user floods the text field with hundreds or thousands of characters in the hope that it will trigger a server error that will provide them with information that they can use to reach the database. You’ll want to make sure that there are input limits in place to prevent this kind of vulnerability.

Size limits are also important with file uploads. What is the maximum file size that your application will accommodate? Not knowing this can result in users seeing their upload fail without knowing why, or malicious users crippling the application by uploading files your software can’t handle.

Type Limits:
Whenever a file can be uploaded to an application, you’ll want to discover exactly which file types are allowed and which are prohibited. Again, one reason for this is that you’ll want to make sure that the user receives an appropriate error message if they try to upload a file type that is not allowed. You’ll also want to validate that all allowed file types can be processed properly. It won’t matter if your application says that it allows .png files if the application doesn’t actually process them! Finally, it’s important to make sure that the application is not vulnerable to malicious file uploads. If a malicious user can upload a file with embedded code, you are opening up your application to attack.

When you are testing file types, don’t believe the file extension. A hacker could disguise an .exe file as a .txt file to try to bypass the file upload checks. You’ll want to make sure that there is file-checking software in place that validates that the file is actually what it says it is.

Performance Limits:
A third kind of limit you should know about is the performance limit. You should know how many requests your application can handle per second before performance is degraded, and you should also know what the user will experience when the performance is degraded. To gain this information, you’ll want to conduct load and stress testing. Find out how many users your application is expected to support; then run load tests with that number of simulated users to confirm that the application works as expected. Next, increase the number of simulated users gradually until the response time slows or the requests start backing up. Make note of what your real users will experience if this happens: will they get an error message? Will the application freeze or crash? Or will they be staring at a progress wheel for several minutes? Once you have this information, you can bring it back to your team and discuss any needed changes.

The great benefit of software testers is that they are always thinking about what could possibly go wrong with an application. Be sure to know your limits in every area of your software!

Usually when I learn new things, like a programming language or a new automation tool, I like to take a complete course. Courses are structured for logical, incremental learning. But sometimes when I want to learn something quickly, I use what I call “The Ripping-Out-Code Method”. I don’t know if anyone else uses this method, but I’d like to think that my name for it is original!

The Ripping-Out-Code Method involves taking a code project written by someone else and working to understand it by gradually removing lines of code until you are left with the basics. Here’s how it works:

Step 1: Find some code you want to understand
This could be some test automation code in a new framework, some production code that you will be testing soon, or some kind of script that you’d like to use.

Step 2: Pull down the code
Pull down the code from the repository where it is located.

Step 3: Save a copy of the code

When using the Ripping-Out-Code Method, you might occasionally find that you’ve ripped out too much code, and then can’t remember how to put it all back again! If this happens, it’s nice to have a fresh copy of the code to switch to so you can start over.

Step 4: Read through the code
Before you make any changes at all, take a look at the code to see if you can figure out what’s going on. See whether you can locate functions that are being called, whether you can find the values of variables that are being used, and so on.

Step 5: Run the code

Once you have a sense of what the code is doing, it’s time to run the code. Watch what happens when the code runs, either through the log statements or through the UI. If the code consists of automated tests, watch the tests run and pass.

Step 6: Make a small change to the code

Now that you have confirmed that the code is working, make a very small change. For example, if there is an automated test that enters a value, change the value and run the code again.

Step 7: Start removing lines of code

At this point, you should probably have a rough idea of how the code works. Now it’s time to start ripping out the code! Look for things you don’t need. For example, maybe the code contains some UI tests, and you’d just like to focus on API tests. Start ripping out the UI tests. Don’t rip out too much, though! Make a note of what you are doing, perhaps even copying the code into a separate document, so that you can put the code back in if you need to.

Step 8: Run the code again

Once you’ve ripped out some lines of code, run the code again to make sure it still works. If the code doesn’t work any more, you know you’ve ripped out too much! Add the removed code back in, and take a look at the code to determine why it was needed. However, if the code does work, it’s time to rip out more code.

Step 9: Repeat steps 7 and 8 until you have removed as much code as you can

In the example above, where you wanted to focus only on API tests, you might look through the code to find classes called by the UI tests but not the API tests. Rip out those classes one at a time, running the API tests each time to make sure they still pass.

Step 10: Use what you’ve learned to write some new code
Once you’ve ripped out as much code as you can, you should be left with the basics of what you need. Moreover, you should have a pretty good idea of why the leftover code is still needed. Using this knowledge, try creating a new project and writing the code on your own.

The Ripping-Out-Code Method does not work for everything. If you are learning a new programming language, for example, it’s probably best to start with a beginner’s course. But if you already know a bit about the code or tool you are working with, this method can be a quick way to learn. If you try it out and find success with it, please let me know in the comments!

There’s nothing like a shiny new year to make people think about how they can improve their careers! Here are five ideas for software testers in 2025:

I. Learn Test Automation

Will this be the year that you finally- FINALLY- learn to create automated tests? Coding skills make you more employable and provide better-paying roles. I’ve made learning test automation easy through my completely FREE YouTube course, Monday Morning Automation! The lessons start with the command line, then move into Git commands, then teach JavaScript basics, and finally get you started on automating tests with Cypress. Each lesson takes about ten minutes, and when you have completed the lessons, you will know enough to create automated tests for your own company’s product.

II. Read a Book

Books are a great way to dive deeply into understanding a topic. Here are four books I recommend:

• The Complete Software Tester: I wrote this book to help beginning and middle-level testers improve their testing in all areas, including API testing, security testing, and performance testing. It’s available in Kindle and in paperback, and is also available in Russian!

• Logical Fallacies for Testers: This is a super-short book that you could read in one weekend. I wrote it to help testers think critically when evaluating their test projects.

• Explore It! by Elisabeth Hendrickson: I recommend this book to every tester I know, and to everyone who is considering a career in testing. Exploratory testing is so important, because this is where the tricky bugs are found.

• Perfect Software and Other Illusions About Testing by Gerald Weinberg: This book is a classic. It’s filled with amusing anecdotes from the author’s career and helps testers think about why they are testing what they are testing.

III. Take a Course

Whenever I’m struggling to understand something, I know it’s time for me to take a hands-on course. Following along with the course and actually doing the activities is a great way to solidify your understanding of concepts. Here are four courses I recommend:

• Postman Essential Training: I created this LinkedIn Learning course to help people understand how to create and execute API tests. API tests are faster and more reliable than UI automation and also provide a great way to test business logic and find potential security issues.
• Playwright: Web Automation Testing From Zero to Hero by Artem Bondar: This Udemy course is a great introduction to Playwright. Playwright is a powerful test automation tool, but it’s a little tricker than Cypress. I found this course extremely helpful for understanding Playwright concepts.
• The Complete Node.js Developer Course by Andrew Mead: I took this Udemy course when I needed to learn how to create a test application. It is super clear and provides exercises that force you to think about what you’re doing. Understanding Node and JavaScript is very useful for Web testing.
• Android 14 App Development Bootcamp 2024 by Vin Norman: This is another Udemy course that I took when I was doing mobile testing, and wanted to understand more about how mobile apps are developed. When I took the course, I was surprised to find that the course was incomplete, but it still provided me with a good understanding about how Kotlin development works.

IV. Attend a Meetup or Conference

Meetups and conferences can be really helpful in introducing you to new concepts. It’s so great that modern technology allows us to attend virtual meetups and conferences anywhere in the world! Here are a few meetup and conference ideas:

• Next Wednesday (January 8, 2025), I’ll be presenting at the Testomat.io Test Automation Meetup. My topic will be “Configuring Playwright for Test Automation Success”. If you are just getting started with Playwright, or are thinking about using it, this presentation should be very informative!
• Automation Guild ’25: This online conference is coming up in February and is a great way to improve your test automation skills. Once you have registered, the training sessions are available for you to attend in real-time, or to view anytime via recordings and transcripts.
• Testflix: I love this conference because each presentation lasts less than 15 minutes- great for my short attention span! The conference features presenters from around the world on a wide variety of themes, such as API Testing, Career Development, and AI and ML in Testing. The conference usually happens in October, so be sure to watch for it later this year!

V. Learn a New Skill

Is there an area of testing that you have always wanted to know more about, but you either haven’t had the time or the will to learn it? For me in 2024, that was API Contract Testing. I had always wanted to learn it, but reading about it in blog posts wasn’t providing me with enough understanding. Fortunately, Marie Cruz and Lewis Prescott came out with a great book in 2024 called Contract Testing in Action! Between the examples in that book, and the hands-on learning project at pact.io, I was able to finally figure it out.

Learning new skills can be daunting, and sometimes frustrating! But it’s worth the journey. You will become a better thinker and a better tester in the process and the more you learn, the more your skills will be in demand! Let’s make 2025 our best year ever!

We live in a time where it is easy to measure things. Websites measure visits from users all over the world; YouTube videos measure views and likes; mobile apps measure crash statistics. So it makes sense that software managers would want to measure quality activities.

Unfortunately, we don’t always have clear language to describe what is being measured. You may have heard a manager talk about getting to “100% Test Coverage”. But what do they mean by this statement? Here are a few things this could mean, and one thing that this cannot mean.

Test Coverage can actually mean Code Coverage

Sometimes when people refer to test coverage, what they really mean is code coverage. Code coverage is the number of lines of code that are executed when automated tests run against the product or feature. Areas that are not touched by automated tests could indicate gaps in testing. Unit tests are a great way to increase code coverage, because they are designed to test the code directly. And there are number of tools available to measure code coverage, such as DotCover, Coverlet, Cobertura, and SonarQube.

Test Coverage does not mean the percent of every possible test

A common mistake made by those who have never tested software is thinking that there are a finite number of things that can be tested in an application, so it is possible to have 100% test coverage. This is simply impossible. There is no limit to the number of tests that could be created even for the simplest applications. Consider a simple web form with five fields, none of which are required. A user could fill in just one field (five different possibilities), just two fields (ten different possibilities), three fields (ten different possibilities), four fields (five different possibilities), or all five fields. That’s 31 test cases, and that isn’t even considering negative testing. What if one field has an error and two fields are correct? What if three fields have an error and one field is correct? It’s easy to see how the number of test scenarios can quickly move to the millions of test cases. So no team will ever reach 100% test coverage as defined here.

Test Coverage can refer to Automation Coverage

It’s possible to use the term test coverage to refer to the percent of manual test cases that have been automated. This is useful for showing the status of an automation project. If you have 500 documented manual test cases, and you have automated 200 of them, your automation coverage is 40%. However, this meaning is less useful if you don’t already have a documented suite of manual tests. If only a handful of manual tests have been documented, a team could claim to have 100% automation coverage after automating those tests. Meanwhile, hundreds of other test cases could be neglected.

Test Coverage can refer to Feature Coverage

Another possible meaning for test coverage could be the measurement of which application features have existing tests. If a product has ten features, and the tester executes tests for five of those features, it would be possible to say that 50% of the features have been tested. In this usage, it’s also important to distinguish what is meant by “tested”. Tested could mean that exploratory testing has been conducted and documented, but no regression test suite has been created. Or it could mean that manual regression tests have been created, or that test automation has been written. Also keep in mind that it’s not possible to say that a feature has 100% coverage, for the same reason listed above.

What can Test Coverage tell us?

Test coverage can tell us how many lines of code are executed by tests, how much testing has been done on a feature, how many regression tests have been created, or how many automated tests have been written. But it is only useful if all members of the team or organization have agreed upon what the term means. If your team is asked to start measuring “Test Coverage”, share this blog post with the requester and ask them to clarify what information they are looking for.