Think Like a Tester

As software testers, we have a lot of different things to think about.  We need to test new features and existing features.  We need to make sure different features work correctly together.  We need to run manual tests and make sure that our test automation is running correctly.  And we need to test on different browsers and platforms.  

Because of this, it’s often easy to get bogged down in our day-to-day work and forget that the whole point of our testing is to make sure that our end users have a good experience with our product.  This point was really driven home to me recently when I had an experience as a customer.  Here’s my sad story.

I was making a change to a financial account, and the change required that I fill out an online form that was several pages long.  I took the time to fill out the entire form, and when I got to the end, there was a Submit button.  I clicked the button, and…nothing happened.

I was hoping this was just a fluke, so I waited a day and tried submitting my form again.  No luck.  Then I tried a different browser.  No luck.  Then I submitted a bug report on the company’s website and waited a few more days.  I tried the Submit button again, and still nothing happened. 

Then I called customer service.  I spent an hour and a half on the phone talking with four different representatives: two were customer service reps and two were tech service reps.  The service reps told me to try all sorts of things, including rebooting my router, which seemed like a very odd suggestion to me.  Finally the last customer service rep told me there was nothing they could do; I was going to have to print out the form and fill it out manually, which was a further waste of my time.

Ultimately, it turned out that the problem was that I was filling out the wrong form for my account type.  But where was the validation for this?  When I started to fill out the form and I added my account number, there should have been some validation that determined that I was using the wrong form and returned an error telling me that.  Instead I wasted hours of my time trying to submit the wrong form. 

So what can we learn from my experience?  The testing team at that financial institution probably has hundreds of tests that exercise the functionality of this form and other forms. The testers might be tempted to say, “It’s just one button”.   But to me, the end user, this one button represented hours of aggravation and wasted time.

The moral of the story is to remember to think about customer workflows when you are testing.  What are the customers doing when they interact with your application?  If you don’t know, talk to the Product Owners at your company and find out!  What kinds of things do the customers typically do wrong?  How can you help the customer via the application when they do something wrong?

One button seems like a little thing when you have so many things to test, but to your customer, that button could mean everything.

People often ask me, “What’s the ideal tester-developer ratio?” My answer is always, “It depends.” There are a number of factors that determine what a good tester-developer ratio should be. Things to consider are: whether you are working on cutting-edge technology or a legacy product, the talent and experience of your team members, and what kind of release cadence you are expected to maintain. The truth is that there are all different kinds of ratios that can work, but there are pros and cons to each. Let’s take a look at some examples.

1 tester: 1 developer

The 1:1 ratio is great when you have developers who don’t know much about testing and testers who don’t know much about development. A developer-tester pair can work together to release a new feature, and because they are both so focused on that one feature, they may be able to find and fix all the bugs. However, the developer probably won’t contribute to any test automation, and the tester will likely be the only one who knows how to run and fix the automation. This will mean that with any future development on the feature, the tester will become a bottleneck, slowing down the work.

1 tester: 2 developers
This ratio is good for a situation where there is a front-end and a back-end developer working on a feature. The tester can be responsible for testing the integration between the front and back ends. Like the 1:1 ratio, these three will become the experts on the feature. However, this can result in silos, where it’s difficult later in the project for someone else to come in and assist with the work.

2 testers: A team of developers
This is a very common scenario. The testers can divide up the stories to test according to their skillset and availability. If both testers are talented and organized, they will usually be able to keep up with the manual testing and the test automation work. They can also swap features to see if one tester has missed a bug that the other finds. However, this ratio will occasionally result in bottlenecks when there’s a feature that needs a lot of testing or if one tester goes on vacation.

1 tester: A team of developers
In this scenario, the tester becomes a “quality coach”. They are not responsible for all of the testing or all of the test automation. They guide and train the developers to understand what should be tested and automated. In this way, the whole team owns quality. Any time the tester is not available, the developers are able to fill in the gaps by creating test plans and testing each other’s work. Because the developers are contributing to and helping to maintain the automated tests, test automation never becomes a bottleneck.

0 testers: A team of developers
Some might cringe in horror at this idea, but a team of very well-trained software developers is capable of doing all their own testing. In order for this to be successful, developers need to understand the importance of exploratory testing and how to create test plans. They need to understand what kinds of tests should be automated, and they should be committed to maintaining their test code as carefully as they maintain their production code. Although they will do some initial testing on their own features, they will also create “test buddy” pairs where one developer will act as the tester for another developer’s work. In this way, they’ll have two sets of eyes on each feature and will be more likely to catch bugs.

These ratios all have a few things in common. First, and most importantly, at least one person on the team has excellent testing skills. Those skills are necessary to find elusive bugs. Next, good communication skills are needed. There is no “throwing software over the wall to be tested”; testers and developers are working together. Finally, there is the willingness to be a team player. Testers and developers alike need to be willing to step forward and do testing tasks whether or not it’s part of their assigned role. When all three things are present on a team, any of these ratios can bring success.

For this month’s post, I’m doing something a little different! Usually my posts are aimed at software testers who want to improve their skills and improve their thinking about what to test. But this month, I want address the people who hire the software testers!

Making the right hiring decisions is crucial. Being saddled with an ineffective tester is often worse than having no tester, for the following reasons:
• They may fail to automate any tests, meaning that all testing will be manual and will be a huge bottleneck to releasing software
• They may automate their tests poorly, resulting in flaky tests that no one trusts and that require tons of maintenance
• They may fail to grasp technical concepts, meaning that other team members will have to waste time explaining those concepts to them again and again
• They may be inept at creating test strategies, resulting in the wrong things tested and the right things not tested at all

So, to avoid hiring an ineffective tester, I recommend that you look for the following ten skills:

Able to find bugs
If the tester can’t find bugs, there’s really no reason to employ them. Any developer on the team can do “happy-path” testing and put together automated tests for their code. What sets testers apart from the others is their ability to think of things to test that the developers might miss: negative tests, strange edge cases, interactions between features, and so on. To determine whether a candidate can find bugs, give them a buggy application and ask them to find and report on the bugs. You will be surprised how many “experienced” testers can’t do it! And these are the people you will not want to hire.

Makes good decisions about what to test
A tester that can’t prioritize what should be tested will not be an effective worker on your team. Testers should be able to determine what should go in a smoke test, what areas of an application should have regular regression tests, when it’s the right time to search for an elusive bug, and when it’s time to save the search for later. To determine whether your candidate can think strategically, present them with an example application or system and ask them to design a test plan for it. One easy example is a soda vending machine. While your candidate may not know the mechanisms involved in delivering the soda, they should be able to identify the different tasks of the machine and come up with systematic ways to to test them.

Understands API testing
If your application uses APIs at all- and most modern applications do- you’ll want to make sure that your candidate understands how to test them. I continue to be surprised at how many testers still don’t understand how to test APIs (a problem that could be easily solved by taking my Postman course or reading my new book) when they are so prevalent today. A poorly developed and tested API can result in serious security holes and a poor user experience. To make sure that your candidate knows how to execute API tests, give them a sample API to test. See if they are capable of creating both positive and negative test scenarios.

Communicates clearly
A tester who can find bugs, but can’t explain to anyone how to reproduce them is not going to be particularly helpful. Software testers need to be excellent communicators. To check if your candidate is a clear communicator, ask them to explain a complicated bug they’ve found in their current position. This actually accomplishes two things: it verifies that they actually find bugs in their current position, and it also gives you a sense of how the candidate explains a complicated situation. If you can’t follow along with their explanation, this might not be the right tester for you.

Writes good test automation
According to Robert C. Martin, the author of the widely-respected book Clean Code, test code is just as important as production code. This is because test code is used to validate that changes in production haven’t broken anything. If the test code is unreliable, then all changes to production code have to be manually tested in addition to being tested with automation, which slows the entire development process down. Because of this, you want a software tester who writes clean automated tests: common tasks should be separated out into methods or classes, and the code should be well-organized. To check whether the candidate can write good test automation, ask them to write some automated tests for a simple application. Then ask them to run the tests for you and explain why they wrote the tests they did, and why they organized the code the way they did.

Understands databases
Whether you use relational databases like SQL Server or non-relational databases like Mongo DB, you’ll want your candidate to be able to interact with the databases to get the information they need for testing. If you use relational databases, ask them to write a simple query or a join. For non-relational databases, you can ask them how they would get a specific record from the database.

Understands the challenges of mobile testing
If you have a mobile application, or if your application has a mobile version, you’ll want to make sure that the candidate understands the the challenges of mobile testing. Ask them what those challenges are; you should hear things like differences in screen size, operating system, version, carrier, and so on. If your application is primarily mobile, you’ll also want to make sure that your candidate has experience with mobile automated testing.

Understands the basics of security and performance testing
Even if you have security and performance test teams at your organization, you’ll still want to make sure that your candidate understands some basic security concepts like privilege escalation and IDOR, and some simple performance concepts like measuring page load times and API response times. For smaller companies that don’t have security and performance test teams, understanding these basics is even more important.

Understands the importance of visual and accessibility testing
The candidate should be able to identify some reasons why visual automated testing might be needed. For example, ordinary UI test automation does not validate that an image is appearing correctly in a browser, but a visual testing tool can do that. They should also understand that accessibility is an important aspect of web applications today; they should be able to give you some examples of accessibility needs, such as the ability to zoom in on text, use a screen reader, and view videos with captions.

Can be an advocate for quality on your team
Finally, you will want a candidate who is able to speak up when the situation calls for it. A tester who can find bugs, but who can’t advocate for those bugs to be fixed, will not be much help to your team. In today’s Agile software teams, the tester acts as more of a quality coach, helping all the team members to think like testers, do exploratory testing, and contribute to the automation code.

A poor software tester can be a drag on the entire team, but a good software tester can spur the team on to new heights of quality and productivity! With these skills in mind, you will be able to hire testers you will be grateful to work with.

When I was first introduced to JavaScript automated testing, I was working with a test framework that one of the developers I worked with had set up. I started the tests the way he told me to, with this command: npm run protractor. Later, when I was working with a different project, the command to run the tests was npm run test. Why was it “test” sometimes and “protractor” other times? It’s because these commands referred to npm scripts!

Npm scripts can be set in the package.json file of your project. They are like shortcuts that you can use to execute commands to minimize the amount of typing you need to do. Let’s say you want to run some Cypress tests, and you have a couple of different configuration files to choose from when you run your tests that represent your development and production environments. To run your tests in your development environment, you could type in this command: npx cypress run -C cypress/config/dev-config.json, or you could type npm run dev. Chances are you’ll be running your tests over and over again; which method would you prefer?

Let’s do a simple exercise to see how npm scripts work. You’ll need to have Node installed for this exercise; you can download and install it here. We’ll be working with the command line; if you need some command line basics, you can find them in this post.

Step One: Set up a Node project
In your command window, navigate to the folder where you want to save your project. Then type mkdir npm-script-project. A new folder should be created with this name. Next, navigate to that new project by typing cd npm-script-project. Now initialize the project as a Node project by typing npm init –y.

Step Two: Open your Node project in a code editor
Now that your Node project has been initialized, open it up in your favorite code editor. My favorite is Visual Studio Code, which is available for free on Windows and Mac. When you open the npm-script-project folder in your editor, you will see that a package.json file has been generated for you. This is the file where you will add your script.

Step Three: Add your script
Open the package.json file in your code editor. You will see that there is a “scripts” section in this file, and that there is currently a “test” command listed in the script. You can run the test command by opening up a command window (you can do this within VS Code by choosing “New Terminal” from the “Terminal” menu) and typing npm run test. You’ll get the error message that is specified in the script, which is expected.
Let’s change the “test” script to do something different! Change the “test” name to “hello”, and change the script to “echo \”This is my test script!\””. Now execute the script with this command: npm run hello. You will see the message “This is my test script!” returned in the command window.

Step Four: Make your script do something useful
Now that you know how to make npm scripts, it’s time to make them do something useful! For example, if you install Cypress in your project, you can create a “test” command to run your Cypress tests. Let’s try this out. In your command line, type npm i cypress. This will install Cypress in your project. Next, start Cypress by typing npx cypress open. You’ll see the Cypress test window open, and a cypress folder with some example tests will be installed in your project. You can run the tests from the Cypress window to watch them work and close the windows when they have finished, or you can simply close the test window.
To create an npm script to run those Cypress tests, add a new line to the “scripts” section of the package.json file: “test”: “npx cypress run”. (Be sure to add a comma at the end of your “hello” script so that the JSON will be correct.) Now try running the Cypress tests by using your new script: npm run test. You should see the Cypress tests run in the command window!

This is a pretty simple example, where the command we were replacing was just as short as the script we replaced it with. But hopefully this illustrates how easy it is to replace longer commands with a script that is easy to type. Happy scripting!

It’s a new year, which is a great time to take a look at your career and plan what you’d like to accomplish. One of the things I love about software testing is that it’s a continually evolving craft. The software that we test changes over time, as do the methods and tools we use. Because of this, we must always learn new things: new languages, new tools, and new strategies. Let’s all learn something new in 2022!

Here are some resources you can use to learn new things:

Take a Course:
In 2019, I took this awesome course on Node.js and I learned enough to create a simple API and website to accompany my new book. This year I’m going to be taking this course on Android development with the idea that understanding more about mobile app development will help me be a better mobile tester. Here are some great resources for online learning:
Test Automation University– all of the courses are related to test automation, and they are completely free!
W3 Schools– another free resource, this site teaches HTML, CSS, SQL, and a whole host of software languages.
LinkedIn Learning– If you have a LinkedIn Pro membership, you can take LinkedIn Learning courses for free. If you don’t have a Pro membership, you can always purchase courses individually. Check out my course on API testing with Postman!
Udemy– the price tags on these courses look daunting, but they very frequently go on sale. This is where I took that great Node.js course.
Pluralsight– many workplaces offer memberships to Pluralsight as a perk of employment.

Read a Book:
Books are a great way to learn about software testing, because they dive deeper into testing concepts. Books are also easy to take with you when you are on a train, on your lunch break, or on vacation. Here are some software testing books I recommend:
The Complete Software Tester by Kristin Jackvony: Of course I’m going to recommend my new book! This book is a comprehensive look at all aspects of software testing, from manual testing through API testing, from coding basics through test automation, and from creating test plans through working effectively on a team.
Agile Testing Condensed by Janet Gregory and Lisa Crispin: This is a great introduction in how to test on an Agile team and how to help your whole team own quality. I’ve written a review of the book here.
Perfect Software, and Other Illusions About Testing by Gerald Weinberg: This book gives a fun, unbiased look at what happens when people test software. Read my review here.
The Way of the Web Tester by Jonathan Rasmusson: This is my favorite book for helping people get started with test automation. My review can be found here.
Continuous Testing for DevOps Professionals by Eran Kinsbruner: Once you’ve learned test automation, it’s time to set up continuous testing. Reading this book is a great way to get started. Read my review here.

Listen to a Podcast:
We all have hours in our day where we can’t read or look at a computer screen: when we’re driving, exercising, or folding laundry, for example. This is a great time to listen to podcasts, and there are a lot of testing podcasts out there! Here are some I recommend:
The Test Guild Podcasts: The gold standard of podcasts, Joe Colantonio offers the original Test Guild Automation podcast as well as two newer podcasts: the Test Guild Performance & SRE podcast, and the Test Guild Security podcast.
AB Testing: This podcast is hosted by the creators of the Modern Testing Principles, Alan Page and Brent Jensen. It’s always interesting to hear them talk about testing philosophy!
The Guilty Tester: A fun and irreverent look at the challenges and triumphs of software testing, presented by Dave Duke.
The QA Lead Podcast: Hosted by Jonathon Wright, this podcast looks at recent developments in software testing, with a special emphasis on leading QA teams.
The Testing Show: Presented by Qualitest and hosted by Matt Heusser and Michael Larson, this podcast features great guests and fascinating discussions.

Read a Blog:
There are SO many good software testing blogs to choose from! One great way to discover which blogs you enjoy the most is to read one of the weekly digests that aggregate the best posts of the week. Here are some good sites to try:
Software Testing Weekly by Dawid Dylowicz
CodingJag by LambdaTest
Test Automation Weekly
And here are some blogs I read regularly:
A Tester’s Journey by Lisi Hocke
A Seasoned Tester’s Crystal Ball by Maaret Pyhajarvi
Beth the Tester by Beth Marshall
Marie Drake’s blog
Satisfice blog by James Bach
Developsense blog by Michael Bolton

Attend a Conference:
One of the unexpected boons of COVID-19 was that many conferences went online. This means that you can attend conferences all over the world without plane fare, a hotel room, or expensive restaurant meals! Here are some conferences to attend:
Automation Guild: Joe Colantonio’s conference has been online from the beginning, and it offers really high-quality content. The 2022 conference is just a few weeks away (and I will be presenting)!
QA Global Summit: Presented by Geekle, this conference focuses on all aspects of software testing, from agile and devops to the tester’s role and test automation. I’m speaking at this conference as well, along with some well-known presenters!
TestFlix: This is a fun annual offering from the Test Tribe with bite-sized presentations about all areas of software testing. Click here to watch my presentation at the 2021 TestFlix.
TestBash: Brought to you by the Ministry of Testing, these online presentations, courses, and conferences cover a wide variety of topics from navigating your software testing career to writing test automation.

This has been a fairly extensive list of resources, and this is just scratching the surface! We are so blessed to be software testers in a time when there are so many great ways to share information. Don’t let your skills stagnate in 2022; learn something new!

Anyone who has focused on improving their health has learned that health begins with good habits: taking care of one’s teeth, exercising regularly, eating a healthy diet, and so on. Quick fixes like diet pills and one jog around the block might provide temporary improvement, but for permanent results, healthy habits are the way to go.

It occurred to me recently that this is true for software quality as well! It’s not enough to splurge on the latest test case management system or adopt the newest test framework; real software quality is the result of healthy testing habits! Below are six healthy testing habits you’ll want your team to adopt.

Habit 1: Check your overnight tests and fix any failures

Many software testers create automated tests that are designed to run overnight. But how many testers take the time to check and fix any failing tests? Too often our nightly test runs become repositories of mediocrity; if most of the tests pass, then we assume everything is OK and dismiss the flaky tests. With a test suite like that, how will you be alerted to actual problems?

Make a commitment to having zero flaky tests. Check your overnight test runs in the morning and rerun any failures. If you notice that you are getting false failures in specific tests, fix those tests so they won’t be flaky. Then you will have overnight tests that your team can truly rely upon.

Habit 2: Run unit and integration tests with every build

Hopefully the developers on your team have created unit tests for their code! When unit tests are run with every code commit, they provide incredibly fast feedback. Integration tests are also extremely helpful because they can alert the team to potential lost connections to the database or to another team’s API.

Set up your build system so that every build will run your unit and integration tests and fail if any tests fail. You’ll be pleased to see how well this practice keeps bugs from escaping to production.

Habit 3: Set up regular security checks

Security problems can pop up at unexpected times. Recently it’s become more common for malicious users to find and exploit package vulnerabilities. One way to avoid having vulnerable packages or other security holes in your code is to scan your code periodically. There are many automated scanning tools- both free and paid- available for this purpose. You can set your automated scanning tools to run once a week and alert you to any possible vulnerabilities. And don’t forget that you can also set up your own security automation to check for things like access control and security misconfiguration.

Habit 4: Run load tests before releasing

Do you know how your application behaves under load? Hopefully you have run load tests on your application at some point. But remember that every time there is a change to your application’s behavior- for example, adding a new API or altering a database query- the performance of your application may change. A healthy load testing habit involves load testing your application before you release any changes. If there is a significant slowdown in your app’s performance you’ll want to investigate that and make whatever changes are needed.

Habit 5: Ping your APIs at regular intervals.

Just because your API was working correctly last night, and the last time you deployed your software, doesn’t mean that it’s working correctly right now. All kinds of things can cause your API to be unresponsive. Perhaps someone in IT accidentally changed a firewall rule. Maybe a third-party API that your API relies upon has gone down. When things like this happen, you’ll want to find out about it before your customers do.

A great healthy habit is to set up a ping check on your APIs. You can set up a specific health endpoint that returns positive or negative results, or you can use an existing GET request. Whichever method you use, you’ll want to set your check to run every few minutes and alert you if something is wrong.

Habit 6: Set up monitors and alerts

How many problems has your team encountered with your application that could have been prevented if you were alerted to them ahead of time? Perhaps you ran into an issue where a server had reached maximum CPU and stopped responding to requests. Or maybe a database was filled to capacity and started rejecting all additions of new data. Setting up monitoring and alerting means that you can be notified of a problem before it becomes a disaster and affects your customers.

It’s important to make sure that you aren’t alerting too often, though! If you and your team get too many alerts, you’ll begin to suffer from “alert fatigue” and stop paying attention to them entirely. Work with your team to determine exactly what you should alert on and make sure that you and your team take turns responding to the alerts, so no one person bears all the responsibility.

It’s easy to see how healthy testing habits like these can greatly contribute to software quality! They aren’t as exciting as buying a new testing tool, but they are cheaper and more effective.

Web Accessibility means making a website easier to use and understand for people with visual, auditory, physical, or cognitive difficulties. Did you know that there are specific guidelines for how to make a website accessible? The guidelines are called the Web Content Accessibility Guidelines, or WCAG for short. The guidelines were created by the Web Accessibility Initiative, which is a subset of the World Wide Web Consortium (W3C). You can see all of the Accessibility Guidelines and learn how to meet them in this Quick Reference Guide.

When you first look at the guidelines, they can seem daunting because there’s so many of them! But don’t despair: it’s really easy to get started with accessibility testing. In this post, I’ll show you how to check a web page for accessibility and how you can make two quick fixes to your application’s code to make your page more accessible!

The easiest way to audit your website for accessibility is by using the WAVE tool. This extension is available for Chrome, Firefox, and Edge, and it’s completely free. To get the extension, simply go to https://wave.webaim.org/extension/ and click on the browser you’d like to use. Once the extension is installed, it will be available in your browser’s toolbar. To use the extension, begin by navigating to the page you’d like to check. Then click on the extension to turn it on. Instantly you will get a series of icons on your page that will show you where you are complying with WCAG and where you are in violation of the guidelines. If you click on the icons, you’ll get more information about what guideline is being violated or complied with, and there will be a link to the WCAG reference page and a link that will take you directly to your code.

Fixing accessibility violations is often very easy as well! Here are two easy fixes that anyone who is familiar with HTML can make:

Adding an alt-text to an image

When people with impaired vision use the Web, they usually rely on a screen reader. The screen reader tells them what elements are on the page. When there is an image on the page, if it doesn’t have an alt-text, the screen reader doesn’t know how to describe it. All that’s required is to add an alt-text that describes what’s in the image. Here’s an example of an image that’s missing an alt-text:
<img src=”/img/thinkingTesterLogo.png”>
And here’s an example of the same image with an alt-text added:
<img src=”/img/thinkingTesterLogo.png” alt=”Thinking Tester logo”>

Adding a “for” setting to a label:
When a user who relies on a screen reader fills out a web form, they need to know what all of the input fields on the form represent. It’s not enough just to have a label; having a “for” setting makes it every clear what the purpose of the input field is. Here’s an example of an input with a label that is missing the “for” setting:
<label>Email</label>
<input id=”email” placeholder=”Email”>
And here’s what that same field would look like with the “for” setting added:
<label for=”email address”>Email</label>
<input id=”email” placeholder=”Email”>

These simple changes have no impact on the visual aspects of the page, but they have a big impact on someone who relies on a screen reader! And they are so easy to do that anyone who can type and submit a pull request can make the changes. Many other fixes, such as making sure that your header elements are in the proper order, are easy to do as well.

Over 7 million people in the United States regularly use a screen reader, and of course they are also used frequently throughout the world. With just a few simple changes, it’s possible to give people with visual impairments a much better user experience.

Recently I was meeting with some coworkers who were looking to improve our Continuous Delivery practices. They were thinking of ways to measure our progress with automated testing, and one of the first suggestions was to measure code coverage.

“Nope,” I said. “Measuring code coverage isn’t helpful because it doesn’t indicate whether we have good tests; only that the tests execute certain parts of the code.”

The next suggestion was measuring the lines of test code. “No,” I said. “That won’t help either. Just as the number of pages in a book don’t indicate how good it is, the number of lines of test code won’t indicate how good the tests are.”

“OK,” they said. “How about the number of tests? Surely that would indicate our progress.”

“Nope, that won’t do it either,” I responded. “Because you could have hundreds of tests, and every one of them could be unreliable or testing the wrong things.”

At this point, they asked: “Well, then what DOES indicate a good test?” The answer to that question is the topic of this blog post! Below are six indications that you have a good automated test.

1. It tests something important

It’s possible to create automated tests to do all kinds of things, but you want to make sure that what you are actually testing is worth it. Code of any type requires maintenance, so it’s not a good idea to create tests just for the sake of having them.

Here’s an example of something you might not want to automate: let’s say that when you are testing a new page in your application, you discover that the “Last Name” header is misspelled “Lsat Name”. You log a bug, the developer fixes the issue, and you verify the fix. What are the chances that bug will appear again? Probably fairly low; so it makes no sense to write an automated test to check the spelling.

2. It fails when it should

Test engineers new to automation often forget to check that their automated test fails when it’s supposed to. This happened to me when I first started writing automation in JavaScript. I wrote a test to check an uploaded record and was so happy that it was passing; then my developer asked me to check if it would fail if the record didn’t match. It didn’t fail! It turned out that I didn’t understand promises; I was actually validating that the promise existed rather than validating the value of the web element. Having tests that pass 100% of the time regardless of the circumstances might look good in a test report, but they provide no value whatsoever.

3. It’s reliable

A test should provide accurate information 100% of the time. This level of perfection is probably not attainable, but the creator of the test should strive for this ideal. Flaky tests should be fixed or eliminated. A flaky test cannot provide reliable information: did it fail because there’s a bug present, or because of the flakiness?

4. It’s maintainable

An automated test suite filled with spaghetti code will be an incredible chore to maintain. Every time a change is made to the feature code, the test suite will take hours or days to update. Test code should be clean code that makes sense to read, is well-organized, and doesn’t repeat itself.

5. It runs quickly

An automated regression suite that takes eight hours to run is not going to provide fast feedback to a development team. We want tests that run as quickly as possible to let the team know if there’s a problem. There are a lot of ways to speed up test automation; my favorite way is to run more API tests than UI tests. I like to have as few UI tests as possible, testing all of the feature logic through the API instead. Other ways to speed up automation include running tests in parallel, and running a test setup once before running the test suite instead of running it before every test.

6. It runs at appropriate times

As software testers, we often want to test as much as possible, as often as possible. But this isn’t always the most efficient thing to do. Smoke tests are a good example of efficient testing. A smoke test is often used during a deployment. We want very few smoke tests so that we can get a clear indication immediately of how the deployment is behaving. It’s OK to run a longer suite of regression tests at some other time, perhaps overnight.

As you can see, determining whether a test is a good test is not simply a matter of looking at metrics. You need to be familiar with the product, understand what the product’s most critical features are, and be able to read the test for code quality and validity.

In my last post, I introduced the concept of the Quality Maturity Model: a series of behaviors that help teams attain various attributes of quality in their software. One of the things it’s important to note is that adopting a Quality Maturity Model requires the whole team to contribute to quality. Quality is not something to be thrown “over the wall” to testers; rather it is a goal that both developers and testers share.

But how can you get the whole team to own quality? One way is through the creation of a Quality Strategy. A Quality Strategy is a document that the whole team agrees on together. It’s like a contract that describes how quality software will developed, tested, and released by the team. In this post, I’ll discuss some of the questions you may want to answer in your team’s Quality Strategy.

Creating and Grooming Stories:
How does the team decide what stories to work on? This could be a decision by the whole team, or by the product owner only. Or the prioritization could be done by someone outside the team.

Who grooms the stories to get them ready for development? This could be the whole team or a subset of the team. Ideally, you’d want to have at least the product owner, one developer, and one tester participating.

Development Process:
How does the team decide who works on which story? It could be that the developers can pick any story from the board, or it could be that developers each have stories in specific feature areas that they can choose from. On some teams, software testers work on simple development stories as well, such as changing words or colors on a webpage or adding in automation ids to make automation easier.

What does “Done” look like for the story? Is it measured by meeting all of the acceptance criteria in the story? Is the developer required to add unit tests before the story can be considered done? How do you know that a feature is ready for testing? On many teams, it’s expected that the developer will do some initial testing to verify that what they coded is ready for further testing.

Feature Handoff:
How will a story be handed off for testing? On some teams this is done by simply moving the story into the “Testing” column on the story board. On other teams, a more formal handoff ceremony is required, where the developer demonstrates the working story and provides suggestions for further testing.

Who deploys the code to the test environment? This seems like a trivial thing, but it can actually be the cause of many misunderstandings and much wasted time. If the developer thinks it’s the tester’s job to deploy the code to the test environment, and the tester assumes that the developer has done it, the tester could begin testing and not realize that the new code is missing until after they have spent a significant amount of time working with the application.

Who will be doing the testing? On some teams, developers can pick up simple testing stories to help improve product velocity, while the more complex stories are left for the testing experts.

Test Plan Creation:
Who will create the test plans? How will they be created? Where will the test plans be stored? Some teams might prefer to do ad-hoc exploratory testing with minimal documentation. Other teams might have elaborate test case management systems that document all the tests for the product. And there are many other options in between. Whatever you choose should be right for your team and right for your product.

Who will write the test automation? On some teams, the developers write the unit tests, and the testers write the API and UI tests. On other teams, the developers write the unit and API tests, and the testers create the UI tests. Even better is to have both the developers and the testers share the responsibility for creating and maintaining the API and UI tests. In this way, the developers can contribute their code management expertise, while the testers contribute their expertise in knowing what should be tested.

Who will be doing other types of testing, such as security, performance, accessibility, and user experience testing? Some larger companies may have dedicated security and performance engineers who take care of this testing. Small startups might have only one development team that needs to be in charge of everything.

Test Tools:
What tools will be used for manual and automated testing? Selecting test tools is very important when you want the whole team to own testing. Developers will most likely want to use tools that use the same language they are using for development, because this minimizes how much context switching they’ll need to do.

Test Maintenance:
Who is responsible for maintaining the tests? It’s amazing how fast test automation can become out of date. One word change on a page can mean a failed test. Ideally, a team should have a “you break it, you fix it” policy where tests are fixed by the person who checked in the code that broke them. If that’s not possible, at least make sure that everyone on the team understands how the tests work and how to fix them in a situation where a fix is needed quickly.

Bugs and Tech Debt:
How are bugs handled when found in testing? Are they discussed by the developer and the tester, triaged by the whole team, or logged on a backlog to be looked at later? It’s often a good idea to fix bugs as soon as they are found, because the developer is already working in that section of code.

How will the team deal with tech debt? Does the team have an agreement to take on a certain amount of tech debt per sprint? Some teams have a policy that when a developer has run out of stories to work on, they pick up tech debt items from the backlog.

Releases:
What kind of testing will you do before a release? Will there be a regression test plan that the whole team can execute together? How about exploratory testing? One high-performing team I know gets together for exploratory testing right before they release. Using this strategy, they’ve uncovered tricky bugs and fixed them before they were released to production.

How will the software be released? In some companies there is a release manager who takes care of executing the release. In other companies, the team members take turns releasing the software. One very helpful technique is Continuous Deployment, where the software is automatically deployed and tests automatically run to verify the deployment to each environment, saving everyone time and effort.

Maintenance:
How will you measure the success of the release? Once software has been released, it’s easy for development teams to forget about it; but this is the time the users begin working with it. What kinds of metrics might you use to measure how well your product is working? You could keep track of defects reported by customers, or look at logs for unexpected errors.

How will you monitor the health of your application? It would be a good idea to have alerts set up so that you can find out about problems with your application before your users do. What kinds of behaviors should you be looking for?

Quality Strategies can be as varied as snowflakes. Imagine the differences between a small startup of ten people who are making a mobile chat app and a company of twenty thousand people who are designing software that flies airplanes. These two companies will need very different strategies! You can design a Quality Strategy that works well for your team by discussing these questions together and drafting a strategy that you can all agree upon.

One year ago, my company adopted something we call the “Quality Maturity Model”. We created the model to help teams measure how they are doing with behaviors that support creating quality applications. The project has been a big success, so I’ve decided to share some details about it with the world!

We started out by coming up with a definition of quality. Using this excellent blog post as a jumping-off point, we defined the seven Attributes of Quality at Paylocity.
A Quality application is:
Valuable: It meets the customer’s needs
Functional: It does what we say it does, and we can measure those interactions
Reliable: It is available when needed
Secure: It protects customer and company information
Performant: It responds within an acceptable time
Usable: It is easy and intuitive to use
Maintainable: It is easy to test, deploy, automate, monitor, update, and scale

After we had defined these attributes, we created a list of behaviors development teams could do that would help ensure those attributes were part of our products. For each of those behaviors, we determined what a minimum version of that behavior would look like, what a standard version would look like, and what excellent would look like. From there, we created the Quality Maturity Model.

Here are some examples of the behaviors defined in the Quality Maturity Model:
Valuable: Team identifies and investigates customer needs
Functional: Team creates, executes, monitors, and maintains reliable test automation
Reliable: Team actively monitors the health of their applications and takes appropriate action as needed
Secure: Team creates and adheres to a security strategy following security best practices
Performant: Team consistently meets SLO standards for their product
Usable: Team ensures the product is usable on multiple devices and supported browsers/platforms when applicable
Maintainable: Team manages and owns their deployments following the release management process

We rolled out the Quality Maturity Model to all the teams and asked them to identify which of these behaviors they were already demonstrating. From there, we asked teams to create quarterly goals to adopt more of the behaviors. Quality Leaders were each assigned a group of teams to meet with monthly to help answer any questions and hold teams accountable.

After a year of working on adoption of the model, we’ve made significant progress! Here are some examples:

A team committed to having the whole team own the test automation. The team works together to make sure that tests aren’t duplicated; for example, if a unit test already covers what’s needed to test a feature, there’s no need to write a UI test. This saves the team significant time in test creation and maintenance, freeing them up to focus on new features.

Another team made sure that the whole team knew how to use the UI automation framework. A developer was able to do a complete regression test on the UI work he was doing and fix all the bugs he found without involving anyone else on the team.

One tester on a team created a reusable test plan so the developers would be able to determine what to test. When both the testers on the team were on leave at the same time, the developers were able to carry on with feature development and testing with no problems.

A team was able to use the progress they made in test automation to speed up their release times from once a month to twice a month.

If you are looking for a way to enhance the quality of your product, minimize escaped defects, and speed up your delivery time, the Quality Maturity Model may be a great way to help. I recommend starting a discussion with the leaders at your company about what quality behaviors you’d like to see in your teams!