What Makes a Good Automated Test?

Recently I was meeting with some coworkers who were looking to improve our Continuous Delivery practices. They were thinking of ways to measure our progress with automated testing, and one of the first suggestions was to measure code coverage.

“Nope,” I said. “Measuring code coverage isn’t helpful because it doesn’t indicate whether we have good tests; only that the tests execute certain parts of the code.”

The next suggestion was measuring the lines of test code. “No,” I said. “That won’t help either. Just as the number of pages in a book don’t indicate how good it is, the number of lines of test code won’t indicate how good the tests are.”

“OK,” they said. “How about the number of tests? Surely that would indicate our progress.”

“Nope, that won’t do it either,” I responded. “Because you could have hundreds of tests, and every one of them could be unreliable or testing the wrong things.”

At this point, they asked: “Well, then what DOES indicate a good test?” The answer to that question is the topic of this blog post! Below are six indications that you have a good automated test.

1. It tests something important

It’s possible to create automated tests to do all kinds of things, but you want to make sure that what you are actually testing is worth it. Code of any type requires maintenance, so it’s not a good idea to create tests just for the sake of having them.

Here’s an example of something you might not want to automate: let’s say that when you are testing a new page in your application, you discover that the “Last Name” header is misspelled “Lsat Name”. You log a bug, the developer fixes the issue, and you verify the fix. What are the chances that bug will appear again? Probably fairly low; so it makes no sense to write an automated test to check the spelling.

2. It fails when it should

Test engineers new to automation often forget to check that their automated test fails when it’s supposed to. This happened to me when I first started writing automation in JavaScript. I wrote a test to check an uploaded record and was so happy that it was passing; then my developer asked me to check if it would fail if the record didn’t match. It didn’t fail! It turned out that I didn’t understand promises; I was actually validating that the promise existed rather than validating the value of the web element. Having tests that pass 100% of the time regardless of the circumstances might look good in a test report, but they provide no value whatsoever.

3. It’s reliable

A test should provide accurate information 100% of the time. This level of perfection is probably not attainable, but the creator of the test should strive for this ideal. Flaky tests should be fixed or eliminated. A flaky test cannot provide reliable information: did it fail because there’s a bug present, or because of the flakiness?

4. It’s maintainable

An automated test suite filled with spaghetti code will be an incredible chore to maintain. Every time a change is made to the feature code, the test suite will take hours or days to update. Test code should be clean code that makes sense to read, is well-organized, and doesn’t repeat itself.

5. It runs quickly

An automated regression suite that takes eight hours to run is not going to provide fast feedback to a development team. We want tests that run as quickly as possible to let the team know if there’s a problem. There are a lot of ways to speed up test automation; my favorite way is to run more API tests than UI tests. I like to have as few UI tests as possible, testing all of the feature logic through the API instead. Other ways to speed up automation include running tests in parallel, and running a test setup once before running the test suite instead of running it before every test.

6. It runs at appropriate times

As software testers, we often want to test as much as possible, as often as possible. But this isn’t always the most efficient thing to do. Smoke tests are a good example of efficient testing. A smoke test is often used during a deployment. We want very few smoke tests so that we can get a clear indication immediately of how the deployment is behaving. It’s OK to run a longer suite of regression tests at some other time, perhaps overnight.

As you can see, determining whether a test is a good test is not simply a matter of looking at metrics. You need to be familiar with the product, understand what the product’s most critical features are, and be able to read the test for code quality and validity.