Easy Free Automation Part V: Visual Tests

Visual tests are more than just UI tests; they verify that what you are expecting to see in a browser window is actually rendered.  Traditional UI tests might verify that a page element exists or that it can be clicked on, but they don’t validate what the element looks like.  Fortunately, there are a variety of ways to do visual validation.  My favorite way is through Applitools, but since this series of posts is on “Easy Free Automation”, I needed to look elsewhere for visual validation this week.

I settled on a fun tool called BackstopJS.  It was pretty easy to set up, and while I haven’t yet discovered everything that it can do, I created a simple example with two tests that you can clone or download here.

Once you have cloned or downloaded these files, it’s time to install BackstopJS.  I’m assuming that you already have Node and npm installed; if you don’t, see last week’s post for instructions.  To install BackstopJS, simply open your command window and type npm install -g backstop.js.  

After the installation completes, navigate using the command line to the folder where you have downloaded the easyFreeVisualTests files, and type backstop test.  This will run the visual tests for the first time, and when the tests complete, a browser window will pop up with the test results.  You’ll notice that the tests all failed; this is expected, and we’ll fix this in a moment.  Before we do, take a look at the test results.  You should see four tests: two tests with screenshots of Google’s homepage, and two tests with screenshots of my picture (taken from my personal website).  The tests failed because at the moment, there is nothing to compare the screenshots to.  We can accept the screenshots and use them for future comparisons by typing backstop approve in the command line.  Now run backstop test again, and you will see the four tests run and pass.

Let’s take a look at the backstop.json file to see how these tests are configured.  Near the top of the file, we see the viewports section:

This section is showing what screen sizes we want to test with.  In this case, we are testing with a phone screen size and a laptop browser size.

A little farther down, we see the scenarios section.  The first scenario is for the Google Search Homepage:

We see the label of the test, along with the url.  Also of note is the “misMatchThreshold”.  Setting this to a higher number means that the test will allow for a higher percentage of pixel mismatches before failing the test.

The second scenario is for my personal webpage:

Note that this scenario is using the “selectors” section.  In this section, I’m specifying the specific element I want to look at: my picture.  When the test runs, rather than taking a screenshot of the entire page, it takes a screenshot of this single element and compares it to the baseline screenshot.  This is great for pages that have content that changes frequently; you can set your test to look only at the elements that stay the same.

If you want to add BackstopJS to your own Javascript project, it’s easy to do!  Simply navigate to your project folder in the command line and run backstop init. This will add a backstop.js file to your existing project, and you can configure it as needed.

It would certainly be possible to have both Backstop and other UI automated tests installed in the same project.  When your automation runs, you could run through all of your UI tests first, then run your Backstop tests to verify all your visual elements are appearing as expected.

Next week, we’ll move on to automated security tests!

Easy Free Automation Part IV: UI Tests

I’ll be honest: UI tests are my least favorite automated tests.  This is because they are often so hard to set up.  There are dozens of different ways to run automated UI tests, but this can make things more confusing because it’s hard for someone new to automation to figure out what to do.

So when I prepared to write this week’s post, my primary goal was to make it as easy as possible to get started with UI testing.  And of course, I also wanted the framework to be free, with no need to purchase a tool.

I’ve arrived at a way to get up and running with UI automation using Node and Selenium Webdriver in just six steps.  While I have only tested this process on two computers, I believe these steps will be effective for most people.  The one prerequisite is that you need to have Chrome installed, because that’s the browser that we will be using for the test.

Setting Up Automated UI Testing in Six Easy Steps:

1. Open up a command window and verify that you have Node.js installed by typing
node –version
If you get a version number in response, you have Node installed.  If it’s not installed, you can install it here: https://nodejs.org/en/.

2. If you needed to install Node in Step 1, add Node to your system’s PATH (see instructions for Windows here and instruction for Mac here).  After you’ve done this, reboot your computer so the new PATH will be recognized.  Check one more time that Node is installed by typing
node –version again.

3. When you install Node, the npm package manager should be installed automatically.  Verify that npm is installed by typing
npm –version
If you get a version number in response, npm has been installed.  If not, you can check this link for instructions about installing npm.

4. Open a browser and go to this GitHub repo.   If you have Git installed, you can clone the project.  If not, you can download a zipfile of the project and extract it.

5. In the command window, navigate to the folder where the project has been installed.  The project folder should contain a test.js file and a package.json file.  (See instructions here about navigating in the command line.)  Type this command:
npm install
This will install everything you need to run the automated test.

6. Type this command:
node test
This will run the test.js file, which has one test.  You should see an instance of Chrome browser open, run the test, and close again!

Let’s take a look at what the test.js file does:

var webdriver = require(‘selenium-webdriver’),
    By = webdriver.By,
    until = webdriver.until;
var chrome = require(‘selenium-webdriver/chrome’);
var path = require(‘chromedriver’).path;

var service = new chrome.ServiceBuilder(path).build();
chrome.setDefaultService(service);

var driver = new webdriver.Builder()
    .withCapabilities(webdriver.Capabilities.chrome())
    .build(); 

The purpose of all of this code is to require webdriver and chrome driver, setting up the Chrome driver, and to set up the “By” and “until” classes, which are helpful for automated UI testing.

driver.get(‘http://www.google.com’);

This command tells the driver to navigate to the Google home page.

driver.findElement(By.name(‘q’)).sendKeys(‘webdrivern’);

This command tells the driver to find the web element named “q”, which is the search box; type “webdriver” into the search box; and click the Return key.

driver.findElement(By.partialLinkText(“seleniumhq”)).click();

This command looks through the search responses for the element that has “seleniumhq” in the link text, and once the element has been found, it clicks on it.

driver.wait(until.elementLocated(By.id(‘sidebar’)));

This is waiting for the Selenium Webdriver page to load by watching for the element with the id called ‘sidebar’.

driver.getTitle().then(function(title) {
    if(title === ‘Selenium WebDriver’) {
      console.log(‘Test passed’);
    } else {
      console.log(‘Test failed’);
    }
    driver.quit();
});

Once the element has been located, then the driver looks at the title of the page and checks to see if it is what was expected.  If the title matches “Selenium Webdriver”, it logs to the console that the test passed, and if it does not match, it logs to the console that the test failed.  Finally, the driver closes the browser window.

Hopefully this post has helped you with the most difficult part of automated UI testing- the setup!  Once you are up and running, there are lots of great tutorials that describe how to locate and interact with browser elements using Webdriver.  You can find the documentation for the Webdriver By class here, and I have some old blog posts about element locators here, here, and here.   The posts were written for Java, but the same concepts apply when you are locating elements in Node.

The most important thing to remember about automated UI testing is that it should be done sparingly!  Whatever you can test with unit and services tests should be tested that way instead.  UI testing is best for validating that elements are on a web page, and for running through simple user workflows.  Next week, we’ll go on to an important addition to UI testing: visual testing.

UPDATE: If you are experiencing an issue where you get an “unhandled promise rejection”, try running this command:  npm install [email protected] and then try running the test again.

Easy Free Automation Part III: Services Tests

This week we are looking at my favorite type of automated tests: services tests.  The reason I love services tests is because they test so much of the application without the hassle of the UI.  There are many types of services, but the most widely used service is the REST API.  And my favorite way to test a REST API is with Postman

I’m not going to get into too many details about REST verbs or about Postman, because I have already done so in earlier blog posts.  If you’d like to catch up on what I’ve written, you can start with this post and work your way forward week by week.  You can also read this tutorial that I wrote for Simple Programmer.  
This post will focus on how you can automate Services Tests.  To demonstrate, I’ve created a collection of Postman tests that you can download from Github here.  The API I’m testing is called Restful-Booker: it’s a great easy API created by Mark Winteringham to help people learn how to test APIs.  
Once you have downloaded the two files from Github, you can upload them in Postman to see what they do.  To upload the collection, click on the Import button on the top left of the Postman window:
When the Import window pops up, click the Choose Files button and then navigate to the location of the file you downloaded called restfulbooker.collection.json.  Click on that file name, and the collection will be imported:  
Next, you’ll need to upload the environment file.  The environment file is what contains the variables that are used in the requests and assertions.  To upload the environment file, click on the gear icon in the top right of the Postman window:
When the environment window pops up, click on the Import button, then on the Choose Files button, then navigate to the location of the downloaded file called restfulbooker.env.json.  Click on that file name and the environment will be imported: 
Finally, in the dropdown menu in the top right, select the Restful Booker environment: 
You are now ready to run the requests from within Postman.  Keep in mind that the tests are not idempotent: some of the requests depend on previous requests for setting variables and creating bookings, so while you are exploring you should run the requests in order.
Let’s take a look at one of the requests and its assertions:
This is a GET request.  The URL of the request is https://restful-booker.herokuapp.com/booking/1.  The request is asking for the booking with the id of 1.  You can see two tests in the Tests section: the first test verifies that the response code of the request is 200, and the second test verifies that “firstname” is one of the fields in the response.  
Click the Send button to run the request, then scroll down to the bottom of the page:
Here you see the body of the response, which has returned a booking.  You can also see a tab called Test Results, which shows that 2 out of 2 tests pass.  If you click on this tab, you can see your results:
Now that you understand how the tests and assertions work, let’s run them from the command line!  In order to do this, you will need to have node.js installed.  Installing node will also install npm, which is the node package manager.  Then you can use npm to install Newman, the package that is used to run Postman tests.  To install Newman, simply open your command-line window and type: 
npm install -g newman

Once Newman is installed, navigate in the command-line window to the place where your restfulbooker.collection.json and restfulbooker.environment.json files are located.  Now you can run your tests with this command::

newman run restfulbooker.collection.json -e restfulbooker.environment.json

(This command should be in a line with no returns; it’s OK if it wraps around to a second line, though.)

The “-e” in this command stands for “environment”.  If all goes well, you should get a result that looks like this:  
You can also run your tests from other file locations.  To do this, simply specify the path to your files in your command.  For example, if my files were in a folder called NightlyTests, which was in my Documents folder, and if I were using a Windows computer, I would use this command:
newman run C:UsersKJackvonyDocumentsNightlyTestsrestfulbooker.collection.json -e C:UsersKJackvonyDocumentsNightlyTestsrestfulbooker.environment.json

Now that you know how to run your tests from the command line, you can set up your tests to run automatically with a cron job or in a CI/CD platform such as Jenkins.  Just be aware that whatever machine you run your tests on will need to have Newman installed.  

You can also have your test results write to a file with the -r command.  For example, you can have your results written in junit with this command:

newman run restfulbooker.collection.json -e restfulbooker.environment.json -r junit

When the tests are finished running, a Newman folder will be created with your test results inside.

I hope that this tutorial has given you some ideas about how you can automate your API testing!  Next week we’ll move on to UI tests with Selenium.  

Easy Free Automation Part II: Component Tests

Last week, I started an eight-part series to demonstrate in a free and very easy way how to write automation for each test type in the Automation Test Wheel.  This week, we’re taking a look at component tests.

As with every term in software testing, component tests mean different things to different people.  I like to define a component test as a test for a service that an application is dependent on.  For example, an application might need to make calls to a database, so I would have a component test that made a simple call to that database and verified that it received data in response.  Another example of a component would be an API that the application doesn’t own.  In this scenario, I would make a simple call to the API and verify that I got a 200-level response.

Coming up with a free and easy example of automated component tests was, unfortunately, not that easy!  But I have created a very simple Node.js application, which you can download from Github here.

In order to run this application and have my two tests pass, you’ll need to have Node, npm, and MongoDB installed, and you’ll need to create a very simple Mongo database with just one item in it.  I used most of the instructions in this really awesome tutorial by Chris Buecheler at CloseBrace.com to create this application.  You can use my application along with the instructions in Part 3 of the tutorial to make your Mongo database.  Or you can just clone my application and run it, with the understanding that one of the tests will fail. Or if this seems like too much work, you can just read on and look at my test code!

My extremely simple app is dependent on two things: my Mongo database, and an external API (the really great Restful-Booker API, which I’ll be using in next week’s blog).  For my component tests, I want to test those two dependencies: that I can make a request to the database and get a positive response, and that I can make a call to the Restful-Booker API and get a positive response. 

I am using Jest and Supertest for these tests.  I have limited experience with Jest, but from what I have seen so far, it is very easy to set up.  Supertest is a library that enhances Javascript testing by making it easier to call APIs. 

I’ve put my tests in a file called index.test.js, and this is what it looks like:

const request = require(‘supertest’);

describe(‘Database Connection Test’, () => {
  it(‘Returns a 200 with a call to the DB’, async () => {
    const res = await request(‘http://localhost:3000’)
      .get(‘/userlist’)
      .expect(200)
  });
});

describe(‘Restful-Booker Connection Test’, () => {
  it(‘Returns a 201 with a health check’, async () => {
    const res = await request(‘https://restful-  booker.herokuapp.com’)
      .get(‘/ping’)
      .expect(201)
  });
});

The first line of my file is invoking Supertest, so I will be able to do HTTP requests.  Let’s take a look at each part of the first test so we can see what it’s doing:

describe(‘Database Connection Test’, () => {
  it(‘Returns a 200 with a call to the DB’, async () => {
    const res = await request(‘http://localhost:3000’)
      .get(‘/userlist’)
      .expect(200)
  });
});

The “describe” section comprises the entire test. 

describe(‘Database Connection Test’, () => {
  it(‘Returns a 200 with a call to the DB’, async () => {
    const res = await request(‘http://localhost:3000’)
      .get(‘/userlist’)
      .expect(200)
  });
});

‘Database Connection Test’ is the title of the test.

describe(‘Database Connection Test’, () => {
  it(‘Returns a 200 with a call to the DB’, async () => {
    const res = await request(‘http://localhost:3000’)
      .get(‘/userlist’)
      .expect(200)
  });
});

The “it” section is where the assertion is called.

describe(‘Database Connection Test’, () => {
  it(‘Returns a 200 with a call to the DB’, async () => {
    const res = await request(‘http://localhost:3000’)
      .get(‘/userlist’)
      .expect(200)
  });
});

‘Returns a 200 with a call to the DB’ is the title of the assertion.

describe(‘Database Connection Test’, () => {
  it(‘Returns a 200 with a call to the DB’, async () => {
    const res = await request(‘http://localhost:3000’)
      .get(‘/userlist’)
      .expect(200)
  });
});

Here is where we are making a GET request to ‘http://localhost:/3000/userlist’.

describe(‘Database Connection Test’, () => {
  it(‘Returns a 200 with a call to the DB’, async () => {
    const res = await request(‘http://localhost:3000’)
      .get(‘/userlist’)
      .expect(200)
  });
});

Here is where we are expecting that we will get a 200 response.

If you’d like to try to run the tests on your own, assuming you have cloned the application, you can do so with these commands:

cd easyFreeComponentTests (this will move you to the directory where you cloned the application)
npm install (this will install all the components you need to run the application)
npm start (this will start the application)

Then go to a new command-line window, cd to where the application is located, and run:

npm test (this will run the tests)

That was a lot of information for just two component tests!  But remember it will be easier to get started when there is already a project to test.  How many tests you have in this area will depend on how many external systems your application is dependent on.  It may be a good idea to create a “health check” that will run your component tests whenever your code is deployed.  That way you will be alerted if there is any error when calling your external systems. 

Next week, we’ll move on to my favorite test type: services tests!

Easy Free Automation Part I: Unit Tests

This post is the beginning of an eight-part series on easy, free ways to automate each area of the Automation Test Wheel.  It’s been my experience that there are a number of barriers to learning test automation.  First, the team you are on might not need certain types of automation.  For example, my team has been solely API-focused, so for the last two years I haven’t had much reason to do UI automation.  Second, your company may already have invested in specific automation tools, so when you want to learn to use a different tool, you need to do it on your own.  Third, there are many tools that have barriers to using them, such as a high cost or a complicated setup.  And finally, there is not always good documentation out there to help people get started.

In this series, I’m hoping to provide simple, free examples that will demonstrate each area of the Automation Test Wheel in practice, which you can use as a jumping-off point in your own automation journey.  We’ll begin with unit tests.

Unit tests are usually written by developers, but it’s a good idea for all software testers to understand what they are and how they work.  Unit tests test just one method or function, and they aim to exercise as many paths of that method or function as possible.  The major benefit of unit tests is that they provide extremely fast, accurate feedback.
I’m going to use Python and Pytest to demonstrate how unit tests work.  I am admittedly not an expert in either, but I managed to put together a little project with just one function.  If you would like to try the code out yourself, I have added it to GitHub here.  (If you have never cloned a Git repository before, you can find some instructions here.)
To run the tests, you will need to have Python installed.  There are some great directions for installing Python here.  Installing Python 3 will most likely install pip, the Python package installer, but if it doesn’t, you can get installation instructions here.  Finally, you’ll need to use pip to install Pytest; there are instructions for that here.  (If all this seems like too much work, you can just read on and look at the examples.)
In the file called _init_.py, I’ve written a simple function called isItADozen that determines whether a number of objects equals a dozen.  Here’s the code:
def isItADozen(input): 
if type(input) != int:
result = “This is not a number”
elif input == 12:
result = “Yup, it’s a dozen!”
elif input < 12 and input > 0:
result = “Nope, you have less than a dozen”
elif input > 12:
result = “You have more than a dozen here”
elif input <=0:
result = “You don’t have any at all!”
return result

The function takes the input and first checks to see if it’s an integer.  If not, it returns the message “This is not a number”.  Then it checks to see whether the number is 12, or is more than 12, or is between 0 and 12, or is less than 0.  Depending on which statement is true, it will return an appropriate message.  
Before we look at the unit tests, let’s think about how we would test this function manually, assuming it had a UI interface.  We’d check to make sure it recognized 12 as a dozen, we’d check to make sure it recognized when a number was more than or less than a dozen, and we’d also try some of the usual QA tricks, like passing in a negative number or “FOO”.  
That’s exactly what we’ll do with our unit tests!  Here is the code in my test.py file:
import unittest

from my_test import isItADozen

class TestDozen(unittest.TestCase):
    def test_a_dozen(self):
        result = isItADozen(12)
        self.assertEqual(result, “Yup, it’s a dozen!”)

    def test_more_than_a_dozen(self):
    result = isItADozen(15)
    self.assertEqual(result, “You have more than a dozen here”)

    def test_less_than_a_dozen(self):
    result = isItADozen(10)
    self.assertEqual(result, “Nope, you have less than a dozen”)

    def test_less_than_zero(self):
    result = isItADozen(-1)
    self.assertEqual(result, “You don’t have any at all!”)

    def test_not_a_number(self):
    result = isItADozen(“FOO”)
    self.assertEqual(result, “This is not a number”)

if __name__ == ‘__main__’:
    unittest.main()

I have five test cases here, aptly named:
test_a_dozen
test_more_than_a_dozen
test_less_than_a_dozen
test_less_than_zero
test_not_a_number

In each test, I call the isItADozen function with the number I want to test with.  Then I assert that the result I got matches the result I was expecting.  

To run my tests, I simply go to the command line, navigate to the unitTestProject folder, and type:

python3 test.py –verbose

(If you don’t know how to use your command line, see my blog post from several years ago.)

You may not need to use the “3” in “python3”- I need to do this because I have Python 2.7.1 installed as well and I want to run on the newer version.  You don’t need to do the “–verbose” command; I like to do this because then it shows me the names of my tests as they pass instead of “…..”, which isn’t much fun.  
Once you have the tests running, try making one fail by changing one of the assert statements like this:
self.assertEqual(result, “This is not the real result message”)
Then see if you can change a test so that you are passing in a different value.  For example, you could pass the number 13 into the “test_more_than_a_dozen” test instead of 15.  
Once you have mastered that, you may want to copy my entire test folder and see if you can write your own simple function and unit tests.  If you are more familiar with another programming language, you can try writing unit tests in that language as well.  
Hopefully this has been an easy way to get you started with writing unit tests!  We’ll continue with the Easy Free Automation series next week.  

Automation Wheel Strategy: Moving from What to How to When to Where

Last week, we talked about how I would decide what to test in a simple application in terms of testing every segment of the Automation Test Wheel.  I find it’s very helpful to answer the question “What do I want to test?” before I think about how I’m going to test it.  This week we’ll look at how to take the “What” of automated testing and continue on with how I want to test, when I want to test it, and where (what environment) I’m going to test it in.  As a reminder, my hypothetical application is a simple web app called Contact List, which allows a user to add, edit, and delete their contacts.


How I’m Going to Test:

I’m going to run my unit and component tests directly in the code.  Unit tests are designed to specifically run in the code, because they are the smallest possible unit and test the code directly.  My component tests are very simple- just one call to the database and one call for authentication- so I will run those directly from my code as well.
For my services tests, I’m going to use Postman, which is my favorite API testing tool.  I’ll run the Postman tests using Newman, which is the command-line tool for Postman.  I’ll also include some security tests in Postman, validating that any requests without appropriate authentication return an appropriate error, and I will also do some performance checks here, verifying that the response times to my API requests are within acceptable levels.
For my UI tests, I’m going to use Selenium and Jasmine, because I like the assertion style that Jasmine uses.  I’ll be adding a few security tests here, making sure that pages do not load when the user doesn’t have access to them.  I’ll also be integrating my visual tests into my Selenium tests, using Applitools, and I’ll be using both Selenium and Applitools to run my accessibility tests. 
Finally, I would set up a performance testing tool such as Pingdom that would consistently monitor my web page loading times and alert me when load times have slowed.  
When I’m Going to Test:

Now that I’ve figured out how I’m going to test, it’s time to think about when I’m going to run my tests.  I’m going to organize my tests into four times.
With every build: every time new code is pushed, I’m going to run my unit tests, component tests, and Newman tests.  These tests will give me very fast feedback.  I’m not going to run any UI tests at this time, because I don’t want to slow my feedback down.
With every deploy: every time code is deployed to an environment, I’m going to run all my Newman tests, and a small subset of my Jasmine tests.  My Jasmine tests will include at least one visual check and one security check.  This will ensure that the API is running exactly as it should and that there are no glaring errors in the UI.
Daily: I’ll want to run all of my Newman tests and all of my Jasmine tests early in the morning, before I start my workday.  When I begin my workday I’ll have a clear indication of the health of my application.
Ongoing: As mentioned above, I’ll have Pingdom monitoring my page load times throughout the day to alert me of any performance problems.  I’ll also set up a job to run a small set of Newman tests periodically throughout the day to alert me of any server downtime.
Where I’m Going to Test:
Now that I’ve decided how and when to test, I need to think about where to test.  Let’s imagine that my application has four different environments: Dev, QA, Stage, and Prod.  My Dev environment is solely for developers.  My QA environment is where code will be deployed for manual and exploratory testing.  My Stage environment is where a release candidate will be prepared for Production.  Let’s look at what I will test in each environment.
Dev: My unit and component tests will run here whenever a build is run, as well as my Newman and Jasmine tests whenever a deploy is run.
QA: I’ll run my full daily Newman and Jasmine suites here, and I’ll run my full Newman suite and a smaller Jasmine suite with a deploy.
Stage: I’ll run the full sets of Newman and Jasmine tests when I deploy.  This is because the Stage environment is the last stop before Prod, and I’ll want to make sure we haven’t missed any bugs.  I’ll also run my Pingdom monitoring here, to catch any possible performance issues before we go to Prod.
Prod: I’ll run a small set of daily Newman and Jasmine tests here.  I’ll also point my Pingdom tests to this environment, and I’ll have those tests and a set of Newman tests running periodically throughout the day.
Putting it All Together:

When viewed in prose form, this all looks very complicated.  But we have actually managed to simplify things down to four major test modalities tested at four different times.  Are we covering all the areas of the Automation Test Wheel?  Let’s take a look:
Code Newman Jasmine Pingdom
Unit Services UI Performance
Component Security Security
Performance Visual
Accessibility
We are covering each different area with one or more testing modalities.  Now let’s visualize our complete test plan:
Hourly Daily Build Deploy
Dev Unit Newman
Component Jasmine
Newman
QA Newman Newman
Jasmine Jasmine
Stage Pingdom Newman
Jasmine
Prod Pingdom Newman Newman
Newman Jasmine Jasmine
Viewed in a grid like this, our plan looks quite simple!  By considering each question in turn:
  • What do we want to test?
  • How are we going to test it?
  • When will we run our tests?
  • Where will we run them?

We’ve been able to come up with a comprehensive plan that covers all areas of the testing wheel and tests our application thoroughly and efficiently.  

The Automation Test Wheel in Practice

Last week’s blog post, “Rethinking the Pyramid: The Automation Test Wheel“, sparked many interesting discussions on LinkedIn, Twitter, and in the comments section of this blog!  The general consensus was that the Test Pyramid is still useful because it reminds us that tests closest to the code are the fastest and most reliable to run, and that the Automation Test Wheel reminds us to make sure to include categories such as security, accessibility, and performance testing.  Also, a reader pointed us to Abstracta’s Software Testing Wheel, which looks at the definition of quality from a number of different perspectives.

This week I’m talking about how to put the Automation Test Wheel into practice.  Let’s imagine that I have a simple web app called Contact List.  It allows a user to log in, view a list of their contacts, and add new contacts.  I want to design a complete automation strategy for this application that will enable my team to deploy all the way up to production confidently.  In order to feel confident about the quality of my application, I’ll want to be sure to include tests from every segment of the Automation Test Wheel.

Unit Tests: I will make sure that every function of my code has at least one unit test.  I’ll run these tests using mock objects.  For example, I will create a list of mock contacts and a mock new contact, add the new contact, and verify that the new contact has been added to the list of mock contacts.  I’ll update a contact with new data and verify that the contact has been updated in the list.  I’ll create a mock contact with invalid data and verify that attempting to add the contact results in an appropriate error.  These are just some examples; for each function in my app, I’ll want to have several tests which exercise all possible code paths.

Component Tests:  My application is very simple and relies on just one database.  The database is used for both authentication and for retrieving the contact data.  I will include one test for each function; I’ll send an authentication request for a valid user and verify that the user is authenticated, and I’ll make one request to the database to retrieve a known contact, and verify that the contact is retrieved.

Services Tests: My application has an API which allows me to do CRUD operations (Create, Read, Update, Delete) on my contacts.  I have a GET endpoint which allows me to retrieve the list of contacts, and a GET endpoint which allows me retrieve one specific contact.  I have a POST endpoint which allows me to add a contact to the contact list.  I have a PUT endpoint which allows me to update the data for an existing contact, and I have a DELETE endpoint which allows me to delete an existing contact.  For each one of these endpoints, I will have a series of tests.  The tests will include both happy paths and error paths.  I’ll verify that in each request, the response code is correct and the response body is correct.  For example, with the GET endpoint where I retrieve one contact, I’ll verify that a GET on an existing contact returns a 200 response and the correct data for the contact.  I’ll also verify that a GET on a contact that doesn’t exist returns a 404 Not Found response.

User Interface (UI) Tests: This is where I will be testing in the browser, doing activities that a real user would do. A real user will want to fetch their list of contacts, add a new contact, update an existing contact, and delete a contact.  I will have one test for each of these activities, and each test will have a series of assertions.  To take one example, when I add a new contact, I will navigate to the new contact page, fill in all the form fields, and click the Save button.  Then I will navigate to the list page and verify that my new contact appears on the page.

Visual Tests: This is where I will verify that elements are actually appearing on the page the way I want them to.  I will navigate to the list page and verify that all of the columns are appearing on the page.  I will navigate to the add contact page and verify that all of the form fields and their labels are appearing appropriately on the page.  I will trigger all possible error messages (such as the one I would receive if I entered an invalid zip code), and verify that the error appears correctly on the screen.  And I will verify that all of the buttons needed to use the application are rendering correctly.

Security Tests: I will run security tests at both the Services layer and the UI layer.  I will test the API operations relating to authenticating a user, verifying that only a user with the correct credentials will be authenticated.  I will test every request endpoint to make sure that only those requests with a valid token are executing; requests without a valid token should return a 401.  For the UI layer, I will conduct a series of login tests that validate that only a user with correct credentials is logged in, and I will verify that I cannot navigate to the list page or the add contact page without being logged in.

Performance Tests: I will set benchmarks for both the server response time and the web page load time.  To measure the server response, I will add assertions to my existing Services tests that will verify that the response was returned within that benchmark.  To measure the web page load time, I will run a UI test that will load each page and assert that the page was loaded within the benchmark time.

Accessibility Tests:  I want to make sure that my application can be used by those with visual difficulties.  So I will run a set of UI and Visual tests on each page where I validate that I can zoom in and out on the text and that scroll bars appear and disappear depending on whether they are needed.  For example, if I zoom in on the contact list I will now need a vertical scrollbar, because some of the contacts will now be off the page.

With this series of automated tests, I will feel confident that I’ll be able to deploy changes to my application and discover any problems quickly.

I’ve received a few questions over the last week about what percentage of total tests each of spokes in the Automation Test Wheel should have.  The answer will always be “It depends”.  It will depend on these and many other considerations:

  • How many other services does your application depend on?  If it depends on many external services, you’ll need more Component tests.
  • How complicated is your UI?  If it has just a page or two, you’ll need fewer UI and Visual tests.  If it has several pages with many images, you’ll need more UI and Visual tests.
  • How complicated is your data structure?  If you are dealing with large data objects, you’ll need more Services tests to validate that CRUD operations are being handled correctly.
  • How secure does your application need to be?  An application that handles personal banking will need many more Security tests than an application that saves pictures of kittens.
  • How performant does your application need to be?  A solitaire game doesn’t need to be as reliable as a heart monitor.
The beauty of the Automation Test Wheel is that it can be tailored to all types of software applications!  By considering each spoke in the wheel, we’ll be sure that we are creating great automated test coverage.

Rethinking the Pyramid: The Automation Test Wheel

Anyone who has spent time working on test automation has likely heard of the Test Automation Pyramid.  The pyramid is typically made of three horizontal sections: UI Tests, API Tests, and Unit Tests.  The bottom section is the widest section, and is for the unit tests.  The idea is that there should be more unit tests run than any other kind of tests.  The middle section is for the API tests, and the idea is that fewer API tests should be run than unit tests.  Finally, the top section is for the UI tests, and the idea is that the least number of tests run should be UI tests, because they take the most time and are the most fragile.

In my opinion there are two things wrong with the pyramid: it leaves out many types of automated tests, and it assumes that the number of tests is the best indicator of appropriate test coverage.  I propose a new way of thinking about automated testing:  the Automation Test Wheel.

Each of these test types can be considered as spokes in a wheel; none is more important than another, and they are all necessary.  The size of each section of the wheel does not indicate the quantity of the tests to be automated; each test type should have the number of tests that are needed in order to verify quality in that area.  Let’s take a look at each test type.

Unit Tests:  A unit test is the smallest automated test possible.  It tests the behavior of just one function or method.  For example, if I had a method that tested whether a number was zero, I could write these unit tests:

  • A test that passes a zero to the method and validates that it is identified as a zero
  • A test that passes a one to the method and validates that it is identified as non-zero
  • A test that passes a string to the method and validates that the appropriate exception is thrown
Because unit tests are independent of all other services and because they run so quickly, they are a very effective way of testing code.  They are often written by the developer who wrote the method or function, but they can also be written by others.  Each method or function should have at least one unit test associated with it.
Component Tests: These tests check the various services that the code is dependent on.  For example, if we had code that called the GitHub API, we could write a component test that would make a call to the API and verify that the API was running.  Other examples of component tests are pinging a server or making a call to a database and verifying that a response was received.  There should be at least one component test for each service the code relies on.
Services Tests:  These are tests that check the web services that are used in our code.  In today’s applications, web services often use API requests.  For example, if we have an API with POST, GET, PUT, and DELETE requests, we will want to have automated tests that check each request type.  We will want to have both “happy path” tests that check that a valid request returns an appropriate response, and also negative tests that verify that an invalid request returns an appropriate error code.
User Interface (UI) Tests: UI tests verify that end-user activities work correctly.  These are the tests that will fill out text fields and click on buttons.  As a general rule, anything that can be tested with a unit, component, or service test should be tested by those methods instead.  UI tests should focus solely on the user interface.
Visual Tests: Visual tests verify that elements are actually appearing on the screen.  This is slightly different from UI tests, because the UI tests are focusing on the functionality of the user interface rather than the appearance.  Examples of visual tests would be: verifying that a button’s label is rendered correctly, and verifying that the correct product image is appearing on the screen.
Security Tests:  These are tests that verify that security rules are being respected.  These tests can overlap with services tests, but should still be considered separately.  For example, a security test could check to make sure that an authorization token cannot be generated with an invalid username and password combination.  Another security test would be to make a GET request with an authorization token for a user who should not have access to that resource, and verify that a 403 response is returned.
Performance Tests: Automated performance tests can verify that request response times happen within an appropriate time period.  For example, if your company has decided that GET requests should never take longer than two seconds, test requests can be set to return a failure state if the request takes longer than that time.  Web page load times can also be measured with performance tests.
Accessibility Tests:  Automated accessibility tests can check a variety of things.  When combined with UI tests, they can verify that images have text descriptions for the visually impaired.  Visual tests can be used to verify that the text on the screen is the correct size. 

You may have noticed that the above descriptions often overlap each other.  For example, security tests might be run through API testing, and visual tests might be run through UI testing.  What is important here is that each area is tested thoroughly, efficiently, and accurately.  If there is a spoke missing from the wheel, you will never be comfortable relying on your automation when you are doing continuous deployment. 

Next week, I’ll discuss how we can fit all these tests into a real-world application testing scenario!

Fifteen Free Tools to Help With Testing

There are a great many articles, blog posts, and presentations that discuss automation frameworks and strategies.  But even the most robust automation framework won’t eliminate the need to do exploratory testing.  There will always be situations where we need to generate a large amount of text to test a text field or where we need to encode a string in HTML to test for cross-site scripting.  In this week’s post, I share fifteen of my favorite free tools that make testing faster and easier.  

Text Tools:
1. Letter Count:  This tool will count the characters or words in a block of text.  I use it for creating strings with a specific character count when I test text fields.
2. Lorem Ipsum Generator: I use this tool when I need to generate large amounts of text for text fields where a user will be able to enter several paragraphs of text.
3. Convert Case: This tool comes in handy when I’m testing with Postman and my assertions are expecting the exact casing for string comparison.  Convert Case will set all the characters in a string to lower case, upper case, sentence case, alternating case, and more.
JSON Tools:

4. Pretty Print: JSON objects need indentation to be easily readable.  This tool will take care of all of the indentation and spacing for you.  This is especially helpful when you receive flattened JSON in a response and you want to be able to read through it.
5. Online JSON Viewer: This tool will flatten your JSON for you by removing all the white spaces, which is helpful when you need to use your JSON in code.
6. JSON Lint: I use this tool whenever I’m using JSON in a test and getting 400 errors.  I paste the JSON into the tool and it will tell me whether my JSON is valid.  If it’s not valid, it shows me the line with the error.
Encoding and Decoding Tools:

7. URL Encoder/Decoder: This tool is great for testing cross-site scripting.  Simply paste in your script and it will encode it in valid HTML format for you.
8. JWT.IO: This official JWT documentation includes a tool that will decode a JWT.  It’s really helpful for testing authorization, because you can see exactly what information your JWT is sending.  
9. Base 64 Decode and Encode: When you have an image or string that’s been encoded into Base 64, this is an easy way to decode it.  
GUID Tools:

10. GUID Generator: I often need random GUIDs for my testing, and this tool will generate as many as I need.
11. GUID Test and Validate: I use this tool to check GUIDs to make sure that they are valid. 
Miscellaneous Tools:
12. Rubular: This is a regex editor that allows you to put in a regex, add a string to test, and find out whether your string matches the regex.  It is specifically designed for Ruby but can be used with other languages as well.  
13. NumVerify: This tool is helpful when you are testing international phone numbers.  It lets you know whether a given number is valid, and which country it is valid in.  
14. WePay Testing: I use this site when I need a fake bank account to test with.  It also lists fake credit card numbers for use in testing.  Some of the features may be specific to Chase Bank.  
15. Online Random File Generator: When you need to test files of a very specific size, this tool uses random strings to generate a file with exactly the size you need.
Just as we use tools in our daily life to make our tasks easier, these free online tools can help speed up our testing, giving us more time to write test automation or do further exploratory testing.  
What are your favorite free tools for testing?  Let me know in the comments below!  

Six Steps to Writing an Effective Test Report

As testers, we know how important it is to test our software thoroughly and document our findings meticulously.  But all of our talent will be useless if we can’t effectively communicate our test results to others!  If your test results are written in a giant, poorly organized spreadsheet with tiny text and lots of unnecessary details, even the most dedicated test manager will find her eyes glazing over with boredom when she looks at it.  In this post, I’ll describe six steps to take to make sure that you can communicate your findings to others efficiently and effectively.

Step One: Determine what goal you are trying to accomplish with your report

Why are you creating your report?  What do you want people to understand when they read it?  You might be creating a report to demonstrate that you have tested all the acceptance criteria in the user story.  You could be showing your manager exactly which areas of the application are not functioning properly.  You could be demonstrating that after a recent code change several bugs were fixed.  
There are all kinds of reasons to create a report, but if you don’t stop and think about why you are creating it, it’s probably not going to be very clear to your readers.  Simply sending over your test notes is not enough to communicate effectively.
Step Two: Focus on the viewer’s needs

Who will be reading your report?  Will it be a developer, your test manager, your product owner, your team lead, or the CTO of your company?  You will want to tailor your test report for your audience.  The CTO of your company is probably very busy and will not care how many permutations of your form data you ran, or how you developed the permutations.  She will want to see that you ran 100 tests and that 99 passed.  Your product owner will want to see that you have tested the user stories and that the outcomes were as expected.  Your test manager might be interested in how many different permutations of the test you ran, and what logic you used to create them.  Your developer might want to know what test data you used and what your server response times were.  
You can see how the interests of your reader will vary quite a bit depending on their role, so think about how you can best present the information that he or she wants.
For steps three through six, we’ll once again use the example of the Superball Sorter.  This is a hypothetical software feature that sorts out superballs of varying sizes and colors to four children according to a set of rules.  

Step Three: Avoid Extraneous Information

Make sure that your report contains only the information that your reader needs.  Extraneous information means that a reader has to sift through your report to find the important results.  Consider this report, which shows the results of two tests where two children have a sorting rule:
Rules Amy  Bob Carol Doug
Amy-small blue balls; Doug- large green balls small blue ball; small blue ball; small blue ball; small blue ball large red ball; small orange ball; large yellow ball; small purple ball large purple ball; small green ball; large yellow ball; small red ball large green ball; large green ball; large green ball; large green ball
Bob- large red balls; Carol- small yellow balls large orange ball; small purple ball; large yellow ball; small green ball large red ball; large red ball; large red ball; large red ball; large red ball small yellow ball; small yellow ball; small yellow ball; small yellow ball small blue ball; small green ball; large purple ball; small orange ball
Did the tests pass?  How long did it take for you to determine that by looking at the test report?  There is a lot of information here that is unnecessary.  It doesn’t matter that Bob got the large red ball first and the small orange ball second.  What matters is that Amy only got small blue balls and Doug got large green balls.  Contrast that test report with this one:
Rules Rules respected?
Amy-small blue balls; Doug- large green balls Yes
Bob- large red balls; Carol- small yellow balls Yes
Here you can very quickly see what rules were set, and whether those rules were respected.  If a reader of your report needs to know what balls Bob got, they can ask you for those details and you can look them up in your notes.
Step Four: Make the report visually immediate

We are all busy people; developing and testing software is fast-paced and time-consuming!  Your manager or CTO is probably sent dozens of reports and emails a day, so make your report so easy to read that just a glance at it will give them information.  Take a look at this test report:
Number of Rules Pass/Fail
0
1
2
3
4
How many seconds does it take you to see that a test failed?  This report is pretty immediate!  It’s also really easy to see how many rules were used when the test failed.  Compare that with this test report, which shows the same tests and the same results:
Test Case Result
None of the children have rules The balls are sorted evenly amongst the children
One child has a rule The child’s rule is respected
Two children have rules The two children’s rules are respected
Three children have rules The three children’s rules are respected
Four children have rules None of the balls are sorted
A reader needs to read through all the text of this report in order to see that the fourth test failed.
Step Five: Make the report easy to read

In addition to making the report visually immediate, it also needs to be easy to read. Take a look at this example, where two tests are run where three children are given rules:

Rules Rules respected?
Amy- small blue; Bob- large blue; Carol- small purple Amy gets only small blue balls, and Bob gets only large blue balls, but Carol gets balls other than the small purple balls
Amy- large blue; Bob- small purple; Carol- small yellow Amy gets only large blue balls, Bob gets only small purple balls, and Carol gets only small yellow balls
This report shows very quickly that one of the tests failed, but in order to see why the test failed, it’s necessary to read through the whole description to see that Carol’s rule was not respected.  This report conveys exactly the same information:
Rules Amy Bob Carol
Amy- small blue; Bob- large blue; Carol- small purple PASS PASS FAIL
Amy- large blue; Bob- small purple; Carol- small yellow PASS PASS PASS
With this report, it is so easy to see that Carol’s rule was not respected, and it’s also easy to look to the left to see which rule she had.
Step Six: Make the report readable without any additional explanation

How long does it take for you to figure out what this report means? 
Rules Result
A-SB; B-LO; C-L; D-S A-Y; B-Y; C-Y; D-N
A-L; B-S; C-Y; D-P A-Y; B-N; C-Y; D-Y
A-LY; B-L; C-S; D-SG A-Y; B-Y; C-N; D-Y
It’s fine to use all sorts of abbreviations when you are testing and taking notes for yourself, but you shouldn’t need to provide a key to your reader in order to have them interpret it.  Who but the most interested testers are going to take the time to see where the bug is here?

This report conveys exactly the same information:

Test One
Amy- small blue
Bob- large orange
Carol- large
Doug- small
Rule respected?
Yes
Yes
Yes
No
Test Two
Amy- large
Bob- small
Carol- yellow
Doug- purple
Rule respected?
Yes
No
Yes
Yes
Test Three
Amy- large yellow
Bob- large
Carol- small
Doug- small green
Rule respected?
Yes
Yes
No
Yes

It’s easy to see exactly what rules each child was given for each test.  Through the use of color, the report demonstrates very clearly where the bug is: whenever a child is given a rule that they should get only small balls, that rule is not respected.

Conclusion:

In today’s fast-paced world, we all have vast amounts of information coming to us every day.  If we are going to make a difference with our testing and influence decision-making where we work, we need to be able to convey our test results in ways that clearly show what is going on with our software and what should be done to improve it.