Think Like a Tester

In discussions about security testing, you have probably heard about Cross-Site Scripting (XSS), but you may not have a good definition of what it is.  Cross-Site Scripting is an attack in which a malicious user finds a way to execute a script on another user’s website.  Today we’ll learn about two different kinds of XSS attacks, do a hands-on demo of each, and discuss why they are harmful to the end user.

Reflected XSS:

Reflected XSS is an attack that is executed through the web server, but is not stored in the code or the database.  Because the attack is not stored, the owner of a site may have no idea that the attack is happening.

In order to demonstrate this attack, we’ll go to a great training site from Google called the XSS Game. This site has a series of challenges in which you try to execute XSS attacks.  The challenges become increasingly more difficult as they progress.  Let’s try the first challenge.

On this page, you see a simple search field and button.  To execute the attack, all you need to do is type
<script>alert(“XSS here!”)</script>
into the text field, and click the button.  You will see your message, “XSS here!”, pop up in a new window.

What is happening here is that you are sending a script to execute a popup alert to the server.  The client-side code does not have appropriate safeguards in place to prevent a script from executing, so the site executes the script.

You might be thinking “This is a fun trick, but how could a malicious user use this to hack me?  I’m typing into my own search window.”  One way this is used is through a phishing link.  Let’s say that you are the owner of a website.  A malicious user could create a link that goes to your site, but appends a script to the end of the URL, such as
?query=%3Cscript%3Ealert%28%22XSS%22%29%3C%2Fscript%3E. 
(This is simply the attack we used earlier, with HTML encoding.) The malicious user could send this link in an email to an unsuspecting visitor to your site, making the email look like it came from you.  When the person clicks on the link, the script will navigate to your site, and then execute the popup script.  The malicious user will craft the script so that instead of containing the message “XSS here!”, it contains a message that encourages the visitor to interact with it, in order to obtain the user’s account number, or other sensitive information.

Stored XSS:

Stored XSS is an attack where the malicious script is actually stored in the database or code of a website, so it executes whenever a user navigates to the page or link. This could happen if the creator of the site did not put adequate input sanitization in the back-end database.

We’ll take a look at how to craft this attack by looking at the second challenge of the XSS Game.  (In order to see this challenge, you’ll need to have solved the first challenge, so follow the instructions above.)

In the second challenge, you are presented with a chat app.  To solve the challenge, you need to add some text to the application that will execute a script.  You can do this by typing in
<img src=’foobar’ onerror=’alert(“xss”)’>

As soon as you submit this entry, you should see a popup window with the “XSS alert!” message.  And not only that, if you navigate away from this page and return to it, you will see the popup window again.  The attack has been stored in your comment on the chat page, where it will cause a popup for any users who navigate to it.

Let’s parse through the script we entered to see what it’s doing:

<img src=’foobar’ onerror=’alert(“xss”)’>
The items in red indicate that we are passing in an image element.

<img src=’foobar’ onerror=’alert(“xss”)’>
The section in blue is telling the server what the source of the image should be.  And here’s the trick- there is no URL of foobar, so the image cannot load.

<img src=’foobar’ onerror=’alert(“xss”)’>
The section in green is telling the server that if there is an error, that a popup window should be generated with the “xss” text.  Because we have set things up so that there will always be an error, this popup will always execute.

One way that stored XSS might be used is to spoof a login window.  When a user navigates to a hacked site, they will be presented with a login window that has been crafted to look authentic.  When they enter their login credentials, their credentials will be sent to the malicious user, who can now use them to log in to the site, impersonating the victim.

Next week, we’ll discuss more ways to test for XSS attacks!

In this week’s post, we will learn how to test for IDOR.  IDOR stands for Insecure Direct Object Reference, and it refers to a situation when a user can successfully request access to a webpage, a data object, or a file that they should not have access to.  We’ll discuss four different ways this vulnerability might appear, and then we’ll actually exploit this vulnerability in a test application using Chrome’s Developer Tools and Postman.

One easy way to look for IDOR is in a URL parameter.  Let’s say you are an online banking customer for a really insecure bank.  When you want to go to your account page, you login and you are taken to this URL:  http://mybank/customer/27.  Looking at this URL, you can tell that you are customer number 27.  What would happen if you changed the URL to http://mybank/customer/28?  If you are able to see customer 28’s data, then you have definitely found an instance of IDOR!

Another easy place to look is in a query parameter.  Imagine that your name is John Smith, and you work for a company that conducts annual reviews for each of its employees.  You can access your review by going to http://mycompany/reviews?employee=jsmith.  You are very curious about whether your coworker, Amy Jones, has received a better review than you.  You change the URL to http://mycompany/reviews?employee=ajones, and voila!  You now have access to Amy’s review.

A third way to look for IDOR is by trying to get to a page that your user should not have access to.  If your website has an admin page with a URL of http://mywebsite/admin, which is normally accessed by a menu item that is only visible when the user has admin privileges, see what happens if you log in as a non-admin user and then manually change the URL to point to the admin page.  If you can get to the admin page, you have found another instance of IDOR.

Finally, it’s also possible to exploit an IDOR vulnerability to get files that a user shouldn’t have access to.  Let’s say your site had a file called userlist.txt with the names and addresses of all your users.  If you can log in as a non-admin user and navigate to http://mywebsite/files?file=userlist.txt, then your files are not secure.

Let’s take a look at IDOR in action, using Postman, Chrome Developer Tools, and an awesome website called the OWASP Juice Shop!  The OWASP Juice Shop is an application created by Bjorn Kimminich to demonstrate the most prevalent security vulnerabilities.  You can download it and run it locally by going to https://github.com/bkimminich/juice-shop, or you can access an instance of it by going to https://juice-shop.herokuapp.com.  For this tutorial, we’ll use the heroku link.  Once you navigated to the site on Chrome, create a login for yourself.  Click the Login button in the top left, and then click the “Not yet a customer?” link. You can use any email address and password to register (don’t use any real ones!).  Log in as your new user, and click on any of the juices on the Search page in order to add it to your shopping basket. 

Before you take a look at your basket, open up the Chrome Developer Tools by clicking on the three dots in the top right corner of the browser, selecting “More Tools”, and then “Developer Tools”.  A new window will open up on either the right or the bottom of your browser.  In the navigation bar of the tools, you should see a “Network” option.  Click on this.  This network tool will display all of the network requests you are making in your browser. 

Click on the “Your Basket” link in the top right of the Juice Shop page.  You will be taken to your shopping cart and you should see the juice that you added to the basket.  Take a look in the Network section of the Developer Tools.  The request that you are looking for is one that is named simply with a number, such as “6” or “7”.  Click on this request, and you should see that the request URL is https://juice-shop.herokuapp.com/rest/basket/<whateverYourAccountIdIs>, and that the request type is a GET.  Scrolling down a bit, you’ll see that in the Request Headers, the Authorization is set to Bearer and then there is a long string of letters and numbers.  This is the auth token.  Copy everything in the token, including the word “Bearer”. 

Next, we’ll go to Postman.  Click on the plus tab to create a new request.  The request should already be set to GET by default.  Enter https://juice-shop.herokuapp.com/rest/basket/<yourAccountId> into the URL.  Now go to the Headers section, and underneath the Key section, type “Authorization”, and underneath the Value section, paste the string that you copied.  Click to Send the request, and if things are set up correctly, you will be able to see the contents of your shopping basket in the response. 

Now for the fun part!  Change the account id in the URL to a different number, such as something between 1 and 5, and click Send.  You will see the contents of someone else’s basket!  Congratulations!  You have just exploited an IDOR vulnerability! 

Until a few years ago, security testing was seen as something separate from QA; something that an InfoSec team would take care of.  But massive data breaches have demonstrated that security is everyone’s responsibility, from CEOs to product owners, from DBAs to developers, and yes, to software testers.  Testers already verify that software is working as it should so that users will have a good user experience; it is now time for them to help verify that software is secure, so that users’ data will be protected.

The great news is that much of what you already do as a software tester helps with security testing!  In this post, I will outline the ways that testers can use the skills they already have to start testing with security in mind, and I will discuss the new skills that testers can learn to help secure their applications.

Things you are probably already testing:

• Field Validation: It’s important to make sure that fields only accept the data types they are expecting, and that the number and type of characters is enforced. This helps ensure that SQL injection and cross-site scripting can’t be entered through a data field.
• Authentication: Everyone knows that it’s important to test the login page of an application. You are probably already testing to make sure that when login fails, the UI doesn’t provide any hints as to whether the username or password failed, and testing to make sure that the password isn’t saved after logout or displayed in clear text.  This serves to make it more difficult for a malicious user to figure out how to log in.
• Authorization: You are already paying attention to which user roles have access to which pages.  By verifying that only authorized users can view specific pages, you are helping to insure that data does not fall into the wrong hands.

New API testers will often be mystified by the assortment of curly braces, colons, and commas that they see in the body of the response to their GET requests.  Trying to create a valid JSON body for a POST request is even more puzzling.  In this week’s post, I’ll discuss how JSON data is formed and offer up some resources that will make working with JSON easier.

"Name": "Dino"
"Color": "Purple"

"FirstName": "Fred",
"LastName": "Flintstone",
"City": "Bedrock"

{
  "Name": "Dino",
  "Type": "Dinosaur",
  "Age": "5",
  "Color": "Purple"
}

"residents": [
  {
    "FirstName": "Fred",
    "LastName": "Flintstone"
  },
  {
    "FirstName": "Barney",
    "LastName": "Rubble"
  },
  {
    "FirstName": "Wilma",
    "LastName": "Flintstone"
  }
]

{
  "residents": [
    {
      "firstName": "Fred",
      "lastName": "Flintstone",
      "contactInfo": {
        "phoneNumber": "555-867-5309",
        "email": "[email protected]"
      }
    },
    {
      "firstName": "Wilma",
      "lastName": "Flintstone",
      "contactInfo": {
        "phoneNumber": "555-423-4545",
        "email": "[email protected]"
      }
    }
  ],
  "pets": [
    {
      "name": "Dino",
      "type": "dinosaur",
      "color": "purple"
    },
    {
      "name": "Hoppy",
      "type": "hopparoo",
      "color": "green"
    }
  ]
}

Last week, we talked about running API tests from the command line using Newman, and how to add Newman into your Continuous Integration system so that your API tests run automatically.  But knowing how to run your tests isn’t that helpful unless you make good choices about what tests to run, and when to run them.  Let’s first think about what to automate.

Let’s imagine that we have an API with these requests:

POST user
GET user/{userId}
PUT user/{userId}
DELETE user/{userId}

The first category of tests we will want to have are the simple Happy Path requests.  For example:

POST a new user and verify that we get a 200 response
GET the user and verify that we get a 200 response, and that the correct user is returned
PUT an update to the user and verify that we get a 200 response
DELETE the user, and verify that we get a 200 response

The next category of tests we want to have are some simple negative requests.  For example:

POST a new user with a missing required field and verify that we get a 400 response
GET a user with an id that doesn’t exist and verify that we get a 404 response
PUT an update to the user with an invalid field and verify that we get a 400 response
DELETE a user with an id that doesn’t exist and verify that we get a 404 response

You’ll want to test 400 and 404 responses on every request that has them.  It’s not necessary to test every single trigger of a 400- for example, you don’t need to have automated tests for every single missing required field- but you will want to have one test where one required field is missing.

The third category of tests we want to have are more Happy Path requests, with variations. For example:

POST a new user with only the required fields, rather than all fields, and verify that we get a 200 response
GET a user with query parameters, such as user/{userId}?fields=firstName,lastName, and verify that we get a 200 response, and the appropriate values in the response
PUT a user where one non-required field is replaced with null, and one field that is currently null is replaced with a value, and verify that we get a 200 response

It’s worth noting that we might not want to test every possible combination in this category.  For example, if our GET request allows us to filter by five different values: firstName, lastName, username, email, and city, there are dozens of possibilities of what you could filter on.  We don’t want to automate every single combination; just enough to show that each filter is working correctly, and that some combinations are working as well.

Finally, we have the category of security tests.  For example, if each request needs an authorization token to run, verify that we get an appropriate error:

POST a new user without an authorization token, and verify that we get a 401 response
GET a user with an invalid token, and verify that we get a 403 response
PUT an update to the user with an invalid token, and verify that we get a 403 response
DELETE a user without an authorization token, and verify that we get a 401 response

For each request, you’ll want to test for both a 401 response (an unauthenticated user’s request) and a 403 response (an authorized user’s request).

There may be many more tests than the ones that have been listed here that are appropriate for an API you are testing.  But these examples serve to get you thinking about the four different types of tests.

Now let’s take a look at how we might use these four types in automation!  First, we want to have some Smoke Tests that will run very quickly when code is deployed from one environment to another, up the chain to the Production environment.  What we want to do with these tests is simply verify that our endpoints can be reached.  So all we need to do is run the first category of tests: the simple Happy Path requests.  In our example API, we only have four request types, so we only need to run four tests.  This will only take a matter of seconds.

We’d also like to have some tests that run whenever new code is checked in.  We want to make sure that the new code doesn’t break any existing functionality.  For this scenario, I recommend doing the first two categories of tests: the simple Happy Path requests, and the simple negative requests.  We could have one positive and one or two negative tests for each request, and this will probably be enough to provide accurate feedback to developers when they are checking in their code.  In our example API, this amounts to no more than twelve tests, so our developers will be able to get feedback in about one minute.

Finally, it’s also great to have a full regression suite that runs nightly.  This suite can take a little longer, because no one is waiting for it to run.  I like to include tests of all four types in the suite, or sometimes I create two nightly regression suites: one that has the first three types of tests, and one that has just the security tests.  Even if you have a hundred tests, you can probably run your full regression suite in just a few minutes, because API tests run so quickly.

Once you have your Smoke, Build, and Regression tests created and set to run automatically, you can relax in the knowledge that if something goes wrong with your API, you’ll know it.  This will free you up to do more exploratory testing!

Next week we’ll take look at API request formats: JSON structure and query parameters!

Running a test collection in Postman is a great way to test your APIs quickly.  But an even faster way to run your tests is to run them automatically!  In order to automate your Postman tests, we first need to learn how to run them from the command line.

Newman is the command-line running tool for Postman.  It is a NodeJS module, and it is easy to install with npm (the Node Package Manager).  If you do not already have Node installed, you can easily install it by going to https://nodejs.org.  Node contains npm, so once Node is installed, it is extremely easy to install Newman.  Simply open a command line window and type:

npm install -g newman

The “-g” in this command is telling npm to install newman globally on your computer.  This means that it will be easy to run Newman from the command line no matter what folder you are in.

Now that Newman is installed, let’s try running a collection!  We will use the Pet Store collection that we created in this blog post and updated with assertions in this post.  In order to run Newman, you will need to export your collection and your environment as .json files.

Find your collection in the left panel of Postman, and click on the three dots next to it.  Choose “Export”, then choose Collection v.2.1 as your export option, then click on the “Export” button.  You’ll be prompted to choose a location for your collection.  Choose any location you like, but make a note of it, because you’ll need to use that location in your Newman command.  I have chosen to export my collection to my Desktop.

Next, click on the gear button in the top right of the screen to open the Environments window.  Next to the Pet Store environment, click on the down arrow.  Save the exported environment to the same location where you saved your collection.

Now we are ready to run the Newman command.  In your command window, type:

newman run <pathToYourFile> -e <pathToYourEnvironment>

Obviously, you will want to replace the <pathToYourFile> and the <pathToYourEnvironment> with the correct paths.  Since I exported my .json files to my Desktop, this is what my command looks like:

newman run “Desktop/Pet Store.postman_collection.json” -e “Desktop/Pet Store.postman_environment.json”

The -e in the command specifies what environment should be used with the collection.

An alternative way of pointing to the .json files is to cd directly into the location of the files.  If I ran cd Desktop, then my command would look like:

newman run “Pet Store.postman_collection.json” -e “PetStore.postman_environment.json”

Also note that the quote marks are not necessary if your file name doesn’t have any spaces.  If I were to rename my files so there was no space between Pet and Store, I could run my collection like this:

newman run PetStore.postman_collection.json -e PetStore.postman_environment.json

And, once you have exported your .json files, you can really name them whatever you want.  As long as you call them correctly in the Newman command, they will run.

When you make the call to Newman, you will see your tests run in the command line, and you will wind up with a little chart that shows the test results like this:

If you are running your tests against your company’s test environment, you may find that running Newman returns a security error.  This may be because your test environment has a self-signed certificate.  Newman is set by default to have strict SSL, which means that it is looking for a valid SSL certificate.  You can relax this setting by sending your Newman command with a -k option, like this:

newman run “Desktop/Pet Store.postman_collection.json” -e “Desktop/Pet Store.postman_environment.json” -k

Another handy option is the -n option.  This specifies the number of times you want the Newman collection to run.  If you’d like to put some load on your application, you could run:

newman run “Desktop/Pet Store.postman_collection.json” -e “Desktop/Pet Store.postman_environment.json” -n 1000

This will run your collection 1000 times.  In the results chart, you can see what the average response time was for your requests.  While this is not exactly a load test, since the requests are running one at a time, it will still give you a general idea of a typical response time.

Once you have your Newman command working, you can set it to run in a cron job, or a Powershell script, or put the command into a continuous integration tool such as Jenkins, or Travis CI, or Docker. There is also an npm module for reporting Newman results with TeamCity.

Whatever tool you choose, you will have a way to run your Postman API tests automatically!  Next week, we’ll talk about which API tests to automate, and when to run them.

One of the things that makes me happy about API testing is how easy it is to organize tests and environment variables.  I love having test suites ready at a moment’s notice; to run at the push of a button when regression testing is needed, or to run automatically as part of continuous integration.

This week I’ll be talking about some organizational patterns you can use for your API tests.  I’ll be discussing them in the context of Postman, but the concepts will be similar no matter what API testing platform you are using.

First, let’s discuss environments.  As you recall from last week’s post, an environment is a collection of variables in Postman.  There are two different ways I like to set up my Postman environments.  In order to explain them, I’ll use some scenarios.  For both scenarios, let’s assume that I have an application that begins its deployment lifecycle in Development, then moves to QA, then Staging, and then Production.

In my first scenario, I have an API that gets and updates information about all the users on my website.  In each product environment (Dev, QA, Staging, Prod), the test users will be different.  They’ll have different IDs, and different first and last names.  The URLs for the product environments will each be different as well. However, my tests will be exactly the same; in each product environment, I’ll want to GET a user, and PUT a user update.

So, I will create four different Postman environments:
Users- Dev
Users- QA
Users- Staging
Users- Prod

In each of my four environments, I’ll have these variables:
environmentURL
userId
firstName
lastName

Then my test collection will reference those variables.  For example, I could have a test request like this:

GET https://{{environmentURL}}/users/{{userId}}

Which environment URL is called and which userId is used will depend on which Postman environment I am using.  With this strategy, it’s easy for me to switch from the Dev environment to the QA environment, or any other environment.  All I have to do is change the Postman environment setting and run the same test again.

The second scenario I use is for situations like this one: I have a function that delivers an email.  The function uses the same URL regardless of the product environment. I like to pass in a timestamp variable, and that stays the same (it shows the current time) regardless of what environment I am using.  But I like to change the language of the email depending on what product environment I am in.

In this case, I am creating only one Postman environment:
Email Test

My one Postman environment has this variable:
timestamp

My test collection, however, has one test for each product environment.  So I have:
Send Email- Dev
Send Email- QA
Send Email- Staging
Send Email- Production

Each request includes the timestamp variable, but has a variation in what is sent in the body of the email. For the Dev environment, I use a request that contains “Dev” in the message body, for the QA environment, I use a request that contains “QA” in the message body, and so on.

When deciding which of these two environment strategies to use, consider the following:

• what stays the same from product environment to product environment?
• what changes from product environment to product environment?

This week, we’ll be talking about the many ways to use variables in Postman. We’ll be using the collection that we created a few weeks ago, so you may want to check that tutorial out before reading on.

The first thing to understand about variables in Postman is that they are organized into environments. A Postman environment is simply a collection of variables that can be used to run against a Postman collection. Creating an environment is easy! Click on the gear button in the top right corner of the window, and then click the “Add” button. Give your environment a name, such as “Pet Store”, and then click “Add”.

An environment isn’t helpful at all unless you give it some variables, so let’s add a variable now. After you added the new environment, you should have been returned to the “Manage Environments” popup window; click on your new environment to edit it. In the “key” section, add petId, and in the “value” section, add 100. Click the “Update” button, and your new variable has been added.

Close the “Manage Environments” window. In the upper right corner, you’ll see a dropdown selector, which currently says “No Environment”. Click on this selector and choose “Pet Store”. Now when you run requests in your Pet Store collection, it will run them with this Pet Store environment.

We have one variable set up now: petId, which equals 100. Let’s see all the different ways we can use this variable!

First, we can use it in the URL. In your “Verify Add Pet” request, change the URL from http://petstore.swagger.io/v2/pet/100 to http://petstore.swagger.io/v2/pet/{{petId}}.  When you add in the variable, you will see {{petId}} turn to orange. This is because Postman recognizes the variable from the current environment. If you ever forget to set your environment (an easy thing to do), you will see that the {{petId}} will be red. Save your changes, and run your request (you may need to run the “Add Pet” request first, so there will be a pet with an id of 100). You will see that the request was made for the pet with the id of 100, and that is what is returned in the response.

You can also use a variable in this way in the request headers. We won’t be doing that in this collection, but one example of how this can be used is if you need to send an authentication token with a request. Rather than typing in the token every time, you can save it as a variable named “token”, and then pass it in with the header like this: {{token}}.

Getting back to our petId variable, we can use it in the request body as well. Let’s open up the Add Pet request, and change the body of the request so it looks like this:

{
  “id”: “{{petId}}”,
  “category”: {
    “id”: 1,
    “name”: “Cat”
  },
  “name”: “Grumpy Cat”,
  “photoUrls”: [
    “https://pbs.twimg.com/profile_images/948294484596375552/RyGNqDEM_400x400.jpg”
  ],
  “tags”: [
    {
      “id”: 1,
      “name”: “Mixed breed”
    }
  ],
  “status”: “available”
}

We’ve replaced the id of 100 with our petId variable. You can see how using a variable like this would be helpful in testing. Suppose we decided that we didn’t want to add a pet with an id of 100, and we preferred to use some other number instead. It will be easy to change this, because now all we have to do is change the variable in one place (the Pet Store environment) rather than in every request where 100 was used. Be sure to save your Add Pet request.

Another place we can use this petId is in our assertions. Return to the Verify Add Pet request, and add this assertion to the Tests section:

var jsonData = JSON.parse(responseBody);
tests[“Correct pet ID is returned”] = jsonData.id == environment.petId;

You may notice that this assertion is different from the other assertions we have in this request. The other assertions are written in the new Postman assertion style, and this assertion is written in the old style. (If any of my readers knows how to do this assertion in the new style, please let me know in the comments section!) What this assertion does is compare the id value returned in the body of the response (jsonData.id) with the id value set as an environment variable (environment.petId). Save this request and run it, and you should see that this assertion passes.

Finally, we can set environment variables based on the body of a response. Let’s duplicate our Add Pet request (see instructions in Creating a Postman Collection for how to do this), and we’ll rename our copied request “Add Pet With No Id”. We’re going to use this request to add a pet without specifying the id, letting the program assign an id for us. Change the body of the request to:

{
  “category”: {
    “id”: 1,
    “name”: “Cat”
  },
  “name”: “Grumpy Cat”,
  “photoUrls”: [
    “https://pbs.twimg.com/profile_images/948294484596375552/RyGNqDEM_400x400.jpg”
  ],
  “tags”: [
    {
      “id”: 1,
      “name”: “Mixed breed”
    }
  ],
  “status”: “available”
}

Note that the id that was at the top of the request (the pet id) has now been removed. Now go to the Tests tab, and add in this instruction:

var jsonData = JSON.parse(responseBody);
postman.setEnvironmentVariable(“newPetId”, jsonData.id);

This instruction is parsing the JSON data and setting the id value returned to a new environment variable called “newPetId”. (Note that this is also in the old Postman assertion style. If you know how to do this in the new style, please add it in the comment section!) Let’s save this request and run it. After you’ve run the request, click on the eyeball icon in the top right corner of the screen.  This is the Environment Quick Look button, which allows you to quickly see what variables you have in your environment. You should see the newPetId variable, with a value of whatever id your new pet was assigned!  What’s nice about this feature is that you don’t even need to create the new variable in the environment first; the Postman request will create it for you.

We have only been looking at one variable- the pet id- in these instructions. You could also set variables for the pet name, the pet’s image URL, the status type, and so on. You may want to practice this in your Postman collection.The more variables you create in your environment, the easier it will be to maintain your tests in the future. 

Next week, we’ll talk about one of my favorite things- organizing tests and environments!

This week, we’ll be talking about adding assertions to our Postman requests.  In last week’s post, we discussed what various API request responses mean, and how to write assertions for them in Postman requests.  Now we’ll be adding some more assertions that will test our requests more fully. 

We will be using the Postman collection that we created two weeks ago, so if you haven’t yet created that, you may want to set it up before you continue. 

The first type of assertion we’ll discuss is the Response Time assertion.  This verifies that the response to an API request is returned within an acceptable amount of time. Let’s say that the product owners of the Swagger Pet Store have decided that their end users should be able to add a new pet to the store in under one second.  We’ll add an assertion for this to our Add Pet request.  Click on the Add Pet request in the collection list on the left of the screen so it will open up in the main window. Then click on the “Tests” tab.  You should already have the Response Code assertion that you added last week.  Click underneath that assertion, and then click on the “Response time is less than 200 ms” code snippet on the right.  This will be added to the Tests window:

pm.test(“Response time is less than 200ms”, function () {
    pm.expect(pm.response.responseTime).to.be.below(200);
});

Currently the test is asserting that the response time should be less than 200 milliseconds.  We don’t need it to be that fast, so let’s change the assertion.  First, change the “Response time is less than 200ms” to be “Response time is less than one second”.  This is the title of the test.  Now let’s change the actual assertion.  Change the value in “to.be.below” from 200 to 1000 (this is one thousand milliseconds, or one second).  Save your request and click the Send button.  You should see in the Test Results section at the bottom of the screen that your assertion passed.  You can also look at the top of the results window and see the response time next to the response status code. 

Another helpful assertion is the Response Contains String assertion.  This simply asserts that the body of the request response has a certain string.  I often use this assertion when I am expecting an error message.  To see this in action, let’s add a new negative test to our collection.  We will trigger a 500 response by trying to add a pet to the store with a non-integer id. The easiest way to create this request will be to copy the existing Add Pet request and rename it to “Add Pet With Invalid ID”. (If you don’t know how to copy and rename a request, see Creating a Postman Collection.) 

We have a few changes to make to this request before we add in our new assertion.  In the body of the request, we’ll change the id of pet from “100” to “FOOBAR”.  Because we copied this request, we already have a couple of assertions in it: one that asserts that the response code will be 200, and one that asserts that the response time is less than 1000ms.  We know that our response code won’t be 200 any more, so let’s change that test to read:

pm.test(“Status code is 500”, function () {
    pm.response.to.have.status(500);
});

We can leave the Response Time assertion as is, or we can remove it.  Now we can add in our new assertion.  Look in the snippet list for “Response body:Contains string”, and click on it.  This will be added to the Test window:

pm.test(“Body matches string”, function () {
    pm.expect(pm.response.text()).to.include(“string_you_want_to_search”);
});

Let’s rename the assertion from “Body matches string” to “Error response is returned”, and change the response text from “string_you_want_to_search” to “something bad happened”. It’s worth noting at this point that Postman assertions are case-sensitive, so you won’t want to capitalize the word “Something”, since it’s not capitalized in the actual response.

Click the Send button and verify that you get Pass messages for both your Status Code test and for your Response Contains String test.

Now let’s look at a more powerful assertion: the JSON Value Check.  We can actually assert that the values returned in a response are the values that we are expecting.  Let’s look at the Get Pet request.  We are already asserting that the response code is 200, but that doesn’t really tell us much.  It’s possible that when we make our request that we are getting the wrong pet in the response!  We’ll want to assert that we are getting the right pet back without having to physically look at the response. So we’ll add a JSON Value Check.  Click below the Response Code assertion, and then look for the code snippet called “Response body: JSON value check”. Click on this snippet.  This will be added to your test window:

pm.test(“Your test name”, function () {
    var jsonData = pm.response.json();
    pm.expect(jsonData.value).to.eql(100);
});

Change the test name from “Your test name” to “Body contains pet id”. Notice that the next line of code creates a variable called jsonData. This is the parsed version of the JSON response. On the third line, we’ll change “jsonData.value” to “jsonData.id”. This is because we want to find out what the id of the response is.  Finally, we don’t need to change the value in “to.eql”, because we were looking for an id of 100, and that’s what it’s already set to. When you are done making your changes, the assertion should look like this:

pm.test(“Response contains id”, function () {
    var jsonData = pm.response.json();
    pm.expect(jsonData.id).to.eql(100);
});

Save your request changes and then click the Send button.  Both your Response Code assertion and your JSON Value Check tests should pass.

You may have noticed that there are other ids in the body of the Get Pet response:

{
    “id”: 100,
    “category”: {
        “id”: 1,
        “name”: “Cat”
    },
    “name”: “Grumpy Cat”,
    “photoUrls”: [
        “https://pbs.twimg.com/profile_images/948294484596375552/RyGNqDEM_400x400.jpg”
    ],
    “tags”: [
        {
            “id”: 1,
            “name”: “Mixed breed”
        }
    ],
    “status”: “available”
}

 How would we assert on the category id or the tag id?  We do this by parsing through the JSON response. In order to demonstrate this, copy the test you just created, and edit it to look like this:

pm.test(“Response contains category id”, function () {
    var jsonData = pm.response.json();
    pm.expect(jsonData.category.id).to.eql(1);
});

We’ve changed the test name in the top line of the code to reflect what we are actually asserting.  The second line of code remains the same. In the third line of code, we’ve changed “jsonData.id” to “jsonData.category.id”. This means that instead of looking at the id, we are looking at the category section, and then at the id within that section. Finally we changed the value that we were expecting from 100 to 1. 

Save your request and click the Send button again. Now you should see all three assertions on the request pass.

Now that you know how to create assertions, you can go through all of the requests in your Pet Store collection and add your own! You may have noticed that your POST and PUT requests return the pet in the body of the response, so you can add similar assertions to those requests.  You’ll definitely want to make sure that after you have changed your pet data with the PUT (Update Pet) request, the data you get back from your GET (Verify Update Pet) request has the updated information that you are expecting. 

This is just the beginning of what you can do with assertions!  Next week, we’ll take a look at how to set variables and use them in both your requests and your assertions. 

If you have ever made a REST request or looked in the developer tools section of a browser, you have likely seen the three-digit response code that is returned with an HTTP request.  This week, we’ll be talking about the different types of response codes you might receive when doing API testing, and what those codes mean.