What I Learned at POST/CON Part I: Examples and Mocking

I’ve just returned from POST/CON, the annual Postman users’ conference, and I am so excited about everything I learned there!  So excited, in fact, that I’m going to devote not one, but TWO blog posts to sharing my findings.

If you aren’t already using Postman for your API testing, why on earth not?  It’s the best API testing tool out there!  My opinion was reinforced this week, when I learned just how easy it is to create API examples and mock responses.  I’ll be teaching you how to do both things in today’s post.

The instructions in this post will be for the free version of Postman, and I’ll be using version 7.  Your results will look slightly different in version 6; and the Pro version of Postman will have more functionality for the documentation and mocks than will be described here.

First, examples: why would you want to create an example of a request?  Because it’s a great way to show other people on your team how the request is supposed to work.  It can also be used to create documentation for your API. 

The request I’ll be using for today’s post is one of the GET requests from Mark Winteringham’s wonderful API, Restful-Booker.  Here is the request and the response:

You can see that this is a GET request that is asking for the booking with the ID of 1, and that the response returns the booking with the first and last name, checkin and checkout dates, and other details. 

If we wanted to teach someone else on our team how this request works, we could simply share this request with them.  But- what if the API is still being created and doesn’t work yet?  Or what if our teammate doesn’t have the correct authentication access to the request, and can’t run it?  We can show them exactly what’s supposed to happen with the request by creating an example.  In the upper right corner of the Postman window, just underneath the environment dropdown, is a link that says “Examples”:

Click on this link, then click “Add Example”.  A new request tab will open up with the same name as the original GET request, prefaced by “e.g“:

The request will already have the HTTP verb, the URL, and any headers or request parameters set.  All you need to do to finish the example is to paste in an example of what the request response should be:

and set the appropriate response code:

Save the example, and return to the original GET request.  You’ll see that now there is an example request listed in the top right:

Now anyone who sees the request can click on the Examples link to see exactly how the request is supposed to behave. 

Examples are also great for showing how an API request should behave in negative scenarios, such as when an id is not found, or when a user is not authenticated.

Another great feature of examples is that you can use them in your documentation.  You don’t need to have examples to create documentation, but it makes your documentation much easier to understand.  To create documentation in Postman, you need a collection with requests.  Click on the three-dot menu beside the collection, and choose “Publish Docs”:

A “Publish Collection” web page will open.  Select an environment if you like, and click the Publish Collection button.  Once published, you’ll see a URL that you can go to view your documentation.  Go to that URL, and you’ll see your request with your example response!

You can also use your example requests to create a mock server.  This can be used whenever you don’t have access to the actual API server, such as when a feature is still being developed or when you are doing contract testing with another API. 

To set up a mock server, simply click on the three-dot menu beside the collection name and choose “Mock Collection”.  You’ll be presented with a pop-up window like this:

Give your mock collection a name, and click the “Create mock server” button.  You’ll be assigned a special mock server that looks like this: https://<some guid>.mock.pstmn.io. 

This mock server is designed to return the response you created in your GET example whenever you make a request with an endpoint that matches the GET example.  Copy the URL for the mock server and paste it in a GET request, and then add the appropriate endpoint for your example: /booking/1.  Click send, and you should get this response:

Now you can save this request, naming it something like MOCK Get Booking, and you can save it to a collection called something like MOCK Restful Booker. 

You can create examples and mock requests like this for every request in an API, and when you have finished, you will have a complete documentation of your API as well as a mock server that will allow you call an API and get an appropriate response without actually connecting to the API’s server!

I hope you find this helpful in your work with APIs.  Next week, I’ll have more great knowledge from POST/CON to share!

Your Test Cases Are Slowing You Down

One of the first QA jobs I had was a position at a company that made software that could be used to create mobile applications.  It was a very complex application, with so many features that it was often hard to keep track of them all.  Shortly before I started working there, the company had adopted a test tracking system to keep track of all of the possible manual tests the team might want to run.  This amounted to thousands of test cases.

Many of the test cases weren’t written well, leaving those of us who were new to the team confused about how to execute them.  The solution to this problem was to assign everyone the task of revising the tests as they were run.  This helped a bit, but slowed us down tremendously.  Adding to the slowdown was the fact that every time we had a software release, our manager had to comb through all the tests and decide which ones should be run.  Then there was the added confusion of deciding which mobile devices should be used for each test.

We were trying to transition to an Agile development style, but the number of test cases and the amount of overhead needed to select, run, and update the tests meant that we just couldn’t adapt to the pace of Agile testing.

You might be thinking at this point, “Why didn’t they automate their testing?”  Keep in mind that this was back when mobile test automation was in its infancy.  One of our team had developed a prototype for an automated test framework, but we didn’t have the resources to implement it because we were so busy trying to keep up with our gigantic manual test case library.

Even when you have a robust set of automated tests in place, you’ll still want to do some manual testing.  Having a pair of eyes and hands on an application is a great way to discover odd behavior that you’ll want to investigate further.  But trying to maintain a vast library of manual tests is so time consuming that you may find that you don’t have time to do anything else!

In my opinion, the easiest and most efficient way to keep a record of what manual tests should be executed is through using simple spreadsheets.  If I were to go back in time to that mobile app company, I would toss out the test case management system and set up some spreadsheets.  I would have one smoke test spreadsheet; and one regression test spreadsheet for each major feature of the application.  Each time a new feature was added, I’d create a test plan on a spreadsheet, and once the feature was released, I’d either add a few test cases to a regression test spreadsheet (if the feature was minor), or I’d adapt my test plan into a new regression test spreadsheet for that feature.

This is probably a bit hard to imagine, so I’ll illustrate with an example.  Let’s say we have a mobile application called OrganizeIt!  Its major features are a To-Do List and a Calendar.  Currently the smoke test for the app looks like this:

Test iOS phone iOS tablet Android phone Android tablet
Log in with incorrect credentials
Log in with correct credentials
Add an event
Edit an event
Delete an event
Add a To-Do item
Edit a To-Do item
Complete a To-Do item
Mark a complete item as incomplete
Delete a To-Do item
Log out

And then we also have a regression test for the major features: Login, Calendar, and To-Do List.  Here’s an example of what the regression test for the To-Do List might look like:

Test Expected result
Add an item to the list with too many characters Error message
Add an item to the list with invalid characters Error message
Add a blank item to the list Error message
Add an item to the list with a correct number of valid characters Item is added
Close and reopen the application Item still exists
Edit the item with too many characters Error message, and original item still exists
Edit the item with invalid characters Error message, and original item still exists
Edit the item so it is blank Error message, and original item still exists
Mark an item as completed Item appears checked off
Close and reopen the application Item still appears checked off
Mark a completed item as completed again No change
Mark a completed item as incomplete Item appears unchecked
Mark an incomplete item as incomplete again No change
Close and reopen the application Item still appears unchecked
Delete the item Item disappears
Close and reopen the application Item is still gone

This test would also be run on a variety of devices, but I’ve left that off the chart to make it more readable in this post.
Now let’s imagine that our developers have created a new feature for the To-Do List, which is that items on the list can now be marked as Important, and Important items will move to the top of the list.  In the interest of simplicity, let’s not worry about the order of the items other than the fact that the Important items will be on the top of the list.  We’ll want to create a test plan for that feature, and it might look like this:

Test Expected result
Item at the top of the list is marked Important Item is now in bold, and remains at the top of the list
Close and reopen the application The item is still in bold and on the top of the list
Item at the middle of the list is marked Important Item is now in bold, and moves to the top of the list
Item at the bottom of the list is marked Important Item is now in bold, and moves to the top of the list
Close and reopen the application All important items are still in bold and at the top of the list
Every item in the list is marked Important All items are in bold
Close and reopen the application All items are still in bold
Item at the top of the list is marked as normal The item returns to plain text, and moves below the Important items
Close and reopen the application The item is still in plain text, and below the Important items
Item in the middle of the Important list is marked as normal The item returns to plain text and moves below the Important items
Item at the bottom of Important list is marked as normal The item returns to plain text and is below the Important items
Close and reopen the application All important items are still in bold, and normal items are still in plain text
Delete an important item Item is deleted
Close and reopen the application Item is still gone
Add an item and mark it as important The item is added as important, and is added to the top of the list
Add an item and mark it as normal The item is added as normal, and is added to the bottom of the list
Close and reopen the application The added items appear correctly in the list
Mark an important item as completed The item is checked, and remains in bold and at the top of the list
Close and reopen the application The item remains checked, in bold, and at the top of the list
Mark an important completed item as incomplete The item is unchecked, and remains in bold and at the top of the list
We would again test this on a variety of devices, but I’ve left that off the chart to save space.  
Once the feature is released, we won’t need to test it as extensively, unless there’s some change to the feature.  So we can add a few test cases to our To-Do List regression test, like this:

Test Expected result
Add an item to the list with too many characters Error message
Add an item to the list with invalid characters Error message
Add a blank item to the list Error message
Add an item to the list with a correct number of valid characters Item is added
Close and reopen the application Item still exists
Add an important item to the list Item is in bold, and is added to the top of the list
Edit the item with too many characters Error message, and original item still exists
Edit the item with invalid characters Error message, and original item still exists
Edit the item so it is blank Error message, and original item still exists
Mark an important item as normal Item returns to plain text and is moved to the bottom of the list
Mark an item as completed Item appears checked off
Mark an important item as completed Item remains in bold text and appears checked off
Close and reopen the application Item still appears checked off
Mark a completed item as completed again No change
Mark a completed item as incomplete Item appears unchecked
Mark an incomplete item as incomplete again No change
Close and reopen the application Item still appears unchecked
Delete the item Item disappears
Close and reopen the application Item is still gone
Delete an important item Item disappears
The new test cases are marked in red, but they wouldn’t be in the actual test plan.  
Finally, we’d want to add one test to the smoke test to check for this new functionality:

Test iOS phone iOS tablet Android phone Android tablet
Log in with incorrect credentials
Log in with correct credentials
Add an event
Edit an event
Delete an event
Add a To-Do item
Add an important To-Do item
Edit a To-Do item
Complete a To-Do item
Mark a complete item as incomplete
Delete a To-Do item
Log out
With spreadsheets like these, you can see how it is easy to keep track of a huge amount of tests in a small amount of space.  Adding or removing tests is also easy, because it’s just a matter of adding or removing a line to the table.  
Spreadsheets like this can be shared among a team, using a product like Google Sheets or Confluence.  Each time a smoke or regression test needs to be run, the test can be copied and named with a new date or release number (for example, “1.5 Release” or “September 2019”), and the individual tests can be divided among the test team.  For example, each team member could do a complete test pass with a different mobile device.  Passing tests can be marked with a check mark or filled in green, and failing tests can be marked with an X or filled in red.
And there you have it!  An easy to read, easy to maintain manual test case management system.  Instead of taking hours of time maintaining test cases, you can use your time to automate most of your tests, freeing up even more time for manual exploratory testing.  

Break Your App With This One Weird Trick

I missed a bug recently, more than once, and I’m kicking myself about it.  This post is about that bug and how you should make sure to always run a test for it. It’s also about how to keep from repeating your mistakes.

Here’s what happened: as I mentioned in a previous post, I’m currently testing file uploads and downloads.  When the developers on my team first coded the upload functionality, I dutifully tested all kinds of file names: long names, short names, names without extensions, names with capital letters, etc.  Everything looked great, but I missed one important test: testing file names with spaces.  Every file name I had tested with was one single word, like “sunrise.jpg”.  I forgot to test with a name like “Grand Canyon.jpg”, and as it turned out, uploads with spaces in them weren’t working correctly.

Spaces are SO easy to forget to test, because they are literally invisible!  Testing with spaces applies to any text fields, not just file names.  For example, when you are testing a first name field, make sure to test with two first names, like “Mary Jo”.

It’s also important to remember to test with a space at the beginning of the text and at the end of the text.  Your developer should be trimming these whitespaces when processing input, but if he or she forgets, this can mean trouble.  You may wind up with a situation where you have a list of names sorted by last name, and the name “Smith” is appearing at the top.  This is probably because someone entered ” Smith” with a leading space.

Similarly, your users might have problems logging in to your application, even though they are sure they have the right username.  This may be because when they set their username, they accidentally put a space at the end of the username, and that space was not trimmed by the developer.  So they are trying to log in with “catLover” when they should really be logging in with “catLover “.

Back to my story: after the bug with file uploads was discovered and fixed, I carefully retested uploads again, this time being sure to include file names with spaces in the beginning, middle, and end of the file.

Our next development task was to be able to resize a file upon request.  When this functionality was ready, I started running all kinds of tests: resizing by height only, width only, height and width, resizing various file types, etc.  While I was testing, the developer who worked on the feature mentioned that he had just discovered a bug: the files wouldn’t resize if they had spaces in the names, because the spaces weren’t being encoded properly.  I’d like to think I would have discovered this eventually, but who knows?  I was more focused on the new functionality than I was on regression testing.

The bug was fixed, and again I carefully tested it.  I added spaces in the beginning, middle, and end of the file name, and I used every kind of special character on my keyboard.  Surely, I thought, this must be the end of this bug.

Our next development task was to do file size and type checking.  We didn’t want to upload a file if it was larger than the application was told to expect, or if it had the wrong file type.  I tested with files with all kinds of sizes and types, and all kinds of mismatches.  With each mismatch, I verified that the file was checked and rejected.  I did a great deal of regression testing as well.  It occurred to me to test with different file names, but since this I had already verified that file names of all kinds were working with resizing, it seemed like overkill, so I chose not to do it.

I was wrong!  As it turned out, the system that was doing the file checking was different from the system that was doing the file resizing, and once again, spaces in the file name weren’t being encoded properly.  A developer noticed that every file that had spaces in the name wasn’t being checked by the system.  I was bitten by the file space bug once again.

I hate making mistakes, and I hate missing bugs!  Fortunately these bugs were caught before they impacted any end users.  But I like to learn from my mistakes, and I’ve taken the following steps to make sure I don’t miss this bug ever again:

  • I’ve put a sticky note that says “Spaces” on my desk.  Whenever I see it, I will be reminded to test for spaces in my inputs.  I did this years ago when I kept forgetting to test on different browsers.  I had a “Test on IE” note on my monitor, and it helped me develop the habit.
  • I’ve added file names with spaces to my saved Postman collections.  This way, even if I forget to test with spaces, the saved requests will remember for me.
  • I’ve written this blog post to serve as a cautionary tale to myself and others.
One final lesson I’ve learned from this is that different parts of the same feature can process information differently.  The feature is using new technologies that I’ve never tested before, and I assumed file names would be handled in the same way for uploads, resizes, and type checking. But that was not the case.  When you are dealing with new technologies, be sure to regression test everything, even things you think might not need to be covered.
It’s amazing how much one simple space can break!  When you are testing anything that can accept text, from a simple form field to a file upload, be sure to remember those invisible spaces.

Let’s Go Deep! Part III: Internet Routing

In the last two posts, we’ve been going deep, learning about how information is transmitted over the Internet.  In Part I, we learned how data is divided into packets, sent to its destination and reassembled.  In Part II, we learned how data sent over the Internet can be encrypted and protected.  But how does data know how to get from one IP address to another?  This is the subject of today’s post.

Every device that can be connected to the Internet has a network interface card (NIC), which is a piece of hardware installed on the device that allows it to make a network connection.  Each NIC has a media access control (MAC) address that is specific to only that device.

A modem is a device that connects to the Internet.  Other devices can connect to the modem in order to receive Internet transmissions.  A wireless router is capable of receiving wifi transmissions.  The router connects to the modem, and other devices connect wirelessly to the router in order to receive data from the Internet.  Many Internet service providers (ISPs) provide customers with a combination modem/router, which connects to the Internet and sends and receives wireless signals.

In Part I, we learned that every device connected to the Internet has an IP address.  An IP address is different from a MAC address in that the MAC address is assigned by the manufacturer of the device, and an IP address is assigned by the network.  An IP address has two sections, called the subnet and the host.  The subnet refers to one subsection of the entire Internet.  The host is the unique identifier for the device on the network.  The IP address combines these two numeric values using bitwise operations.  You can’t look at an IP address and say that the numbers on the left make up the subnet and the numbers on the right make up the host; it doesn’t work that way.  The IP address is more like a sum of two long numbers.

As mentioned in Part I, when a packet of data is sent from a server to a client, it is sent with the IP address of the destination.  In order to get to that destination, the packet will hop from one network to another.  The routing protocol of the Internet is called the Border Gateway Protocol (BGP).  This is a system that helps determine a route that will traverse the least number of networks to get to the destination.  Every router in a network has a series of routing tables, which are sets of directions for how to get from one network to another.

When a packet of information is first sent to the network’s router, it looks at the IP address of the destination and determines if the directions to the destination are available in the routing tables.  If they are not, the BGP is used to determine the next logical network where the packet should be sent.  A gateway is an entrance to a network, and a default gateway is the address that the request is sent to if there’s no knowledge of a specific address in that network.  When a packet arrives at the new gateway, the BGP calculates the next appropriate destination.

After traversing through networks in this way, eventually the packet arrives at the router for the network that contains the IP address of the destination.  The router determines the MAC address of the destination and sends the packet to that address.

One more important feature of networking is the use of a proxy server.  A proxy server is a server that is positioned between the client and the destination server.  It is configured so that any requests your client makes will go through the server before it gets to its destination.  There are many uses for a proxy server; the main use is to keep the actual address of a site or a router private.  Proxy servers can also be used by hackers to intercept requests, especially on a public network.  Finally, proxy servers are a great way to do security testing!  Using a tool like Fiddler or Burp Suite, you can intercept the requests that you make to your application and the responses you receive in return.  You can learn more about how to use Burp Suite in this post.

This concludes my three-part series on how the Internet works.  I hope that you have found it helpful, and that you can apply these concepts when testing!

Let’s Go Deep! Part II: Encryption, Tokens, and Cookies

In last week’s post, we talked about how HTTP works to pass information from a server to a browser.  But when information is passed back and forth between systems, we need to make sure that it’s protected from being intercepted by others for whom it was not intended.  That’s why HTTPS was created.  In this week’s post, we’ll talk about how encryption is used in HTTPS, what the difference is between cookies and tokens, the different types of cookies, and how cookies can be protected.

How HTTPS Works:
When two systems communicate with each other, we refer to them as the client and the server.  The client is the system making the request, such as a browser, an application, or a mobile device, and the server is the system that supplies the information, such as a datastore.  HTTPS is a method of securely transmitting information between the client and the server.  HTTPS uses SSL and TLS to encrypt the data being transmitted and decrypt it only when it arrives at its destination.  SSL (Secure Sockets Layer) and TLS (Transport Layer Security) are both tools for the encryption and decryption of data; TLS is a newer version of SSL.  
Here’s how TLS works: before any data is transmitted, the client and the server first perform a handshake.  The handshake begins with the client contacting the server with a suggested encryption method and the server responding back agreeing to use that encryption method.  It then continues with the client and the server swapping certificates.  A certificate is like an ID card; it verifies the identity of the client or server.  The certificates are checked against the CA (Certificate Authority), which is a third-party entity that creates certificates specifically for the purpose of HTTPS communication.  
Once the certificates are swapped and verified, the client and the server swap decryption keys, and the handshake is completed.  Now the data is encrypted, transmitted from the server to the client, and decrypted once it arrives at the client safely.  
Session Cookies and Tokens:

Another important way data is secured is through the use of session cookies or tokens.  Session cookies and tokens are strings that are passed in with a client’s request to verify that the person making the request has the right to see the data requested.  The main difference between session cookies and tokens is that a session cookie is stored both on the client and the server, and a token is only stored on the client.  
In systems that use tokens, the token is created when a user logs in.  The token is made up of encrypted information that identifies the user.  The token is stored in local storage in the client’s browser and is sent with every HTTPS request to the server.  When the server receives the token, it decrypts it, validates the user’s information, and then returns the response.  
The most popular system of tokens in use today is JWT (JSON Web Token).  You can read more about JWTs in this helpful documentation.  
A session cookie is a unique string that is created by the server when the user logs in.  It is saved in the server’s datastore as a session id.  The server returns the cookie to the client, and the client saves it in the browser while the session is active.  Whenever the client makes a request to the server, it sends the cookie with the request.  The server then compares the cookie with the one it has saved to make sure it matches before returning the response.  
Tokens and session cookies are usually set to expire after a period of time or after an event.  For example, a token issued might be good for one hour.  Just before the hour is up, a request can be made for a new token (called a refresh token) in order to keep the user signed in.  Session cookies usually expire when the user logs out or when the browser is closed.
Persistent Cookies:

Another type of cookie used is the persistent cookie, which is a bit of data saved on the server about the user’s preferences.  For example, if a user goes to a website and chooses German as the language they would like on the site, a persistent cookie will remember that information.  The next time the user goes to the site, the cookie will be examined and the site will load in German.  
Securing Cookies:

Because they are stored on the server, cookies are more vulnerable to being intercepted and used by someone other than the user than tokens are.  To help protect cookies, these flags (attributes) can be added to them at the time of creation:
  • Secure flag: ensures that the cookie can only be transmitted over HTTPS requests and not over HTTP requests.
  • HttpOnly flag: keeps a cookie from being accessed via JavaScript, which helps protect it from Cross-Site Scripting (XSS) attacks.  
  • SameSite flag: ensures that the cookie can only be sent from the original client that requested the cookie, which helps protect it from Cross-Site Request Forgery attacks.  

Hopefully this post has helped you learn how HTTPS works, and how tokens and cookies ensure a client’s validity.  Researching this information certainly helped me!  But I want to go deeper: how does a message know how to get from one IP address to another?  How are HTTP requests intercepted?  And how does a router work?  I’ll have answers to these questions in next week’s post!  

Let’s Go Deep! Part I: How HTTP Requests Work

Recently, an astute reader of my blog pointed me to a great post about the importance of having technical skills as a software tester.  The author makes an excellent analogy: a software tester who doesn’t understand technical concepts is like a surgeon who doesn’t understand anatomy.  If we are going to test our applications thoroughly, we should understand the underlying systems that make them work.

I freely admit that I am not an expert in networking, or even in how the Internet works.  But I’m willing to learn, and pass that information on to you!  So let’s go deep!  We’ll begin this week with how HTTP requests work.

When you type an website’s address into a Web browser, you are typing a URL.  A URL (Uniform Resource Locator) is simply a fancy name for a web address.  The URL contains a domain name.  The domain name identifies a specific grouping of servers on the Internet.  Examples of domain names would be google.com or amazon.com.
Once the browser has the domain name, it uses it to look up the associated IP address in the DNS (Domain Name System), which is a database that contains all the mappings of domain names and IP addresses.  An IP address (Internet Protocol address) is a unique series of numbers that is assigned to every device that is connected to the Internet.  
Once the IP address is known, a connection is opened to that address using HTTP.  HTTP (HyperText Transfer Protocol) is an application protocol that allows information to be transmitted electronically.  
Once the connection is opened to the server at the IP address, a request can be made to get information from that server.  Information sent over the Internet is called a message.  The request uses TCP (Transmission Control Protocol), which is a system of delivering messages over the Internet.  
The TCP divides a message into a series of packets, which are fragments of between 1000 and 3000 characters.  Each packet has a series of headers, which include the address of the packet’s destination, information about the ordering of the packets, and other important information.  
If for any reason a packet doesn’t make it to its destination, the client (the address making the request) can request a packet to be resent.  Once all the packets have arrived, the client reassembles them according to the instructions in the header.
It’s no secret that information sent over the Internet can be vulnerable to security attacks.  A malicious user can intercept HTTP requests and get sensitive information from them or manipulate them to make requests that will return sensitive information.  For this reason, data sent over the Internet is often encrypted or protected.  We’ll learn more about this in my next post, which will be on encryption, tokens, and cookies!

Six Tips and Four Tools for File Upload Testing

I’ve been testing file uploads lately, which is always fun.  It’s also important, because uploading a malicious file is one of the ways that a bad actor can exploit your application, either by taking down your application, or by extracting sensitive data from it.  In this week’s post, I’ll offer six tips and four tools to help you be successful with testing file uploads.

Tip One: Upload Files With Allowed and Forbidden Extensions

The first step in testing file uploads is to find out what kinds of files will be allowed to be uploaded.  These files should be in the form of a whitelist, NOT a blacklist.  A whitelist specifies that only certain extensions will be allowed, whereas a blacklist specifies what is not allowed.  You can imagine that when a blacklist is used, there are dozens and dozens of file types that will be allowed, some of which you will not want in your application!  Therefore, it’s important to use a whitelist instead, which will be limited to the very few types of files that you want interacting with your application.  If your developers are not using a whitelist, please share this information with them.
Once you know what the whitelisted file types are, try uploading each type.  Then try uploading a wide variety of files that are not whitelisted.  Each of those files should be rejected with an appropriate error message for the user.
Tip Two: Upload Files With Inaccurate Extensions

One of the tricks malicious users employ to upload forbidden files is to rename a malicious file with an allowed extension.  For example, a bad actor could take a .js file and rename it as a .jpg file.  If .jpg files are allowed in your application, the file might be uploaded and then executed when opened by an unsuspecting user.  So it’s important for your application to have checks in place to not only verify the extension, but also to scan the file to verify its type.  
It’s easy to test this by simply taking a forbidden file, renaming it to have an allowed extension, and attempting to upload the file.  The file should be rejected with an appropriate error message.  The attempt should also be logged by the application, so if there is ever an upload attempt of this kind in production your security team can be alerted.
Tip Three: Test for Maximum File Size

Your application should specify a maximum file size.  Files that are too big can cause damage to your application either by slowing it down or causing it to crash, and can even cause data to be accidentally exposed, such as in a buffer overflow exploit.  
Find out what your application’s maximum file size is, and verify that files equal to and less than that size are uploaded appropriately.  Then verify that files over that maximum size are rejected with an appropriate error message.  Be sure to test with files just over the maximum size, and with files well over the maximum size.  
Tip Four: Test With Animated GIFs

Often when image uploads are allowed in an application, the .gif extension is one of the allowed types.  GIFs can sometimes contain animation.  Verify with your team whether your application will allow animated GIFs, and if not, verify what should happen if a user uploads one.  Will the file just display as a static image, or will the file be rejected?  Make sure that uploading an animated GIF does not result in a broken image on the page.  If animated GIFs are accepted, verify that it loads and displays the animation properly (see the next tip).  
Tip Five: Verify That the File Was Uploaded Correctly

It’s not enough to verify that you don’t get an error message when you upload a whitelisted file.  You also need to verify that the file was saved to the database correctly.  The easiest way to do this is to download the file and make sure it looks the same way it did when you uploaded it.  If your file should be displayed in the UI, you should make sure that the file looks correct in a browser or on a mobile device.  If an image that you uploaded should be resized on the page, make sure that it has resized correctly.  You don’t want to have other data obscured because someone uploaded an image that’s too large!  If you are expecting a video or audio file to play, make sure it’s playable. 
Tip Six:  Have a Folder With File Examples for Testing

My favorite tip is to have a folder filled with files of all kinds for use in testing.  I have a folder with tons of files with different extensions and a wide variety of sizes.  This way whenever I need to test file uploads, I’m ready to go with test files and I don’t have to waste time combing the internet for good examples to use.  
This brings me to my Four Tools for File Upload Testing!

I recently discovered this site when I realized that I needed some files with a .doc rather than a .docx extension.  This site definitely delivered, and it has many other example files as well.
I mentioned this tool in my Fifteen Free Tools to Help With Testing post.  When you need to test file size limitations, you can use this tool to create files of all different sizes.
This is an extensive, easy-to-read list of all the MIME types and their extensions.  It’s very helpful when you want to identify less common file types to test with, or when you are wondering what MIME type goes with a certain extension.  
Tool Four: Eicar Test File

If your application has virus-checking for uploaded files, you will want to use this test file.  It is a file that is designed to look like it has a virus, but it is actually virus-free.  You may find, however, that if your computer has virus-checking you won’t actually be able to download the file!  I was able to get around this by having someone send me the text of the file through chat, and then I pasted the text into the raw input window of Postman for my upload request.  
File uploads are one of my favorite things to test.  If you follow these tips and use these tools, it may become one of your favorites as well!

One Request, Sixteen Assertions

Anyone who has been reading my blog for a while knows that I’m passionate about API testing.  I’m also passionate about using Postman for API testing, because in my opinion it’s the easiest way to test API requests.  So it makes me sad when I see people testing an API and only asserting that they got a 200 response!  In this week’s post, I’m going to take a simple GET request and show examples of 16 assertions that could be run on the request.

For the GET request, I’ll be using the wonderful Restful-Booker application created by Mark Winteringham to teach people how to do API testing.  The request URL I’ll be using is https://restful-booker.herokuapp.com/booking/1 which will retrieve the booking with the ID of 1. If you run this GET request in Postman, you’ll get a response like this:


The response has the first and last name of a hotel guest, the total price of their room, whether or not their deposit has been paid, and what their checkin and checkout dates are.

You may see different details in the response from those in the above image, because this is a site that is under frequent use as people practice making API requests to change booking information.

Let’s look at all the different assertions we can do on this response. To add these assertions yourself, click on the “Tests” tab just underneath the request URL.

1. First, we can add the most common assertion: asserting that the response code is correct.

pm.test(“Status code is 200”, function () { pm.response.to.have.status(200); });

The pm.test refers to “Postman test”. “Status code is 200” is the name of our assertion. And pm.response.to.have.status(200) is what we are expecting from the response.

2. We can also assert that the response comes back within a reasonable time frame, like this:

pm.test(“Response time is less than 1000ms”, function () { pm.expect(pm.response.responseTime).to.be.below(1000); });

This asserts that the response time is less than one second.

3-8. Next we can assert that expected headers are present:

pm.test(“Server header is present”, function () { pm.response.to.have.header(“Server”); }); pm.test(“Connection header is present”, function () { pm.response.to.have.header(“Connection”); });

pm.test(“Content-Length header is present”, function () { pm.response.to.have.header(“Content-Length”); }); pm.test(“Etag header is present”, function () { pm.response.to.have.header(“Etag”); }); pm.test(“Date header is present”, function () { pm.response.to.have.header(“Date”); }); pm.test(“Via header is present”, function () { pm.response.to.have.header(“Via”); });

These assertions are looking for the presence of a particular header rather than checking that the header has a specific value. The specific assertions above might not be particularly necessary, but sometimes it’s a good idea to assert that certain headers are in place, such as X-XSS-Protection (not in this API), which indicates that there are measures in place to protect from cross-site scripting.  

9-10.  We can also have assertions that validate that a specific header value is being returned:

pm.test(“X-Powered-By header value is Express”, function () {
    pm.response.to.be.header(“X-Powered-By”, “Express”);
});

pm.test(“Content-Type header value is application/json”, function () {
    pm.response.to.be.header(“Content-Type”, “application/json; charset=utf-8”);
});

The first assertion is validating that the server used is Express, and the second assertion is validating that the Content-Type of the response is application/json.

11.  We can assert that certain text is included in the body of the response:

pm.test(“Response contains last name”, function () {
    pm.expect(pm.response.text()).to.include(“lastname”);
});

In this assertion, I am validating that the text in the body of the response contains “lastname”.

12-13.  I can also assert that specific json fields exist in the response.  To do this, I first need to parse out the json in the response and save it to a variable:

var jsonData = pm.response.json();

Then I can assert on specific fields in the jsonData variable.  Here I am checking to make sure that the response has both a checkin field and a checkout field:

pm.test(“Response contains checkin date”, function () {
    pm.expect(jsonData.bookingdates.checkin).to.exist;
})

pm.test(“Response contains checkout date”, function () {
    pm.expect(jsonData.bookingdates.checkout).to.exist;

})

14-15. I can also assert that specific values are returned in the response:

var jsonData = pm.response.json();

pm.test(“Correct first name is returned in response”, function () { pm.expect(jsonData.firstname).to.eql(“Mark”); });

pm.test(“Deposit has been paid”, function () { pm.expect(jsonData.depositpaid).to.be.true; });

In this first assertion, I am validating that the value of the first name returned in the response is “Mark”. In the second assertion, I’m validating that “depositpaid” is true. Keep in mind that these values may actually change because others are using this API. When you add specific value assertions, you’ll want to be sure that your test data does not change; otherwise your tests will be flaky!

16. Finally, I can assert that a value is greater or less than a certain number:

var jsonData = pm.response.json();

pm.test(“Total price is greater than 100”, function () { pm.expect(jsonData.totalprice).to.be.above(100); });

In this case, I am asserting that the total price paid is greater than 100.

The types of assertions that you run on your API responses will vary depending on what is important for you to test. While you may not decide that all types are appropriate for your API, I hope that these examples show that there are many more things to assert on than simply “200 OK”!



Stop Writing So Many UI Tests!

If you were to guess the importance of various types of automated tests by looking at the number of tutorials and articles about them on the Web, you’d think that UI tests were the most important.  But this is not the case- so much of an application can be tested through other means, especially API tests.  API tests are faster and much less flaky than UI tests, and they’re easier to write as well!  Below are four types of tests that are better suited for API testing.

Login Tests:  It’s easy to cycle through all kinds of username and password combinations with API tests.  The response time from a POST to a login endpoint is lightning fast, as opposed to a UI test which has to wait for the username and password fields to be populated and wait for the login response to reach the browser.  To prove this, I created a Postman collection that had sixteen login tests with various user and password combinations.  The sixteen tests ran in less than three seconds!  Imagine how long the equivalent UI tests would take.  
However, you should have two automated UI tests: one that validates that the login page looks correct, and that the user is able to log in, and one that validates that an appropriate error message is displayed when the user attempts to log in with bad credentials.
CRUD Tests:  When you’re testing CRUD functionality, you’re testing how your application interacts with the underlying data store.  This is best done at the API level.  It’s easy to create GET, POST, PUT, and DELETE tests using a tool like Postman.  You can assert on both the response codes and the body of the response (if any), and you can also do GETs to assert that your POSTs, PUTs, and DELETEs have saved correctly to the database.
The only UI tests you need in this area are one that demonstrates that form fields can be filled out and submitted, and one that shows that data is displayed correctly on the page.
Negative Tests:  API testing is great for negative tests, because not only can you run through all kinds of negative scenarios very quickly, but you can also run tests that aren’t possible in the UI.  For example, let’s say that your form has a required field.  In the UI, you can’t do a test where you submit a new record without that required field, because the UI simply won’t let you.  But in the API, you can do a POST without the required field and verify that you are getting a 400-level response.  API testing is also great for checking application security, because malicious users are likely to try to attack the application at this level.  
Here is just a sampling of the types of negative tests you can run with API testing:
  • sending in an inaccurate URL
  • trying a request without appropriate authentication
  • testing for IDOR
  • sending in incorrect headers
  • sending in a request without a required field
  • trying a request with a field value that violates type or length constraints
  • verifying that the correct 400-level error is displayed when a request is invalid
For UI testing, you’ll simply want to verify that appropriate errors are displayed on the page when you leave a required field blank or violate a field constraint.  Everything else can be covered by API tests.
Tests of Complicated Business Logic:  If you have an area of your application that requires an extensive setup of data and complicated business logic, it’s much easier to test with an API than with the UI.  Consider my hypothetical Superball Sorter, which sorts balls of various colors and sizes among four children.  Setting up the rules through the UI in an automated test would be tedious; assuming each child had a dropdown picker for size and color, you’d need to do a lot of element selection.  But if the Superball Sorter had an API that could set all the rules for the children in a single POST, it would take milliseconds to prepare the test.  
Similarly, after the sorting has been run, a UI test would need to grab all the responses on the page to validate that the balls have been sorted correctly, where an API could do a GET request for each child and validate that the appropriate balls are returned.  Four GET requests will most likely be returned and validated before a UI test could validate a single child’s values.  
Now that you have seen the many ways that API tests can be used, I hope that you will take the time to look at your current UI test suite to see which tests could be shifted to API testing.  Your automation suite will be faster, more reliable, and easier to maintain as a result!

The Easiest MongoDB Tutorial on the Web

MongoDB is one of the most popular non-relational databases in use today.  Its versatility, speed, and scalability make it popular with applications that need to store data in a JSON-like format.  It’s very easy to install MongoDB and create a database, but the query language it uses is quite different from the SQL query language.  When I first started using MongoDB, I was frustrated by the documentation I found on queries; either they tried to explain too much or the examples were too complicated.  More than once I said things in frustration like “How can I simply ask for ‘select lastName from contacts where id = 3?!!'”.

It is because of this frustration that I have created this tutorial.  In the tutorial, I’ll be including several really simple examples of queries that you are probably used to running in SQL. 

Installing Mongo:

Because this post is really about writing queries, I’m going to skip the installation instructions, and instead send you here for MacOSX and here for Windows.  Once you have finished the installation instructions, you should have a command window open that is running mongo.

Creating a Database:
Creating a new database is unbelievably easy.  We’re going to name our new database tutorial.  To create it, simply type use tutorial.  You’ll get the response switched to db tutorial.  Tada!  Your database is created.

Adding a Document:
Of course, right now your database is completely empty.  Let’s change that by typing
db.tutorial.insertOne( { _id: 1, firstName: “Prunella”, lastName: “Prunewhip” } ).  
You will get a response of 
{ “acknowledged” : true, “insertedId” : 1 }. 
You have just added your first document!  Note that a “document” is the equivalent of a “record” in a SQL database.  Your document has an id (which is preceded by an underscore, by convention), a first name, and a last name.

Retrieving All Documents:
Let’s make sure that your document was really added by asking for it.  Type 
db.tutorial.find() 
and you should get this as a result: 
{ “_id” : 1, “firstName” : “Prunella”, “lastName” : “Prunewhip” }. 
The empty find() command will find all of the documents in the database.  At the moment, we only have one document, so that’s all that was returned.

Adding Multiple Documents:
To add several documents at the same time, use the InsertMany command, like this:

db.tutorial.insertMany([ { _id: 2, firstName: “Joe”, lastName: “Schmoe” }, { _id: 3, firstName: “Amy”, lastName: “Smith” }, { _id: 4, firstName: “John”, lastName: “Smith” }, { _id: 5, firstName: “Joe”, lastName: “Bagadonuts” }, { _id: 6, firstName: “Carol”, lastName: “Jones” }, { _id: 7, firstName: “Robert”, lastName: “Johnson” } ]). 

Note that each document is wrapped in curly braces, separated by commas.  You’ll get a response like this: 
{ “acknowledged” : true, “insertedIds” : [ 2, 3, 4, 5, 6, 7 ] }. 
Now you have seven records in your database.

If you retrieve all documents at this point using db.tutorial.find(), you’ll get a result like this:
{ “_id” : 1, “firstName” : “Prunella”, “lastName” : “Prunewhip” }
{ “_id” : 2, “firstName” : “Joe”, “lastName” : “Schmoe” }
{ “_id” : 3, “firstName” : “Amy”, “lastName” : “Smith” }
{ “_id” : 4, “firstName” : “John”, “lastName” : “Smith” }
{ “_id” : 5, “firstName” : “Joe”, “lastName” : “Bagadonuts” }
{ “_id” : 6, “firstName” : “Carol”, “lastName” : “Jones” }
{ “_id” : 7, “firstName” : “Robert”, “lastName” : “Johnson” }

Retrieving a Single Document:
To retrieve a single document, use this syntax:
db.tutorial.find( { _id: 1 } ).  
This will return the document with the id of 1: 
{ “_id” : 1, “firstName” : “Prunella”, “lastName” : “Prunewhip” }

Search for All Documents With a Single Value:
The previous search on id will always return just one document, because the id is unique.  If you want to search for all documents that have a particular value, you can use 
db.tutorial.find({ lastName: “Smith”}).  
This will return all documents that have the last name Smith:
{ “_id” : 3, “firstName” : “Amy”, “lastName” : “Smith” }
{ “_id” : 4, “firstName” : “John”, “lastName” : “Smith” }

Search for One Value in One Document:
Let’s say you want to find the last name of the document with the id of 3.  To do this, type:
db.tutorial.find({ _id: 3}, {lastName:1, _id:0}).  
You will get this result:  
{ “lastName” : “Smith” }. 
The _id:0 is there to specify that you don’t want the id returned in the response; returning the id in the response is a default behavior in MongoDB.

Return All the Values for a Specific Field:
If you wanted to get a list of all the last names in your database, you would use
db.tutorial.find({}, {lastName:1, _id:0}).  
This would return
{ “lastName” : “Prunewhip” }
{ “lastName” : “Schmoe” }
{ “lastName” : “Smith” }
{ “lastName” : “Smith” }
{ “lastName” : “Bagadonuts” }
{ “lastName” : “Jones” }
{ “lastName” : “Johnson” }

Search with “Starts With”:
MongoDB uses regex to search on field values.  To search for all the documents that have last names that begin with S, you’d do this search:
db.tutorial.find({ lastName: /^S/}).  
This will return 
{ “_id” : 2, “firstName” : “Joe”, “lastName” : “Schmoe” }
{ “_id” : 3, “firstName” : “Amy”, “lastName” : “Smith” }
{ “_id” : 4, “firstName” : “John”, “lastName” : “Smith” }

Search with “And”:
If you wanted to search for a document that had a specific first name AND a specific last name, you’d search like this:
db.tutorial.find( {$and: [{ lastName: “Smith” },{ firstName: “Amy”}]} )
which would return 
{ “_id” : 3, “firstName” : “Amy”, “lastName” : “Smith” }.

Search with “In”:
To search for all the documents that have either the last name Smith or the last name Jones, you’d use:
db.tutorial.find({lastName :{$in :[“Smith”,”Jones”]}}).  
This will return
{ “_id” : 3, “firstName” : “Amy”, “lastName” : “Smith” }
{ “_id” : 4, “firstName” : “John”, “lastName” : “Smith” }
{ “_id” : 6, “firstName” : “Carol”, “lastName” : “Jones” }

Update a Document:
If you’d like to change an existing document, you can use the Update command.  For example, to change the last name of the third document from Smith to Jones, you would type:
db.tutorial.updateOne({_id: 3 }, {$set: {lastName: “Jones”}}).  
You’ll get this response: 
{ “acknowledged” : true, “matchedCount” : 1, “modifiedCount” : 1 }.

To verify that the record was updated correctly, you can use db.tutorial.find( { _id: 3 } ), which will return { “_id” : 3, “firstName” : “Amy”, “lastName” : “Jones” }.

Delete a Document:
Finally, there may be times where you want to delete a document.  This can be done with
db.tutorial.deleteOne({_id: 4 })
which will return a response of 
{ “acknowledged” : true, “deletedCount” : 1 }.

To verify that the document has been deleted, you can run db.tutorial.find() and get this response:
{ “_id” : 1, “firstName” : “Prunella”, “lastName” : “Prunewhip” }
{ “_id” : 2, “firstName” : “Joe”, “lastName” : “Schmoe” }
{ “_id” : 3, “firstName” : “Amy”, “lastName” : “Jones” }
{ “_id” : 5, “firstName” : “Joe”, “lastName” : “Bagadonuts” }
{ “_id” : 6, “firstName” : “Carol”, “lastName” : “Jones” }
{ “_id” : 7, “firstName” : “Robert”, “lastName” : “Johnson” }
and you can see that the document with the id of 4 is no longer in the database.

This is by no means a complete record of everything that you can do with MongoDB, but it should be enough to get you started.  You can also refer to last week’s post to get a few examples of interacting with nested values in MongoDB.  I hope that you will find today’s post helpful in understanding how Mongo works, and that you will use it as a reference whenever you need it.  Happy querying!