Think Like a Tester

In my last two positions, I have been the sole Software Engineer in Test for a brand-new software platform.  This means that I have been able to set up an entire arsenal of automated tests for both the backend and the front-end.  This article will describe the suite of tools I used to test a web application.

API Tests:
First, I set up tests that checked that every API call worked correctly.  I did this through Runscope, an API testing tool I highly recommend.  I was fortunate enough to have engineers who set up Swagger, a framework that lists API calls in a really easy-to-read format.  I used the information from Swagger to set up my Runscope tests.  For testing against our development environment, I created tests for every GET, POST, PUT, and DELETE request.  I set this suite of tests to run once a day as a general health check for the development environment.  For testing against our production environment, I set up tests for every GET request that ran hourly.  This way we would be notified quickly if any of our GET requests stopped working.  Runscope is also good for monitoring the response times of API calls.

Backend Tests:
The backend of our platform was written in Java, and we developed a suite of integration tests using JUnit and AssertJ.  Generally the developers would create a few basic tests outlining the functionality of whatever new feature they had coded, and I would create the rest of the tests, using the strategy that I outlined in my previous post.

Front-End Tests:
The front-end of our platform was written in Angular, so I used Protractor to write a series of end-to-end tests.  The UI developers were responsible for writing unit tests, so I focused on testing typical user interactions.  For example, in my Contact-Info tests, I tested adding and editing a phone number, adding and editing an email address, and adding and editing a mailing address.  If it was possible to delete contact info in our UI, I would have added those tests to the suite as well.  I set up two suites of tests: a smoke suite and a full suite.  The smoke suite was for basic feature testing and was integrated into the build process, running whenever a developer pushed new code.  If they pushed a change that broke a feature, they would be notified immediately and the code would not be added to the build.  The full suite was set to run every test, and was configured in Jenkins to run overnight against the development environment using SauceLabs.  If anything happened to cause the tests to fail, I’d be notified by email immediately.

Since the UI also included uploaded images, I integrated Applitools visual testing into my Protractor tests.  Their platform is extremely easy to integrate and use.  I wrote tests to check that the image I added to a page appeared on the screen, and tests to check that if I changed an image, the new image would appear on the screen.

Load Tests:
In my previous position, I set up a series of load tests using SoapUI.  This tool is similar to Runscope in that it tests API calls, but it has the added functionality of being able to configure load testing.  I set up the following test scenarios: a Baseline test, where a small group of simulated users would run through a series of API requests; a Load test, where a small group of users would make a more frequent series of requests; a Stress test, where a large group of users would make a frequent series of requests; and a Soak test, where a small group of users would make a frequent series of requests for a long test time.  We were able to use this suite of tests to monitor response times, check for memory leaks, and test our load balance setup.

Every application is different, so you may find that you need a different set of tools for your test toolbox!  But I hope that this blog post gives you a jumping-off point to think about how to test your application thoroughly.

In this post, I’ll be switching gears from discussing automated tests with Webdriver to discussing integration tests that are included as part of an application’s back-end Java code. 

In my new Software Engineer in Test position, I have been writing integration tests to test every API call we have.  This means that I have dozens of test classes, and over a thousand possible individual tests.  Today I’m sharing with you the strategy I came up with for organizing the tests. 

I sorted out the tests into four test types: 

Resource Tests:  these are the “Happy Path” tests- the tests where you expect things to work correctly.  Examples include:  POST requests with valid parameters, GET requests where the correct object is returned,  PUT requests where the new parameters are valid , and DELETE requests where the correct object is deleted.

Security Tests:  this is where I put all the tests that should fail because the user is not logged in.  For every method that is tested in the Resource Tests, there should be a similar test where the method is called, but the user is not logged in.

Validation Tests:  this is where I put all the tests that should fail because a value being sent is invalid.  Examples include: sending an null value where a value is required, sending an empty value where a value is required, sending a value with invalid characters (such as letters in a numeric field), and sending a value with too many or too few characters.  It’s important to include tests for both Create and Update requests. 

Conflict Tests:  this is where I put tests that should fail because the request has been incorrectly formed.  Examples include: 
any POST, GET, PUT, or DELETE request where the required id is null
any POST, GET, PUT, or DELETE request where the required id is non-existent
any POST request where an id of the object about to be created is included
any PUT request where the id in the request does not match the id of the existing object

I created a package for each of these test types.  Then for each domain object, I created four test classes: one for each package.  For example, our code has a domain object called Phone.  Here are the test classes I created:

PhoneResourceTest- verifies that it is possible to add, get, edit, and delete a phone number

PhoneSecurityTest- verifies that it is not possible to add, get, edit, or delete a phone number if the user is not logged in

PhoneValidationTest- verifies that the phone number being added or edited is in a valid format (ten digits, no letters, etc.)

PhoneConflictTest- verifies that it is not possible to add or edit a phone number if the id of the associated contact is missing or incorrect, that it is not possible to add a phone number if there is a phone number id in the request, that it is not possible to edit a phone number if the phone number id in the request does not match the id of the phone number being edited, etc.

I hope this post will help you think about how to best organize your own integration tests!

Today we will tackle the most difficult ways to find an element: by css and xpath.  These are the most difficult because it’s so easy to miss a slash or a period in the description.  They are also the most difficult because they are so brittle- every time a developer makes a change to the webpage structure, the navigation path will change and your findElement command will no longer work.

However, finding an Element by css or xpath is often very needed, because developers often don’t take the time to give their elements easy and unique ids or names.  I would recommend using these strategies as a last resort to find the element you need, and I would try css first before resorting to xpath.

First, let’s find an element with css.  Take a look at the table that was described in the last blog post:

In order to find and use web elements using By.cssSelector or By.xpath, it is very helpful to understand the DOM.  The DOM (Document Object Model) is simply the interface that is used to interact with HTML and XML documents.  When a JavaScript program manipulates elements on a page, it finds them using the DOM.  If you already understand HTML, it will be very easy to understand how the DOM is organized.  Rather than explaining the differences between the DOM and HMTL source code, I’ll simply direct you to this very helpful webpage:  

https://css-tricks.com/dom.  

If you have little knowledge of HTML, here’s an example of how web elements are organized on a page:

<html>

<head>

<title>My Web Page</title>

</head>

<h1>My Web Page</h1>

<p>This is a paragraph</p>

<a href=”http://www.google.com” >Here’s a link to Google</a>

<table border=”2″>

<tr>

<td>Row 1, Column 1</td>

<td>Row 1, Column 2</td>

</tr>

<tr>

<td>Row 2, Column 1</td>

<td>Row 2, Column 2</td>

</tr>

</table>

</html>

If you’d like to see what this webpage would look like, you can simply copy and paste this html code into a notepad file and save it with the .html extension.  Then find the file in the folder where you’ve saved it and double-click on it.  It should open up as a webpage.  

Now let’s take a look at the various elements on the page.  Note that each element has a beginning and ending tag.  For example, the line that says “This is a paragraph” has a <p> tag to indicate the beginning of the paragraph and a </p> tag to indicate the end of the paragraph. Similarly, the title of the page has a beginning tag: <title> and an ending tag: </title>.  

Notice that elements can be nested within each other.  Look at the <table> tag, and then find the </table> tag several lines below it.  In between the tags, you will see row tags (<tr>) and table data tags (<td>).  Everything in between the <table> and </table> tags is part of the table.  Now look at the first <tr> tag and the first </tr> tag.  Notice that there are two pairs of <td></td> tags in between. Everything between the first <tr> tag and the first </tr> tag is a row of the table.  The <td></td> pairs in the row are elements of data in the row.

Now imagine that this data is organized into tree form:

If you were going to traverse the DOM in order to get at the data in Row 1, Column 1, you’d start by finding the <table> element, then by finding the first <row> element, then you’d find the first <data> element.  This is how we will use css selectors and the xpath to find elements in the next blog post.

If you’d like to find out more about HTML and CSS, I highly recommend the w3schools website.  

In my last post, I discussed how to find an element with Webdriver using the element id, class name, or name.  In this post, I will discuss how to find an element using link text, partial link text, or tag name.

Return to www.google.com, and look at the links on the bottom of the page.  Right-click on the link that says “Advertising” and select “Inspect Element”.  You should see something like this:
<a class=”_Gs” href=”//www.google.com/intl/en/ads/?fg=1″>Advertising</a>

The link text is what is found between the >reverse brackets<.  In this case, the link text is “Advertising”.

Return to myFirstTest in Eclipse and replace the findElement command in your test with:
driver.findElement(By.linkText(“Advertising”)).click();

This command tells Webdriver to:
1.  Find the element with the link text “Advertising”
2.  Click on the link

If you followed the instructions in the last post, you should still have a breakpoint set in your test. If not, right-click in the left margin of the “driver.quit();” command and select “Insert Breakpoint”. Then run the test by right-clicking on the test and selecting Debug As->JUnit Test.  The test should run and will stop just after it clicks on the Advertising link.  If the test has run correctly, you should now be on the Google Ads page.  Click on the green arrow to finish the test.  

Now we will find an element using a partial link text.  Return to www.google.com, right-click on the link at the bottom of the page that says “Business”, and select “Inspect Element”.  You should see something like this: 
<a class=”_Gs” href=”//www.google.com/services/?fg=1″>Business</a>
The link text is “Business”.  We will find this link using just a portion of the text: “ness”.  

Return to myFirstTest in Eclipse and replace the findElement command in your test with:
driver.findElement(By.partialLinkText(“ness”)).click();

Run the test by right-clicking on the test and selecting Debug As->JUnit Test.  The test should run and will stop just after it clicks on the Business link.  If the test has run correctly, you should now be on the Google Business Solutions page.  Click on the green arrow to finish the test.  

Now we will find an element using a tag name.  Go to www.pinterest.com, right-click on the button that says “Continue with Facebook” and select “Inspect Element”.  You should see something like this:
<button class=”Button Module btn hasText large rounded unAuthFacebookConnect registerLoginButton unauthHomeRegisterButton multiStepRedesign” type=”button”>    

Note that the first word that appears after the “<” is “button”.  This is the tag name of the element.

Return to myFirstTest in Eclipse and replace the String URL line with:
String URL = ‘http://www.pinterest.com’
Next, replace the findElement command in your test with:
driver.findElement(By.tagName(“button”)).click();

This command tells Webdriver to:
1.  Find the element with the tag name “button”
2.  Click on the button

Run the test by right-clicking on the test and selecting Debug As->JUnit Test.  The test should run and will stop just after clicking on the button.  If the test has run correctly, you will see a popup prompting you to log in with Facebook.  Click on the green arrow to finish the test.

You have successfully located a web element by using an link text, a partial link text, and a tag name! Before we proceed to finding a web element by css and xpath, it is first necessary to understand the DOM.  This will be the topic of my next blog post.  

The most useful method of Webdriver is the findElement() method.  It can also be the most frustrating:  you can see the element you want on your webpage, but it’s often difficult to figure out how to tell Webdriver to find it!

There are many different ways to find an element with findElement():

By.id
By.className
By.name
By.linkText
By.partialLinkText
By.tagName
By.cssSelector
By.xpath

I will include examples of these in this blog entry, and in the two entries following.  But first you need to know how to inspect an element in order to make the best choice about which By method to use.  Go to www.google.com, right-click in the search text field, and choose “Inspect Element”.  A new window will open up at the bottom of the screen, and you will see the element highlighted:

<input id=”lst-ib” class=”gsfi” type=”text” value=”” autocomplete=”off” name=”q” style=”border: medium none; padding: 0px; margin: 0px; height: auto…; width: 100%; background: url(‘” dir=”ltr” spellcheck=”false”></input>

Look at the id attribute:  id = “lst-ib”.  This is what we will use for our first findElement() exercise.
(Note that the id for this element changes periodically.  Substitute whatever you see listed as the element id in the steps below.)

Return to myFirstTest in Eclipse and add the following text below the driver.get(URL) command:
driver.findElement(By.id(“lst-ib”)).sendKeys(“Hello World”);

This command tells Webdriver to:
1.  Find the element with the id “lst-ib”
2.  Type “Hello World” into the element

You will also need to add the following to the import section of your test:
import org.openqa.selenium.By;

Insert a breakpoint into your test so that you can see if the commands were executed correctly.  To do this, right-click in the left margin of the “driver.quit();” command and select “Insert Breakpoint”.  Then to run the test, right-click on the test, and select Debug As->JUnit Test.  The test should run and will stop just after it has typed “Hello World” into the text field.  If this is what you see, then you have used findElement() correctly!  Click on the green arrow at the top of the screen to finish the test.

We now have an example of findElement(By.id()).  Now let’s try By.className.  Look back at the html that describes the text field.  You should see the class attribute:  class=”gsfi”.  (Note that the class name for this element changes periodically.  Substitute whatever you see listed as the class in the steps below.)Let’s change our test to find the element by className.  Replace the findElement command in your test with:
driver.findElement(By.className(“gsfi”)).sendKeys(“Hello World”);

Run the test in debug mode again, and you should again see that “Hello World” has been typed into the text field.  The result is the same, but this time the element was located by the class name instead of by the id! Click on the green arrow to finish the test.

Finally, let’s try to find the element by its name.  Look at the html again and see the attribute name=”q”. Replace the findElement command in your test with:
driver.findElement(By.name(“q”)).sendKeys(“Hello World”);

Run the test in debug mode again, and you should see that once again, “Hello World” has been typed into the text field.  Click on the green arrow to finish the test.

You have successfully located a web element by using an id, a class name, and a name!  The next blog entry will discuss finding elements by link text, partial link text, and tag name.

Webdriver is an API that enables testers to find and use web elements in their automated tests.  To get set up, you will need to find out what the latest version of Webdriver is.  Navigate to this link:  http://docs.seleniumhq.org/download and in the section called Selenium Client and Webdriver Language Bindings, find the Java entry and make a note of the latest client version.

Next, you will need to add this text to your project’s POM file:
<dependency>
   <groupId>org.seleniumhq.selenium</groupId>
   <artifactId>selenium-java</artifactId>
   <version>2.39.0</version>  //replace the version number here with the most current version number
 </dependency>

Add the text after the JUnit dependency, so that the dependencies section now looks like this:

<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.8.1</version>
    </dependency>
    <dependency>
<groupId>org.seleniumhq.selenium</groupId>
<artifactId>selenium-java</artifactId>
<version>2.39.0</version>
</dependency>
  </dependencies>

In the command line, navigate to your project folder.  Then type:
clean install
This will download all of the dependencies needed to run Webdriver in your project.

Let’s set up a new test using Webdriver:

1. Right-click on the myFirstTests package and select New->Class
2. Give the class a name of secondTest
3. The SecondTest class should open in the center pane, and should look like this:
package myFirstTests;
public class SecondTest {
}

4. In between the brackets, add the following:
@Test
public void test() {
}

5. Add the following to the import section:
import org.junit.Test;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.firefox.FirefoxDriver;

6.  In between the brackets following test(), add this code:
driver = (WebDriver) new FirefoxDriver();
String URL = “http://www.google.com”;
driver.get(URL);
driver.quit();

7.  Right-click on the test name secondTest and choose Run As-> JUnit Test.  You should see a Firefox browser open, navigate to Google, and close again, and on the JUnit tab of Eclipse, you should see that the test has passed.

Congratulations!  You are now all set up to begin writing your own automated Webdriver tests!  In my next post, I will discuss the Webdriver syntax.

A POM file is necessary for any automated testing with Maven.  It can seem daunting at first, especially if you don’t usually work with XML files, but it’s actually very simple.  

In order to use Webdriver, you will need to have Maven installed and set up.  Fortunately, the helpful people at Apache Software have created an excellent tutorial about how to set up Maven:

http://maven.apache.org/guides/getting-started/maven-in-five-minutes.html

It probably won’t take you five minutes as advertised, but it will take you less than a day.  Don’t worry too much about what you are doing as you are running these commands; once Maven is set up, we only have to think about the commands we need to run our automated tests.

Now let’s return to Eclipse.  The last time we were in Eclipse we created a project called MyFirstJavaProject.  Right-click on this project and select “Configure-> Convert to Maven project”.  A popup window will appear.   Click “Finish” on the window, and wait for the configuration to complete.  When it’s done, you’ll notice that your project icon now has a little M next to the J.  You’ll also notice that you have a pom.xml file in your project folder.  We’ll learn more about the POM in a future blog post.

Let’s create a JUnit test.  First we’ll run it from Eclipse, then we’ll run it from the command line using Maven.

1. Right-click on the MyFirstJavaProject and select New->Source Folder
2. Give the new folder a name of src/test/java (click the “Update exclusion filters” checkbox if needed)
3 Right-click on the new folder and select New->Package
4. Give the package a name of myFirstTests
5. Now right-click on the myFirstTests package and select New->Class
6. Give the class a name of FirstTest
7. The FirstTest class should open in the center pane, and should look like this:
package myFirstTests;
public class FirstTest {
}
8. In between the brackets, add the following:
@Test
public void test() {
int x = 2;
int y = 3;
int total = x+y;
assertEquals(total, 5);
}
9. Right-click on the Project name (MyFirstJavaProject) and choose Build Path->Add Libraries
10. Select JUnit
11. Choose JUnit 4 and click Finish
12. Look again at the lines of code you added to the FirstTest class.  There will be two lightbulb icons on the left side of the code.
13. Right-click on the one next to “@Test” and choose Add JUnit4 to Build Path
14. Right-click on the one next to “assertEquals(total,5)” and choose Import org.junit.Assert;
15. Now near the top of the page, you will see
import static org.junit.Assert.*;
import org.junit.Test;
16. Save your changes by clicking on the disk icon in the toolbar
17. Right-click on the FirstTest.java file in the left pane, and choose Run As-> JUnit Test
18. Your test should run, and a new tab named JUnit should appear in the left pane
19. Click on the left tab and notice that the window displays what test has been run, and that there is a green bar across the pane indicating that the test has passed

Now let’s try running the test from the command line. First we’ll need to make a change to the pom.xml file:

1. Double-click on the pom.xml file to open it in the center pane
2. Click the tab at the bottom of the pane that says pom.xml
3. You should now be viewing the file in xml format
4. Add this text to the xml file, right underneath </build>:
<dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.8.1</version>
    </dependency>
  </dependencies>
5. Save your changes
6. Now open the command window
7. Change directories until you are in your project folder (MyFirstJavaProject)
8. Run this command:
mvn clean test
9. Your project should build, your test should run from the command line, and you should be notified that the build is successful

Congratulations!  You have run your first test from Maven!

In order to run Maven (which we will discuss in the next post), it is necessary to know how to run commands from the command line.  If you are not familiar with using the command window, this post will explain the two most important commands.  My instructions will refer to the Windows command line, but they can be adapted without too much difficulty to Linux or Mac.

To bring up the command line, go to the Start menu and type “cmd” in the search field.  Click the return key and a command window should open.

cd:
cd stands for “change directory”.  You will use this to navigate through your file system.  When you opened your command window, you probably saw something like this:  C:UsersYourName>.  This is your user folder.  If you were to navigate to it in the File Explorer, you would go to Computer->Windows7_OS (C:)
->Users->YourName.  (The OS will vary based on your computer, and the YourName will be your user name.)

Let’s try navigating to the Users folder.  In the command window, type “cd Users” and click return key.  Note that the command prompt now says C:Users>.

There is another way to navigate up one folder.  First, let’s return to your folder.  Type “cd YourName” (replacing the words YourName with your actual user name) and click Return.  Now the command prompt should say C:UsersYourName.  Now type “cd ..” and click Return.  Notice that you have returned to the Users folder.  The two dots send the instruction to go up one level.  If you type “cd..” and Return again, you will see C:> in the command prompt.

You can also navigate through more than one folder at a time.  Type “cd Users/YourName” (replacing the YourName with your user name) and click Return.  Now the command prompt will say C:UsersYourName.  You’ve navigated down two levels, from the C: level to the YourName level.

dir:
dir is the command used to list all the files in a particular location.  This really comes in handy when you can’t remember what you named a file.

Let’s try using the dir command.  In the command window, verify that the command prompt currently reads C:UsersYourName.  If it doesn’t, use the cd command to navigate there.  Now type “dir” and click Return.  You should get a list of all the files in your personal folder.

Let’s add a new file to this location.  It’s possible to do this through the command line, but that is beyond the scope of this tutorial.  Instead, create a simple Word or Notepad file called MyCommandFile, and save it in your folder.  Now return to the command window and type the “dir” command again.  You should see your new file in the directory.

Another shortcut to navigation when you have forgotten a file name is the tab key.  In the command line, type “cd My” and then click the tab key.  You should see the file name auto-complete for you.

Bonus command:  
One more helpful command to use is the up arrow key.  This key will go back through your most recent commands.  This really comes in handy if you have typed a particularly long command and would rather not type it again.

The command window is very helpful, and there are a number of good tutorials available if you would like to learn more about what it can do for you.