The Power of Pretesting

Having been in the software testing business for a few years now, I’ve become accustomed to various types of testing: Acceptance Testing, Regression Testing, Exploratory Testing, Smoke Testing, etc.  But in the last few weeks, I’ve been introduced to a type of testing I hadn’t thought of before: Pretesting.

On our team, we are working to switch some automatically delivered emails from an old system to a new system.  When we first started testing, we were mainly focused on whether the emails in the new system were being delivered.  It didn’t occur to us to look at the content of the new emails until later, and then we realized we had never really looked at the old emails.  Moreover, because the emails contained a lot of detail, we found that we kept missing things: some extra text here, a missing date there.  We discovered that the best way to prevent these mistakes was to start testing before the new feature was delivered, and thus, Pretesting was born.

Now whenever an email is about to be converted to the new system, we first test it in the old system.  We take screenshots, and we document any needed configuration or unusual behavior.  Then when the email is ready for the new system, it’s easy to go back and compare with what we had before.  This is now a valuable tool in our testing arsenal, and we’ve used it in a number of other areas of our application.

When should you use Pretesting?

  • when you are testing a feature you have never tested before
  • when no one in your company seems to know how a feature works
  • when you suspect that a feature is currently buggy
  • when you are revamping an existing feature
  • when you are testing a feature that has a lot of detail

Why should you Pretest?

Pretesting will save you the headache of trying to remember how something “used to work” or “used to look”.  If customers will notice the change you are about to make, it’s good to make note of how extensive the change will be.  If there are bugs in the existing feature, it would be helpful to know that before the development work starts, because the developer could make those bug fixes while in the code.  Pretesting is also helpful for documenting how the feature works, so you can share those details with others who might be working on or testing the feature.

How to Pretest:

1. Conduct exploratory testing on the old feature.  Figure out what the Happy Path is.
2. Document how the Happy Path works and include any necessary configuration steps
3. Continue to test, exploring the boundaries and edge cases of the feature
4. Document any “gotchas” you may find- these are not bugs, but are instead areas of that feature that might not work as someone would expect
5. Log any bugs you find and discuss the with your team to determine whether they should be fixed with the new feature or left as is
6. Take screenshots of any complicated screens, such as emails, messages, or screens with a lot of text, images, or buttons. Save these screenshots in an easily accessible place such as a wiki page.

When the New Feature is Ready:

1. Run through the same Happy Path scenario with the new feature, and verify that it behaves the same way as the old feature
2. Test the boundaries and edge cases of the feature, and verify that it behaves in the same way as the old feature
3. Verify that any bugs the team has decided to fix have indeed been fixed
4. Compare the screenshots of the old feature with the screenshots of the new feature, and verify that they are the same (with the exceptions of anything the team agreed to change)
5. Do any additional testing needed, such as testing new functionality that the old feature didn’t have or verifying that the new feature integrates with other parts of the application

The power of Pretesting is that it helps you notice details you might otherwise miss in a new feature, and as a bonus, find existing bugs in the old feature.  Moreover, testing the new feature will be easy, because you will have already created a test plan.  Your work will help the developer do a better job, and your end users will appreciate it!

Automating Tests for a Complicated Feature

Two weeks ago, I introduced a hypothetical software feature called the Superball Sorter, which would sort out different colors and sizes of Superballs among four children.  I discussed how to create a test plan for the feature, and then last week I followed up with a post about how to organize a test plan using a simple spreadsheet.  This week I’ll be describing how to automate tests for the feature.

In case you don’t have time to view the post where I introduced the feature, here’s how it works:

  • Superballs can be sorted among four children- Amy, Bob, Carol, and Doug
  • The balls come in two sizes: large and small
  • The balls come in six colors: red, orange, yellow, green, blue, and purple
  • The children can be assigned one or more rules for sorting: for example, Amy could have a rule that says that she only accepts large balls, or Bob could have a rule that says he only accepts red or orange balls
  • Distribution of the balls begins with Amy and then proceeds through the other children in alphabetical order, and continues in the same manner as if one was dealing a deck of cards
  • Each time a new ball is sorted, distribution continues with the next child in the list
  • The rules used must result in all the balls being sortable; if they do not, an error will be returned
  • Your friendly developer has created a ball distribution engine that will create balls of various sizes and colors for you to use in testing

When automating tests, we want to keep our tests as simple as possible.  This means sharing code whenever we can.  In examining the manual test plan, we can see that there are three types of tests here:

  • A test where none of the children have any rules
  • Tests where all the children have rules, but the rules result in some balls not being sortable
  • Tests where one or more children have rules, and all of the balls are sortable
We can create a separate test class for each of these types, which will make it easy for us to share code inside each class.  We’ll also have methods that will be shared among the classes:

child.deleteBalls()- this will clear all the distributed balls, getting ready for the next test

child.deleteRules()- this will clear out all the existing rules, getting ready for the next test

distributeBalls(numberOfBalls)- this will randomly generate a set number of balls of various sizes and colors and distribute them one at a time to the children, according to the rules

verifyEvenDistribution(numberOfBalls)- this is for scenarios where none of the children have rules; it will take the number of balls distributed and verify that each child has one-fourth of the balls

child.addRule(Size size, Color color)- this will set a rule for a child; each child can have more than one rule, and either the size or color (but not both) can be null

child.verifyRulesRespected()- for the specified child, this will iterate through each ball and each rule and verify that each ball has respected each rule

child.addFourthRuleAndVerifyError(Size size, Color color)- for the tests in the InvalidRules class, it will always be the fourth child’s rules that will trigger the error, because it’s only with the fourth child that the Sorter realizes that there will be balls that can’t be sorted.  So this method will assert that an error is returned.

Each test class should have setup and cleanup steps to avoid repetitive code.  However, I would use child.deleteBalls() and child.deleteRules() for each child in both my setup and cleanup steps, just in case there is there is an error in a test that causes the cleanup step to be missed.

For the DistributionWithNoRules test class, each test would consist of:

distributeBalls(numberOfBalls);
verifyEvenDistribution(numberOfBalls);

All that would vary in each test would be the number of balls passed in.

For the DistributionWithRules test class, each test would consist of:

child.AddRule(Size size, Color color); –repeated as many times as needed for each child
distributeBalls(numberOfBalls);
child.verifyRulesRespected(); –repeated for each child that has one or more rules

Finally, for the InvalidRules test class, each test would consist of:

child.AddRule(Size size, Color color); –repeated as many times as needed for the first three children
child.addFourthRuleAndVerifyError(Size size, Color color); –verifying that the fourth rule triggers an error

The nice thing about organizing the tests like this is that it will be easy to vary the tests.  For example, you could have the DistributionWithNoRules test class run with 40, 400, and 800 balls; or you could set up a random number generator that would generate any multiple of four for each test.

You could also set up your DistributionWithRules and InvalidRules test classes to take in rule settings from a separate table, varying the table occasionally for greater test coverage.

Astute readers may have noticed that there are a few holes in my test plan:

  • how would I assert even distribution in a test scenario where no children have rules and the number of balls is not evenly divisible by four?
  • how can I show that a child who doesn’t have a rule is still getting the correct number of balls, when there are one or more children with a rule?  For example, if Amy has a rule that she only gets red balls, and Bob, Carol, and Doug have no rules, how can I prove that Bob, Carol and Doug get an even distribution of balls?
  • will the Superball Sorter work with more or less than four children? How would I adjust my test plan for this?
I’ll leave it to you to think about handling these things, and perhaps I will tackle them in future posts.

I’ve had a lot of fun over the last few weeks creating the idea of the Superball Sorter and thinking of ways to test it!  I’ve started to actually write code for this, and someday when it’s finished I will share it with you.

Organizing a Test Plan

In last week’s post, we took a look at a hypothetical software feature which would sort out Superballs among four children according to a set of rules.  I came up with forty-five different test cases from simple to complicated that would test various combinations of the rules. 

But a blog post is not a very easy way to read or execute on a test plan!  So in this week’s post, I’ll discuss the techniques I use to organize a test plan, and also discuss ways that I find less effective.

What I Don’t Do

1. I don’t write up step by step instructions, such as

Navigate to the login page
Enter the username into the username field
etc. etc.

Unless you are writing a test plan for someone you have never met, who you will never talk to, and who has never seen the application, this is completely unnecessary.  While some level of instruction is important when your test plan is meant for other people, it’s safe to assume that you can provide some documentation about the feature elsewhere. 

2. I don’t add screenshots to the instructions.

While having screenshots in documentation is helpful, when they are in a test plan it just makes the plan larger and harder to read.

3. I don’t use a complicated test tracking system.

In my experience, test tracking systems require more time to maintain than the time needed to actually run the tests.  If there are regression tests that need to be run periodically, they should be automated.  Anything that can’t be automated can be put in a one-page test plan for anyone to use when the need arises. 

What I Do:

1. I use a spreadsheet to organize my tests.

Spreadsheets are so wonderful, because the table cells are already built in.  They are easy to use, easy to edit, and easy to share.  For test plans that I will be running myself, I use an Excel spreadsheet.  For plans that I will be sharing with my team, I use a table in a Confluence page that we can all edit. 

You can see the test spreadsheet I created for our hypothetical Superball sorter here.  I’ll also be including some screenshots in this post. 

2. I keep the instructions simple. 

In the screenshot below, you can see the first two sections of my test:

In the second test case, I’ve called the test “Amy- Large balls only”.  This is enough for me to know that what I’m doing here is setting a rule for Amy that she should accept Large balls only.  I don’t need to write “Create a rule for Amy that says that she should accept Large balls only, and then run the ball distribution to pass out the balls”.  All of that is assumed from the description of the feature that I included in last week’s post. 

Similarly, I’ve created a grouping of four columns called “State at End of Test Pass”.  There is a column for each child, and in each cell I’ve included what the expected result should be for that particular test case.  For example, in the third test case, I’ve set Amy, Carol, and Doug to “any balls”, and Bob to “Small balls only”.  This means that Amy, Carol, and Doug can have any kind of balls at all at the end of the test pass, and Bob should have only Small balls.  I don’t need to write “Examine all of Bob’s balls and verify that they are all Small”.  “Small balls only” is enough to convey this.

3. I use headers to make the test readable. 

Because this test plan has forty-five test cases, I need to scroll through it to see all the tests.  Because of this, I make sure that every section has good headers so I don’t have to remember what the header values are at the top of the page. 

In the above example, you can see the end of one test section and the beginning of the final test section, “Size and Color Rules”.  I’ve put in the headers for Amy, Bob, Carol, and Doug, so that I don’t have to scroll back up to the top to see which column is which. 

4. I keep the chart cells small to keep the test readable. 

As you can see below, as the test cases become more complex, I’ve added abbreviations so that the test cases don’t take up too much space:

After running several tests, it’s pretty easy to remember that “A” equals “Amy” and “LR” equals “Large Red”. 

5. I use colors to indicate a passing or failing test.  The great thing about a simple test plan is that it’s easy to use it as a report for others.  I’ll be talking about bug and test reports in a future post, but for now it’s enough to know that a completed test plan like this will be easy for anyone to read.  Here’s an example of what the third section of the test might look like when it’s completed, if all of the tests pass: 

If a test fails, it’s marked in red.  If there are any extra details that need to be added, I don’t put them in the cell, making the chart hard to read; instead, I add notes on the side that others can read if they want more detail. 

6. I use tabs for different environments or scenarios. 

The great thing about spreadsheets such as Google Sheets or Excel is that they offer the use of tabs.  If you are testing a feature in your QA, Staging, and Prod environments, you can have a tab for each environment, and copy and paste the test plan in each.  Or you can use the tabs for different scenarios.  In the case of our Superball Sorter test plan, we might want to have a tab for testing with a test run of 20 Superballs, one for 100 Superballs, and one for 500 Superballs. 

Test plans should be easy to read, easy to follow, and easy to use when documenting results.  You don’t need fancy test tools to create them; just a simple spreadsheet, an organized mindset, and an ability to simplify instructions are all you need.

Looking over the forty-five test cases in this plan, you may be saying to yourself, “This would be fine to run once, but I wouldn’t want to have to run it with every release.”  That’s where automation comes in!  In next week’s post, I’ll talk about how I would automated this test plan so regression testing will take care of itself. 

How to Design a Test Plan

Being a software tester means much more than just running through Acceptance Criteria on a story.  We need to think critically about every new feature and come up with as many ways as we can to test it.  When there are many permutations possible in a feature, we need to balance being thorough with testing in a reasonable amount of time.  Automation can help us test many permutations quickly, but too many people jump to automation without really thinking about what should be tested.

In this week’s post, I’m going to describe a hypothetical feature and talk about how I would design a test plan for it.

The feature is called the Superball Sorter, and here is how it works:

  • Superballs can be sorted among four children- Amy, Bob, Carol, and Doug
  • The balls come in two sizes: large and small
  • The balls come in six colors: red, orange, yellow, green, blue, and purple
  • The children can be assigned one or more rules for sorting: for example, Amy could have a rule that says that she only accepts large balls, or Bob could have a rule that says he only accepts red or orange balls
  • Distribution of the balls begins with Amy and then proceeds through the other children in alphabetical order, and continues in the same manner as if one was dealing a deck of cards
  • Each time a new ball is sorted, distribution continues with the next child in the list
  • The rules used must result in all the balls being sortable; if they do not, an error will be returned
  • Your friendly developer has created a ball distribution engine that will create balls of various sizes and colors for you to use in testing

Here’s a quick example:  let’s say that Carol has a rule that she only accepts small balls.

The first ball presented for sorting is a large red ball.  Amy is first in the list, and she doesn’t have any rules, so the red ball will go to her.

The next ball presented is a small blue ball.  Bob is second on the list, and he doesn’t have any rules, so the blue ball will go to him.

The third ball is a large purple ball.  Carol is next on the list, BUT she has a rule that says that she only accepts small balls, so the ball will not go to her.  Instead the ball is presented to Doug, who doesn’t have any rules, so the large purple ball will go to him.

So what we have after a first pass is:
Amy: large red ball
Bob: small blue ball
Carol: no ball
Doug: large purple ball

Since Doug got the most recent ball, we’d continue the sorting by offering a ball to Amy.

How should we test this?  Before I share my plan, you may want to take a moment and see what sort of test plan you would design yourself.  Then you can compare your plan with mine.

My test plan design philosophy always begins with testing the simplest possible option, and then gradually adding more complex scenarios.  So, I will begin with:

Part One:  No children have any rules

  • If no children have any rules, we should see that the balls are always evenly distributed between Amy, Bob, Carol, and Doug in that order.  If we send in twenty balls, for example, we should see that each child winds up with five.  

Next,  I will move on to testing just one type of rule.  There are only two parameters for the size rule, but six parameters for the color rule, so I will start with the size rule:

Part Two: Size rules only

We could have anywhere from one child to four children with a rule.  We’ll start with one child, and work up to four children.  Also, one child could have two rules, although that would be a bit silly, since the two rules would be accepting large balls only and accepting small balls only, which would be exactly the same as having no rules.  So let’s write up some test cases:

A.  One child has a rule

  • Amy has a rule that she only accepts large balls.  At the end of the test pass, she should only have large balls.
  • Bob has a rule that he only accepts small balls.  At the end of the test pass, he should only have small balls.
B. Two children have rules
  • Amy and Bob both have rules that they only accept large balls.  At the end of the test pass, they should only have large balls.
  • Carol has a rule that she only accepts large balls, and Doug has a rule that he only accepts small balls.  At the end of the test pass, Carol should have only large balls, and Doug should have only small balls.
C. Three children have rules
  • Amy, Bob, and Carol have rules that they only accept small balls. At the end of the test pass, they should only have small balls.
  • Amy and Bob have rules that they only accept small balls, and Carol has a rule that she only accepts large balls.  At the end of the test pass, Amy and Bob should have only small balls, and Carol should have only large balls.
  • Amy has a rule that she accepts both large balls and small balls, and Bob and Carol have rules that they only accept large balls.  At the end of the test pass, Amy should have both large and small balls (assuming that both were distributed during the test pass), and Bob and Carol should have only large balls.
D. Four children have rules
  • Amy and Bob have rules that they only accept large balls, and Carol and Doug have rules that they only accept small balls.  
  • All four children have a rule that they only accept large balls- this rule should return an error
Now that we have extensively tested the size rule, it’s time to test the other rule in isolation:
Part Three: Color rules only
As with the size rule, anywhere from one to four children could have a color rule.  But this rule type is a bit more complex, because each child could have from one to six color rules.  Let’s start simple with one child and one rule:
A. One child has one rule
  • Bob accepts only red balls
  • Bob accepts only orange balls
  • Bob accepts only yellow balls
  • Bob accepts only green balls
  • Bob accepts only blue balls
  • Bob accepts only purple balls
This tests that each color rule will work correctly on its own.  
B. One child has more than one rule
  • Carol accepts only red and orange balls
  • Carol accepts only red, orange, and yellow balls
  • Carol accepts only red, orange, yellow, and green balls
  • Carol accepts only red, orange, yellow, green, and blue balls
  • Carol accepts only red, orange, yellow, green, blue and purple balls (which, again, is sort of silly, because it’s like having no rule at all)
C. Two children have color rules
  • Amy and Bob both accept only red balls
  • Amy accepts only red balls and Bob accepts only blue balls
  • Amy accepts only red and green balls and Bob accepts only blue and yellow balls
  • Carol accepts only red, orange, and yellow balls, and Doug accepts only green balls
Note that there are MANY more possibilities here than what we are actually testing.  We are merely trying out a few different scenarios, such as one where one child has three rules and one child has one.
D. Three children have color rules
  • Amy, Bob, and Carol accept only red balls
  • Amy accepts only red balls, Bob accepts only orange balls, and Carol accepts only yellow balls
  • Amy accepts only red balls, Bob accepts only red and orange balls, and Carol accepts only red, orange, and yellow balls
The last scenario above exercises a path where the children share one rule but not other rules.
E. Four children have color rules
  • Amy, Bob, Carol, and Doug only accept purple balls- this should return an error
  • Amy only accepts red and yellow balls, Bob only accepts orange balls, Carol only accepts yellow and blue balls, and Doug only accepts green balls- this should also return an error, because no one is accepting purple balls
  • Amy only accepts red balls, Bob only accepts red and orange balls, Carol only accepts yellow balls, and Doug only accepts yellow, green, blue, and purple balls
Now that we’ve exercised both rule types separately, it’s time to try testing them together!  Here’s where things get really complicated.  Let’s try to start simply with scenarios where each child has either a color rule or a size rule, but not both, and move on to more complex scenarios from there:
Part Four: Size and color rules

A. Children have one size rule or one color rule
  • Doug only accepts large balls, and Bob only accepts red balls
  • Doug only accepts large balls, Bob only accepts red balls, and Carol only accepts small balls
  • Doug only accepts large balls, Bob only accepts red balls, and Carol only accepts small balls, and Amy only accepts yellow balls
  • Amy and Doug only accept large balls, Bob only accepts small balls, and Carol only accepts purple balls
  • Amy and Doug only accept large balls, and Bob and Carol only accept purple balls- this should return an error, because there’s no one to accept any small balls other than the small purple balls
B. Children have both a size and a color rule
  • Amy only accepts large red balls
  • Amy only accepts large red balls, and Bob only accepts small blue balls
  • Amy only accepts large red balls, Bob only accepts small blue balls, and Carol only accepts large green balls
  • Amy only accepts large red balls, Bob only accepts small blue balls, Carol only accepts large green balls, and Doug only accepts small yellow balls- this should return an error
C. Children have a size rule and more than one color rule
  • Amy only gets large red, orange, and yellow balls, Bob only gets small red, orange, and yellow balls, Carol only gets large green, blue, and purple balls, and Doug only gets small green, blue and purple balls
  • Try the above scenario, but remove the large yellow ball from Amy’s list- this should return an error, because there’s no one to accept the large yellow balls

D. Children have more than one size rule and more than one color rule

  • Amy: large red, large blue, small yellow; Bob: large orange, large purple, small green; Carol: large yellow, small red, small blue; Doug: large green, small orange, small purple
  • Try the above scenario, but add a small purple ball to Amy’s rules
  • Try the first scenario, but change Doug’s small purple rule to a large purple rule- this should return an error, because now there’s no one to accept the small purple balls
And there you have it!  Forty-five rules that exercise a great many of the options and permutations that the Superball Sorter offers.  If you have read this far, you must enjoy testing as much as I do!  
How did my test plan compare to yours?  Did you think of things I didn’t?  Be sure to let me know in the comments section!
You may have noticed that while this test plan is complete, it’s not that easy to read, and it doesn’t provide any way to keep track of test results.  Next week I’ll talk about how to organize the test plan so you can run it quickly and record results easily.  

Localization Testing

If your app is used anywhere outside your country of origin, chances are it uses some kind of localization strategy.  Many people assume that localization simply means translation to another language, but this is not the case.  Here are some examples of localization that your application might use:

Language: different countries speak different languages.  But different regions can speak different languages as well.  An example of this would be Canada: in the province of Quebec, the primary language spoken is French, and in the other provinces, the primary language spoken is English.

Spelling: even when two areas speak the same language, the spelling of words can be different.  For example, “color” in the US, as opposed to “colour” in Canada and the UK.

Words and idioms: words can vary even in a common language.  In the UK, a truck is a lorry, and a car’s trunk is a boot.  In the US, to “table” a topic means to stop talking about it until a later meeting.  But in the UK and Canada, to “table” a topic means to start talking about it in the current meeting- the complete opposite of what it means in the US!

Currency: different countries will use different currencies.  But this doesn’t just mean using a different symbol in front of the currency, like $ or £.  The currencies can also be formatted differently.  In the US, fractions of a dollar are separated with a dot, and amounts over one thousand are separated with a comma.  In the UK, it’s the opposite.  So what would be written as 1,000.00 in the US would be written as 1.000,00 in the UK.

Date and Time Formats:  in the US, dates are written as month/day/year, but in the UK, dates are written as day/month/year.  The US generally writes times using AM and PM, but many other countries use 24-hour time, so what would be 1:00 PM in the US would be 13:00 elsewhere.

Units of Measure: the US usually uses US Customary units, such as pounds for weight and feet and inches for height.  Most other countries will use the metric system for these measurements.  Most countries measure air temperature in Celsius, while the US uses Fahrenheit.

Postal Codes and Phone Numbers: these vary widely from country to country.  See my posts on international phone numbers and postal codes for some examples.

Images: pictures in an application might need to be varied from country to country, but there are some considerations.  For example, if your application was to be used internationally, you might not want to include a picture of a building with an American flag in the front.  Or if your app were to be used in religiously conservative countries, you might not want a picture of a person in a sleeveless shirt.

Testing for Localization

The first step in localization testing is to determine exactly what will be localized.  Your company may decide to localize for date and time, postal codes, and phone numbers, but not for language.  Or a mobile app may choose to only use other languages that are built into the device, so that the text of the app would be in one language, but the buttons for the app would be in the user’s language.

If your app will be using other languages, gather all the texts you will need to be checking.  For example, if your app has menu items such as “Home”, “Search”, “Your Account”, and “About Us”, and your app will be localized for French and Spanish, find out what those menu items should be in French and Spanish.  It goes without saying that whoever has done the translations should have consulted with a native speaker to make sure that the translations are correct.

Next, create a test plan.  The simplest way to do this would be to create a spreadsheet, where the left column lists the different localization types you need to test and the top row lists the different countries.  Here is a very basic example:

Once your matrix is created, it should be very simple to run through your tests.  If you are testing on mobile, here’s a helpful hint: When you switch your mobile device to a different language, make sure you know exactly how to switch it back if you don’t recognize the words in the language you are switching to.  When I was testing localization for Mandarin, this was especially important; since I didn’t know any of the characters, I had no idea what any of the menu items said.  I memorized the order of the menu items so I knew which item I needed to click on to get back to English.

Another important thing to watch for as you are testing is that translated items fit well in the app.  For example, your Save button might look perfectly fine in English, but in German it could look like this:

Once you have completed your localization testing, you’ll want to automate it.  This could be done with UI automation tools such as Selenium.  You could have a separate test suite for each language, where the setup step would be to set the desired country on the browser or device, and each test would validate one aspect of localization, such as verifying that button texts are in the correct language, or validating that you can enter a postal code in the format of that country.  It would be very helpful to use a tool like Applitools to validate that buttons are displaying correctly or that the correct flag icon is displaying for the location.

Localization is a tricky subject, and just like software, it’s hard to make it perfect.  But if you and your development team clarify exactly what you want to localize, and if you are methodical in your testing, you’ll ensure that a majority of your users will be satisfied with your application.

Usability and Accessibility Testing

Two often-overlooked types of application testing are Usability Testing and Accessibility Testing.  Usability testing refers to the user experience: whether the app is intuitive and easy to use.  Accessibility testing refers to how easy it is for users with limited ability to use the application.  I’ll be discussing both in today’s post.  

Usability Testing

Usability testing is often called User Experience (UX) testing, and some larger companies have dedicated UX designers whose goal is to make their company’s application pleasing to customers.  Even if you have UX people at your company, it’s still a good idea to test your application with the user experience in mind.  Here are four ways to do that:
  • Learn what the expected “User Journeys” are.  Usually when we are testing an application, we use it in ways that users won’t, focusing on one feature or page at a time.  A good strategy is to find out how real users will be using the application and run through those scenarios.  For example, if you had an application that allowed users to order tickets for a movie theater, a user journey might be to log in, browse through the movies, look at the show times for one movie, select a show time, click the button to place the order, add the credit card information, and complete the sale.  By running through scenarios like this, you’ll discover which areas might not be offering the best user experience.
  • As you run through your user journeys, look for tasks that require many clicks or steps.  Could those tasks be done with fewer clicks?  This week, my husband was searching online for a new car.  He went to a website for a car manufacturer and was browsing through the different models.  Every time he changed to a new page, he was prompted to enter his Zip Code again.  That’s not a great user experience!
  • Test a new feature before you know what it’s supposed to do.  This is a strategy that doesn’t get used much any more, now that Test-Driven Development (TDD) is popular.  Even if your company isn’t using TDD, you are probably present for the feature planning.  But I have found that it is sometimes helpful to take a look at a feature while knowing very little about it.  That is what your customers will be doing, so anything you find frustrating or complicated will probably also be frustrating or complicated for your customers.  An alternative to testing without knowing anything about the feature is to get someone who has never used the application to try it out.  Spouses, roommates, friends, and people from non-technical teams in your company are good candidates for this testing.  By watching them navigate through your site, you will find the tasks that might not be intuitive.  
  • When you are testing, see if you can do everything you want to do in the application with the keyboard alone, or with the mouse alone.  People who use applications extensively want to be able to use them as quickly as possible.  A good example of this would be a customer service representative who has to fill out a form for every person who calls them.  If they are typing into each field, and they have to keep moving their hand over to the mouse to click the Submit button, this is a waste of their time.  If they can instead submit the form with the Enter key, their hands don’t need to leave the keyboard.  

Accessibility Testing
Accessibility testing is important because fifteen percent of the population has some kind of disability, and we want our applications to be used by as many people as possible.  The three main types of accessibility testing you will want to do are Visual, Dexterity, and Auditory.  Here are some strategies for each. 
Visual Testing:
  • Is the text large enough for most users?  Could users zoom and enlarge the text if needed?
  • Do the images have text descriptions, so that visually impaired users who use text-to-speech features will know what the image is?
  • Are the colors in the application distinctive enough that color-blind users won’t be confused by them?  This helpful website allows you to load in a screenshot of your application and see what it will look like to a color-blind person.  I put a screenshot from an earlier blog post into the site, and here’s what the buttons will look like to a red-green color-blind individual:
Dexterity Testing:
  • Does your app require any complicated click-and-drag or highlight-and-click scenarios?  Imagine how difficult those would be to a person who had only one arm, or who had limited use of their hand.  Can you change the app so that those tasks can be accomplished in a simpler way?
  • Are your buttons and links easy to click on?  If the buttons are too small, it may be difficult for someone with limited dexterity to click in the correct place.
Auditory Testing:
  • Are there any places in your application that have videos?  If so, do those videos have captions so the hearing impaired will know what is being said?
  • Are there places in your application that rely solely on sound effects in order for the user to know what is happening?  Try running the application with the sound turned off.  Do you miss any information while you are running through your test scenarios?
As software testers, we want our users to have as pleasant an experience as possible when using our application.  Usability and accessibility testing will help us ensure that our users will be able to accomplish what they want with our apps efficiently and easily.  

How to Design a Load Test

Last week, we talked about Performance Testing and various ways to measure the reliability and speed of your application.  This week we’ll be talking about Load Testing.  Load testing is simply measuring how your application will perform under times of great demand.  This could mean testing scenarios of reasonable load, or it could mean testing with scenarios of extreme stress to find the limits of the application.

It’s easy to find a load testing tool, create a few tests, and run them with a few hundred users to create metrics.  But this isn’t particularly helpful unless you know why you are testing and how your results will help you.

So before you begin any load testing, it’s important to ask the following questions:

  • What kind of scenarios are you testing for?
  • What is the expected behavior in those scenarios?

Let’s imagine that you have a new website that sells boxes of chocolates.  You have noticed that your site is most active on Saturday mornings.  Also, Valentine’s Day is coming, and you anticipate that you will have many more orders in the week leading up to that day.  In March, your company will be reviewed by a popular cable TV show, and you are hopeful this will lead to thousands of visits to your site.

With this in mind, you come up with the following expected behaviors:

  • You would like your web pages to load in less than two seconds under a typical Saturday morning load
  • You would like to be able to process five hundred orders an hour, which will get you through the busy Valentine’s Day season
  • You would like your site to be able to handle ten thousand visitors in an hour, which is how many people you expect to visit right after the TV show airs
The next step is to figure out what test environment you will use.  Testing in Production would obviously provide the most realistic environment, but load testing there might be a bad idea if your testing results in crashing your site!  The next most realistic option would be a test environment that accurately mimics your Prod environment in terms of the number of servers used and the size of the back-end database.  An ideal situation would be one in which this test environment was only used for your load testing, but this is not always an option; you may need to share this environment with other testers, in which case you’ll need to be aware of what kinds of tests they are running and how they will impact you.  You’ll also want to let other testers know when you are conducting your load tests, so they won’t be surprised if response times increase.  
Once your test environment is ready, you can conduct some simple baseline tests.  You can use some of the strategies mentioned in last week’s post to find out how long it takes for your web pages to load, and what the typical response times are for your API endpoints.  Knowing these values will help you gauge how large an impact a load scenario will have on your application.
Now it’s time to design your tests.  There are a couple of different strategies to use in this process:
  • You can test individual components, such as loading a webpage or making a single request to an API
  • You can test whole user journeys, such as browsing, adding an item to a cart, and making a purchase
You’ll probably want to use both of these strategies, but not at the same time.  For instance, you could measure how long it takes to load the home page of your website when you have ten thousand requests for the page in an hour.  In a separate test, you could create a scenario where hundreds of users are browsing, adding items to their cart, and making a purchase, and you could monitor the results for any errors.  
For each test you design, you’ll want to determine the following:
  • How many users will be interacting with the application at one time?
  • Will those users be added all at once, or every few seconds?
  • Will the users execute just one action and then stop, or will they execute the action continuously for the duration of the test?
  • How long will the test last?
Let’s design a test for the Valentine’s Day scenario.  We’ll assume that you have created test steps that will load the webpage, browse through three product pages, add one product to the cart, and make a purchase.  We already mentioned above that you’ll want to be able to handle five hundred orders an hour, so we’ll set up the test to do just that.  It’s unlikely that in a real-world scenario all five hundred users would start the ordering process at the same time, so we’ll set the test to add a new user every five seconds.  Each user will run through their scenario once and then stop.  The test will run for one hour, or until all five hundred users have completed the scenario.

Before you run your test, be sure that your test steps will return errors if they don’t result in the expected response.  When I first got started with load testing, I ran several tests with hundreds of requests before I discovered that all of my requests were returning an empty set.  Because the requests were returning a 200 response, I didn’t notice that there was anything wrong!  Make sure that your steps have assertions that will validate that your application is really behaving as you want it to.

Once you have the test steps in place, you’ve made sure that the steps have good assertions, and you have your test parameters set up with the number of users, the ramp-up time (how frequently a new user will be added to the test), and the test duration, it’s time to run the test!  While the test is running, watch the response times and the CPU usage of your application.  If you start seeing errors or high CPU spikes, you can stop the test and make note of how high the load was when the spikes occurred.

Whether you need to stop the test early or whether the test completed successfully, you’ll want to run a few test passes to make sure that your data is fairly consistent.  At the end of your testing, you’ll be able to answer the question: can my website handle 500 orders in an hour?  If all of the purchases were completed with no errors, and if all of the response times were reasonable, then the answer is yes.  If you started seeing errors, or if the response times increased to several seconds, then the answer is no. If the answer is no, you can take the data you collected and share it with your developers, showing them exactly how many users it took to slow the system down.

Load testing should never be an activity you do to simply check it off of a list of test types.  When you take the time to consider what behaviors you want to measure, how you want your application to behave, what scenarios you can run to test those behaviors, and how you can analyze your results, you will ensure that load testing is a productive and informative activity.  

  

Introduction to Performance Testing

Performance Testing, like many other phrases associated with software testing, can mean different things to different people.  Some use the term to include all types of tests that measure an application’s behavior, including load and stress testing.  Others use the term to mean the general responsiveness of an application under ordinary conditions.  I will be using the latter definition in today’s post.

Performance Testing measures how an application behaves when it is used.
This includes reliability:

  • Does the page load when a user navigates to it?  
  • Does the user get a response when they make a request?

and speed:

  • How fast does the page load?  
  • How fast does the user get a response to their request?
Depending on the size of the company you work for, you may have performance engineers or DevOps professionals who are already monitoring for metrics like these.  But if you work for a small company, or if you simply like to be thorough in your testing, it’s worth learning how to capture some of this data to find out how well your application is behaving in the real world.  I know I have stopped using an application simply because the response time was too slow.  You don’t want your end users to do that with your application!

Here are five different ways that you can monitor the health of your application:
Latency– this is the time that it takes for a request to reach a server and return a response.  The simplest way to test this is with a ping test.  You can run a ping test from the command line on your laptop, simply by entering the word ping, followed by a website’s URL or an IP address.  For example, you could run this command
ping www.google.com

And get a response something like this:

(To stop a ping test, simply use CTRL-C)
Let’s take a look at the response times.  Each ping result shows how long it took in milliseconds to reach that server and return a response.  At the bottom of the test results, we can see the minimum response time, the average response time, the maximum response time, and the standard deviation in the response time.  In this particular example, the slowest response time was 23.557 milliseconds.
API Response Time- this is a really helpful measurement, because so many web and mobile applications are using APIs to request and post data.  Postman (my favorite tool for API testing) has response time measurements built right into the application.  When you run a request, you will see a Time entry right next to the Status of the response:
Here we can see that the GET request I ran took 130 milliseconds to return a response.  You can include an assertion in your Postman tests which will verify that your response was returned in less than a selected time, such as 200 milliseconds.  The assertion will look like this:

pm.test(“Response time is less than 200ms”, function () {
    pm.expect(pm.response.responseTime).to.be.below(200);
});

(If you would like to learn more about API testing with Postman, I have several blog posts on this topic.)

Another great tool for testing API response time is Runscope.  It is easy to use, and while it does not offer a free version, it has a very helpful feature: you can make API requests from locations all over the world, and verify that your response times are good no matter where your users are.  You can easily set up automated checks to run every hour or every day to make sure that your API is up and running.  Runscope also offers real-time monitoring of your APIs, so if your application suddenly starts returning 500 errors for some users, you will be alerted.
Web Response Time- Even if your API is responding beautifully, you’ll also want to make sure that your web page is loading well.  There’s nothing more frustrating to a user than sitting around waiting for a page to load!  There are a number of free tools that you can use to measure how long it takes your application’s pages to render.  With Pingdom, you can enter your website’s URL and it will crawl through your application, measuring load times.  Here are the results I got when I used my website’s URL and requested that it be tested from Melbourne, Australia:
Pingdom also provided suggestions for improving my site’s performance, such as adding browser caching and minimizing redirects.  Paid customers can also set up monitoring and alerting, so you can be notified if your page loading times slow down.  

Mobile Application Monitoring- If you have a native mobile application, you’ll want to make sure that it’s responding correctly and quickly.  Crashlytics is free software that can be added to your app to provide statistics about why your app crashed.  New Relic offers paid mobile monitoring for your app, allowing you to see data about which mobile devices are working well with your app and which might be having problems.

Application Performance Monitoring (APM) Tools- For more advanced monitoring of your application, you can use an APM tool such as ElasticSearch or Sharepath.  These tools track every single transaction your application processes and can provide insights on CPU usage, server response times, and request errors.  

Whether you work for a big company with a web application that has millions of users, or a small startup with one little mobile app, performance testing is important.  It may mean the difference between happy customers who keep using your application, and disaffected users who uninstall it.

Mobile Testing Part IV: An Introduction to Mobile Security Testing

Mobile security testing can be problematic for a software tester, because it combines the challenges of mobile with the challenges of security testing.  Not knowing much about mobile security testing, I did some research this week, and here are some of the difficulties I discovered:

  • Mobile devices are designed to be more secure than traditional web applications, because they are personal to the user.  Because of this, it’s harder to look “under the hood” to see how an application works.  
  • Because of the above difficulty, mobile security testing often requires tools that the average tester might not have handy, such as Xcode Tools or Android Studio.  Security testing on a physical device usually means using a rooted or jailbroken phone.  (A rooted or jailbroken phone is one that is altered to have admin access or user restrictions removed.  An Android phone can be rooted; an iPhone can be jailbroken. You will NOT want to do this with your personal device.)
  • It’s difficult to find instructions for mobile security testing when you are a beginner; most documentation assumes that you are already comfortable with advanced security testing concepts or developing mobile applications.
I’m hoping that this post will serve as a gentle introduction for testers who are not already security testing experts or mobile app developers.  Let’s first take a look at the differences between web application security testing and mobile app security testing: 
  • Native apps are usually built with the mobile OS’s development kit, which has built-in features for things like input validation, so SQL injection and cross-site scripting vulnerabilities are less likely.
  • Native apps often make use of the data storage capabilities on the device, whereas a web application will store everything on the application’s server.
  • Native apps will be more likely than web applications to use biometric data, such as a fingerprint, for authentication.

However, there are still a number of vulnerabilities that you can look for in a mobile app that are similar to the types of security tests you would run on a web application.  Here are some examples:

  • For apps that require a username and password to log in, you can check to make sure that a login failure doesn’t give away information.  For example, you don’t want your app to return the message “invalid password”, because that lets an intruder know that they have a correct username.
  • You can use a tool such as Postman to test the API calls that the mobile app will be using and verify that your request headers are expected to use https rather than http.
  • You can test for validation errors. For example, if a text field in the UI accepts a string that is longer than what the database will accept, this could be exploited by a malicious user for a buffer overflow attack.
If you are ready for a bigger testing challenge, here are a couple of mobile security testing activities you could try:
  • You can access your app’s local data storage and verify that it is encrypted.  With Android, you can do this with a rooted phone or an Android emulator and the Android’s ADB (Android Debug Bridge) command line tool.  With iPhone, you can do this with Xcode Tools and a jailbroken phone or an iPhone simulator.  
  • You can use a security testing tool such as Burp Suite to intercept and examine requests made by the mobile app.  On Android, unless you have an older device running the Lollipop OS or earlier, you’ll need to do this with an emulator.  On iPhone, you can do this with a physical device or a simulator.  In both instances, you’ll need to install a CA certificate on the device that allows requests to be intercepted.  This CA certificate can be generated from Burp Suite itself.  
These two testing tasks can prepare you to be a mobile security testing champion!  If you are ready to learn even more, I recommend that you check out the online book OWASP Mobile Security Testing Guide.  This is the definitive guide to making sure that your application is free of the most common security vulnerabilities.  Happy hacking!

Mobile Testing Part III: Seven Automated Mobile Testing Tips (and Five Great Tools)

Walk into any mobile carrier store and you will see a wide range of mobile devices for sale.  Of course you want to make sure that your application works well on all of those devices, in addition to the older devices that some users have.  But running even the simplest of manual tests on a phone or tablet takes time.  Multiply that time by the number of devices you want to support, and you’ve got a huge testing burden!

This is where automated mobile testing comes in.  We are fortunate to be testing at a time where there are a whole range of products and services to help us automate our mobile tests.  Later in this article, I will discuss five of them.  But first, let’s take a look at seven tips to help you be successful with mobile automated testing.

Tip 1: Don’t test things on mobile that could be more easily tested elsewhere
Mobile automation is not the place to test your back-end services.  It’s also not the place to test the general logic of the application, unless your application is mobile only.  Mobile testing should be used for verifying that elements appear correctly on the device and function correctly when used.  For example, let’s say you have a sign-up form in your application.  In your mobile testing, you’ll want to make sure that the form renders correctly, that all fields can be filled in, that error messages display appropriately, and that the Save button submits the form when clicked on.  But you don’t want to test that the error message has the correct text, or that the fields have all saved correctly.  You can save those tests for standard Web browser or API automation.
Tip 2: Decide whether you want to run your tests on real devices or emulators
The advantage of running your tests on real devices is that the devices will behave like the devices your users own, with the possibility of having a low battery, connectivity issues, or other applications running.  But because of this, it’s more likely that your tests will fail because a phone in the device farm froze up or was being used by another tester.  Annoyances like these can be avoided by using emulators, but emulators can’t completely mimic the real user experience.  It’s up to you decide which choice is more appropriate for your application.  
Tip 3: Test only one thing at a time
Mobile tests can be flaky, due to the issues found in real devices discussed above and other issues such as the variations found in different phones and tablets.  You may find yourself spending a fair amount of time taking a look at your failed tests and diagnosing why they failed.  Because of this, it’s a good strategy to keep your tests small.  For example, if you were testing a login screen, you could have one test for a successful login and a second test for an unsuccessful login, instead of putting both scenarios into the same test.
Tip 4: Be prepared to re-run tests
As mentioned in Tip 3, you will probably encounter some flakiness in your mobile tests.  A test can fail simply because the service that is hosting the emulator loses connectivity for a moment.  Because of this, you may want to set up a system where your tests run once and then re-run the failed tests automatically.  You can then set up an alert that will notify you only if a test has failed twice.
Tip 5: Don’t feel like you have to test every device in existence

As testers, we love to be thorough.  We love to come up with every possible permutation in testing and run through them all.  But in the mobile space, this can quickly drive you crazy!  The more devices you are running your automated tests on, the more failures you will have.  The more failures you have, the more time you have to spend diagnosing those issues.  This is time taken away from new feature testing or exploratory testing.  Do some research on which devices your users own and come up with a list of devices to test with that covers most, but not all, of those devices.  
Tip 6: Take screenshots
Nothing is more frustrating than seeing that a test failed and not being able to figure out why.  Screenshots can help you determine if you were on the correct screen during a test step and if all the elements are visible.  Some mobile testing companies take a screenshot of every test step as the test progresses.  Others automatically take a screenshot of the last view before a test fails.  You can also code your test to take screenshots of specific test steps.  
Tip 7: Use visual validation

Visual validation is essential in mobile testing.  Many of the bugs you will encounter will be elements not rendering correctly on the screen.  You can test for the presence of an element, but unless you have a way to compare a screenshot with one you have on file, you won’t really be verifying that your elements are visible to the user.  Applitools makes an excellent product for visual comparison.  It integrates with common test software such as Selenium, Appium, and Protractor.  With Applitools, you can build visual verification right into your tests and save a collection of screenshots from every device you test with to use for image comparison. 

Now let’s discuss some good tools for test automation.  I’ve already mentioned Applitools; below are four other tools that are great for mobile test automation.  The mobile landscape is filled with products for automated testing, both open-source and paid.  In this post, I am discussing only the products that I have used; there are many other great products out there. 

Visual Studio App Center:  A Microsoft product that allows you to test Android and iOS applications on real devices.  A screenshot is taken of every test step, which makes it easy to figure out where a test started to go wrong. 

Appium:  An open-source product that integrates with Selenium and provides the capability to test on device emulators (or real devices if you integrate with a device farm). 

Sauce Labs:  Sauce Labs is great for testing on both mobile devices and web browsers on all kinds of operating systems.  You can run tests on real devices or emulators, and you can run tests in parallel.  They integrate well with Selenium and Appium.  A screenshot is taken whenever a test fails, and you can even watch a video of your test execution.

Perfecto: Uses real devices and integrates with Visual Studio, Appium, and Selenium.  They can simulate real-world user conditions such as network availability and location.

Whatever automated test tools you choose to use, remember the tips above, and you will ensure that you are comprehensively testing your application on mobile without spending a lot of time debugging. 

I initially said this series on Mobile Testing was going to be three blog posts long.  On reflection, I’ve realized that we need a fourth post: Mobile Security Testing.  This is a topic I know very little about.  So I’ll be doing some research, and you can expect Mobile Testing Part IV from me next week!