Automated Form Testing

Now that we have looked at all the different ways we can manually test forms, it’s time to think about automating those tests so you don’t have to run through all the tests again every time you need to do a regression test!  But before you jump into automation, think for a while about what tests you really want to run.  Let’s take last week’s form as our example again.

What sorts of things would we want to make sure still work when we do our regression test?

  • We’ll want to make sure that every field can be populated with data and saved
  • We’ll want to make sure that all of the required fields are still required, and that we get an appropriate error message when we leave a required field blank
  • We’ll want to make sure that validation rules are respected in each field
  • We’ll want to verify that both the Save and the Cancel buttons work correctly
As I’ve discussed in a previous post, automated tests involving the GUI can often be slow and flaky.  So we’ll want to limit the number of times we spin up a browser for testing.  But we’d also like our tests to only assert on one thing at a time.  Here are my suggestions for a suite of tests that could be run on this form.  I’ve written them in Cucumber for ease of reading:
Scenario: Happy Path
Given I am adding a new user
When I fill out all the fields and click the Save button
And I navigate to the Users page
Then I verify that all of the fields are displayed on the page
Scenario: Required Fields
Given I am adding a new user
When I leave all fields empty
And I click the Save button
Then I verify that an error message is present for each required field
Scenario: Validation Rules
Given I am adding a new user
When I give each field an invalid value
And I click the Save button
Then I verify that an error message is present for each invalid field
Scenario: Cancel Button
Given I am adding a new user
When I give each field a value
And I click the Cancel button
Then I verify that all fields are now empty
So, we can cover a lot of territory with just four automated tests!  
Our Happy Path tests that values in all of the fields will save correctly, and it also tests the Save button.  If one of the values does not save, we will know we have an error.  If all of the values do not save, we will know there is a problem either with the Save button or with the underlying data store. 
Our Required Fields test checks to make sure that every field that is required displays an appropriate error message if it is not filled out.  If even one required field does not display the message, we will know there is a problem.
Our Validation Rules test violates one rule for each text field.  It would be a good idea to mix and match the different types of rule violations.  For example:
First Name- send a value with numbers and non-letter characters, such as $23.00
Last Name- send a value with not enough letters, such as A
Personal Phone- send a value with the wrong number of digits, such as 008675309
Work Phone- send a value with non-number characters, such as 800*555-!@#$
Street Address 1- send a value with way too many characters, such as all the lyrics to “Frosty the Snowman” (my favorite song to test with)
Street Address 2- send a value with one too many characters
City- try sending a script, such as <script>alert(“XSS Here!”)</script>
State- send an invalid state, such as XX
Zip Code- send a Zip Code with the wrong number of digits, such as 0377
While this does not test every single way that the rules could be violated in every text field, it covers a wide variety of possibilities, and verifies that for each error, you receive an appropriate error message.  You could always add more tests here if you felt that more coverage was needed.
Finally, we verify that the Cancel button works correctly.  
With just these four automated tests, we’re able to make sure no functionality is broken.  Running these regression tests will free you up to do more exploratory testing in your application, and to test new and more interesting features!

Testing Forms

Today is the day that we’ll be putting it all together!  In the past few posts, we’ve been looking at different types of text fields and buttons; now we’ll discuss testing a form as a whole.

There are as many different ways to test forms as there are text field types!  And unfortunately, testing forms is not particularly exciting.  Because of this, it’s helpful to have a systematic way to test forms that will get you through it quickly, while making sure to test all the critical functionality.

Let’s take a look at the form below, and I will walk you through a systematic approach to testing it.  For simplicity, we’ll assume that the users will all be US-based.  (See my previous posts for discussions on testing international postal codes and phone numbers.)

Step One:  Required Fields

The first thing I do when I test a form is to make note of which fields are required.  This particular form denotes required fields with a red asterisk.  We can see that First Name, Last Name, Personal Phone, Street Address 1, City, State, and Zip Code are all required.  I want to verify that the form cannot be submitted when a required field is missing and that I’m notified of which field is missing when I attempt to submit the form.
1. I click the Save button when no fields have been filled out. I make sure that all of the required fields have error messages.
2. I fill out the two non-required fields, click the Save button, and verify that all the required fields have error messages.
3. I fill out one required field and click the Save button, and I verify that all the other fields have error messages.  I start with submitting just the First Name, then just the Last Name, then just the Personal Phone, etc., until I have cycled through all of the required fields.
4. I try various combinations of two, three, four, and five required fields to make sure the appropriate error messages are displayed.  I don’t concern myself with testing every possible combination, since it’s highly likely that the required field will behave similarly in these situations.
5. I try filling out all the required fields but one, and verify that I still receive an error for that field.  I cycle through all the possibilities of having one missing required field.
6. I fill out all of the fields- required and non-required- except for one required field, and verify that I still receive an error.
7. I fill out all of required fields and click the Save button, and I verify that no error messages appear and that the entered fields have been saved correctly to the database.
8. Finally, I fill out all of the fields- required and non-required, and I verify that no error messages appear and that the entered fields have been saved correctly to the database.

Step Two: Field Validation

The next thing I do is verify that each individual text box has appropriate validation on it.  First, for each text field, I discover what the upper and lower character limits are.  I enter in all the required fields except for the one text box I am testing.  Then in that text box:
1. I try entering just one character
2. I try entering the lower limit of characters, minus one character
3. I try entering the upper limit of characters, plus one character
4. I try entering a number of characters far beyond the upper limit
In all of these instances, the form should not save, and I should receive an appropriate error message.
Next:
5. I enter the lower limit of characters, and verify that the form is saved correctly
6. I enter the upper limit of characters, and verify that the form is saved correctly

Now that I have confirmed that the limits on characters are respected, it’s time to try various letters, numbers and symbols.  For each text field, I find out what kinds of letters, numbers, or symbols are allowed.  For example, the First and Last Name fields should allow apostrophes (ex. O’Connor) and hyphens (ex. Smith-Clark), but should probably allow no numbers or symbols.  For each text field:
1. I try entering all of the allowed letters, numbers and symbols
2. I try entering the letters, numbers, or symbols that are not allowed, one at a time, until I have verified that they are all not allowed

For fields that have very specific accepted formats, I test those formats now.  For instance, I should be able to enter a Zip Code of 03773-2817, but not one of 03773-28.  For the State field, the two-letter code I put into the field should be a valid state, so I should be able to enter MA, but not XX.

Even though I’ve already tested all the forbidden numbers and symbols, I try a few cross-site scripting and SQL injection examples, to make sure no malicious code gets through.  (More on this in a future post.)

Step Three: Buttons

Even though I have already used the Save button dozens of times at this point, there are still a few things left to test here.  And I have not yet used the Cancel button.  So:
1. I click the Save button several times at once, and verify that only one instance of the data is saved
2. I click the Save button and then the Cancel button very quickly, and verify that the data is saved and that there are no errors
3. I click the Cancel button when no data has been entered, and verify that there are no errors
4. I click the Cancel button when data has been entered, and verify that the data is cleared.  I’ll try this several times, with various combinations of required and non-required fields.
5. I click the Cancel button several times quickly, and verify that there are no errors

Step Four: Data

Finally, it’s time to look at the data that is saved.  In earlier steps, I’ve already done some quick checks to determine if the data that I am entering is saved to the database.  Now I’ll check:
1. That bad data I’ve entered is not saved to the database
2. That good data I’ve entered is saved to the database correctly
3. That good data I’ve entered is retrieved from the database correctly, and displayed correctly on the form

If the form I am testing has Edit and Delete functionality, it’s important to test these as well, but I’ve covered these in my post CRUD Testing Part II- Update and Delete.

Once I’ve gone through a form in this systematic way, it’s almost certain I will have found a few bugs to keep the developers busy.  It’s tedious, yes, but once the bugs have been fixed I can move on to automation testing with confidence, knowing that I have really put this form through its paces!

Testing Buttons

Buttons tend to be something that it’s easy to forget about.  The “Save” button is so ubiquitous that it seems like it would just work.  But overlooking testing buttons on a page can also mean overlooking bugs.  Recently someone told me about new functionality she was testing on an existing web page.  The new feature worked great, but her team forgot to test the “Delete” button.  It turned out that the developers had forgotten to account for the delete action in their new feature, and now “Delete” did nothing! 

Here are a few things to think about when testing buttons:

1. Test the “happy path” of the button.  Usually buttons have some sort of message on them that tell you what they are supposed to do.  So try it out, and make sure that the button delivers on its promise!  Did the “Save” button really save your data?  Did the “Delete” button delete it?  Did the “Clear” button clear it?  Did the “Search” button execute a search?  (Note: a “Back” button is also common, but back buttons are tricky enough that I will be saving them for a separate post.)

2. Misuse the button.  For example, quickly press the “Save” button twice when adding data.  Were two records saved instead of one?  Does a button get confused when you press it twice?  What about when you quickly press one button and then another?  One of the most amusing bugs I have found in my career was a “Refresh” button that redrew the screen, making the button larger whenever I pressed it.

3. Is the button there when it’s supposed to be?  When we have spent a lot of time testing an application, it’s easy to get used to the page and not notice when things are missing.  Think about your user stories when looking at the page.  What would your user need when doing certain activities on that page?  Follow the path that the user would take, and check for the buttons as you go.

4. Think about when the button is enabled and when it is disabled.  Does it make sense?  For example, is the “Save” button only enabled when all the required fields on your form have been filled out?  Is the “Clear” button only enabled when a field has been dirtied?  How do you know the button is enabled?  Can you tell by looking at it?  Does the button look enabled when it isn’t?  Does the button look disabled when it’s really active?  What happens when you press the button and it’s not enabled?  Do the enabled and disabled rules for the button make sense?

5. Finally, see if you can hack your buttons!  For example, if you have a “Save” button that is disabled on your form because some of the required fields are missing, can you edit the html on the page so that it is enabled, and can you then use the button?  If you view the html and you see that there is a hidden button on the page, can you make it visible and active?  At best, that bug is an annoyance that will get inaccurate data into the database.  At worst, the bug represents a way that a hacker can infiltrate your system!  Imagine that you have a button that should only be visible and enabled if a user is an administrator.  If a malicious user can make that button appear on the screen and be active, they will have access to pages or features that only an admin should have. Your developer should include checks whenever a button is clicked on to make sure that the user has the rights to do what the button does. 

If you have never edited the html on a web page while testing, here are a few simple instructions for the Chrome browser:
a. Click on the three-button menu on the top right of your screen, and choose “More Tools -> Developer Tools”.  A new panel will appear on the bottom or the right side of your screen.
b. Right-click on the button you want to test, and click “Inspect”.  In the developer tools panel, you will now see the html for that button highlighted.
c. Right-click on that highlighted text and choose “Edit as HTML”.  An editable text window will open up. 
d. If you see text such as “‘disabled’=disabled” delete the text.  Click away from the editable field, and see if your button is now enabled. If it is, click on it and see what happens!
e. To find hidden buttons, look at the html in the developer tools, and use the search bar to search for “button”.
f. If you find a button with the markup “ng-hide”, try changing it to “ng-show”.  See if you can get the button to appear on the page!

Buttons are one of the most important things to test.  Just imagine a user’s frustration if the button they are trying to use is disabled, or doesn’t do what they expect it to.  By being diligent in our testing, we can ensure that our users will want to work with our application. 

Testing Phone Fields- Part II- International Phone Numbers

If I were to be given three wishes for the world, my third wish (after world peace and ending world hunger) would be to standardize international phone numbers, because testing them is so complicated!  Let’s take a look at some phone number patterns from around the world:

Mexico:
Phone numbers are ten digits, plus an area code of either two digits or three digits.  So the local number will be either twelve or thirteen digits.

Italy:
Landline numbers are generally nine to eleven digits, but some can be as short as six.  Mobile numbers are usually ten digits, but there are also some nine-digit numbers.

Japan:
Telephone numbers in Japan have an area code, an exchange number, and a subscriber number.  Area codes can have between two and five digits.  Generally the length of the entire number is limited to nine digits, so if the area code is longer, the exchange and subscriber numbers will be shorter.

Now let’s think about international calling codes.  Calling codes can range from one digit (such as the US’s +1) to three digits (such as +351 for Portugal).  So between the international calling codes and the phone number itself, there is a wide range of the number of digits that might be expected for a phone number.

Finally, let’s think about number separation.  While US numbers are always separated in a 3-3-4 pattern, in other countries the numbers can be grouped in various ways.  Here are just some of the ways that phone numbers are grouped in England:

(xxx) xxxx xxxx
(xxxx) xxx xxxx
(xxxxx) xxxxxx
(xxxx xx) xxxxx

Therefore, it is extremely difficult to validate on whether the pattern of numbers, parentheses, and dashes entered by a user is correct.

So how on earth are we to validate international phone numbers?  The most important thing to do is to make sure that the developer doesn’t try to come up with a validation regex on their own- that will make them (and you) go crazy!  Fortunately an international phone number formatting standard called E.164 has been developed.  Many companies such as Microsoft and Google have come up with regex patterns that can be used for E.164 validation.  For testing, there are free websites that will let you know if an international phone number is valid.  Most of these sites will require some sort of registration.  You can come up with a list of international numbers to test, verify with the website to see if they are valid, and then try them in your application.  For negative testing, simply use a sampling of invalid numbers and verify that you get an appropriate error message.

What format should a user use when entering in their phone number?  Some users might want to put parentheses, spaces, or dashes into their number.  Then there is the plus sign, which is used to indicate a country code.  For simplicity’s sake, it would be best to expect nothing but digits and a plus sign.  It would also be good to prompt the user for what format the application is expecting.  For example, in the United States you might have a message such as: “Enter your phone number with no dashes, parentheses, or spaces.  For numbers outside the US, a plus sign (+) and the country code should be used.”  Then for validation, if the number does not have a plus sign, simple US number validation rules can be applied, and if the number does have a plus sign, international validation rules can be applied.

Similarly, you may want to store the number in the database with the plus sign, to indicate that the number is an international one.  How should the numbers be displayed when retrieved in the application?  For many countries it’s nearly impossible to know how the numbers should be spaced, unless the validation code you are using returns that to you.  If your country has a standard number formation, you can display the number in that standard.  If your country does not have a standard formation, or if you are displaying a number from outside your country, you can simply display the number as is, such as: +442073238299.

When it comes to international phone numbers, there really are no easy answers, which is why I am wishing for world standardization!  There are other ways to (mostly) solve the problem, such as 1) expecting only between 6 and 20 digits, not validating the number beyond making sure that it’s within that number of digits, and displaying the number as entered, or 2) having a separate field for the country code, validating the number for that specific country, and formatting the number according to that validation.  What is most important is that you communicate closely with the developer to decide on a clear strategy and test it accordingly.

Testing Phone Fields- Part I- US Phones

Phone number fields are without a doubt the most time-consuming thing I have ever tested.  This seems so unlikely- phone numbers are standardized to ten digits in the US, so how can they cause so many headaches?  The answer is that, as we saw with Zip codes, there are so many ways to get them wrong.

If you are fortunate enough to be testing a new application without any existing data, you can fend off most phone number headaches by clearly establishing validation rules for phone number fields.  The easiest type of validation is one that accepts ten digits only, and no other characters.  The developer can rely on UI formatting to put the number in an easily-readable format rather than expecting the user to do it themselves.  The reason why this is the best course of action is that allowing the user to add in formatting creates all sorts of ambiguity.  If my phone number is 800-867-5309, how might I type that in?  Just with dashes, or will I use parentheses, such as (800) 867-5309?  What about the older format of 1-800-867-5309?  With that format, I now have eleven digits.  
Many developers try to accommodate all styles in the mistaken notion that this will make things easier for the user, but this actually makes things much more complicated.  If both parentheses and dashes are allowed, there needs to be some sort of regex used for the validation.  Otherwise, a user could type in something like ((800)–867 5309 and the number will be accepted.  If the developer you are working with has decided that he or she would like to make it easy on the user by accepting parentheses and dashes, try entering values like 800–867–5309, 800(867)5309, and 8-0-0-867-5309.  After you have demonstrated that it’s possible to enter a badly formatted phone number, the developer’s strategy may change.

As with Zip codes, it’s important to consider how the phone number is saved to the database.  The best strategy for US phone numbers is to strip out any formatting and save the number as just the ten digits.  That way the number can be displayed in any way the UI designer deems appropriate.

Next, it’s time to think about how bad phone numbers will be displayed to the user.  If a number has been saved with formatting, will it be displayed with that formatting?  Will some attempt be made to strip the formatting?  Will it be displayed at all?  I have seen all kinds of bad numbers in databases.  One of my favorites was “dad’s office number”.  No one can format that!  A good strategy would be to format any ten-digit number by stripping out all of the non-number characters, and then adding in whatever format is desired for display.  If the phone number is eleven digits and starts with a 1, the leading 1 could be stripped off and then the number could be formatted for display.  With any other values, the “number” could be displayed as-is, or it could not be displayed at all.  Whatever is decided, it’s important to test with whatever bad numbers are currently in the database. 

Another wrinkle not often considered is phone extensions.  Many offices still use extensions to direct a call to a specific person.  Sometimes phone fields allow extensions as part of the number, but this is a bad idea, for the same reasons that allowing the user to format their own number is a bad idea.  Phone extensions can be any number of digits, so allowing them as part of the phone number means that now the validation can’t expect a specific number of digits.  Also, how might the user indicate that an extension is part of the number?  Will they enter 800-867-5309 ext. 1234?  800-867-5309×1234?  Allowing these types of variations will mean allowing letters and other characters, which it makes it more difficult to validate.  A far better solution is to include a separate field for an extension.  If your developer expresses interest in including the extension as part of the phone field, some testing with entries like 8008675309extextext…1234 will probably dissuade them. 

Finally, remember to test editing phone numbers, especially numbers that are in an incorrect format.  What will happen when the user has a phone number like 800-867-530 and tries to edit another field on the form?  Will they be notified upon saving that the number is incorrect and needs fixing?  This can be a good way to involve users in cleaning their own data.  What should happen when the user tries to fix an invalid number?  Ideally they should be able to save the value with a new valid number. 

I hope that this post has emphasized the importance of having good phone field validation.  If you are fortunate enough to be testing a brand-new application with an empty database, taking good care to ensure that only valid phone numbers are saved to the database, you will prevent many testing headaches in the future! 

This post only tackled United States phone numbers.  In my next post, I will take on the even greater challenge of international phone numbers! 

Testing Date Fields

Date fields are another data type that seems simple to test.  After all, dates are standard throughout the world: there’s a month, a day of the month, and a year.  But as you will see below, there are many factors to consider and many scenarios to test.

There are three main areas to think about when testing a date field:

  • What format will be accepted?
  • How will the date be stored in the database?
  • How will the saved date be displayed?  

There are many ways to format a date when entering it into a form.  One very important factor to consider is whether the system is expecting an American format: month, day, year; or a European format: day, month, year.  For example, if I try to enter next Wednesday’s date (12/13/2017) into some forms, it won’t be accepted if the European format was what the form was expecting, since there is no 13th month.  And if I enter July 4 into this form as 7/4/18, it may be saved to the database as April 7th.  
Beyond the American/European formatting, there are still a myriad of ways to format dates.  For example, will all four digits of the year be expected, or is it possible to put in just the last two digits?  Will single-digit months and days be allowed, or is it necessary to precede those digits with zeroes, such as in 07/04/2018?  What about spelling out the month, as in December 13, 2017?  The developer should have a clear idea of what format or formats will be allowed, and should clearly communicate that to the user.  For example, a tooltip such as “mm/dd/yyyy” can be used inside the field to help the user know what format to use.  Regardless of whether the tooltip is in place, it is important to test with a number of different date formats to ensure that an appropriate error message is displayed when the expected format is not followed.  Also be sure that the date itself is checked for whether it is a valid date; for example, 2/30/2016 should never be allowed, nor should 18/18/18.  
Next, let’s consider how the date will be stored in the database.  Some developers, not knowing better, might store the date as a string, rather than as a datetime value.  This is nearly always a bad idea for a number of reasons.  If the accepted format changes over time, or if good validation was not in place in the past, there could be strings where the date is saved as “7/4/2017” and strings where the date is saved as “December 13, 2017”.  This will make it difficult to consistently display the data.  Also, if the dates are stored as strings it will be difficult to sort correctly by ascending or descending date.  “2017-12-13” might wind up coming before “3/7/2017” in an ascending sort.  As testers, we should verify that the date is stored as a datetime value, and advocate for a change if it is not. 
It’s also important to look at how dates are displayed.  When a user calls up saved information, they should be able to read the dates easily. “2017-12-13T00:00:00” might be the way the date is saved in the database, but a user won’t be able to interpret this date quickly.  The developer or designer should decide what date format would be best for display purposes and should stick to it consistently throughout the application.  It’s also important to consider what should happen in the case of bad data.  What if there is a date saved in the database that says simply “December 13”?  Should it be displayed as 12/13/0000?  Should it not be displayed at all?  These are important scenarios to consider and test.  
There is one final consideration to make when testing date fields, and that is what the upper and lower limits of the date are.  For example, are dates in the future ever allowed?  What about dates from 100 years ago?  Remember that the future and the past change every day!  Let’s say that our application doesn’t allow dates in the future.  That means that today, 12/13/17 is not an allowed date in our application.  But by next Wednesday, 12/13/17 will be allowed.  And of course, you may be reading this blog post years in the future, at which time 12/13/17 will be a thing of the past!  

Testing Postal Codes

A text field with a postal code looks so simple, and yet it can be one of the most complex things to test on a form.  There are two important questions to ask before you start testing postal codes:

1. What countries does our application support?
2. What formats will we be accepting for the postal codes?
It always comes as a bit of a relief when I find out that the application I am testing will only be supporting US Zip codes.  Zip codes traditionally come in a five-digit format, such as 10012.  But there are also Zip +4 codes to consider, such as 10012-1234, and this is where the second question comes into play.  Will the application be accepting Zip +4 codes?  Will it be requiring the hyphen between the first five digits and the next four digits, or will a space be accepted as well?  What about just nine straight digits, with no hyphen or space in between?  If nine straight digits will be accepted, it’s important to also verify that 6, 7, and 8 digits will not be accepted.  
And there is another very important thing to test with US Zip codes: the leading zero.  There are many fields which will strip leading zeros off of a number upon submission.  It’s very important to submit some Zip codes with leading zeros and verify that they have been saved correctly to the database.  
The next likely postal code type you may encounter (if you are based in the US) is the Canadian postal code.  All Canadian postal codes are six characters in this pattern of letters and numbers:  A1A 1A1.  It’s important to clarify with the developer whether the space between the two groups of three characters will be expected, or whether you can submit the code with no space.  Hopefully the validation will expect the correct letter-number pattern, and will reject postal codes like 1A1 A1A.
What if your application expects both US and Canadian postal codes?  If this is the case, the third important question to ask is:
3. Will there be a separate validation pattern for each type?
If there is a separate validation pattern, the code may first look to see what country the address contains, and then use the appropriate validation.  In this case, it’s a good idea to test that you can’t choose “United States” for your country and then add “A1A 1A1” as the postal code.  Or the validation pattern may be chosen by the number of characters submitted.  If six or seven (including the space) characters are submitted, a Canadian validation pattern could be used.  If five, nine, or ten (including the hyphen) characters are submitted, a United States validation pattern could be used.  Understanding what validations the developer is using will allow you to craft appropriate test cases.  For example, in a scenario where only US and Canadian postal codes are used, there should never be a scenario where eight characters are accepted.  
When we move into postal codes for other countries, things can get more confusing.  Many countries have five-digit codes, which are easy to validate.  Other countries, such as Russia, have six-digit codes.  But consider Great Britain, which has postcodes in two sections: the first section can have between two and four characters, and the second section always has three characters. There is a space between the two sections, and the postcode will always start with a letter, not a number.   In this case, it’s best if the developer has ways to recognize if a British postcode is being used (perhaps by looking at what country has been entered in the form) and uses a separate validation regex for those codes.  For testing, be sure to try several different valid postcodes from various places in Great Britain, with two, three, and four characters in the first section.  You can also test with codes that have the right number of characters, but have the space in the wrong place, or with a code that has a number as the first character.
Finally, remember to check that the valid postal codes that you have submitted have been saved correctly to the database, and remember to verify that retrieved postal codes are displaying correctly.  Also be sure to check that any invalid codes in the database (that may have been added before the correct validation was in place) are displaying as added, or not at all, and that it’s possible to edit both invalid and valid postal codes.  
I hope that this post has helped you see that postal codes are not as simple as they seem!  Asking the three questions listed above can help you and your developer to recognize potential issues even before testing has begun.  

Automating CRUD Testing

There are a number of different patterns we can use to automate CRUD testing.  At the very minimum, we want to test one operation of each: Create, Read, Update, and Delete.  For the purposes of this discussion, let’s assume the following:

1. We are testing the simple form used in this post
2. We are doing UI automation (API automation will be discussed in a future post)

This is the pattern I like to use when testing CRUD.  I’m writing in Specflow/Cucumber for simplicity:

Scenario: Adding a user
Given I am adding a new user
When I add a first name and save
Then I navigate to the Users page
And I verify that the first name is present

Scenario: Updating a user
Given I am updating a user
When I change the first name and save
Then I navigate to the User page
And I verify that the first name has been updated

Scenario: Deleting a user
Given I am deleting a user
When I delete the first name and save
Then I navigate to the Users page
And I verify that the first name is not present

These three tests have tested Create, Update, and Delete. The first two tests are also testing Read, because we are retrieving the user for our assertions.  Therefore, with these three tests I’m testing the basic functionality of CRUD.

Some may argue that these tests are not idempotent.  Idempotency means that a test can be run again and again with the same results.  I can’t run the third test over and over again with the same results, for example, because after the first time I run it, the user is no longer there to delete.

If we wanted to solve the idempotency issue, we could write a test like this:

Scenario: CRUD testing of user
Given I am testing CRUD
When I add a first name and save
And I verify that the first name is present
When I change the first name and save
And I verify that the first name has been updated
When I delete the first name and save
Then I verify that the first name has been deleted

This one scenario tests all the CRUD functionality of the form.  But it also has three different assertions.  I prefer to keep my UI tests down to one or two assertions.

When my original three scenarios are looked at as a group, they are idempotent together.  My tests are responsible for creating and deleting my data.

It would also be a good idea to test some negative scenarios with our CRUD testing, such as creating a user with an invalid first name, and updating a user with an invalid first name.  These tests could look like this:

Scenario: Creating a user with an invalid first name
Given I am adding a new user
When I enter an invalid first name and save
Then I verify that I receive the appropriate error message on the page
And I navigate to the User page
And I verify that the user has not been added

Scenario: Updating a user with an invalid first name
Given I am updating an existing user
When I update the first name with an invalid value and save
Then I verify that I receive the appropriate error message on the page
And I navigate to the User page
And I verify that the existing first name has not been updated

The first scenario is idempotent, because nothing is actually being added to the database.  The second scenario is also idempotent, but it requires an existing user.  We could assume that our user will always be present in the database, but if someone inadvertently deletes it, our test will fail.  In this case, it would be good to add in a Before step and an After step that will create the user at the beginning of the test suite and delete it when the suite is over.

These five scenarios- the three to test the happy path, and the two negative tests- would be a great regression suite for our simple form.  This is a very simple form, with just one field, which is not exactly a real-world scenario.  But it is a good way to start thinking about automated UI test patterns.

CRUD Testing Part II- Update and Delete

In last week’s post, we started looking at CRUD testing.  As you recall, CRUD stands for Create, Read, Update, and Delete.  Last week we discussed testing Create and Update operations, and now we will continue by looking at Update and Delete.

In our discussion of the Read operation last week, I mentioned how important it is to test scenarios where the data in the database is invalid.  This is also true for Update operations.  Just because a text field is supposed to be required and have a certain number of characters doesn’t mean that’s how it is in the database!

Below is a matrix of testing scenarios for editing a text field.  Remember from last week that our hypothetical text field has the following validation rules:  it is a required field, it must have at least two characters, it must have 40 or fewer characters, and it should only have alphanumeric characters or hyphens and apostrophes; no other symbols are allowed.  As with the Create operation, be sure to test that the newly edited field is correct in the UI and in the database after the update.

For the bad values that you are testing with, be sure to vary them so you are covering a number of different validation scenarios.  For the good values that you are testing with, make sure to test hyphens and apostrophes, numbers and letters, and the upper and lower limits of the character count. 
Finally, let’s discuss Delete.  The main thing to test with Delete is that the value has been deleted from both the UI and the database.  But just as with Read and Update, you’ll want to make sure that you can delete bad values.  For example, if your First Name field has forty-one characters in it (violating the rule that only 40 characters or less should be added), you’ll want to make sure that you can delete it in the UI.  
You may be wondering how to find all these invalid values for testing.  While it is possible to find them by searching through the existing records, the easiest way to test them is to put them in yourself!  This is why it’s important to know the appropriate query language, such as SQL, for your database.  
For those who are wondering why testing with bad values is important, I once had a scenario where I was testing contact information for a person.  There were a number of incorrectly formatted phone numbers in the database.  When I tried to update any of the contact information for the person, I got the message that the record couldn’t be updated, because there was invalid data present.  This was true even if I was trying to update the invalid phone number in question!  Please be sure to check for scenarios like these; your end-user will thank you!

CRUD Testing Part I- Create and Read

In spite of its unappealing name, CRUD testing is extremely important!  CRUD stands for Create, Read, Update, and Delete.  As any tester knows, much of our testing involves these operations.  Today we’ll discuss the best ways to test Create and Read. 

The most important thing to know about testing CRUD is that it’s not enough just to rely on what you are seeing in your UI to confirm that a field’s value has been created or changed.  This is because the UI will sometimes cache a value for more efficient loading in the browser.  What you need to do to be absolutely sure that the value has changed is to check the database where your value is stored.  So you’ll be confirming that your value is set in two places: the UI and the database.  If you are doing API testing as well, you can actually confirm in three places, but we’ll save discussing API testing for another post.

Testing a Create Operation:

This text field looks similar to the one we looked at last week, but now we know what it does!  This is a form to add a user.  We’ll enter the first name of the user into the text field and click Submit.  Next, we’ll take a look at the Users page of our imaginary application and verify that the new user is present:

And there it is!  Finally, we need to query our database to make sure that the value has saved correctly there.  In our imaginary database, this can be done by running

SELECT * from Users

This will give us a result that looks a lot like what we are seeing in the UI, so I won’t include a screenshot here.

To thoroughly test the Create function, you can use some of the same ideas that we talked about in last week’s post.  Verify that valid entries of all kinds are saved to the database.

Testing a Read Operation:

We actually started testing the Read operation when we checked the Users page to verify that our new user was added.  But there is something else that is important to test!  We need to find out what happens when bad data is in the database and we are trying to view it in the UI.

Let’s take a look at what some bad data might look like in the database:

In our imaginary application, there are some constraints in the UI for the First Name field.  It is a required field, it must have at least two characters, it must have 40 or fewer characters, and it should only have alphanumeric characters or hyphens and apostrophes; no other symbols are allowed.  As we can see in our table, we’ve got lots of bad data:

  • User 2 has no entry for First Name
  • User 3 has an empty string for a First Name
  • User 4 is violating the rule that the name must have at least two characters
  • User 5 is violating the rule that the name must have 40 or fewer characters
  • User 6 is violating the rule that only hyphens and apostrophes are allowed for symbols
What should happen when we view the Users list in our application?  That will depend on what the product designers decide.  They may choose to display bad data as long as it is not a security risk, such as the First Name for User 6, which is actually a Stored XSS attack.  Whatever the rules for display are, it’s important to test that those rules are respected.  
You may be saying to yourself (or a developer may be saying to you), “Displaying bad data won’t be an issue, because we are putting good validation in place to make sure that bad data won’t get in the database to begin with.”  While this is absolutely standard practice today, there will always be cases where bad data will slip in.  I once tested a PATCH operation where phone numbers could be inserted into a record.  I discovered that while validation was taking place when the PATCH body was formed correctly, there was an edge case where a PATCH body was formed incorrectly and accepted without validation.  Any developer that incorrectly coded the PATCH operation could inadvertently allow bad phone numbers into the database!  
A good rule of thumb for testing any Create and Read operation is to assume that anything can go wrong and test accordingly.  We’ll continue imagining what can go wrong next week, when we test Update and Delete operations.