The Hierarchy of Quality

About a year ago, I wrote a post suggesting that we could think about automation in terms of a test wheel, where each section of the wheel represented a different type of automation.  A reader who works at Abstracta told me that my wheel reminded her of the wheel they use to think about all of the different facets of quality.  I thought their wheel was so great that I knew I would eventually want to write a post about it.

I’ve been thinking about the different types of quality mentioned in Abstracta’s Software Testing Wheel, and wondering what I would do if I was brought on to a project that had never had any testing and I needed to start from scratch.  Where would I begin my testing?  I thought about what the most important needs are for quality, and I was reminded of Maslow’s Hierarchy of Needs.

For those who are unfamiliar with this psychological concept, this is a theory that all human beings need to have certain basic needs met before they can grow as people.  The needs are as follows:

1. Physiological needs- food, water, shelter
2. Safety needs- security, property, employment
3. Love and belonging- friendship, family
4. Esteem- respect, self-esteem
5. Self-actualization- becoming the best person one can be

Looking at this list, it’s clear that physiological needs are the most important.  After all, it doesn’t matter if you have high self-esteem if you have no water to drink.  Each successive need builds on the more important one before it.

With this in mind, I realized that there is a Hierarchy of Quality- certain conditions of quality that need to be met before a team can move on to the next area of quality.  Here is my perception of where the different areas of the Abstracta test wheel fall in the hierarchy:

1.  Functionality and Reliability

These two areas share the most important spot.  Functionality means that the software does what it’s supposed to do.  This is critical, because without this, the application might as well not exist.  Imagine a clock app that didn’t tell time, or a calculator that didn’t add numbers.

Reliability means the software is available when it’s needed.  It doesn’t really matter if the app works if a user can’t get to it when they need it.

Once these quality needs have been met, we can move on to the next level:

2. Security and Performance

Security is important because users need to feel that their data is being protected.  Even applications that don’t have login information or don’t save sensitive data still need to be protected from things like cross-site scripting, which might allow a malicious user to gain control of someone else’s device.

Performance is also important, because no one wants to wait for sixty seconds for a web page to load.  If an application isn’t responsive enough, the users will go elsewhere.

Now that the application is secure and performant, we can go to the third level:

3. Usability and Compatibility

This is the level where we make sure that as many users as possible have a good experience with the application.  Usability means that the workflows of an application are intuitive so users don’t get confused.  It also means that the application is internationalized, so users all around the world can use it, and that it is accessible, so users with visual, auditory, or physical differences can use it as well.

Compatibility means that users with different operating systems, browsers, or devices can use the application.  Have you ever filled out a form in a browser and had it not save correctly when you clicked the button?  This has happened to me more than once, and I’ve needed to fill out the form again in a different browser to have it save correctly.  It’s important that our users have a positive experience no matter where they are using the software.

Now that we’ve made our application accessible to as many users as possible, it’s time to go on to the next level:

4. Portability

Portability covers how easy it is to move an application from one place to another.  One example of portability would be the way I can access my Google Drive files on my laptop, my tablet, and my phone.  Portability also refers to how easy an application can be installed or updated.  Also, we want our application to keep working when a device has an operating system upgrade.

Finally, we have thought about all of our users’ needs.  Now it’s time for one more level:

5. Maintainability

This is a level of quality that benefits the software team.  Maintainability refers to how easily an application can be updated.  Is it possible to add new APIs or update existing ones?  How easy is it to test the system?  Is it easy to deploy new code?  Is it easy for other teams to use the code?  Is the code clear and easy to understand?

When software is accessible and easy to use for all end users, AND is easy to work with and maintain for the development team, then truly high quality has been achieved.

I hope that this Hierarchy of Quality will help you make decisions about what areas of an application should be focused on first when there are a number of different quality areas competing for your team’s attention.

What do you think of this order?  Do you agree or disagree with where I placed items in the hierarchy?  Are there any missing quality areas?  Let me know in the comments below!

…but TEST like a QA Engineer!

In last week’s post, I wrote about how it is important for software testers to code like a developer.  But there is a second half of the sentence “Code like a developer…”, and that is that software testers should be TESTING. 

I’m not a stickler for using the right word for testing-related concepts, which is why I use the term “test automation”.  But automated testing is really automated checking.  Automated tests serve a very valuable purpose in that they can run regression checks at any hour of any day, without human intervention.  But they do not actually test the software.
A sad casualty of the very important move towards test automation is the QA Engineer.  Many large software companies don’t employ QA Engineers any more, feeling sure that Software Developers in Test are all that’s needed to validate the quality of their software.  And many Software Developers in Test focus solely on the automation, working from acceptance criteria in development stories and looking at the code rather than manually interacting with the software.  How is that trend working out for end users?  
Just this week, I experienced the following:  I received a (legitimate) email that I had some money to accept from PayPal.  The email contained a button to click that said “Accept the Money”.  When I clicked it, I got a message that said “The previous page is sending you to an invalid URL.”  
Last week when I was using a mobile app, a screen that I needed stayed permanently blank.  And in a post I wrote two weeks ago, I mentioned that while I was writing, Blogger had a page load error when I tried to add an image.  
Three weeks, three major companies, three bugs.  This is what comes from not employing people who think and act like testers.  
It’s true that the whole software development team owns quality, and that quality is everyone’s responsibility.  And there are also non-QA people who care deeply about certain areas of an application:
  • Developers write unit tests to check the quality of their code
  • Product Owners care about whether the feature does what it’s supposed to
  • UX Designers care about whether the user journey is intuitive
  • Security testers check the software for vulnerabilities
  • Performance engineers care about the response time the application 
But only QA Engineers care so much about the quality of the application that they’ll do things like:
  • Type ~!@#$%^&*()-=_+{}|[]:”;'<>?,./ into every text field to test for invalid character handling
  • Try to purchase -1, 99999999999, 1.3415, and foo of something
  • Enter a birth year of 3019 to see what happens
  • Click every button twice to check for multiple submissions
  • Click the forward and back button on every single page of a website
  • Test 48 different permutations of feature sets to be as thorough as possible 
  • Create dozens of test users with many varieties of security settings, to have scenarios ready for testing at a moment’s notice
  • Become an expert on a particular feature and provide documentation and assistance to other testers
  • Test the same thing in the QA environment, the Staging environment, the Demo environment, and the Production environment to make absolutely sure that the feature is working everywhere 
  • Test every feature on every supported browser and every supported mobile device
This is why we need software testers who TEST.  We need people who will continually ask themselves “How could we break this?”, “What haven’t we tested yet?”, and “What features will be used with this?”.  We need software testers who don’t rush into writing automation without first interacting with a feature.  We need software testers who remember that the goal of all their efforts is to have a user who has a positive, bug-free experience.  

Code Like a Developer…

I’ll be honest: I don’t love coding.  Don’t get me wrong, I love test automation!  I love the feeling of solving a technical challenge and coming up with a great way to automatically assert that software is doing what it’s supposed to be doing.  I love maintaining and updating my automated test suites.  But the actual writing of the code is not my favorite thing.  Whenever I find myself having to write another nested “for” loop, I sigh inwardly.

However, with all the coding I’ve done over the years, I’ve come to really appreciate the work that software developers do!  Software is complex stuff, and developers have come up with great ways to set standards, share repositories, and review each other’s work.

The test automation code we write is important; just as important as the code the software developers are writing.  Therefore, we should write our code with the same standards the developers use.  Here are a few suggestions for coding practices you should adopt:

Your code should live in the same repository as the developers’ code.
This is for a few reasons: first, the developers’ unit tests reside with the code, so it makes sense to have your integration and UI tests in the same place.  Secondly, it’s easier to maintain one repository instead of two; and finally, having your code in the same place serves to remind the whole team that test automation is everyone’s responsibility.  
Write clean code.
When I first got started with test automation, I had absolutely no idea what I was doing.  All I had was my manual testing experience and a couple of courses in Java and C++.  I did a lot of Googling and a lot of guessing as I put together my first Selenium tests.  After much work, they ran and (mostly) passed, but boy, were they lousy!  I didn’t know anything about how to write clean code.  Fortunately I had great developers around to teach me how to make my code better.
Here are some of the principles of writing clean code:
  • Keep it simple.  Always look over your code and ask yourself if there’s a simpler way of doing what it is that you are trying to do.  Sometimes the obvious solution to a testing problem only becomes clear after you have solved it in a complicated way; now it’s time to go back and solve it more elegantly. 
  • Don’t repeat yourself.  If there’s something you’re doing in more than one test- for example, logging in to the application- write a method that you can call instead of putting those steps into every test.  Similarly, create a file where you save all of your variables and element locators, and have all of your tests refer to that file.  That way if a variable or a locator changes, you can make the change in one place rather than several. 
  • Be consistent.  Consistent code is easier to read.  Be consistent with your casing: if you have a variable for the user’s first name called “firstName”, don’t make the variable for the user’s last name “LastName”.  Follow the conventions that your developers are using: if they indent with two spaces, you should too.  If they put their opening curly braces on a separate line, you should as well.
  • Comment your code.  It’s not always obvious what test automation code is doing at first glance, and while you might be quite used to the syntax you are using for your tests, your developers might not be familiar with it.  Simple comments like “Polling the queue for the delete request” can be really helpful in explaining your intent.  Moreover, what might seem really obvious to you now might not be obvious in three months when you need to update the test!  Your future self will thank you for the comments you write today.  
Solicit feedback.  
Like me, you may not have had a thorough grounding in good coding principles.  Some of the best software testers I’ve had the pleasure of working with did not major in Software Engineering.  If you did not go through rigorous training in software development, it’s important to get feedback from the developers you work with.  On my team, the software testers often review and approve each other’s code, but I also like to have my code checked by developers to make sure I’m not doing anything unusual or creating steps that could possibly result in a race condition.  
Test automation helps the whole team by speeding up the feedback process and freeing testers up to do more exploratory testing.  We owe it to our whole team to write quality code that is readable, runs quickly and consistently, and provides valuable feedback!
You may be wondering why the title of this blog post ends with “…”.  Be sure to check out next week’s blog to read the other half of the story!  

SQL Query Secrets

Have you ever been querying a SQL table, and one of your queries seems to take forever?  And then the next query you run takes milliseconds?  This would frequently happen to me, and I thought it meant that the server that hosted the database was unreliable in some way.  But this week I learned about indexes, and that the way we structure our queries has a huge impact on how long they will take to execute!  In this post, I’ll describe what indexes are and talk about the ways we can use them to optimize our queries.

An index is a database structure that is designed to speed up queries in a table.  An easy way to understand this is to think about the index at the back of a book.  Let’s say you have a book on car repair, and you want to find information about your car’s brakes.  You could look up “brakes” in the index, or you could search through every single page of the book for the word “brakes”.  It’s pretty obvious which would take less time!

Unlike books, databases can have more than one index.  There are two different kinds of indexes: clustered and unclustered.  A clustered index is used to store a table in sorted order.  There can only be one clustered index, because the table is stored in only one order.  Unclustered indexes are stored in the original table order, but they save the location of certain fields in the table.

Let’s take a look at an example.  If we had a table like this, called the Users table:

UserId State LastName FirstName Email Mobile Phone
1 MA Prunewhip Prunella [email protected] 800-867-5309
2 RI Schmoe Joe [email protected] 401-555-8765
3 NH Smith Amy [email protected] 603-555-3635
4 RI Jones Bob [email protected] 401-555-2344
5 MA Jones Amy [email protected] 617-555-2310

and we had a clustered index defined to have UserId as the key, a search on UserId would be very fast, and the data returned would be in order by UserId.

The table could also use unclustered indexes, such as the following:

State- the records in the table are indexed by state
LastNameFirstName- the records in the table are indexed by LastName and FirstName
When you query a database, the query will first look to see if an index can be used to speed up the search.  For example, if I made the request 
select LastName, FirstName from Users where UserId = 5 
the query would use the UserId index and the LastNameFirstName index to find the record.
Similarly, if I made the request
select LastName, FirstName from Users where State = ‘MA’
the query would use the LastNameFirstName index and the State index to find the record.
Of course, with a table of only five records, optimizing in this way won’t make much of a difference.  But imagine that this table had five million records, and you can see how using an index would be very helpful.

Querying a table on a non-indexed field is called a table scan.  The query needs to search through the entire table for the values, just as a person who wasn’t using a book index would have to search through every single page of the book.  

How can you know what indexes a table has?  You can find out with one simple query:
EXEC sp_helpindex “Users” 
where you would replace “Users” with whatever the name of the table is.  This will return a result of all of the clustered and unclustered indexes applied to the table, and the result will include the name of the index, a description of the index, and all the keys used in the index.
If you want to optimize your SQL queries, only ask for the data that you really need, rather than asking for select *.  Because not every field in the table is indexed, looking for every field will take longer.  
Let’s say that you want to query the Users table to find the email addresses of all of the users who live in Massachusetts (MA).  But you also would like to have some more information about those users.  You could ask for 
select FirstName, LastName, Email from Users where State = ‘MA’.
To find the records, the query will use the FirstNameLastName index and the State index.  Only the Email will be a non-indexed field.
But if you asked for
select * from Users where State = ‘MA’
now the query needs to look for two different non-indexed fields: Email and Mobile Phone.
Another helpful tip is to specify all the keys in an index when you want to use that index to make a query.  For example, if you wanted to find the Email for Prunella Prunewhip, you should ask for 
select Email from Users where LastName = ‘Prunewhip’ and FirstName = ‘Prunella’
rather than asking for
select Email from Users where LastName = ‘Prunewhip’.
In the second example, the LastNameFirstName index won’t be used.

And when you want to use an index, the query will run faster if you specify the keys in the order they appear, so it’s better to say
where LastName = ‘Prunewhip’ and FirstName = ‘Prunella’
than it is to say
where FirstName = ‘Prunella’ and LastName = ‘Prunewhip’

Here’s one more tip: when you want to use an index, be sure not to manipulate one of the index keys in your query, because this will mean that the index won’t be used.  For example, if you had a table like this, called Grades:

StudentId LastName FirstName Grade
1 Miller Kara 89
2 Smith Carol 56
3 Jones Bob 99
4 Davis Frank 78
5 Green Doug 65
and you had an unclustered index called LastNameGrade, and you executed a query like:
select LastName from Grades where (Grade + 100) = 178
the LastNameGrade index wouldn’t be used, because the Grade value was being manipulated.  It’s necessary for the query to go through the entire table and add 100 to each Grade field in order to search for the correct value.

Armed with this knowledge, you should be able to create queries that will run as fast as possible, getting you the data you need.  I’d like to extend my thanks to my colleague Cindy Gall, whose informative workshop inspired this post!

Six Ways Chrome DevTools Can Help With Testing

Did you know that there is a wealth of testing tools right in your browser?  Web browsers like Chrome and Firefox have developer tools that are available for free, for everyone.  And these tools are not just for developers!  In this post, I’ll be sharing six ways that Chrome DevTools can help you with your testing.

To access Chrome DevTools, simply click on the three-dot menu in the upper right corner of your browser, choose “More Tools”, and then choose “Developer Tools”.  DevTools will open up alongside your browser window.  You can customize where you would like the tools to display by clicking on the three-dot menu in the DevTools nav bar and selecting an option for “Dock Side”.  You can choose to have the DevTools display on the left, on the right, on the bottom, or in a separate window.

Here are some of the things that Dev Tools can do:

1. Inspect an HTML Element
Have you ever been writing UI automation and you just can’t figure out how to access an element?  With DevTools, you can right-click on the element and choose “Inspect”, and the Elements pane of DevTools will show you the element in the HTML.  You can then use this information to figure out the best way to access the element.

2. Edit HTML Elements
Not only can you find an element in the HTML, you can also edit it!  This is great for security testing.  Imagine that there is a page with a button that is hidden for users who are not admins.  A malicious user could find that element using DevTools, remove the “hide” tag, and use the button.  So it’s helpful to try this while testing to verify that there’s an additional check for user permissions when the button is used.

To edit an element, right-click on it in the HTML displayed in the Elements pane, and choose “Edit as HTML”.  Make whatever edits to the element you want, then click out of the edit box.  You should see the element on the page change as a result of your edits.

3. View HTTP requests
If you click on the Network tab of DevTools, you can see all of the requests made to the server while using a web page.  This includes API calls, which you can then copy and use in a tool like Postman.  This feature is helpful for determining if your page is making the API calls that you are expecting, and it’s also great for security testing.  For example, just because the front-end of a web page doesn’t allow a user to submit a field with more than 50 characters doesn’t mean that it can’t be done.  If a malicious user copies the API call and submits it through Postman, through a curl command, or through some other tool, they may be able to send more than 50 characters directly to the server.  This is why it’s important to have both front-end and back-end validation on a website.

4. Simulate device frames
When you are testing a webpage, it’s important to make sure that the page appears correctly on both laptops and mobile devices.  But even the most well-equipped tester doesn’t have access to every single device in use today.  So DevTools comes with a simulator that shows roughly what your webpage will look like on various devices.  To access this feature, click on the device logo

in the toolbar.  This will open the simulator in the webpage side of the browser.  Then you can use the dropdown to select specific devices (which seem to be a bit obsolete), or you can choose the “Responsive” setting and then manually expand or contract the window to get the size you want.  The exact size is displayed in the navbar at the top.

5. Simulate performance on slower networks
Testing a webpage while in your office usually means you are using a great high-speed network.  But what about your users who have slower connections?  You can use DevTools to simulate slower connections and throttled CPU, which could help uncover race conditions in your application.  To use this feature, go to the Performance tab in the navbar.  In the Network dropdown, you can choose “Fast 3G”, “Slow 3G”, or “Offline”, and in the CPU dropdown, you can choose “No throttling”, “4x slowdown” or “6x slowdown”.  Don’t forget to reverse your changes when you are done testing!

6. Investigate page load errors
As I was creating this post, I was reminded of one more way that DevTools are helpful.  I was trying to upload the Chrome logo to my post, and the popup that I usually use to add an image was completely blank.  I went to the Console tab of DevTools and saw that there was a 404 “File not found” error when I clicked on the Add Images button in Blogger.  When you are testing your team’s application and you’ve found a bug on a page, checking for errors in the console can help you give more information to your developers so they can get to the root of the problem more quickly.

Sometimes the most useful testing tools are right there in front of you!  I hope this post has inspired you to take a look at DevTools to see how it can help you in your testing.

The Power of Not Knowing

Recently I saw a tweet from Ben Simo (@QualityFrog) that mentioned that he sometimes likes to practice what he calls “intentional ignorance”- where he doesn’t read some of the documentation or code for a new feature to see what he can find while doing exploratory testing.  His tweet reminded me that I used to do this too!

I haven’t done this in a while, because the team I work on is a great Agile team.  The testers are invited to the feature grooming sessions, each story has acceptance criteria written, and the developers do a feature handoff with the testers when each story is ready for testing.

But at previous companies, I was often given a story to test with no feature handoff and no acceptance criteria.  Sometimes the story wouldn’t even have a description, and would have some cryptic title, like “Endpoint for search”.  I would usually be annoyed by this, and I would ask for clarification, but I would first use it as an opportunity to do some exploratory testing while I had no pre-conceived notions what the feature could do or not do.   And while testing in this fashion, I would often find a bug, show it to the developer, and have him or her say, “Oh, it never even occurred to me to test the feature in that way.”

Of course I don’t want to go back to the days of cryptic story titles with no description!  But testing without knowing what the feature does can have some benefits:
  • You approach the application the same way a user would.  When your users see your new feature for the first time, they don’t have the benefit of instructions.  By trying out the feature without knowing how it works, you could discover that an action button is hard to find, or that it’s difficult to know what to do first on a multi-part form.  
  • You might try entering data that no one was expecting.  For example, there could be a form field where the date was supposed to be entered with month and day only, but you enter in the month, day, and year, which breaks the form.  
  • Without any instructions from the developer, you might think of other features to test the new feature with, besides those the developer thought of.  Those feature combinations might yield new bugs.

So how can we add these advantages back into our testing without skipping reading the acceptance criteria and having feature handoffs?  Here are a few ways:

  • Pair test with someone on another team.  At my company we have many teams, each of which often has no idea what the other teams are building.  Four times a year, the software testers get together in pairs where the two testers are from very different teams, and they swap applications and start testing.  This is a great way to find bugs and user experience issues!
  • When you start testing, spend some time just playing around with the new feature before writing a test plan.  By exploring in this way, you might come up with some unusual testing ideas.
  • After you’ve tested the acceptance criteria, take some time to think about what features might be used with the new feature.  What happens when you test them together?  For example, if you were testing a new page of data, you could test it with the global sort feature that already exists in your application.
Of course, there are also times where not knowing all the details about a feature is detrimental.  There have been times in my testing career where I tested a feature and completely missed something that the feature could do, because no one told me about it.  That’s why I’m glad that we have acceptance criteria and feature handoffs.  But there are also times when not knowing can yield some of the most interesting bugs.

Your Flaky Tests Are Destroying Trust

Anyone who has ever written an automated test has experienced test flakiness.  There are many reasons for flaky tests, including:

  • Environmental issues, such as the application being unavailable
  • Test data issues, where an expected value has been changed
  • UI issues, such as a popup window taking too long to appear

All of these reasons are valid explanations for flaky tests.  However, they are not excuses!  It should be your mission to have all of your automated tests pass every single day, except of course when an actual bug is present.

This is important not just because you want your tests to be reliable; it’s important because when you have flaky tests, trust in you and in your team is eroded.  Here’s why:

Flaky tests send the message that you don’t care
Let’s say you are the sole automation engineer on a team, and you have a bunch of flaky tests.  It’s your job to write test automation that actually checks that your product is running correctly, and because your tests are flaky, your automation doesn’t do that.  Your team may assume that this is because you don’t care about whether your job is done properly.

Flaky tests make your team suspect your competence
An even worse situation than the previous example is one where your team simply assumes that you haven’t fixed the flaky tests because you don’t know how.  This further erodes their trust in you, which may spill over into other testing.  If you find a bug when you are doing exploratory testing your colleagues might not believe that you have a bug, because they think you are technically incompetent.

Flaky tests waste everyone’s time
If you are part of a large company where each team contributes one part of an application, other teams will rely on your automation to determine whether the code they committed works with your team’s code.  If your tests are failing for no reason, people on other teams will need to stop what they are doing and troubleshoot your tests.  They won’t be pleased if they discover that there’s nothing wrong with the app and your tests are just being flaky.

Flaky tests breed distrust between teams
If your team has a bunch of flaky tests that fail for no good reason, and you aren’t actively taking steps to fix them, other teams will ignore your tests, and may also doubt whether your team can be relied upon.  In a situation like this, if Team B commits code and sees that Team A has failing tests, they may do nothing about it, and may not even ask Team A about the failures.  If there are tests that fail because there are real issues, your teams might not discover them until days later.

Flaky tests send a bad message your company’s leadership 
There’s nothing worse for a test team than to have test automation where only 80% (or less) of the tests pass on a daily basis.  This sends a message to management that either test automation is unreliable, or you are unreliable!

So, what can we do about flaky tests?  I’d like to recommend these steps:

1. Make a commitment to having 100% of your tests pass every day.  The only time a test should fail is if a legitimate bug is present.  Some might argue that this is an impossible dream, but it is one to strive for.  There is no such thing as perfect software, or perfect tests, but we can work as hard as we can to get as close as we can to that perfection.

2. Set up alerts that notify you of test failures.  Having tests that detect problems in your software doesn’t help if no one is alerted when test failures happen.  Set up an alert system that will notify you via email or chat when a test is failing.  Also, make sure that you test your alert.  Don’t assume that because the alert is in place it is automatically working.  Make a change that will cause a test to fail and check to see if you got the notification.

3. Investigate every test failure and find out why it failed.  If the failure wasn’t due to a legitimate bug, what caused the failure?  Will the test pass if you run it again, or does it fail every time?  Will the test pass if you run it manually?  Is your test data correct?  Are there problems with the test environment?

4. Remove the flaky tests.  Some might argue that this is a bad idea because you are losing test coverage, and the test passes sometimes.  But this doesn’t matter, because when people see that the test is flaky they won’t trust it anyway.  It’s better to remove the flaky tests altogether so that you demonstrate that you have a 100% passing rate, and others will begin to trust your tests.

An alternative would be to set the flaky tests to be skipped, but this might also erode trust.  People might see all the skipped tests and see them as a sign that you don’t write good test automation.  Furthermore, you might forget to fix the skipped tests.

5. Fix all the flaky tests you can.  How you fix the flaky tests will depend on why they are flaky.  If you have tests that are flaky because someone keeps changing your test data, change your tests so that the test data is set up in the test itself.  If you have tests that are flaky because sometimes your test assets aren’t deleted at the end of the test, do a data cleanup both before and after the test.

6. Ask for help.  If your tests are flaky because the environment where they are running is unreliable, talk to the team that’s responsible for maintaining the environment.  See if there’s something they can to do solve the problem.  If they are unresponsive, find out if other teams are experiencing the issue, and lobby together to make a change.

7. Test your functionality in a different way.  If your flaky test is failing because of some element on the page that isn’t loading on time, don’t try to solve the issue by making your waits longer.  See if you can come up with a different way to test the feature.  For example, you might be able to switch that test to an API test.  Or you might be able to verify that a record was added in the database instead of going through UI.  Or you might be able to verify the data on a different page, instead of the one with the slow element.

Some might say that not testing the UI on that problematic page is dangerous.  But having a flaky test on this page is even more dangerous, because people will just ignore the test.  It would be better to stick with an automated test that works, and do an occasional manual test of that page.

Quality Automation is Our Responsibility

We’ve all been in situations where we have been dismissed as irrelevant or incompetent because of the reputation of a few bad testers.  Let’s create a culture of excellence for testers everywhere by making sure that EVERY test we run is reliable and provides value!

Why You Should Be Testing in Production

This is a true story; I’m keeping the details vague to protect those involved.  Once there was a software team that was implementing new functionality.  They tested the new functionality in their QA environment, and it worked just fine.  So they scheduled a deployment: first to the Staging environment, then to Production.  They didn’t have any automated tests for the new feature, because it was tricky to automate.  And they didn’t bother to do any manual tests in Staging or Production, reasoning that if it worked in the QA environment, it must work everywhere.

You can probably guess what happened next- they started getting calls from customers that the new feature didn’t work.  They investigated and found that this was true.  Then they tried out the feature in the Staging environment and found that it didn’t work there either.  As it turned out, the team had used hard-coded configuration strings that were only valid in the QA environment.  If they had simply done ONE test in the Staging or Production environment, they would have noticed that something was wrong.  Instead, it was left to the customers to notice the problem.

There are two main reasons why things that work in a QA environment don’t work in a Production environment:

1) Configuration problems- This is what happened with the team described above.  Software is complicated, and there are often multiple servers and databases that need to talk to each other in order for the software to work properly.  Keeping software secure means that each part of the application needs to be protected by passwords or other configuration strings.  If any one of those strings is incorrect, the software won’t work completely.

2) Deployment problems- In this age of microservices, deploying software usually means deploying several different APIs.  In a large organization, there may be different teams responsible for different APIs.  For example, when a new feature in API A needs the new code in API B to work properly, API B will need to be deployed first.  It’s possible that Team B will forget to deploy API B or not even realize that it needs to be deployed.  In cases like this, Team A might assume that API B had been deployed, and they will go ahead and deploy API A.  Without testing, Team A will have no way of knowing that the new feature isn’t working.

By running tests in every environment, you can quickly discover if you have configuration or deployment problems.  It’s often not necessary to go through extensive testing of a new feature in Production if you’ve already tested it in QA, but it is vital that you do at least SOME testing to verify that it’s working!  We never want to have our customers find problems before we do.

Confused? Simplify!

As testers, we are often asked to test complex systems.  Gone are the days when testers were simply asked to fill out form fields and hit the Submit button; now we are testing data stores, cloud servers, messaging services, and much more.  When so many building blocks are used in our software, it can become easy to get overwhelmed and confused.  When this happens, it’s best to simplify what we are testing until our situation becomes clear.

Here’s an example that happened recently on my team: we were testing that push notifications of a specific type were working on an iPhone.  One of my teammates was triggering a push notification, but it wasn’t appearing on the phone.  What could be wrong?  Maybe notifications were completely broken.  Maybe they were broken on the iPhone.  Maybe only this specific notification was broken.  Maybe only notifications of this type were broken.  In a situation where there are a lot of notifications to test and we are working on a deadline, this can become very confusing. 

So, we simplified by asking a series of questions and running a test for each one.  We started with:
Is this push notification working on an Android phone?
We triggered the same notification to go to an Android phone, and the push was delivered.  So we ruled out that the notification itself was broken.

Next, we asked:
Is this push notification working on any other iPhone?
We triggered the same notification to go to a different iPhone, and the push was delivered.  So we ruled out that the notification was broken on iOS devices.

Then we asked:
Is ANY notification working on this specific iPhone? 
We triggered some different notifications to go to the iPhone, and no pushes were delivered.  So we concluded that the problem was not with the notification, or with the push service; the problem was with the phone.

In taking a step back and asking three simple questions, we were able to quickly diagnose the problem.  Let’s take a look at another example, using my hypothetical feature called the Superball Sorter, which sorts small and large colored balls among four children, as described in this post.

Let’s imagine that we are testing a scenario where we are sorting the balls by both size and color.  We have the children set up with the following rules:
Amy gets only large balls
Bob gets only small purple balls and large red balls
Carol gets only small balls
Doug gets only green balls

When we run the sorter, a small purple ball is next in the sorting process, and it’s Bob’s turn to get a ball.  We are expecting that Bob is going to get the small purple ball because his sorting rules allow it, but he doesn’t get the ball- it goes to Carol instead.  What could be wrong here?  Maybe Bob isn’t getting any balls.  Maybe the purple ball isn’t being sorted at all.  Maybe only the small balls aren’t being sorted.  How can we figure out what is going on?

Our first question will be:
Can Bob get ANY sorted balls?  
We’ll set up the sorter so Amy, Carol, and Doug only get large balls, and Bob only gets small balls.  We run the sorter, and Bob gets all the small balls.  So we know this isn’t the problem.

Can anyone get the small purple ball?
Next, we’ll set up the sorter so that Amy will only get small purple balls, and Bob, Carol, and Doug can get any ball at all.  We’ll set up our list of balls so that the small purple ball is first on the list.  When we start our sorting process with Amy, she gets the small purple ball.  So now we know that the small purple ball isn’t the problem.

Can Bob get the small purple ball in some other scenario?
We saw in our initial test that Bob wasn’t getting the small purple ball, but can he EVER get that ball?  We’ll set up our rules so that Amy will only get large balls, and Bob will get only small purple balls.  We won’t give Carol and Doug any rules. Then we’ll set up our list of balls so that the small purple ball is first on the list.  Amy won’t get the small purple ball, because she only gets large balls, so the small purple ball is offered to Bob.  He gets the ball, so now we know that Bob can get the small purple ball in some scenarios.

At this point, we know that the problem is not the small purple ball.  What is different between the original scenario and the one we just ran?  One difference is that in the original scenario, all four children had a rule.  So let’s ask this question:

Can Bob get the small purple ball when it’s his only rule, and the other children all have rules?
We’ll set up the rules like this:
Amy gets only large balls
Bob gets only small purple balls
Carol gets only small balls
Doug gets only green balls
We again set up our list of balls so that the small purple ball is first on the list.  The ball skips Amy, because it doesn’t meet her rule, and Bob gets the ball.  So now we know that the problem is not that all the children have rules.  So now the next logical question is:

What happens when Bob has TWO rules?
We’ll set up the rules like this:
Amy gets only large balls
Bob gets only small purple balls and small yellow balls
Carol gets only small balls
Doug gets only green balls

Our list of balls is the same, where the small purple ball is first.  This time, the ball skips Amy AND Bob, and Carol gets the small purple ball.

AHA!  Now we have a good working theory: when Bob has two rules, the sorting is not working correctly.  We can test out this theory by giving another child two rules, while giving everyone else one rule.  Are the balls sorted correctly?  What about when a child has two rules that specify color only and not size?  Will the two rules work then?  By continuing to ask questions, we can pinpoint precisely what the bug is.

By making your tests as simple as possible, you are able to narrow down the possibilities of where the bug is.  And by proceeding methodically and logically, you will be able to find that bug as quickly as possible, even in a very complex system.  

Toggles, Revisited

A few years ago, I wrote a blog post detailing why I thought toggles were a bad idea.  It made a clever analogy between toggles and the tribbles on Star Trek’s U.S.S. Enterprise.  I think it’s a fun read, so you may want to check it out; but since the time I wrote it, my opinion has changed a bit. In this post I will explain why I think toggles may be helpful, and I’ll propose some rules for their use.

About a year ago, my team was working on a new notification service that would send out emails and messages more efficiently than the current service.  When the new service was ready, we migrated one notification type to the new service to see how it would work.  We tested the notification extensively and we were sure that we had accounted for all scenarios, so we took the new service to Production.

A couple of weeks later, we discovered that there was an odd case that we hadn’t tested.  If two users in the same company had the same id, the wrong user was getting the notification.  We had no idea that it was possible for two users in the same company to have the same id, so we hadn’t thought to test this.

Fortunately, our new service was behind a toggle.  Since we certainly didn’t want the wrong people to get notifications, we quickly toggled off the new service.  There was no impact to any other customers, because they were still getting their notifications; they were just being notified through the old service.  We were able to quickly fix the bug, get the fix into Production, and toggle the service back on.

If we hadn’t had the toggle, the users with the same id would have continued to get the wrong notifications until we were able to fix the bug.  We would have had to rush to get a code patch into Production, and it’s possible that we would have made mistakes along the way.  Because we had the toggle, we could take the time to make sure that the fix was good, and we could do all the regression testing we wanted.

So, I’ve changed my mind about toggles.  I think they can be useful in situations where there’s a significant risk that accompanies a change.  But if you are going to use toggles, please observe the following rules:

1. Toggles are NOT a substitute for high-quality testing.  Being able to toggle something off at the first sign of trouble does not mean that you can skip testing your new feature thoroughly.  Ideally you should have tested so well that you never need to turn your toggle off.

2. Make sure to test your feature with the toggle on AND with the toggle off.  You don’t want to discover in the middle of dealing with a problem in Production that the toggle doesn’t actually work!

3. When the feature has gone to Production and a certain amount of time has passed, remove the toggle so that the feature is on permanently.  Otherwise you could get into a situation where months from now someone inadvertently toggles the feature off.  And the fewer toggles you have in your application, the fewer combinations of toggles you need to test.

As with many things in software development, the best strategies are those that ensure the best possible outcome for our end users.  When they are used wisely, toggles can help mitigate any unexpected issues found in Production.