A Question of Time

Time is the one thing of which everyone gets the same amount.  Whether we are the CEO of a company or we are the intern, we all have 1440 minutes in a day.  I’ve often heard testers talk about how they don’t have enough time to test, and that can certainly happen when deadlines are imposed without input from everyone on the team.  I’ve written a blog post about time management techniques for testers, but today I’m going to tackle the question:

Is it worth my time to automate this task?

Sometimes we are tempted to create a little tool for everything, just because we can.  I usually see this happen with developers more than testers, but I do see it with some testers who love to code.  However, writing code does not always save us time.  When considering whether to do a task manually or to write automation for it, ask yourself these four questions:

1. Will I need to do this task again?

Recently my team was migrating files from one system to another system.  I ran the migration tool manually and did manual checking that the files had migrated properly.  I didn’t write any automation for this, because I knew that I was never going to need to test it again.

Contrast this with a tester from another team who is continually asked to check the UI on a page when his team makes updates.  He got really tired of doing this again and again, so he created a script that will take screenshots and compare the old and new versions of the page.  Now he can run the check with the push of a button.

2. How much time does this task take me, and how much time will it take me to write the code?

Periodically my team’s test data gets refreshed, and that means that the information we have for our test users sometimes gets changed.  When this happens, it takes about eight hours to manually update all the users.  It took me a few hours to create a SQL script that would update the users automatically, but it was totally worth my time, because now I save eight hours of work whenever the data is refreshed.

But there have been other times where I’ve needed to set up some data for testing, and a developer has offered to write a little script to do it for me.  Since I can usually set up the data faster than they can create the script, I decline the offer.

3. How much time will it take to maintain the automation I’m writing?

At a previous job, I was testing email delivery and I wanted to write an automated test that would show that the email had actually arrived in the Gmail test account.  The trouble was that there could be up to a ten minute delay for the email to appear.  I spent a lot of time adjusting the automated test to wait longer, to have retries, and so on, until finally I realized it was just faster for me to take that assertion out of the test, and manually check the email account from time to time.

However, my team’s automated API smoke tests take very little time to maintain, because the API endpoints change so infrequently that the tests rarely need to change.  The first API smoke test I set up took a few days; but once we had a working model it became very easy to set up tests for our other APIs.

4. Does the tool I’m creating already exist?

At a previous company, the web team was porting over many customers’ websites from one provider to another.  I was asked to create a tool that would crawl through the sites and locate all the pages, and then crawl through the migrated site to make sure all the pages had been ported over.  It was really fun to create this tool, and I learned a lot about coding in the process.  However, I discovered after I made the tool that web-crawling software already exists!

But in that particular month I did have the time to create the tool, and the things I learned helped me with my other test automation.  So sometimes it may be worth “reinventing the wheel” if it will help you or your team.

The Bottom Line: Are you saving or wasting time?

All of these questions come down to one major consideration, and that is whether your task is saving or wasting time.  If you are a person who enjoys coding, you may be tempted to write a fun new script for every task you need to do; but this might not always save you time.  Similarly, if you don’t enjoy coding, you might insist on doing repetitive tasks manually; but using a simple tool could save you a ton of time.  Always consider the time-saving result of your activities!

Measuring Quality

The concept of measuring quality can be a hot-button topic for many software testers.  This is because metrics can be used poorly; we’ve all heard stories about testers who were evaluated based on how many bugs they found or how many automated tests they wrote.  These measures have absolutely no bearing on software quality. A person who finds a bug in three different browsers can either write up the bug once or write up a bug for each browser; having three JIRA tickets instead of one makes no difference in what the bug is!  Similarly, writing one hundred automated tests where only thirty are needed for adequate test coverage doesn’t ensure quality and may actually slow down development time.

But measuring quality is important, and here’s why: software testers are to software what the immune system is to the human body.  When a person’s immune system is working well, they don’t think about it at all.  They get exposed to all kinds of viruses and bacteria on a daily basis, and their immune system quietly neutralizes the threats.  It’s only when a threat gets past the immune system that a person’s health breaks down, and then they pay attention to the system.  Software testers have the same problem: when they are doing their job really well, there is no visible impact in the software.  Key decision-makers in the company may see the software and praise the developers that created it without thinking about all the testing that helped ensure that the software was of high quality.

Measuring quality is a key way that we can demonstrate the value of our contributions.  But it’s important to measure well; a metric such as “There were 100 customer support calls this month” means nothing, because we don’t have a baseline to compare it to.  If we have monthly measurements of customer support calls, and they went from 300 calls in the first month, to 200 calls in the second month, to 100 calls in the third month, and daily usage statistics stayed the same, then it’s logical to conclude that customers are having fewer problems with the software.

With last week’s post about the various facets of quality in mind, let’s take a look at some ways we could measure quality.

How many bugs are found in production by customers?
A declining number could indicate that bugs are being caught by testers before going to production.
How many daily active users do we have? 
A rising number probably indicates that customers are happy with the software, and that new customers have joined the ranks of users.

What is our percentage of uptime?  
A rising number could show that the application has become more stable.
How many errors do we see in our logs?  
A declining number might show that the software operations are generally completing successfully.

How many issues were found by penetration tests and security scans?  
A declining number could show that the application is becoming more secure.

What is our average response time?
A stable or declining number will show that the application is operating within accepted parameters.

What are our customers saying about our product?
Metrics like survey responses or app store ratings can indicate how happy customers are with an application.
How many customer support calls are we getting?
Increased support calls from customers could indicate that it’s not clear how to operate the software.

How many support calls are we getting related to browser, device, or operating system?
An increased number of support calls could indicate that the application is not working well in certain circumstances.
What browsers/devices/operating systems are using our software?
When looking at analytics related to app usage, a low participation rate by a certain device might indicate that users have had problems and stopped using the application.

What percentage of customers upgraded to the new version of our software?
Comparing upgrade percentages with statistics of previous upgrades could indicate that the users found the upgrade process easy.
How many support calls did we get related to the upgrade?
An increased number of support calls compared to the last upgrade could indicate that the upgrade process was problematic.

How long does it take to deploy our software to production?
If it is taking longer to deploy software than it did during the last few releases, then the process needs to be evaluated.
How frequently can we deploy?
If it is possible to deploy more frequently than was possible six months ago, then the process is becoming more streamlined.

There’s no one way to measure quality, and not every facet of quality can be measured with a metric.  But it’s important for software testers to be able to use metrics to demonstrate how their work contributes to the health of their company’s software, and the above examples are some ways to get started.  Just remember to think critically about what you are measuring, and establish good baselines before drawing any conclusions.

The Hierarchy of Quality

About a year ago, I wrote a post suggesting that we could think about automation in terms of a test wheel, where each section of the wheel represented a different type of automation.  A reader who works at Abstracta told me that my wheel reminded her of the wheel they use to think about all of the different facets of quality.  I thought their wheel was so great that I knew I would eventually want to write a post about it.

I’ve been thinking about the different types of quality mentioned in Abstracta’s Software Testing Wheel, and wondering what I would do if I was brought on to a project that had never had any testing and I needed to start from scratch.  Where would I begin my testing?  I thought about what the most important needs are for quality, and I was reminded of Maslow’s Hierarchy of Needs.

For those who are unfamiliar with this psychological concept, this is a theory that all human beings need to have certain basic needs met before they can grow as people.  The needs are as follows:

1. Physiological needs- food, water, shelter
2. Safety needs- security, property, employment
3. Love and belonging- friendship, family
4. Esteem- respect, self-esteem
5. Self-actualization- becoming the best person one can be

Looking at this list, it’s clear that physiological needs are the most important.  After all, it doesn’t matter if you have high self-esteem if you have no water to drink.  Each successive need builds on the more important one before it.

With this in mind, I realized that there is a Hierarchy of Quality- certain conditions of quality that need to be met before a team can move on to the next area of quality.  Here is my perception of where the different areas of the Abstracta test wheel fall in the hierarchy:

1.  Functionality and Reliability

These two areas share the most important spot.  Functionality means that the software does what it’s supposed to do.  This is critical, because without this, the application might as well not exist.  Imagine a clock app that didn’t tell time, or a calculator that didn’t add numbers.

Reliability means the software is available when it’s needed.  It doesn’t really matter if the app works if a user can’t get to it when they need it.

Once these quality needs have been met, we can move on to the next level:

2. Security and Performance

Security is important because users need to feel that their data is being protected.  Even applications that don’t have login information or don’t save sensitive data still need to be protected from things like cross-site scripting, which might allow a malicious user to gain control of someone else’s device.

Performance is also important, because no one wants to wait for sixty seconds for a web page to load.  If an application isn’t responsive enough, the users will go elsewhere.

Now that the application is secure and performant, we can go to the third level:

3. Usability and Compatibility

This is the level where we make sure that as many users as possible have a good experience with the application.  Usability means that the workflows of an application are intuitive so users don’t get confused.  It also means that the application is internationalized, so users all around the world can use it, and that it is accessible, so users with visual, auditory, or physical differences can use it as well.

Compatibility means that users with different operating systems, browsers, or devices can use the application.  Have you ever filled out a form in a browser and had it not save correctly when you clicked the button?  This has happened to me more than once, and I’ve needed to fill out the form again in a different browser to have it save correctly.  It’s important that our users have a positive experience no matter where they are using the software.

Now that we’ve made our application accessible to as many users as possible, it’s time to go on to the next level:

4. Portability

Portability covers how easy it is to move an application from one place to another.  One example of portability would be the way I can access my Google Drive files on my laptop, my tablet, and my phone.  Portability also refers to how easy an application can be installed or updated.  Also, we want our application to keep working when a device has an operating system upgrade.

Finally, we have thought about all of our users’ needs.  Now it’s time for one more level:

5. Maintainability

This is a level of quality that benefits the software team.  Maintainability refers to how easily an application can be updated.  Is it possible to add new APIs or update existing ones?  How easy is it to test the system?  Is it easy to deploy new code?  Is it easy for other teams to use the code?  Is the code clear and easy to understand?

When software is accessible and easy to use for all end users, AND is easy to work with and maintain for the development team, then truly high quality has been achieved.

I hope that this Hierarchy of Quality will help you make decisions about what areas of an application should be focused on first when there are a number of different quality areas competing for your team’s attention.

What do you think of this order?  Do you agree or disagree with where I placed items in the hierarchy?  Are there any missing quality areas?  Let me know in the comments below!

…but TEST like a QA Engineer!

In last week’s post, I wrote about how it is important for software testers to code like a developer.  But there is a second half of the sentence “Code like a developer…”, and that is that software testers should be TESTING. 

I’m not a stickler for using the right word for testing-related concepts, which is why I use the term “test automation”.  But automated testing is really automated checking.  Automated tests serve a very valuable purpose in that they can run regression checks at any hour of any day, without human intervention.  But they do not actually test the software.
A sad casualty of the very important move towards test automation is the QA Engineer.  Many large software companies don’t employ QA Engineers any more, feeling sure that Software Developers in Test are all that’s needed to validate the quality of their software.  And many Software Developers in Test focus solely on the automation, working from acceptance criteria in development stories and looking at the code rather than manually interacting with the software.  How is that trend working out for end users?  
Just this week, I experienced the following:  I received a (legitimate) email that I had some money to accept from PayPal.  The email contained a button to click that said “Accept the Money”.  When I clicked it, I got a message that said “The previous page is sending you to an invalid URL.”  
Last week when I was using a mobile app, a screen that I needed stayed permanently blank.  And in a post I wrote two weeks ago, I mentioned that while I was writing, Blogger had a page load error when I tried to add an image.  
Three weeks, three major companies, three bugs.  This is what comes from not employing people who think and act like testers.  
It’s true that the whole software development team owns quality, and that quality is everyone’s responsibility.  And there are also non-QA people who care deeply about certain areas of an application:
  • Developers write unit tests to check the quality of their code
  • Product Owners care about whether the feature does what it’s supposed to
  • UX Designers care about whether the user journey is intuitive
  • Security testers check the software for vulnerabilities
  • Performance engineers care about the response time the application 
But only QA Engineers care so much about the quality of the application that they’ll do things like:
  • Type ~!@#$%^&*()-=_+{}|[]:”;'<>?,./ into every text field to test for invalid character handling
  • Try to purchase -1, 99999999999, 1.3415, and foo of something
  • Enter a birth year of 3019 to see what happens
  • Click every button twice to check for multiple submissions
  • Click the forward and back button on every single page of a website
  • Test 48 different permutations of feature sets to be as thorough as possible 
  • Create dozens of test users with many varieties of security settings, to have scenarios ready for testing at a moment’s notice
  • Become an expert on a particular feature and provide documentation and assistance to other testers
  • Test the same thing in the QA environment, the Staging environment, the Demo environment, and the Production environment to make absolutely sure that the feature is working everywhere 
  • Test every feature on every supported browser and every supported mobile device
This is why we need software testers who TEST.  We need people who will continually ask themselves “How could we break this?”, “What haven’t we tested yet?”, and “What features will be used with this?”.  We need software testers who don’t rush into writing automation without first interacting with a feature.  We need software testers who remember that the goal of all their efforts is to have a user who has a positive, bug-free experience.  

Code Like a Developer…

I’ll be honest: I don’t love coding.  Don’t get me wrong, I love test automation!  I love the feeling of solving a technical challenge and coming up with a great way to automatically assert that software is doing what it’s supposed to be doing.  I love maintaining and updating my automated test suites.  But the actual writing of the code is not my favorite thing.  Whenever I find myself having to write another nested “for” loop, I sigh inwardly.

However, with all the coding I’ve done over the years, I’ve come to really appreciate the work that software developers do!  Software is complex stuff, and developers have come up with great ways to set standards, share repositories, and review each other’s work.

The test automation code we write is important; just as important as the code the software developers are writing.  Therefore, we should write our code with the same standards the developers use.  Here are a few suggestions for coding practices you should adopt:

Your code should live in the same repository as the developers’ code.
This is for a few reasons: first, the developers’ unit tests reside with the code, so it makes sense to have your integration and UI tests in the same place.  Secondly, it’s easier to maintain one repository instead of two; and finally, having your code in the same place serves to remind the whole team that test automation is everyone’s responsibility.  
Write clean code.
When I first got started with test automation, I had absolutely no idea what I was doing.  All I had was my manual testing experience and a couple of courses in Java and C++.  I did a lot of Googling and a lot of guessing as I put together my first Selenium tests.  After much work, they ran and (mostly) passed, but boy, were they lousy!  I didn’t know anything about how to write clean code.  Fortunately I had great developers around to teach me how to make my code better.
Here are some of the principles of writing clean code:
  • Keep it simple.  Always look over your code and ask yourself if there’s a simpler way of doing what it is that you are trying to do.  Sometimes the obvious solution to a testing problem only becomes clear after you have solved it in a complicated way; now it’s time to go back and solve it more elegantly. 
  • Don’t repeat yourself.  If there’s something you’re doing in more than one test- for example, logging in to the application- write a method that you can call instead of putting those steps into every test.  Similarly, create a file where you save all of your variables and element locators, and have all of your tests refer to that file.  That way if a variable or a locator changes, you can make the change in one place rather than several
  • Be consistent.  Consistent code is easier to read.  Be consistent with your casing: if you have a variable for the user’s first name called “firstName”, don’t make the variable for the user’s last name “LastName”.  Follow the conventions that your developers are using: if they indent with two spaces, you should too.  If they put their opening curly braces on a separate line, you should as well.
  • Comment your code.  It’s not always obvious what test automation code is doing at first glance, and while you might be quite used to the syntax you are using for your tests, your developers might not be familiar with it.  Simple comments like “Polling the queue for the delete request” can be really helpful in explaining your intent.  Moreover, what might seem really obvious to you now might not be obvious in three months when you need to update the test!  Your future self will thank you for the comments you write today.  
Solicit feedback.  
Like me, you may not have had a thorough grounding in good coding principles.  Some of the best software testers I’ve had the pleasure of working with did not major in Software Engineering.  If you did not go through rigorous training in software development, it’s important to get feedback from the developers you work with.  On my team, the software testers often review and approve each other’s code, but I also like to have my code checked by developers to make sure I’m not doing anything unusual or creating steps that could possibly result in a race condition.  
Test automation helps the whole team by speeding up the feedback process and freeing testers up to do more exploratory testing.  We owe it to our whole team to write quality code that is readable, runs quickly and consistently, and provides valuable feedback!
You may be wondering why the title of this blog post ends with “…”.  Be sure to check out next week’s blog to read the other half of the story!  

SQL Query Secrets

Have you ever been querying a SQL table, and one of your queries seems to take forever?  And then the next query you run takes milliseconds?  This would frequently happen to me, and I thought it meant that the server that hosted the database was unreliable in some way.  But this week I learned about indexes, and that the way we structure our queries has a huge impact on how long they will take to execute!  In this post, I’ll describe what indexes are and talk about the ways we can use them to optimize our queries.

An index is a database structure that is designed to speed up queries in a table.  An easy way to understand this is to think about the index at the back of a book.  Let’s say you have a book on car repair, and you want to find information about your car’s brakes.  You could look up “brakes” in the index, or you could search through every single page of the book for the word “brakes”.  It’s pretty obvious which would take less time!

Unlike books, databases can have more than one index.  There are two different kinds of indexes: clustered and unclustered.  A clustered index is used to store a table in sorted order.  There can only be one clustered index, because the table is stored in only one order.  Unclustered indexes are stored in the original table order, but they save the location of certain fields in the table.

Let’s take a look at an example.  If we had a table like this, called the Users table:

UserId State LastName FirstName Email Mobile Phone
1 MA Prunewhip Prunella [email protected] 800-867-5309
2 RI Schmoe Joe [email protected] 401-555-8765
3 NH Smith Amy [email protected] 603-555-3635
4 RI Jones Bob [email protected] 401-555-2344
5 MA Jones Amy [email protected] 617-555-2310

and we had a clustered index defined to have UserId as the key, a search on UserId would be very fast, and the data returned would be in order by UserId.

The table could also use unclustered indexes, such as the following:

State- the records in the table are indexed by state
LastNameFirstName- the records in the table are indexed by LastName and FirstName
When you query a database, the query will first look to see if an index can be used to speed up the search.  For example, if I made the request 
select LastName, FirstName from Users where UserId = 5 
the query would use the UserId index and the LastNameFirstName index to find the record.
Similarly, if I made the request
select LastName, FirstName from Users where State = ‘MA’
the query would use the LastNameFirstName index and the State index to find the record.
Of course, with a table of only five records, optimizing in this way won’t make much of a difference.  But imagine that this table had five million records, and you can see how using an index would be very helpful.

Querying a table on a non-indexed field is called a table scan.  The query needs to search through the entire table for the values, just as a person who wasn’t using a book index would have to search through every single page of the book.  

How can you know what indexes a table has?  You can find out with one simple query:
EXEC sp_helpindex “Users” 
where you would replace “Users” with whatever the name of the table is.  This will return a result of all of the clustered and unclustered indexes applied to the table, and the result will include the name of the index, a description of the index, and all the keys used in the index.
If you want to optimize your SQL queries, only ask for the data that you really need, rather than asking for select *.  Because not every field in the table is indexed, looking for every field will take longer.  
Let’s say that you want to query the Users table to find the email addresses of all of the users who live in Massachusetts (MA).  But you also would like to have some more information about those users.  You could ask for 
select FirstName, LastName, Email from Users where State = ‘MA’.
To find the records, the query will use the FirstNameLastName index and the State index.  Only the Email will be a non-indexed field.
But if you asked for
select * from Users where State = ‘MA’
now the query needs to look for two different non-indexed fields: Email and Mobile Phone.
Another helpful tip is to specify all the keys in an index when you want to use that index to make a query.  For example, if you wanted to find the Email for Prunella Prunewhip, you should ask for 
select Email from Users where LastName = ‘Prunewhip’ and FirstName = ‘Prunella’
rather than asking for
select Email from Users where LastName = ‘Prunewhip’.
In the second example, the LastNameFirstName index won’t be used.

And when you want to use an index, the query will run faster if you specify the keys in the order they appear, so it’s better to say
where LastName = ‘Prunewhip’ and FirstName = ‘Prunella’
than it is to say
where FirstName = ‘Prunella’ and LastName = ‘Prunewhip’

Here’s one more tip: when you want to use an index, be sure not to manipulate one of the index keys in your query, because this will mean that the index won’t be used.  For example, if you had a table like this, called Grades:

StudentId LastName FirstName Grade
1 Miller Kara 89
2 Smith Carol 56
3 Jones Bob 99
4 Davis Frank 78
5 Green Doug 65
and you had an unclustered index called LastNameGrade, and you executed a query like:
select LastName from Grades where (Grade + 100) = 178
the LastNameGrade index wouldn’t be used, because the Grade value was being manipulated.  It’s necessary for the query to go through the entire table and add 100 to each Grade field in order to search for the correct value.

Armed with this knowledge, you should be able to create queries that will run as fast as possible, getting you the data you need.  I’d like to extend my thanks to my colleague Cindy Gall, whose informative workshop inspired this post!

Six Ways Chrome DevTools Can Help With Testing

Did you know that there is a wealth of testing tools right in your browser?  Web browsers like Chrome and Firefox have developer tools that are available for free, for everyone.  And these tools are not just for developers!  In this post, I’ll be sharing six ways that Chrome DevTools can help you with your testing.

To access Chrome DevTools, simply click on the three-dot menu in the upper right corner of your browser, choose “More Tools”, and then choose “Developer Tools”.  DevTools will open up alongside your browser window.  You can customize where you would like the tools to display by clicking on the three-dot menu in the DevTools nav bar and selecting an option for “Dock Side”.  You can choose to have the DevTools display on the left, on the right, on the bottom, or in a separate window.

Here are some of the things that Dev Tools can do:

1. Inspect an HTML Element
Have you ever been writing UI automation and you just can’t figure out how to access an element?  With DevTools, you can right-click on the element and choose “Inspect”, and the Elements pane of DevTools will show you the element in the HTML.  You can then use this information to figure out the best way to access the element.

2. Edit HTML Elements
Not only can you find an element in the HTML, you can also edit it!  This is great for security testing.  Imagine that there is a page with a button that is hidden for users who are not admins.  A malicious user could find that element using DevTools, remove the “hide” tag, and use the button.  So it’s helpful to try this while testing to verify that there’s an additional check for user permissions when the button is used.

To edit an element, right-click on it in the HTML displayed in the Elements pane, and choose “Edit as HTML”.  Make whatever edits to the element you want, then click out of the edit box.  You should see the element on the page change as a result of your edits.

3. View HTTP requests
If you click on the Network tab of DevTools, you can see all of the requests made to the server while using a web page.  This includes API calls, which you can then copy and use in a tool like Postman.  This feature is helpful for determining if your page is making the API calls that you are expecting, and it’s also great for security testing.  For example, just because the front-end of a web page doesn’t allow a user to submit a field with more than 50 characters doesn’t mean that it can’t be done.  If a malicious user copies the API call and submits it through Postman, through a curl command, or through some other tool, they may be able to send more than 50 characters directly to the server.  This is why it’s important to have both front-end and back-end validation on a website.

4. Simulate device frames
When you are testing a webpage, it’s important to make sure that the page appears correctly on both laptops and mobile devices.  But even the most well-equipped tester doesn’t have access to every single device in use today.  So DevTools comes with a simulator that shows roughly what your webpage will look like on various devices.  To access this feature, click on the device logo

in the toolbar.  This will open the simulator in the webpage side of the browser.  Then you can use the dropdown to select specific devices (which seem to be a bit obsolete), or you can choose the “Responsive” setting and then manually expand or contract the window to get the size you want.  The exact size is displayed in the navbar at the top.

5. Simulate performance on slower networks
Testing a webpage while in your office usually means you are using a great high-speed network.  But what about your users who have slower connections?  You can use DevTools to simulate slower connections and throttled CPU, which could help uncover race conditions in your application.  To use this feature, go to the Performance tab in the navbar.  In the Network dropdown, you can choose “Fast 3G”, “Slow 3G”, or “Offline”, and in the CPU dropdown, you can choose “No throttling”, “4x slowdown” or “6x slowdown”.  Don’t forget to reverse your changes when you are done testing!

6. Investigate page load errors
As I was creating this post, I was reminded of one more way that DevTools are helpful.  I was trying to upload the Chrome logo to my post, and the popup that I usually use to add an image was completely blank.  I went to the Console tab of DevTools and saw that there was a 404 “File not found” error when I clicked on the Add Images button in Blogger.  When you are testing your team’s application and you’ve found a bug on a page, checking for errors in the console can help you give more information to your developers so they can get to the root of the problem more quickly.

Sometimes the most useful testing tools are right there in front of you!  I hope this post has inspired you to take a look at DevTools to see how it can help you in your testing.

The Power of Not Knowing

Recently I saw a tweet from Ben Simo (@QualityFrog) that mentioned that he sometimes likes to practice what he calls “intentional ignorance”- where he doesn’t read some of the documentation or code for a new feature to see what he can find while doing exploratory testing.  His tweet reminded me that I used to do this too!

I haven’t done this in a while, because the team I work on is a great Agile team.  The testers are invited to the feature grooming sessions, each story has acceptance criteria written, and the developers do a feature handoff with the testers when each story is ready for testing.

But at previous companies, I was often given a story to test with no feature handoff and no acceptance criteria.  Sometimes the story wouldn’t even have a description, and would have some cryptic title, like “Endpoint for search”.  I would usually be annoyed by this, and I would ask for clarification, but I would first use it as an opportunity to do some exploratory testing while I had no pre-conceived notions what the feature could do or not do.   And while testing in this fashion, I would often find a bug, show it to the developer, and have him or her say, “Oh, it never even occurred to me to test the feature in that way.”

Of course I don’t want to go back to the days of cryptic story titles with no description!  But testing without knowing what the feature does can have some benefits:
  • You approach the application the same way a user would.  When your users see your new feature for the first time, they don’t have the benefit of instructions.  By trying out the feature without knowing how it works, you could discover that an action button is hard to find, or that it’s difficult to know what to do first on a multi-part form.  
  • You might try entering data that no one was expecting.  For example, there could be a form field where the date was supposed to be entered with month and day only, but you enter in the month, day, and year, which breaks the form.  
  • Without any instructions from the developer, you might think of other features to test the new feature with, besides those the developer thought of.  Those feature combinations might yield new bugs.

So how can we add these advantages back into our testing without skipping reading the acceptance criteria and having feature handoffs?  Here are a few ways:

  • Pair test with someone on another team.  At my company we have many teams, each of which often has no idea what the other teams are building.  Four times a year, the software testers get together in pairs where the two testers are from very different teams, and they swap applications and start testing.  This is a great way to find bugs and user experience issues!
  • When you start testing, spend some time just playing around with the new feature before writing a test plan.  By exploring in this way, you might come up with some unusual testing ideas.
  • After you’ve tested the acceptance criteria, take some time to think about what features might be used with the new feature.  What happens when you test them together?  For example, if you were testing a new page of data, you could test it with the global sort feature that already exists in your application.
Of course, there are also times where not knowing all the details about a feature is detrimental.  There have been times in my testing career where I tested a feature and completely missed something that the feature could do, because no one told me about it.  That’s why I’m glad that we have acceptance criteria and feature handoffs.  But there are also times when not knowing can yield some of the most interesting bugs.

Your Flaky Tests Are Destroying Trust

Anyone who has ever written an automated test has experienced test flakiness.  There are many reasons for flaky tests, including:

  • Environmental issues, such as the application being unavailable
  • Test data issues, where an expected value has been changed
  • UI issues, such as a popup window taking too long to appear

All of these reasons are valid explanations for flaky tests.  However, they are not excuses!  It should be your mission to have all of your automated tests pass every single day, except of course when an actual bug is present.

This is important not just because you want your tests to be reliable; it’s important because when you have flaky tests, trust in you and in your team is eroded.  Here’s why:

Flaky tests send the message that you don’t care
Let’s say you are the sole automation engineer on a team, and you have a bunch of flaky tests.  It’s your job to write test automation that actually checks that your product is running correctly, and because your tests are flaky, your automation doesn’t do that.  Your team may assume that this is because you don’t care about whether your job is done properly.

Flaky tests make your team suspect your competence
An even worse situation than the previous example is one where your team simply assumes that you haven’t fixed the flaky tests because you don’t know how.  This further erodes their trust in you, which may spill over into other testing.  If you find a bug when you are doing exploratory testing your colleagues might not believe that you have a bug, because they think you are technically incompetent.

Flaky tests waste everyone’s time
If you are part of a large company where each team contributes one part of an application, other teams will rely on your automation to determine whether the code they committed works with your team’s code.  If your tests are failing for no reason, people on other teams will need to stop what they are doing and troubleshoot your tests.  They won’t be pleased if they discover that there’s nothing wrong with the app and your tests are just being flaky.

Flaky tests breed distrust between teams
If your team has a bunch of flaky tests that fail for no good reason, and you aren’t actively taking steps to fix them, other teams will ignore your tests, and may also doubt whether your team can be relied upon.  In a situation like this, if Team B commits code and sees that Team A has failing tests, they may do nothing about it, and may not even ask Team A about the failures.  If there are tests that fail because there are real issues, your teams might not discover them until days later.

Flaky tests send a bad message your company’s leadership 
There’s nothing worse for a test team than to have test automation where only 80% (or less) of the tests pass on a daily basis.  This sends a message to management that either test automation is unreliable, or you are unreliable!

So, what can we do about flaky tests?  I’d like to recommend these steps:

1. Make a commitment to having 100% of your tests pass every day.  The only time a test should fail is if a legitimate bug is present.  Some might argue that this is an impossible dream, but it is one to strive for.  There is no such thing as perfect software, or perfect tests, but we can work as hard as we can to get as close as we can to that perfection.

2. Set up alerts that notify you of test failures.  Having tests that detect problems in your software doesn’t help if no one is alerted when test failures happen.  Set up an alert system that will notify you via email or chat when a test is failing.  Also, make sure that you test your alert.  Don’t assume that because the alert is in place it is automatically working.  Make a change that will cause a test to fail and check to see if you got the notification.

3. Investigate every test failure and find out why it failed.  If the failure wasn’t due to a legitimate bug, what caused the failure?  Will the test pass if you run it again, or does it fail every time?  Will the test pass if you run it manually?  Is your test data correct?  Are there problems with the test environment?

4. Remove the flaky tests.  Some might argue that this is a bad idea because you are losing test coverage, and the test passes sometimes.  But this doesn’t matter, because when people see that the test is flaky they won’t trust it anyway.  It’s better to remove the flaky tests altogether so that you demonstrate that you have a 100% passing rate, and others will begin to trust your tests.

An alternative would be to set the flaky tests to be skipped, but this might also erode trust.  People might see all the skipped tests and see them as a sign that you don’t write good test automation.  Furthermore, you might forget to fix the skipped tests.

5. Fix all the flaky tests you can.  How you fix the flaky tests will depend on why they are flaky.  If you have tests that are flaky because someone keeps changing your test data, change your tests so that the test data is set up in the test itself.  If you have tests that are flaky because sometimes your test assets aren’t deleted at the end of the test, do a data cleanup both before and after the test.

6. Ask for help.  If your tests are flaky because the environment where they are running is unreliable, talk to the team that’s responsible for maintaining the environment.  See if there’s something they can to do solve the problem.  If they are unresponsive, find out if other teams are experiencing the issue, and lobby together to make a change.

7. Test your functionality in a different way.  If your flaky test is failing because of some element on the page that isn’t loading on time, don’t try to solve the issue by making your waits longer.  See if you can come up with a different way to test the feature.  For example, you might be able to switch that test to an API test.  Or you might be able to verify that a record was added in the database instead of going through UI.  Or you might be able to verify the data on a different page, instead of the one with the slow element.

Some might say that not testing the UI on that problematic page is dangerous.  But having a flaky test on this page is even more dangerous, because people will just ignore the test.  It would be better to stick with an automated test that works, and do an occasional manual test of that page.

Quality Automation is Our Responsibility

We’ve all been in situations where we have been dismissed as irrelevant or incompetent because of the reputation of a few bad testers.  Let’s create a culture of excellence for testers everywhere by making sure that EVERY test we run is reliable and provides value!

Why You Should Be Testing in Production

This is a true story; I’m keeping the details vague to protect those involved.  Once there was a software team that was implementing new functionality.  They tested the new functionality in their QA environment, and it worked just fine.  So they scheduled a deployment: first to the Staging environment, then to Production.  They didn’t have any automated tests for the new feature, because it was tricky to automate.  And they didn’t bother to do any manual tests in Staging or Production, reasoning that if it worked in the QA environment, it must work everywhere.

You can probably guess what happened next- they started getting calls from customers that the new feature didn’t work.  They investigated and found that this was true.  Then they tried out the feature in the Staging environment and found that it didn’t work there either.  As it turned out, the team had used hard-coded configuration strings that were only valid in the QA environment.  If they had simply done ONE test in the Staging or Production environment, they would have noticed that something was wrong.  Instead, it was left to the customers to notice the problem.

There are two main reasons why things that work in a QA environment don’t work in a Production environment:

1) Configuration problems- This is what happened with the team described above.  Software is complicated, and there are often multiple servers and databases that need to talk to each other in order for the software to work properly.  Keeping software secure means that each part of the application needs to be protected by passwords or other configuration strings.  If any one of those strings is incorrect, the software won’t work completely.

2) Deployment problems- In this age of microservices, deploying software usually means deploying several different APIs.  In a large organization, there may be different teams responsible for different APIs.  For example, when a new feature in API A needs the new code in API B to work properly, API B will need to be deployed first.  It’s possible that Team B will forget to deploy API B or not even realize that it needs to be deployed.  In cases like this, Team A might assume that API B had been deployed, and they will go ahead and deploy API A.  Without testing, Team A will have no way of knowing that the new feature isn’t working.

By running tests in every environment, you can quickly discover if you have configuration or deployment problems.  It’s often not necessary to go through extensive testing of a new feature in Production if you’ve already tested it in QA, but it is vital that you do at least SOME testing to verify that it’s working!  We never want to have our customers find problems before we do.