Meet Derek and Emma. They are both Software Test Engineers. Derek works for a company called ContactCo, which is building a web application to allow users to add and manage their contacts. Emma works for a competitor of ContactCo, called ContactsRUs. ContactsRUs is building a similar application to the one ContactCo is buildling.
Emma is very proud of her ability to create test automation frameworks. As soon as development begins on the new app, she gets to work on a UI automation suite. She writes dozens of well-organized automated tests, and sets them to run with every build that the developers check in. The tests are all passing, so she feels really good about the health of the application. She also creates a set of smoke tests that will run with every deploy to every environment. If the smoke tests pass, the deployment will automatically continue to the next environment, all the way through Production; if the tests fail, the deployment will be rolled back and the deployment process will stop. After just three weeks, she’s got a CI/CD system in place, and everyone praises her for the great job she’s done.
Derek begins his involvement with ContactCo’s new app by attending the product design meetings and asking questions. He reads through the user stories so he understands the end user and knows what kinds of actions they’ll be taking. As the application takes shape, he does lots of manual exploratory testing, both with the API and the UI. He tries out the application on various browsers and with various screen sizes. At the end of the first two weeks of development, he’s found several UI and API bugs that the developers have fixed.
Next, Derek works with developers to find out what unit and integration tests they currently have running, and suggests some tests that might be missing. He talks with the whole team to determine what the best automated framework would be for API and UI testing, and works with them to get it set up. He spends a lot of time thinking about which tests should run with the build, and which should run with the deployment; and he thinks about which tests should be run solely against the API in order to minimize the amount of UI automation. Once he has a good test strategy, he starts writing his automated tests. At the end of the third week of development, he’s got some automated tests written, but he’s planning to add more, and he doesn’t quite have the CI/CD process set up yet.
At the end of the three weeks, both ContactCo and ContactsRUs release their applications to Production. Which application do you think will be more successful? Read on to find out!
Derek’s application at ContactCo is a big hit with users. They comment on how intuitive the user interface is, and by the end of the first week, no bugs have been reported. Customers have suggestions for features they’d like to see added to the application, and the team at ContactCo gets started with a new round of product design meetings, which Derek attends. When he’s not in meetings, he continues to work on adding to the automated test framework and setting up CI/CD.
Emma’s application at ContactsRUs was released to Production, and the very same day the company started to get calls from customers. Most of the ContactsRUs customers use the Edge browser, and it turns out there are a number of rendering issues on that browser that Emma didn’t catch. Why didn’t she catch them? Because she never tested in Edge!
The next day the company receives a report that users are able to see contacts belonging to other customers. Emma thinks that this can’t be possible, because she has several UI tests that log in as various users, and she’s verified that they can’t see each other’s data. It turns out that there’s a security hole; if a customer makes an API call to get a list of contacts, ALL of the contacts are returned, not just the contacts associated with their login. Emma never checked out the API, so she missed this critical bug.
Developers work late into the night to fix the security hole before anyone can exploit it. They’ve already lost some of their customers because of this, but they release the fix and hope that this will be the last of their problems. Unfortunately, on the third day, Emma gets an angry message from the team’s Product Owner that the Search function doesn’t work. “Of course it works,” replies Emma. “I have an automated test that shows that it works.” When Emma and the Product Owner investigate, they discover that the Search function works fine with letters, but doesn’t work with numbers, so customer’s can’t search their contacts by phone number. This was a critical use case for the application, but Emma didn’t know that because she didn’t attend the product meetings and didn’t pay attention to the feature’s Acceptance Criteria. As a result, they lose a few more customers who were counting on this feature to work for them.
The Moral(s) of the Story
Were you surprised by what happened to ContactsRUs? It might have seemed that they’d be successful because they implemented CI/CD so quickly into their application. But CI/CD doesn’t matter if you neglect these two important steps:
- Understand the product you are testing. Know who your end users are, what they want from the product, and how they will be using it. Pay attention in planning meetings and participate in the creation of Acceptance Criteria for development stories.
- Look for bugs in the product. Many software test engineers jump right to automation without remembering that their primary role is to FIND THE BUGS. If there are bugs in your product, the end users aren’t going to care about your really well-organized code!
Every good fable deserves a happy ending! Hopefully you have learned from Derek and Emma and will make sure that you understand and test your software before writing good automation.