Mobile Testing Part III: Seven Automated Mobile Testing Tips (and Five Great Tools)

Walk into any mobile carrier store and you will see a wide range of mobile devices for sale.  Of course you want to make sure that your application works well on all of those devices, in addition to the older devices that some users have.  But running even the simplest of manual tests on a phone or tablet takes time.  Multiply that time by the number of devices you want to support, and you’ve got a huge testing burden!

This is where automated mobile testing comes in.  We are fortunate to be testing at a time where there are a whole range of products and services to help us automate our mobile tests.  Later in this article, I will discuss five of them.  But first, let’s take a look at seven tips to help you be successful with mobile automated testing.

Tip 1: Don’t test things on mobile that could be more easily tested elsewhere
Mobile automation is not the place to test your back-end services.  It’s also not the place to test the general logic of the application, unless your application is mobile only.  Mobile testing should be used for verifying that elements appear correctly on the device and function correctly when used.  For example, let’s say you have a sign-up form in your application.  In your mobile testing, you’ll want to make sure that the form renders correctly, that all fields can be filled in, that error messages display appropriately, and that the Save button submits the form when clicked on.  But you don’t want to test that the error message has the correct text, or that the fields have all saved correctly.  You can save those tests for standard Web browser or API automation.
Tip 2: Decide whether you want to run your tests on real devices or emulators
The advantage of running your tests on real devices is that the devices will behave like the devices your users own, with the possibility of having a low battery, connectivity issues, or other applications running.  But because of this, it’s more likely that your tests will fail because a phone in the device farm froze up or was being used by another tester.  Annoyances like these can be avoided by using emulators, but emulators can’t completely mimic the real user experience.  It’s up to you decide which choice is more appropriate for your application.  
Tip 3: Test only one thing at a time
Mobile tests can be flaky, due to the issues found in real devices discussed above and other issues such as the variations found in different phones and tablets.  You may find yourself spending a fair amount of time taking a look at your failed tests and diagnosing why they failed.  Because of this, it’s a good strategy to keep your tests small.  For example, if you were testing a login screen, you could have one test for a successful login and a second test for an unsuccessful login, instead of putting both scenarios into the same test.
Tip 4: Be prepared to re-run tests
As mentioned in Tip 3, you will probably encounter some flakiness in your mobile tests.  A test can fail simply because the service that is hosting the emulator loses connectivity for a moment.  Because of this, you may want to set up a system where your tests run once and then re-run the failed tests automatically.  You can then set up an alert that will notify you only if a test has failed twice.
Tip 5: Don’t feel like you have to test every device in existence

As testers, we love to be thorough.  We love to come up with every possible permutation in testing and run through them all.  But in the mobile space, this can quickly drive you crazy!  The more devices you are running your automated tests on, the more failures you will have.  The more failures you have, the more time you have to spend diagnosing those issues.  This is time taken away from new feature testing or exploratory testing.  Do some research on which devices your users own and come up with a list of devices to test with that covers most, but not all, of those devices.  
Tip 6: Take screenshots
Nothing is more frustrating than seeing that a test failed and not being able to figure out why.  Screenshots can help you determine if you were on the correct screen during a test step and if all the elements are visible.  Some mobile testing companies take a screenshot of every test step as the test progresses.  Others automatically take a screenshot of the last view before a test fails.  You can also code your test to take screenshots of specific test steps.  
Tip 7: Use visual validation

Visual validation is essential in mobile testing.  Many of the bugs you will encounter will be elements not rendering correctly on the screen.  You can test for the presence of an element, but unless you have a way to compare a screenshot with one you have on file, you won’t really be verifying that your elements are visible to the user.  Applitools makes an excellent product for visual comparison.  It integrates with common test software such as Selenium, Appium, and Protractor.  With Applitools, you can build visual verification right into your tests and save a collection of screenshots from every device you test with to use for image comparison. 

Now let’s discuss some good tools for test automation.  I’ve already mentioned Applitools; below are four other tools that are great for mobile test automation.  The mobile landscape is filled with products for automated testing, both open-source and paid.  In this post, I am discussing only the products that I have used; there are many other great products out there. 

Visual Studio App Center:  A Microsoft product that allows you to test Android and iOS applications on real devices.  A screenshot is taken of every test step, which makes it easy to figure out where a test started to go wrong. 

Appium:  An open-source product that integrates with Selenium and provides the capability to test on device emulators (or real devices if you integrate with a device farm). 

Sauce Labs:  Sauce Labs is great for testing on both mobile devices and web browsers on all kinds of operating systems.  You can run tests on real devices or emulators, and you can run tests in parallel.  They integrate well with Selenium and Appium.  A screenshot is taken whenever a test fails, and you can even watch a video of your test execution.

Perfecto: Uses real devices and integrates with Visual Studio, Appium, and Selenium.  They can simulate real-world user conditions such as network availability and location.

Whatever automated test tools you choose to use, remember the tips above, and you will ensure that you are comprehensively testing your application on mobile without spending a lot of time debugging. 

I initially said this series on Mobile Testing was going to be three blog posts long.  On reflection, I’ve realized that we need a fourth post: Mobile Security Testing.  This is a topic I know very little about.  So I’ll be doing some research, and you can expect Mobile Testing Part IV from me next week!

Mobile Testing Part II: Manual Mobile Testing

I am a firm believer that no matter how great virtual devices and automated tests are, you should always do some mobile testing with a physical device in your hand.  But none of us has the resources to acquire every possible mobile device with every possible carrier.  So today’s post will discuss how to assemble a mobile device portfolio that meets your minimum testing criteria, and how to get your mobile testing done on other physical devices.  We’ll also talk about the manual tests that should be part of every mobile test plan.

Every company is different, and will have a different budget available for acquiring mobile devices.  Here is an example of how I would decide on which phones to buy if I was allowed to purchase no more than ten.  I am based in the United States, so I would be thinking about US carriers.  I would want to make sure that I had at least one AT&T, Verizon, T-Mobile, and Sprint device in my portfolio.  I would also want to have a wifi-only device.  I would want to make sure that I had at least one iOS device and at least one Android device.  For OS versions, I’d want to have the both the latest OS version and the next-to-latest OS version for each operating system.  For Android devices, I’d want to have Samsung, LG, and Motorola represented, because these are the most popular Android devices in the US.  Finally, I would want to make sure that I had at least one tablet for each operating system.

With those stipulations in mind, I would create a list of devices like this:

In this portfolio, we have three iOS devices and six Android devices.  All four carriers I wanted are represented, and we have one wifi only device.  We have three tablets and six smartphones.  We have the latest iOS and Android versions, and the next-to-latest versions.  And we also have a variety of screen sizes.  It’s easy to modify a device plan like this if for some reason devices aren’t available.  For example, if I went to purchase these devices and found that Sprint wasn’t carrying the iPhone X, I could easily switch my plan around so I could get an iPhone X from AT&T and an iPhone 8 Plus from Sprint instead. 

The benefit of having a physical device portfolio is that you can add to it every year as your budget allows. Each year you can purchase a new set of devices with the latest OS version, and you can keep your old devices on the older OS versions, thus expanding the range of OS versions you can test with.

Once you have a device portfolio, you’ll need to make sure you are building good mobile tests into your test plans.  You can add the following tests:

  • Test the application in the mobile browser, in addition to testing the native app
  • Test in portrait and landscape modes, switching back and forth between the two
  • Change from using the network to using wifi, to using no service, and back again
  • Test any in-app links and social media features
  • Set the phone or device timer to go off during your testing
  • Set text messages or low battery warnings to come in during your testing
What about testing on the dozens of devices that you don’t have?  This is where device farms come in.  A device farm is made of many physical devices housed in one location that you can access through the Web.  From your computer, you can access the device controls such as the Home or Back buttons, swipe left and right on the screen, and click on the controls in your application.  You may even be able to do things like rotate the device and receive a phone call.  With a device farm, you can expand the range of devices you are testing on.  Good ideas for expanding your test plan would be adding devices with older OS versions, and adding devices from manufacturers that you don’t have in your portfolio.  In my case above, this might mean adding in HTC and Huawei devices.  
For manual device farm testing, I have had good experiences with Perfecto.  Other popular device farms with manual testing capabilities are AWS, Sauce Labs, and Browserstack.
You may be saying to yourself at this point, “You’ve got some great devices and carriers for US testing, but my users come from all over the world.  How can I make sure that they are all having a good user experience with my app?”  This is where crowd-testing comes in!  There are testing companies that specialize in using testers from many countries, who are using devices with their local carriers.  They can test your application in their own time zone on a device in their own language.  Popular global test companies include Testlio and Global App Testing.  Another good resource is uTest, which matches up independent testers with companies who are looking for testing on specific devices in specific countries.  
With a mobile device portfolio, a mobile test plan, a device farm, and a crowd-testing service in place, you will be able to to execute a comprehensive suite of tests on your application and ensure a great user experience worldwide.  But all of this manual testing takes a lot of time!  Next week, we’ll discuss how to save time and maximize your mobile test coverage through automated mobile testing.  

Mobile Testing Part I: Twelve Challenges of Mobile

Just over a decade ago, the first iPhone was released.  Now we live in an age where smartphones are ubiquitous.  Our smartphones are like our Swiss Army knives- they are our maps, our address books, our calendars, our cameras, our music players, and of course our communication devices.  Testing software would not be complete without testing on mobile.  There are so many considerations when testing on mobile that I’ve decided to make this into a three part series: this week I’ll discuss the challenges of mobile testing, next week I’ll discuss mobile manual testing, and the following week I’ll talk about mobile automated testing. 

First, the challenges!  Below are twelve reasons why testing on mobile is difficult.  I thought it would be fun to illustrate just what can go wrong with mobile software by describing a bug I’ve found in each area.  Some of these bugs were found in the course of my testing career, and some were found on my personal device as an end user.
1. Carriers: Mobile application performance can vary depending on what carrier the device is using.  In the US, the two major carriers are Verizon and AT&T, and we also have smaller carriers like Sprint and T-Mobile.  In Europe some of the major carriers are Deutche Telekom, Telefonica, Vodaphone, and Orange; in Asia some of the carriers are China Mobile, Airtel, NTT, and Softbank.  When testing software on mobile, it’s important to consider what carriers your end users will be using, and test with those carriers.
Example bug: I once tested a mapping function within an application, and discovered that while the map would update based on my location when I was using one carrier, it did not update when I was using a different carrier.  This had something to do with the way the location was cached after a cell tower ping.  
2. Network or Wifi: Device users have the choice of using their applications while connected to the carrier’s network, or while on wifi.  They can even make a choice to change how they are connecting in the middle of using the application; or their connection can be cut completely if they go out of network range.  It’s important to test an application when connected to a network and when connected to wifi, and to see what happens when the connection changes or is lost completely.
Example bug: I have a wifi extender in my house. When I switch my phone’s wifi connection to use the extender’s IP, Spotify thinks I am offline.  I have to force-close the app and reopen it in order for Spotify to recognize that I am online.
3. Application Type: Mobile applications can be Web-based, native, or a hybrid of the two (developed like a Web app, but installed like a native app).  Some of your end users will choose not to use a native or hybrid app and will prefer to interact with your application in their phone’s browser.  There are also a variety of mobile browsers that could be used, such as Safari, Chrome, or Opera.  So it’s important to make sure that your web application works well on a variety of mobile browsers.
Example bug: There have been many times where I’ve gone to a mobile website and their “mobile-optimized” site doesn’t have the functionality I need.  I’ve had to choose to go to the full site, where all of the text is tiny and navigation is difficult.  
4. Operating System: Mobile applications will function differently depending on the operating system.  The two biggest operating systems are iOS and Android, and there are others, such as Windows Mobile and Blackberry.  It’s important to test on whatever operating systems your end users will be using, to make sure that all of the features in the application are supported in all systems.
Example bug: This is not a bug, but a key difference between Android and iOS: Android devices have a back button, while iOS devices do not.  Applications written for iOS need to have a back button included on each page so users will have the option to move back to the previous page.
5. Version: Every OS updates their version periodically, with new features designed to entice users to upgrade.  But not every user will upgrade their phone to the latest and greatest version.  It’s important to use analytics to determine which versions your users are most likely to have, and make sure that you are testing on those versions.  Also, every version update has the potential to create bugs in your application that weren’t there before.  
Example bug: Often when the version is updated on my phone, I can no longer use the speaker function when making phone calls.  I can hear the voice on the other end, but the caller can’t hear me.
6. Make: While all iOS devices are manufactured by Apple, Android devices are not so simple.  Samsung is one of the major Android device manufacturers, but there are many others, such as Huawei, Motorola, Asus, and LG.  It’s important to note that not every Android user will be using a Samsung device, and test on other Android devices as well.
Example bug: I once tested a tablet application where the keyboard function worked fine on some makes but not others. The keyboard simply wouldn’t pop up on those devices, so I wasn’t able to type in any form fields.
7. Model: Similar to versioning, new models of devices are introduced annually.  While some users will upgrade every year or two to the latest device, others will not.  Moreover, some devices will not be able to upgrade to the latest version of the OS, so they will be out-of-date in two ways.  Again, it’s important to find out what models your end users are using so you can make decisions about which models to test on and to support.
Example bug: This is not a bug, but it was an important consideration: when Apple released a new model of the iPad that would allow a signature control for users to sign their name in a form, the software I was testing included this feature.  But older versions of the iPad weren’t able to support this, so the application needed to account for this and not ask users on older versions to sign a document.
8. Tablet or Smartphone:  Many of your end users will be interacting with your application on a tablet rather than a smartphone.  Native applications will often have different app versions depending on whether they are designed for tablet or phone.  An application designed for smartphone can often be downloaded to a tablet, but an application designed for a tablet cannot be installed on a smartphone.  If a web app is being used, it’s important to remember that tablets and smartphones sometimes have different features.  Test your application on both tablets and phones.
Example bug: I have tested applications that worked fine on a smartphone and simply gave me a blank screen when I tried to test them on a tablet.  
9. Screen Size:  Mobile devices come in many, many different sizes.  While iOS devices fit into a few sizing standards, Android devices have dozens of sizes. Although it’s impossible to test every screen size, it’s important to test small, medium, large, and extra large sizes to make sure that your application draws correctly in every resolution.  
Example bug: I have tested applications on small phones where the page elements were overlapping each other, making it difficult to see text fields or click on buttons.
10. Portrait or Landscape: When testing on smartphones, it’s easy to forget to test in landscape mode, because we often hold our phones in a portrait position.  But sometimes smartphone users will want to view an application in landscape mode, and this is even more true for tablet users.  It’s important to not only test your application in portrait and landscape modes, but also to be sure to switch back and forth between modes while using the application.  
Example bug: I tested an application once that looked great in a tablet when it was in portrait mode, but all of the fields disappeared when I moved to landscape mode.  

11. In-App Integration: One of the great things about mobile applications is that they can integrate with other features of the device, such as the microphone or camera.  They can also link to other applications, such as Facebook or Twitter.  Whatever integrations the application supports, be sure to test them thoroughly.  
Example bug:  I tested an application where it was possible for a user to take a picture of an appliance in their home and add it to their home’s inventory.  When I chose to take a picture, I was taken to the camera app correctly, and I was able to take the picture, but after I took the picture I wasn’t returned to the application.  
12. Outside of App Integration: Even if your application isn’t designed to work with any other apps or features, it’s still possible that there are bugs in this area.  What happens if the user gets a phone call, a text, or a low battery warning while they are using your app?  It’s important to find out.
Example bug:  For a while, there was a bug on my phone where if my device timer went off while I was in a phone call, as soon as I got off the phone, the timer sounded and wouldn’t stop.  
I hope that the above descriptions and examples have shown just how difficult it is to test mobile applications!  It may seem overwhelming at first, but in my next two blog posts, I’ll discuss ways to make testing simpler.  Next week, we’ll be taking a look at writing mobile test plans and assembling a portfolio of physical devices to test on.  

Cross-Browser Testing

In today’s Agile world, with two-week sprints and frequent releases, it’s tough to keep on top of testing.  We often have our hands full with testing the stories from the sprint, and we rely on automation for any regression testing.  But there is a key component of testing that is often overlooked, and that is cross-browser testing.

Browser parity is much better than it was just a few years ago, but every now and then you will still encounter differences with how your application performs in different browsers.  Here are just a few examples of discrepancies I’ve encountered over the years:

  • A page that scrolls just fine in one browser doesn’t scroll at all in another, or the scrolling component doesn’t appear
  • A button that works correctly in one browser doesn’t work in another
  • An image that displays in one browser doesn’t display in another
  • A page that automatically refreshes in one browser doesn’t do so in another, leaving the user feeling as if their data hasn’t been updated
Here are some helpful hints to make sure that your application is tested in multiple browsers:
Know which browser is most popular with your users
Several years ago I was testing a business-to-business CRM-style application.  Our team’s developers tended to use Chrome for checking their work, and because of this I primarily tested in Chrome as well.  Then I found out that over 90% of our end users were using our application in Internet Explorer 9.  This definitely changed the focus of my testing!  From then on, I made sure that every new feature was tested in IE 9, and that a full regression pass was run in IE 9 whenever we had a release.  
Find out which browsers are the most popular with your users and be sure to test every feature with them.  This doesn’t mean that you have to do the bulk of your testing there; but with every new feature and every new release you should be sure to validate all of the UI components in the most popular browsers.
Resize your browsers

Sometimes a browser issue isn’t readily apparent because it only appears when the browser is using a smaller window.  As professional testers, we are often fortunate to be issued large monitors on which to test.  This is great, because it allows us to have multiple windows open and view one whole webpage at a time, but it often means that we miss bugs that end users will see.  
End users are likely not using a big monitor when they are using our software.  Issues can crop up such as: a vertical or horizontal scrollbar not appearing, or not functioning properly; text not resizing, so that it goes off the page and is not visible; or images not appearing or taking too much space on the page.  
Be sure to build page resizing into every test plan for every new feature, and build it into a regression suite as well.  Find out what the minimum supported window size should be, and test all the way down to that level, with variations in both horizontal and vertical sizes.  
Assign each browser to a different tester

When doing manual regression testing, an easy way to make sure that all browsers you want to test are covered is by assigning each tester a different browser.  For example, if you have three testers on your team (including yourself), you could run your regression suite in Chrome and Safari, another tester could run the suite in Firefox, and a third tester could run the suite in Internet Explorer and Edge.  The next time the suite is run, you can swap browsers, so that each browser will have a fresh set of eyes on it.  
Watch for changes after browser updates

It’s possible that something that worked great in a browser suddenly stops working correctly when a new version of the browser is released.  It’s also possible that a feature that looks great in the latest version of the browser doesn’t work in an older version.  Many browsers like Chrome and Firefox are set to automatically update themselves with every release, but some end users may have turned this feature off, so you can’t assume that everyone is using the latest version.  It’s often helpful if you have a spare testing machine to keep browsers installed with the next-to-last release.  That way you can identify any discrepancies that may appear between the old browser version and the new.  
Use visual validation in your automated tests

Generally automated UI tests focus on the presence of elements on a web page.  This is great for functional testing, but the presence of an element doesn’t tell you whether or not it is appearing correctly on the page.  This is where a visual testing tool like Applitools comes in.  Applitools coordinates with UI test tools such as Selenium to add a visual component to the test validation.  In the first test run, Applitools is “taught” what image to expect on a page.  Then in all subsequent runs, it will take a screenshot of the image and compare with the correct image that it has saved.  If the image fails to load or is displaying incorrectly, the UI test will fail.  Applitools is great for cross-browser testing, because you can train it to expect different results for each browser type, version, and screen size.  
Browser differences are something that can greatly impact the user experience!  If you build in manual and automated systems to check for discrepancies, you can easily ensure a better user experience with a minimum of extra work.  
You may have noticed that I didn’t discuss the mobile experience at all in this post.  That’s because next week, I’ll be focusing solely on the many challenges of mobile testing!

A Gentle Introduction to Session Hijacking

We all know that Session Hijacking is bad, and that we should protect ourselves and our applications against it.  But it’s difficult to get easy-to-understand information about what it is, and how to test for it.  In this post, I’ll first describe the different types of session hijacking, and then I’ll provide a walkthrough on how to test for session hijacking using the OWASP Juice Shop and Burp Suite.

Session Hijacking refers to when a malicious user gets access to authentication information, and uses it to impersonate another user or gain access to information they should not have.  There are several types of Session Hijacking:

Predictable Session Token- this happens when the access tokens being generated by an application follow some kind of pattern.  For example, if a token granted for a login for a user was “APP123”, and a token granted for a second user was “APP124”, a malicious user could assume that the next token granted would be “APP125”.  This is a pretty obvious vulnerability, and there are many tools in use today that create non-sequential tokens, so it’s not a very common attack.

Session Sniffing- this takes place when a malicious user finds a way to examine the web traffic that is being sent between a user and a web server, and copies the token for their own use.  This is classified as a passive attack, because the malicious user is not interfering with the operation of the application or with the request.

Client-Side Attack- in this scenario, the malicious user is using XSS to cause an application to display a user’s token.  Then they copy the token and use it.

Man-in-the-Middle Attack- this attack is similar to Session Sniffing in that the malicious user gains access to web traffic.  But this is an active attack, because the malicious user uses a tool such as Burp Suite or Fiddler to intercept the request and then manipulate it for their purposes.

Man-in-the-Browser Attack- this takes place when a malicious user has managed to get code into another user’s computer.  The implanted code will then send all request information to the malicious user.

Now we are going to learn how test for Session Hijacking by using a Man-in-the-Middle Attack!  But first, it’s important to note that we are going to be intercepting requests by using the same computer where we are making the requests.  In a real Man-in-the-Middle Attack, the malicious user would be using some sort of packet-sniffing tool to gain access to requests that someone was making on a different computer.

The instructions provided here will be for Firefox.  Since I generally do my browsing (and blog-writing) on Chrome, I like to use a different browser for intercepting requests.  But Burp Suite will work with Chrome and Internet Explorer as well.

First, we’ll need to download and install Burp Suite.  You can install the free version, which is the Community Edition.  Don’t launch it yet.

Next, we’ll set up proxy settings in Firefox.  Click on the hamburger menu (a button with three horizontal lines) in the top right corner of the browser.  Choose “Preferences” and then scroll down to the bottom of the page and click on the “Settings” button.  Select the “Manual Proxy Configuration” radio button, and in the HTTP Proxy field, type “127.0.0.1”.  In the Port field, type “8080”.  Click the checkbox that says “Use this proxy server for all protocols”.  If there is anything in the “No proxy for” window, delete it.  Then click “OK”.

Now it’s time to start Burp Suite.  Start the application, then click “Next”, then click “Start Burp”.

Next we’ll set up the certificate that will give Firefox the permission to allow requests to be intercepted.  In Firefox, navigate to http://burp.  You’ll see a page appear that says “Burp Suite Community Edition”.  Click on the link in the upper right that says “CA Certificate”.  A popup window will appear; choose “Save File”.  The certificate will download, probably into your Downloads folder.

Click on the hamburger menu again, and choose “Preferences”.  In the menu on the left, choose “Privacy and Security”.  Scroll to the bottom of the page, and click the “View Certificates” button.  A popup window with a list of certificates will appear.  Click on the “Import” button, navigate to the “Downloads” folder, choose the recently downloaded certificate, and click the “Open” button.  A popup window will appear; check the “Trust this CA to identify websites” checkbox, and then click “OK”.  The certificate should now be installed.  Restart Firefox to ensure that the new settings are picked up.

Next, return to Burp Suite and turn the intercept function off.  We’re doing this so that we’re not intercepting web requests until we’re ready.  To turn off the intercept function, click on the Proxy tab in the top row of tabs, then click on the Intercept tab in the second row of tabs, then click on the “Intercept is on” button.  The button should now read “Intercept is off”.

Navigate to the Juice Shop, and create an account.  Once your account has been created, you’ll be prompted to log in.  Before you do so, go back to Burp Suite and turn the intercept function back on, using that “Intercept is off” button.

Now that Burp Suite is set to intercept your requests, log into the Juice Shop.  You will see nothing happen initially in your browser; this is because your request has gone to Burp Suite.  In Burp Suite, click on the Forward button; this will forward the intercepted request on to the server.  Continue to click the Forward button until the Search page of the Juice Shop has loaded completely.

In Burp Suite, click the HTTP History tab, in the second row of tabs.  You will see all the HTTP requests that were made when the Search page was loaded.  Scroll down until you see a POST request with the /rest/user/login endpoint.  Click on this request.  In the lower panel of Burp Suite, you will see the details of your request.  Notice that your username and password are displayed in clear text!  This means that if anyone were to intercept your login request, they would obtain your login credentials, and would then be able to log in as you at any time.

Next, return to the Juice Shop, and click on the cart symbol for the first juice listed in order to add the juice to your shopping cart.  Return to Burp Suite and click on the Intercept tab, and then click the Forward button to forward the request.  Continue to click the Forward button until no more requests are intercepted.

Return to the HTTP History tab in Burp Suite, and scroll down through the list of requests until you see a POST request with the api/BasketItems endpoint.  Right-click on this request and choose “Send to Repeater”.  This will send the request to the Repeater module where we can manipulate the request and send it again.  Return to the Intercept tab and turn the Intercept off.

Click on the Repeater tab, which is in the top row of tabs.  The request we intercepted when we added a juice to the shopping cart is there.  Let’s try sending the request again, by clicking on the Go button.  In the right panel of the page, we get a 400 response, with the message that the Product Id must be unique.  So, let’s return to the request in the left panel.  We can see that the body of the request is {“ProductId”:1,”BasketId”:”16″ (this number will vary by your basket id),”quantity”:1}.  Let’s change the ProductId from 1 to 2, and send the request again by clicking the Go button.  We can see in the response section that the request was successful.

Let’s return to the Juice Shop and see if we were really able to add an item to the cart by sending the request in Burp Suite.  Click on the Your Basket link.  You should see two juices in your cart.  This means that if someone were to intercept your request to add an item to your cart, they could manipulate the request and use it to add any other item they wanted to your cart.

How else might we manipulate this request?  Return to the Repeater tab in Burp Suite.  The request we intercepted is still there.  This time, let’s change the BasketId to a different number, such as 1.  Click Go to send the request again.  The response was successful, which means that we have just added a juice to someone else’s cart!

So, we can see that if a malicious user would be able to intercept a request to add an item to a shopping cart, they might be able to manipulate that request in all kinds of ways, to add unwanted items to the carts of any number of users.  They are able to do this because the request they intercepted has the Authorization needed to make more successful requests.  When you set up Burp Suite to intercept requests in your own application, you will be able to test for Session Hijacking vulnerabilities like this one.

This concludes my series of posts on Security Testing, although I will probably write new ones at some point in the future.  In the next few weeks, we’ll take a look at browser testing, mobile testing, and visual validation!

Hidden in Plain Sight- Using Dev Tools to Find Security Flaws

A common misconception is that all security testing is complicated.  While some testing certainly requires learning new skills and understanding things like networks, IP addresses, and domain names, other testing is extremely simple.  Today we’re going to talk about three security flaws you can find in an application by simply using your browser’s developer tools.  These flaws could be exploited by an average user of your application, not just a well-trained black-hat hacker.  

Editing Disabled Buttons: 
When you are on a web page that has a button that is disabled and only enables when certain criteria are met, such as filling out all the fields in a form, it may be possible to enable it.  I’ve included instructions on how to do this in my blog post on testing buttons.  
A user of your application might use this flaw to get around submitting a form where there are required fields that she doesn’t want to fill in.  Or a user might enable an Edit button and submit edits to data that he shouldn’t be able to edit.  
Viewing Hidden Data:
Recently someone showed me a security flaw in an application that was listing contact details for various members of the site.  Depending on the user’s access rules, there were certain fields for the members, such as their personal address, that were not displayed on the page. But when the developer tools were opened, all of the hidden fields were displayed in the Elements section!  
Any user of this application could open up the developer tools and search through them to find personal data for any of the members of the site.
Finding Hidden Pages:
It’s possible to find links that are not displayed on a web page by looking in the Elements section of the developer tools.  Let’s try this out using the OWASP Juice Shop.  I’ll be providing instructions using Chrome, but you can also do this with Firefox and Internet Explorer.  Navigate to the Juice Shop, then click on the three dots in the top right corner of the browser.  Select “More Tools” and then “Developer Tools”. This will open the Dev Tools section.  Click on the Elements tab if it is not already selected by default.  We’re going to take a look through the HTML of the page to see if we can find links that are not displayed in the browser.  Let’s get our bearings by clicking on a displayed element in the page.  On the Juice Shop main page, there is a nav bar on the top of the page, with links such as “About Us” and “Contact Us”.  Right-click on one of the elements in the nav bar, and choose “Inspect”.  Notice that this highlights the corresponding section of the HTML in Dev Tools.  You’ll see a number of items all tagged with the <li> tag; these are the items in the nav bar.  Open up each one by clicking on the carat on the left side, and you’ll find one that looks like this:
The status of this list item is hidden, as you can see by the “ng-hide” attribute.  The ‘ng-show=”isLoggedIn”‘ attribute tells us that this nav bar item should only display when the user is logged in.  Finally, we see the link that the nav bar item would take us to if we were to click on it when it was displayed: “#/recycle”.  Let’s try navigating to this link by changing the page’s URL from https://juice-shop.herokuapp.com/#/search to https://juice-shop.herokuapp.com/#/recycle.  Click Enter, and you will be taken to the recycle page.  You have successfully navigated to a page that you should only have access to if you have logged in!
This is why it is important to do authorization checks when a user navigates to a page.  It’s not enough to simply hide a link, because it’s so easy to find hidden links by looking in Dev Tools.  Any user could find hidden links in your application and navigate to them, which could give them access to an admin page or a page with data on other users.  
As you can see, testing for these types of security flaws is quick and easy!  I recommend checking for these flaws whenever you test a web page.  And here’s a bonus vulnerability; if you create an account and log into the Juice Shop, and look in the Network section of Dev Tools, you will see your username and password in plain text!  We’ll talk more about this next week, when we take on Session Hijacking.  

How to Craft a Union SQL Injection Attack

Last week, we learned how to craft a basic SQL injection attack.  This week, we’ll learn how to do something more elaborate.  We are going to use a simple search request to get a database to show us all of the records in a table of users!

It’s important to know how to craft attacks like these, because it helps you find vulnerabilities in your company’s application.  When you find vulnerabilities, they can be fixed before a malicious user exploits them.

In order to see a Union attack in action, we will be using the awesome OWASP Juice Shop website.  This site was created by Bjorn Kimminich, and it is so helpful in learning how to test for security vulnerabilities.  Bjorn has written a companion ebook for the site, which provides explanations of various security attacks, hints for the security challenges, and answers to the challenges if you get stuck.  The attack I’ll be walking you through today is from his “Retrieve a list of all user credentials via SQL injection” challenge.

Navigate to the main page of the Juice Shop site using Chrome, then click on the three dots in the top right corner of of the browser.  Choose “More Tools”, then “Developer Tools” from the dropdown menu.  This will open up the Chrome Dev tools on the page.  If the Dev Tools load on the side of the page, you may want to move it to the bottom of the page instead, so the console messages will be easier to read.  You can do this by clicking on the three dots on the right side of the toolbar, and choosing the Dock Side that puts the tools on the bottom of the browser. Click on the Console tab so you will see the console messages logged as you make requests.

We will be using the Search field on the main page of the Juice Shop for our attack.  The first thing we will do is try a simple attack to see what kind of information we can get in response.

Enter into the Search field: ‘;  and click the Search button.
You will get an error in the console. When you click on the carats on the left to expand the console entries, you will see that the SQL query that was run was: SELECT * FROM Products WHERE ((name LIKE ‘%’;%’ OR description LIKE ‘%‘;%’) AND deletedAt IS NULL) ORDER BY name

Now that you know what SQL query is being run when you submit a search, you can manipulate the query.
Enter into the Search field: ‘))–
Let’s take a moment to examine why we are entering these characters.
We are using to make the query think that the search value is completed.
We are using the first ) to close out the parentheses that started right after the word WHERE.
We are using the second ) to close out the second pair of parentheses that started right after the first.
We are using the to comment out all of the remaining text in the query.

When you submit this search value, this is the query that is run:
SELECT * FROM Products WHERE ((name LIKE ‘%‘))– OR description LIKE ‘%‘;%’) AND deletedAt IS NULL) ORDER BY name
Note that everything after the is ignored.

You have probably noticed that this query returned all the juices in the Juice Shop!  This is because the search is running only on the % character, and the % character acts as a wild card.

Now let’s try to do a UNION between this table of Products and another table.  We will make a guess that all of the usernames and passwords are in a table called Users.
Enter into the search field:  ‘)) UNION SELECT * FROM Users– 
Here we are closing out the query as we did before, but now before the we are adding in a UNION command that will join the Users table with the Products table.

When you submit this search, this is the query that is run:
SELECT * FROM Products WHERE ((name LIKE ‘%‘)) UNION SELECT * FROM Users– OR description LIKE ‘%‘;%’) AND deletedAt IS NULL) ORDER BY name

Notice that this did not give you a response in the browser.  Take a look in the Console to see what error you got:
SELECTs to the left and right of UNION do not have the same number of result columns
What this error is saying is that the number of columns in the Products table does not match the number of columns we are asking for in the Users table.  We need to get the number of columns to match before the UNION will be executed.

We can see that there are at least five columns in the Products table, so we will start our query with that.
Enter into the search field: ‘)) UNION SELECT ‘1’,’2′,’3′,’4′,’5′ FROM Users–
The numbers indicate the column numbers that we are requesting.

When you submit this search, this is the query that is run:
SELECT * FROM Products WHERE ((name LIKE ‘%‘)) UNION SELECT ‘1,’2,’3′,’4′,’5′ FROM Users– OR description LIKE ‘%‘;%’) AND deletedAt IS NULL) ORDER BY name

Note that you get the same error message as before, which means that we still don’t have the number of columns right.

At this point, you’ll need to keep guessing at the number of columns.  To make things easier, I’ll tell you that the correct number of columns is eight.

So, submit this search:  ‘)) UNION SELECT ‘1’,’2′,’3′,’4′,’5′,’6′,’7′,’8′ FROM Users–
This is the query that will run:  SELECT * FROM Products WHERE ((name LIKE ‘%‘)) UNION SELECT ‘1’,’2′,’3′,’4′,’5′,’6′,’7′,’8′FROM Users– OR description LIKE ‘%‘;%’) AND deletedAt IS NULL) ORDER BY name

And note that at the bottom of your search results, you have a new row of values!  We are getting closer to success with our attack.

The next step is to remove the product results so it will be easier for us to see the results we really want.  We can do this by searching for something that we know doesn’t exist, so we won’t get any results at all.

Submit this search: FOO’)) UNION SELECT ‘1’,’2′,’3′,’4′,’5′,’6′,’7′,’8′ FROM Users–
This is the query that will run: SELECT * FROM Products WHERE ((name LIKE ‘FOO’)) UNION SELECT ‘1’,’2′,’3′,’4′,’5′,’6′,’7′,’8′ FROM Users– OR description LIKE ‘%‘;%’) AND deletedAt IS NULL) ORDER BY name

We know that there aren’t any juices named FOO, so we won’t get any response.  But because we are doing a UNION, we will still get our row of column numbers as a result.

Now we can try to get some results into our table!  When we look at the columns in our UNION, we can see that the first column is an image field.  It’s unlikely that there are any images in the Users table, and our UNION needs to have each column match up by data type, so we won’t try to replace the first column with any of the Users columns.  We’ll start with column 2.  We can guess that the Users table probably has a column for an id, a column for a username, and a column for a password.  Let’s guess that the names of these columns are “Id”, “Username”, and “Password”.  Replace columns 2, 3, and 4 with Id, Username, and Password, like this:

Search on: FOO’)) UNION SELECT ‘1’,’id’,’username’,’password’,‘5’,’6′,’7′,’8′ FROM Users–

Unfortunately, this search doesn’t give us any results, and we don’t get any clues from the console either.  We can assume that one of our column names is wrong.  What would happen if we used ’email’ instead of ‘username’?

Let’s search on: FOO’)) UNION SELECT ‘1’,’id’,’email’,’password’,‘5’,’6′,’7′,’8′ FROM Users–

And now we get a complete list of ids, email addresses, and passwords for every user in the Users table!

I hope that this example has demonstrated the kind of thinking that is needed when searching for SQL Injection vulnerabilities.  Security testing requires more trial and error and more patience than traditional QA testing, but if it means finding and fixing vulnerabilities and protecting your company’s data, it is definitely worth it!



Introduction to SQL Injection

SQL Injection is another type of security attack that can do serious damage to your application.  It’s important to find SQL Injection vulnerabilities before a malicious user does.

In SQL Injection, a malicious user sends in a SQL query through a form field which interacts with the database in an unexpected way.  Here are four different things that a malicious user might do with SQL Injection:

  • drop a table
  • change the records of another user
  • return records that the user shouldn’t have access to
  • log in without appropriate credentials
To understand how a SQL Injection attack is crafted, let’s take a look at an example.  Let’s say we have a form in our application with a username field.  When the username field is populated with a name such as ‘testerguy’ and submitted to the server, the following SQL query is run:
SELECT * from users where username = ‘testerguy’
If this username exists in the database, results for the users table are returned to the application.  
A malicious user will try to trick the database by 
  1. making it think that the entry has terminated, by passing in testerguy’ 
  2. adding an additional clause, such as OR 1=1
  3. adding a terminating statement such as ; to make sure no other SQL statement will be run
In the above example, what the user would add to the username field would be:
testerguy’ OR 1=1;

And what the database will execute is:

SELECT * from users where username = ‘testerguy’ OR 1=1;

Take a moment to think about the 1=1 clause.  1=1 is always true, so the database interprets this as selecting everything in the table!  So this select statement is asking for all values for all the users in the table. 

Let’s see some SQL Injection in action, using the OWASP Juice Shop.  Click the login button in the top left corner of the page.  We are going to use SQL injection to log in without valid credentials.

We’ll make the assumption that when the login request happens, a request like this goes to the database:

SELECT * from users where username = ‘testerguy’ AND password = ‘mysecretpass’

If the request returns results, then it’s assumed that the user is valid, and the user is logged in.

What we will want to do is try to terminate the statement so that all usernames will be returned, and so that the password isn’t looked at at all. 

So we will send in:

  1. any username at all, such as foo
  2. a single quote to make it look like our entry has terminated
  3. the clause OR 1=1 to make the database return every username in the table
  4. a terminating string of to make the database ignore everything after our request
Taken together, the string we will add to the username field is:
foo’ OR 1=1–
You may notice that the submit button is not enabled yet.  This is because we haven’t added a password.  The UI expects both a username and a password in order to submit the login.  You can add any text at all into the password field, because we are ensuring that it will be ignored. Let’s add bar.
Now when you submit the login request, this is what will be executed on the database:
SELECT * from users where username = ‘foo’ OR 1=1–‘ AND password = ‘bar’

The first part of the request is returning all users, because 1=1 is always true.  And the second part of the request will be ignored, because in SQL everything after the dashes is commented out.  So when the code sees that all users have been returned, it logs us in!
If you hover over the person icon at the top left of the screen, you will see that you have actually been logged in as the admin!  The admin’s email address was the first address in the database, so this is the credential that was used.  Because you are logged in as the admin, you now have elevated privileges on the website that you would not have as a regular user.
Obviously, this sort of scenario is one that you would want to avoid in your application!  Next week we’ll take a look at some other SQL Injection patterns we can use to test for this vulnerability.  

Automated Testing For XSS

Last week, we talked about three different ways to test for Cross-Site scripting.  We looked at examples of manual XSS testing, and talked about how to use the code to formulate XSS attacks.  Today we will look at the third way to test, which is to use automation.  For today’s testing, we’ll be using Burp Suite, which is an oddly-named but very helpful tool, and is available for free (there is also a paid version with additional functionality).  We’ll also be using the Juice Shop and Postman

First, let’s take a look at the field we will be testing in the Juice Shop.  Using the Chrome browser, navigate to the Juice Shop’s home page.  You’ll see a search window at the top of the page.  Open up the Chrome Developer Tools, by clicking on the three-dot menu in the upper right corner, then choosing “More Tools”, then “Developer Tools”.  Once the dev tools are open, click on the Network tab. 

Do a search for “apple” in the search field.  You’ll get your search results on the web page, and you should see the network request in the developer tools.  The request name will be “search?q=apple”.  Click on this request.  A window will open up with the full request URL, which should be https://juice-shop.herokuapp.com/rest/product/search?q=apple. 

Next, open up Postman.  Paste this URL into the URL window, and click the Send button.  You should get a 200 response, and you should see your search results.  Now we’ll set Postman to use a proxy.  Click on the wrench icon in the top navigation bar and select “Settings”.  Click on the Proxy tab, then turn the Global Proxy on.  In the first section of the Proxy Server window, type 127.0.0.1, which is your local IP address.  In the second section, type 8080, which is your local port.  Postman is now set up to send requests to Burp Suite.  You may need to do one more step here, which is to turn SSL verification off.  In the Settings window, click on the General tab, and turn off the “SSL certificate verification” setting. 

Once you have downloaded Burp Suite, start the application and click Next.  Then click Start Burp.  Now Burp Suite is ready to receive requests.  Go back to Postman, and click Send on the search request that you sent earlier.  It will appear that nothing happens; this is because the request has just been sent to Burp!  Go to Burp, and you will see that the Proxy tab is now in an orange font.  Click on the Proxy tab, and then click the Forward button.  Your request is now being sent on to Postman, and if you return to Postman to check, you will see your request results.  It’s a good idea to turn off the Global Proxy in Postman now, because if you forget, the next time you make a Postman request you’ll wonder why you aren’t getting results! 

Return to Burp Suite, and click on the HTTP tab.  (This is a sub-tab of the Proxy tab.  If you don’t see the HTTP tab, make sure the Proxy tab is selected.)  You should see your GET request listed here.  Right-click on your request, and click “Send to Intruder”.  You should see that the Intruder tab is now in an orange font. 

Click on the Intruder tab.  You can see that the attack target has already been set to the Juice Shop URL.  Now click on the Positions sub-tab.  This is where we choose which element of the request we want to replace. Burp Suite has guessed correctly that the element we’d like to replace in our testing is the search field value of “apple”, so we can leave this setting as-is.  Now click on the Payloads sub-tab.  Here is where we will try out a bunch of cross-site scripting payloads! 

Enter in a bunch of XSS attacks into the Payload Options window, using the Add button.  Here are some suggestions:

<script>alert(“XSS here!”)</script>
<script/src=data:,alert()>
<IMG SRC=javascript:alert(‘XSS’)>
<IMG SRC=JaVaScRiPt:alert(‘XSS’)>
<a onmouseover=”alert(document.cookie)”>xxs link</a>

You can find many more suggestions in this XSS Filter Bypass List.  It’s worth noting that if you sign up for the paid version of Burp Suite, a whole list of XSS attacks will be available to use, saving you from having to type them in manually. 

Let’s start our attack!  Click the “Start attack” button in the top right corner of the application.  You’ll get a warning that your requests may be throttled because you are using the free version.  Just click OK, and the attack will start.  You’ll see a popup window, and one by one your XSS attacks will be attempted.

Once the attacks are finished, we can look at the attack results. There are six requests in the popup window.  The first request, request 0, is simply a repeat of our original request.  Requests 1-5 are the requests that we added into the Payload Options window.  We can see that requests 1, 2, and 5 returned a code 200, while requests 3 and 4 returned a 500.  This means that requests 1, 2, and 5 were most likely successful!  You can try them yourself by pasting them into the search field of the Juice Shop page. 

A few notes: 

  • It’s also possible, and more common, to use Burp Suite to intercept web browser requests directly, rather than going through Postman.  I chose to use Postman here because it’s so easy to set up the proxy.  
  • If you have set up Burp Suite to intercept browser requests directly, you may be able to replay your XSS attack responses directly in the browser to see them in action.  
  • Another feature of the paid version of Burp Suite is the Scanner tool, which will scan for a number of vulnerabilities, including XSS.

I hope that this blog post and the two previous ones have helped you to have a greater understanding of what Cross-Site Scripting is, why it’s dangerous, and how to test for it.  Next week, we’ll take a look at SQL Injection!

Three Ways to Test for Cross-Site Scripting

Last week, we explained what Cross-Site Scripting (XSS) is and demonstrated a couple of examples.  But knowing what it is isn’t enough- we need to able to verify that our application is not vulnerable to XSS attacks!  Today we’ll discuss three different strategies to test for XSS.

Strategy One:  Manual Black-Box Testing

This is the strategy to use when you don’t have access to an application’s code, and when you want to manually try XSS.  To implement this strategy, you’ll need to think about the places where you could inject a script into an application:

  • an input field
  • a URL
  • the body of an HTTP request
  • a file upload area
You’ll also need to think about what attacks you will try.  You may want to use an existing list, such as this one:  https://www.owasp.org/index.php/XSS_Filter_Evasion_Cheat_Sheet

This cheat sheet includes lots of different ways to get scripts past any validation filters, including:

  • on error and on mouseover alerts
  • URL encoding
  • using upper and lower case letters (to evade a filter that’s just looking for “javascript” in lower case letters)
  • putting a tab or space into a script so it won’t be detected
  • using a character to end an existing script, and then appending your own
If you are testing manually, a systematic approach is best.  Locate all of the places where you could inject a script, choose a list of attacks you’d like to try, and try each attack in each place.  While you are testing, you may also gain some insight of how you could change the attack by what kind of response you get from the attack.  For example, if your script tag is stripped out by validation, you could try to encode it.

Strategy Two:  Look at the Code

This is the strategy to use if you want to test manually, and you have access to your application’s code.  By looking at the code, you can determine the best way to craft an attack script.  This also works well for testing file uploads; for example, if your application’s code lists file types that are not allowed, you may find some types that have not explicitly been blacklisted, and you can see if you can upload a script using one of those types.
Let’s take a look at how you can use an application’s code to craft an attack.  We’ll be using the XSS Game again; we’ll be doing Challenge 3, and in order to access it you need to have solved Challenges 1 and 2; so take a look at last week’s post to see how to do that.  
As you look at the website in Challenge 3, you see that there are three different tabs, each of which displays a different image when clicked on.  Take a look at what happens in the URL each time you click on one of the tabs.  When you click on the Image 2 tab, “#2” is appended to the URL.  When you click on the Image 3 tab, “#3” is appended to the URL.  What happens when instead of clicking on a tab, you type “#2” into the URL?  Unsurprisingly, you are taken to the Image 2 tab.  What happens when you type “#5” into the URL?  There is no Image 5, but you can see that the page displays the words “Image 5”.  What happens when you type “#FOO”?  The page displays “Image NaN” (short for “Not a Number”).  You have probably figured out now that the end of the URL is the place that you are going to inject your malicious script.  
Now, let’s take a look at the code!  Click on the “toggle” link next to the words “Target Code”.  This will display the code used for the webpage.  Look at line 17; it shows how the URL for the image tag is created:
“<img src=’/static/level3/cloud” + num + “.jpg’ />”;
The “num” part of this image tag is a variable.  The value of the variable is taken from what we are sending in the URL.  If you send in a URL with “#3”, then the image tag will be
cloud3.jpg
If you send in a URL with “#FOO”, then the image tag will be
cloudFOO.jpg

Our task now is to see how we can inject a script using this “num” variable.  Recall that in last week’s post, we did some Cross-Site Scripting where we made it look like we were uploading an image, and we included an alert that would display when there was an error uploading the image. And we set things up so that there would always be an error, because we weren’t really uploading an image at all.  We are going to do the same thing here.  
Let’s craft our URL.  We will begin with
https://xss-game.appspot.com/level3/frame
because this is how the URL always starts.
Next, we’ll add
https://xss-game.appspot.com/level3/frame#3
because we want to make it look like we are following the pattern of choosing an image number.
Now we’ll add
https://xss-game.appspot.com/level3/frame#3
because we want to trick the code into thinking that the image URL is completed. This means that the code will try to load an image called “cloud3” instead of “cloud3.jpg”, which will generate an error.
Now we can add our on-error script:
https://xss-game.appspot.com/level3/frame#3  onerror=’alert(“Hacked!”)’
When the alert is triggered, a popup window will appear with the “Hacked!” message.
Let’s try it!  Take the entire URL:
https://xss-game.appspot.com/level3/frame#3 ‘ onerror=’alert(“Hacked!”)’
Paste it into the URL window, and click the “Go” button.  
You should see the popup window appear, and you have solved the challenge!
Strategy Three: Use a Security Testing Tool
As you can see from the example above, crafting an XSS attack takes a little time.  You may have many places in your application that you need to test for XSS, and not much time to test them.  This is where automated tools come in!  With an automated tool, you can send hundreds of XSS attacks in less than a minute.  In next week’s blog post, we’ll take a look at how to use an oddly-named tool called Burp Suite to automate XSS attacks!