Mobile Testing Part I: Twelve Challenges of Mobile

Just over a decade ago, the first iPhone was released.  Now we live in an age where smartphones are ubiquitous.  Our smartphones are like our Swiss Army knives- they are our maps, our address books, our calendars, our cameras, our music players, and of course our communication devices.  Testing software would not be complete without testing on mobile.  There are so many considerations when testing on mobile that I’ve decided to make this into a three part series: this week I’ll discuss the challenges of mobile testing, next week I’ll discuss mobile manual testing, and the following week I’ll talk about mobile automated testing. 

First, the challenges!  Below are twelve reasons why testing on mobile is difficult.  I thought it would be fun to illustrate just what can go wrong with mobile software by describing a bug I’ve found in each area.  Some of these bugs were found in the course of my testing career, and some were found on my personal device as an end user.
1. Carriers: Mobile application performance can vary depending on what carrier the device is using.  In the US, the two major carriers are Verizon and AT&T, and we also have smaller carriers like Sprint and T-Mobile.  In Europe some of the major carriers are Deutche Telekom, Telefonica, Vodaphone, and Orange; in Asia some of the carriers are China Mobile, Airtel, NTT, and Softbank.  When testing software on mobile, it’s important to consider what carriers your end users will be using, and test with those carriers.
Example bug: I once tested a mapping function within an application, and discovered that while the map would update based on my location when I was using one carrier, it did not update when I was using a different carrier.  This had something to do with the way the location was cached after a cell tower ping.  
2. Network or Wifi: Device users have the choice of using their applications while connected to the carrier’s network, or while on wifi.  They can even make a choice to change how they are connecting in the middle of using the application; or their connection can be cut completely if they go out of network range.  It’s important to test an application when connected to a network and when connected to wifi, and to see what happens when the connection changes or is lost completely.
Example bug: I have a wifi extender in my house. When I switch my phone’s wifi connection to use the extender’s IP, Spotify thinks I am offline.  I have to force-close the app and reopen it in order for Spotify to recognize that I am online.
3. Application Type: Mobile applications can be Web-based, native, or a hybrid of the two (developed like a Web app, but installed like a native app).  Some of your end users will choose not to use a native or hybrid app and will prefer to interact with your application in their phone’s browser.  There are also a variety of mobile browsers that could be used, such as Safari, Chrome, or Opera.  So it’s important to make sure that your web application works well on a variety of mobile browsers.
Example bug: There have been many times where I’ve gone to a mobile website and their “mobile-optimized” site doesn’t have the functionality I need.  I’ve had to choose to go to the full site, where all of the text is tiny and navigation is difficult.  
4. Operating System: Mobile applications will function differently depending on the operating system.  The two biggest operating systems are iOS and Android, and there are others, such as Windows Mobile and Blackberry.  It’s important to test on whatever operating systems your end users will be using, to make sure that all of the features in the application are supported in all systems.
Example bug: This is not a bug, but a key difference between Android and iOS: Android devices have a back button, while iOS devices do not.  Applications written for iOS need to have a back button included on each page so users will have the option to move back to the previous page.
5. Version: Every OS updates their version periodically, with new features designed to entice users to upgrade.  But not every user will upgrade their phone to the latest and greatest version.  It’s important to use analytics to determine which versions your users are most likely to have, and make sure that you are testing on those versions.  Also, every version update has the potential to create bugs in your application that weren’t there before.  
Example bug: Often when the version is updated on my phone, I can no longer use the speaker function when making phone calls.  I can hear the voice on the other end, but the caller can’t hear me.
6. Make: While all iOS devices are manufactured by Apple, Android devices are not so simple.  Samsung is one of the major Android device manufacturers, but there are many others, such as Huawei, Motorola, Asus, and LG.  It’s important to note that not every Android user will be using a Samsung device, and test on other Android devices as well.
Example bug: I once tested a tablet application where the keyboard function worked fine on some makes but not others. The keyboard simply wouldn’t pop up on those devices, so I wasn’t able to type in any form fields.
7. Model: Similar to versioning, new models of devices are introduced annually.  While some users will upgrade every year or two to the latest device, others will not.  Moreover, some devices will not be able to upgrade to the latest version of the OS, so they will be out-of-date in two ways.  Again, it’s important to find out what models your end users are using so you can make decisions about which models to test on and to support.
Example bug: This is not a bug, but it was an important consideration: when Apple released a new model of the iPad that would allow a signature control for users to sign their name in a form, the software I was testing included this feature.  But older versions of the iPad weren’t able to support this, so the application needed to account for this and not ask users on older versions to sign a document.
8. Tablet or Smartphone:  Many of your end users will be interacting with your application on a tablet rather than a smartphone.  Native applications will often have different app versions depending on whether they are designed for tablet or phone.  An application designed for smartphone can often be downloaded to a tablet, but an application designed for a tablet cannot be installed on a smartphone.  If a web app is being used, it’s important to remember that tablets and smartphones sometimes have different features.  Test your application on both tablets and phones.
Example bug: I have tested applications that worked fine on a smartphone and simply gave me a blank screen when I tried to test them on a tablet.  
9. Screen Size:  Mobile devices come in many, many different sizes.  While iOS devices fit into a few sizing standards, Android devices have dozens of sizes. Although it’s impossible to test every screen size, it’s important to test small, medium, large, and extra large sizes to make sure that your application draws correctly in every resolution.  
Example bug: I have tested applications on small phones where the page elements were overlapping each other, making it difficult to see text fields or click on buttons.
10. Portrait or Landscape: When testing on smartphones, it’s easy to forget to test in landscape mode, because we often hold our phones in a portrait position.  But sometimes smartphone users will want to view an application in landscape mode, and this is even more true for tablet users.  It’s important to not only test your application in portrait and landscape modes, but also to be sure to switch back and forth between modes while using the application.  
Example bug: I tested an application once that looked great in a tablet when it was in portrait mode, but all of the fields disappeared when I moved to landscape mode.  

11. In-App Integration: One of the great things about mobile applications is that they can integrate with other features of the device, such as the microphone or camera.  They can also link to other applications, such as Facebook or Twitter.  Whatever integrations the application supports, be sure to test them thoroughly.  
Example bug:  I tested an application where it was possible for a user to take a picture of an appliance in their home and add it to their home’s inventory.  When I chose to take a picture, I was taken to the camera app correctly, and I was able to take the picture, but after I took the picture I wasn’t returned to the application.  
12. Outside of App Integration: Even if your application isn’t designed to work with any other apps or features, it’s still possible that there are bugs in this area.  What happens if the user gets a phone call, a text, or a low battery warning while they are using your app?  It’s important to find out.
Example bug:  For a while, there was a bug on my phone where if my device timer went off while I was in a phone call, as soon as I got off the phone, the timer sounded and wouldn’t stop.  
I hope that the above descriptions and examples have shown just how difficult it is to test mobile applications!  It may seem overwhelming at first, but in my next two blog posts, I’ll discuss ways to make testing simpler.  Next week, we’ll be taking a look at writing mobile test plans and assembling a portfolio of physical devices to test on.  

9 thoughts on “Mobile Testing Part I: Twelve Challenges of Mobile”

  7. Johny Depph

    Hybrid app development has become mainstream for mobile app development since the majority of mobile developers and leading app development companies are preferring it over others. Hybrid apps are popular because they allow developers to write code for a mobile app once and still accommodate multiple platforms. Because hybrid apps add an extra layer between the source code and the target platform, they may perform slightly slower than native or web versions of the same app.

Comments are closed.