We have witnessed the transition from desktop to web and are seeing another transition from web to mobile. I have been thinking about a blog series around testing mobile applications for a while, and this is the first blog post in the series. In the coming few weeks, I will try to cover various topics/products/approaches related to testing mobile applications. I will focus on Android to start with and will move on to other platforms.
Before I dwell deeper into the subject – it is important to understand how testing mobile applications is different from testing browser/desktop applications. If we understand the distinction and challenges of testing mobile apps, it will be a bit easier to tackle them.
1. Supported platforms & devices – you have more combinations to test
Desktop apps were usually targeted for specific platforms, and it was relatively easy to access those platforms. Web-based applications made it a bit more challenging by adding another dimension – browsers.
Mobile applications take the complexity of supported platforms to the next level by adding devices. Ensuring that mobile apps are working on all type of devices (SmartPhone, Tablets, and Phablets) supplied by major brands (various models from Samsung, Sony, Nokia, HTC, Apple, etc.) and on all the platforms (iOS, Android, Windows, BlackBerry, etc.) is challenging. On top of that, new devices are hitting the market so often that it becomes impossible to cover all the major devices.
In the mobile world, it is important to create something on the lines of graded browser support used by Yahoo to ensure that major platforms are covered.
2. Adaptability & Limited space – Screen size is constantly changing
Pretty much all the major players are varying screen sizes of their phones, tablets and phablets to figure out what works or in response to the competition. How applications adapt themselves for various screen sizes, layout and configuration is a challenging task.
Apart from adaptability to different screen sizes, mobile applications have to deal with the limited screen size. Limited screen size means that user can not be given 30 different options on a single screen – usability, similar experience, on-screen help, inability to use search or other applications, etc. pose different challenges. As a tester we, need to think beyond what is developed and always think of who will use it and under what circumstances.
3. Complex user interaction – More than one way to do everything
User interaction in desktop and browser based applications was pretty much limited to mouse and keyboard. Mobile applications, on the other hand, are trying to make user interaction as fluid as possible. We had the touch screen and with new phones from Samsung, you can just wave your hand to give commands. Siri is becoming more and more advanced and gives us a glimpse of future that voice commands may become part of every application in future. Devices are smart enough to understand complex gestures, eye movement, direction, tilt, movement, acceleration, GPS coordinates, surroundings, sound and so on.
As a tester, we need to ensure that application works as expected when the user interacts with the app in different ways.
4. Application Type – HTML5, Native or Hybrid?
In the desktop and browser world, applications were straightforward. They were either desktop or web based applications. However, with the adoption and support of HTML5 – applications are merging. On all the mobile devices, it is not difficult to find HTML5 applications, Native applications, and hybrid applications. Testing for a hybrid application would be different from testing a native application, and it is important to understand that difference.
5. Dependency on emulator/simulator – Get devices
For the desktop and browsers, developers always had access to the platform or browsers they were targeting with their applications. Also, virtualization has become more or less commonplace and can be trusted for desktop and browsers.
Mobile devices, on the other hand, relies on emulator and simulators. However, they are still not an accurate representation of the devices. It is also not possible to replicate advanced user interaction on these simulators. As a tester, we have to be aware of the capabilities and limitations of these emulators/simulators and figure out what can be tested (reliability) on them and what can not.
6. Security & Privacy – You can’t touch me but I can.
Though most mobile applications live in their sandbox, many platform features are accessible to them. For example, applications such as phone book, pictures and videos are available to many other applications. These are all personal user data – and any defect around the misuse (unintentional) of this data can jeopardize trust of the application.
In the mobile world, it is important to ensure that applications are secure from the intruders, and it is equally important to ensure that applications are not intruding or accessing data unintentionally.
7. Dependency on Network/carrier – more variations
In desktop and web world, most users were either on LAN or Wireless. These networks were not predictable, but compare to mobile networks; they were very predictable. Many connected mobile applications rely on the network – how the application responds to 3G, 4G, weak signals, no signals, powerful signals, Switching from cellular to wireless and vice-versa or when the user is moving at different speeds, etc. can affect how the application will behave. It is often not possible to come up with or simulate real life situations for mobile applications.
Apart from the variation in signal strength and type, mobile apps can respond differently to different carriers. As a tester, it is important to understand if there are any difference or not and whether the application works for all the major carriers or not.
8. Installation, removal, and upgrade – Would you come back?
Mobile apps are installed, removed or updated more frequently than desktop applications. Also, underlying OS and platform are updated more frequently as well. As an app developer and tester in the mobile world, you have to be on top of what changes are coming in the next revision of OS / Platform and how it might affect the application.
Usually, for most of the applications, user data is stored on the servers and not on the devices. It makes installation a bit tricky – what if the user has multiple devices, what if multiple devices have a different version of applications and so on.
Things like backward compatibility, simultaneous support for multiple versions, data preservation, restoring state and data, ability to install/upgrade multiple times, etc. are all part of necessary checks for mobile application testing.
9. Session Management & Interruptions – Who’s calling?
Handling Interruptions are the way of life for mobile applications. Apps and users are constantly interrupted by calls, SMS, push notifications and so on. How applications handle these interruptions, how they maintain their state etc. are important, but it is also important to see how much interruptions application is generating and what triggers those interruptions.
As a tester, it is important to ensure that application behaves properly when it is interrupted and it is also important to ensure that application does not interrupt unnecessary and works according to the boundary defined by platform or users.
10. Mobile specific Non-functional testing – and you thought it’s over.
Mobile applications add much more dimensions to the non-functional testing. Old school performance of the application is the obvious one, but there are many other factors as well which should be considered.
How much data your application is consuming? How much would it cost user (data usage) to use this application? How much battery is consumed by applications? Does it behave differently in high battery and low battery conditions? How much space it is occupying? How many trails it is leaving? How it is clearing the trails/logs etc. are important non-functional factors which should be considered as part of the mobile testing strategy.
So these are my top 10 reasons of why testing mobile apps are different from testing web-application or the desktop applications. I am sure there are more reasons. What else makes testing mobile applications different from testing desktop or browser based applications? Let’s discuss.