The landscape for the software development is continuously changing. We have come a long way from the desktop applications to the mobile apps. In a previous post, we discussed top 10 challenges of testing the mobile apps. In this post, we will talk about one of these challenges, device management, in a bit more detail.
Mobile devices – what should you acquire for your test lab?
As far as mobile devices are concerned, there is an unprecedented level of options available to the consumers. According to this report from the IDC, Android is the undisputed leader of the mobile world. Also, it is clear that very few apps have a need to venture beyond the iOS or Android world.
Let us talk about the easy option first. Even with its close system, there are a plethora of choices available for the iOS world. At the time of writing this post, Apple is selling five distinct models (iPhone 7/7+, iPhone 6/6+, iPhone SE) for the iPhone. Not only that, many people are still on the older iPhone models such as iPhone 5s, 5c, iPhone 4 and so on. On top of that, we have 5 distinct models for the iOS tablets (Two models for the pro, two models for the mini, and one model for the air). That is ten actively selling devices in the market. If we combine older devices such as iPhone 4 and other models of iPhone 5, the number of devices to test will reach at around 20.
On top of that, in every few months, there is an iOS update from Apple and every year there is a major iOS update. At any given time, you can find that users of your apps or mobile web are coming from at least 6-10 different OS versions.
Maintaining a device lab for 20+ different devices that can support anywhere between 6-10 different versions of the iOS is a big challenge.
However, when it comes to the Android world, this challenge of maintaining iOS devices seems minuscule in nature. According to this report from the Open Signal, there are more than 25000 types of Android devices from more than 1200 brands are in circulation. Not only that, unlike iOS, Android OS updates are not spread as quickly as iOS, and as a result, it is not uncommon to find devices that are running a really old version of Android such as version 1, 2 or 3.
With this kind of distribution, you can imagine that maintaining a device lab for your testing effort could be a significant pain. It can be a daunting task to decide the devices and OS combinations that you need to acquire for the testing purpose.
How should we select the devices and os combinations for mobile apps?
In my opinion, there is only one way to find the answer. Look at the data from the analytics system (Google Analytics, Fabric or anything else) that your app uses to find devices and OS combinations your users are using to access your mobile apps.
There is no point in thinking about it or purchasing favourite devices of your teammates. You do not want to test on the devices / OSes that you or your teammates like. To remove any subjectivity or opinion based discussions from this, you can follow these steps to decide devices for your test lab
- Look at the analytics to create a list of OS versions and devices that your users are using.
- Arrange this list according to the percentage of users coming from that combination.
- Define a threshold to limit this list. In my experience, Pareto’s 80-20 rule is applicable for the devices as well. That is, you can focus on the devices and OS combinations that represent 80% of your users.
- You can also define a minimum threshold for device and OS combination support. For example, you can specify that if a device / OS combination represents less than 2% of our user base, we will not support it. Once we have a defined threshold, it becomes easier to discuss if we need to support a new device/OS configuration or not.
- Depending on your team size, it might also make sense to have multiple devices for the top 5-7 device / OS combinations to ensure that devices are always available when needed.
This list should give you a high-level view of which OS and devices are necessary. You may want to make it a bit clearer by putting this information on a 2*2 matrix to ensure that you have sufficient devices for the major combinations and none of the important combination is missed.
It is important to clarify that this list does not mean that other combinations will not be tested. It just means that the team will manage only these devices. For all the other combinations, the team will rely on the simulators or emulators. Also, bear in mind that many cloud-based solutions can give you on-demand access to the device and OS combinations you need for your manual or automated testing.
Apart from this, there are plenty of open devices lab around the world that you can use to find the device/OS combination you need.
In this post, I have shared an approach to select the device and OS combinations for setting up a device lab for testing mobile apps and web applications. However, setting up is just one part of the story. Ongoing maintenance of the device lab is equally challenging and in the next post I will share some of the challenges of maintaining a device lab.
In the meantime, it would be great to discuss the approach you take to select the mobile devices for your mobile testing efforts. Please comment and we can discuss.