At the start of my testing career ten years ago, I chose to tackle the challenge of mobile device test coverage. At the time, I tested multiple apps that had been developed for BlackBerry devices. Each carrier in North America had their own set of BlackBerry devices with their own unique version of the BlackBerry OS. It was important to verify that the app functioned properly on both GSM and CDMA carriers, so testing one hardware/OS combination on one network would not be enough as there were multiple screen sizes, not to mention multiple versions of the OS on the same hardware. The combinations were endless and it was quickly realized that, without a QA team of a hundred people and an endless and varied supply of hardware, we were going to have to make some tough decisions about where to focus our testing.
Today, this problem is even more evident. With literally thousands of hardware/OS combinations of Android devices, it’s just not feasible to test each of them and expect to release a product in a timely fashion.
7 Considerations To Achieve The Best Test Coverage
Let’s assume you are developing a mobile app for Android and iOS. Where do you start when it comes to identifying the hardware/OS combinations to focus on? I have identified seven things to think about when working to maximize your test resources.
Business Analysis
If you are on a QA team that is part of a larger organization, you may be given requirements outlining OS versions to support. Questioning whether or not supporting iOS8 or Android 4.3 is a good business decision may not be your concern, but choosing what exactly to test in between the minimum and the latest versions is QA’s business and coming to the right conclusions is extremely important. The BA (if there is one in your organization) might tell you the minimum supported version, but they certainly won’t tell you how much time to spend on each OS/HW combination. For the purpose of this conversation, let’s assume a decision has been made about the minimum version and that’s it. If you can leverage more data from a BA, then do it but oftentimes, QA is on the spot to figure it out.
Location, location, location
Determine the geographic location of your target audience. If your targeted users are in North America, the devices you will want to focus your attention on will differ from if you are targeting an audience in India. Knowing which devices are used where, can narrow things down drastically so I recommend starting here.
Data
Once you have determined the geographic location of your target user base, you’ll need to collect some data. Fortunately, there is plenty available online. For example, the Android developer dashboard provides up-to-date hardware and OS statistics for Android. If 30% of Android users are on version 4.4 and another 30% are on 5.x, then you know you can cover a big chunk of your user base with just a couple versions. (Remember to pay attention to the geographic region when collecting the data.) Now, focus your attention on two or three of those major platform versions and test with those on the most popular devices.
Minimum Version
Determine a minimum version, but don’t spend too much time here. If only 3% of your potential users are on version 4.3 of Android, then don’t spend more than about 3% of your time testing this version. Perform regular sanity tests to make sure the app doesn’t crash at launch, but don’t test every possible scenario on that version. With the minimum version, it’s good to remember that the user base of this version is not going to grow – it’s going to shrink.
Latest and Greatest
When it comes to supporting the latest version of the OS, you will want to approach this differently. The user base may also be just 3% for the latest version but unlike the minimum version, the user base of the latest version is always growing. It’s important to regularly regression test your app on the latest version of an OS as soon as it is available. You want users to continue using the app and the worst case scenario is they uninstall it because bugs appear when they upgrade the OS. It’s important to keep on top of this and remember, if there aren’t many users on that version yet, there soon will be.
Upgrade? Maybe later!
Don’t upgrade all of your devices. This might seem like a simplistic concept but it is very important. A lot of time can be wasted by trying to downgrade a device to test an issue that has been reported on an older OS. If you have the devices, leave as many of them as possible on older versions. It is especially difficult to downgrade iOS devices and the iPhone upgrade notifications make it easy to accidentally upgrade to the latest version. Hit cancel on the upgrade prompt and keep that older OS if you can because you may need it. Users don’t want to fix what isn’t broken, so they will stay on an older OS and you should too. I’ve always found it much easier to upgrade than downgrade.
Emulators
I always feel much better testing an app on real hardware with a real network connection than on an emulator. If you have identified a big percentage of users that are using a hardware/software combination, then test it on the real thing. It’s worth it. But, if you want to make sure your app works on a version only used by 1% of the user base, you can sanity test it on an emulator. It’s most likely not worth the resources to purchase hardware and then find and install the version that almost nobody is using. That being said, we all want as many users as possible. Give it a quick test on an emulator and you may quickly uncover a bug and earn yourself more users.
Relationships
Lastly, leverage partners for usage data. If you are partnering with a provider that has an app or user base in a similar category, reach out to them and see if you can leverage data they have obtained. If they are not a competitor, sharing localytics data may benefit both of you. At some point you might be able to return the favor.
Conclusion
As the number of hardware manufacturers grows and compatibility with different types of devices expands (phones, tablets, watches,) the issue of test coverage will continue to become more and more complex. It’s not efficient to test every possible combination. Use these solutions to narrow your focus and always be ready to adjust.
Happy testing!
About the author
Brian Hamilton is a software tester who lives in Kansas City and contracts for TestLodge, an online test case management tool that helps software testers efficiently write and execute their test cases online. When he’s not testing the latest gadgets, Brian enjoys playing sports and being outdoors.