Dealing with mobile fragmentation in testing
Mobile markets have diversified across the globe. In advanced markets, users run cutting-edge software on advanced Apple and Android devices, while users in developing markets run older versions of software on older devices. Many companies see the growth potential in reaching out to both advanced and developing markets. But in order to do that, those companies need to test against all potential mobile devices in some way.
The challenge is to perform frequent testing on enough real devices to be confident that your app’s user experience is consistently good across all of your potential customers’ devices (just using emulators isn’t going to cut it). Buying or renting a large number of mobile devices can be expensive, so how do you narrow down the specific devices that you test? How do you obtain all those devices for testing? Let’s look at some strategies for testing a broad range of devices without a great deal of cost.
The state of mobile fragmentation
There are millions of mobile phones and tablets around the world, and a wide variety being produced. Windows Mobile and Apple hardware differs slightly from version to version but is tightly controlled due to the closed operating systems. More diversity occurs in Android hardware, since it has an open-source operating system. As of 2014, there were roughly 19,000 different Android devices internationally. That number increased to more than 24,000 in 2015.
Data from OpenSignal's Android fragmentation report.
Both Apple and Android devices are only able to support a certain version of their operating systems before they are no longer able to receive upgrades. Both software and hardware are deprecated at a shorter rate than in the past, with the average user having a phone for only around two years. Each new hardware upgrade typically involves changes in look, feel, and size of the phone, creating a vast array of issues to consider when designing and testing a mobile app.
What devices should you test?
Some developers and even product teams are skeptical of, or fail to even consider, testing older mobile devices or operating systems, at least at first. Because their peer groups might all have devices less than two years old (certainly less than three!) they might assume that all people are operating on new equipment. However, businesses are often looking toward international and developing markets as prime audiences for the adoption of new technology, which creates additional revenue.
While the business concerns itself with adding revenue, developers and testers must concern themselves with meeting the needs of target customers, a broad audience that is likely using many devices to access an app. By using a device grouping strategy, devices can be grouped by shared characteristics, and mobile testing can be made more manageable.
What might a mobile device grouping strategy look like? A grouping strategy could be arranged per project, per sprint, or even per feature, so that testing is more strategic at every level. Ways to group devices include priority, popularity, or other market data gathered from mobile analytics information about the target market.
- Universal Analytics by Google, Flurry by Yahoo, and Apple App Analytics are free tools for analyzing what devices and operating systems are the most common among your users.
- Other market data, such as AppBrain’s top devices by country or InMobi’s mobile advertising statistics, can be used to learn about common device usage across different countries.
A sample mobile device grouping
For this business case, let us consider a company that is rolling out an app to three different markets. Market A uses popular and new devices. Market B uses a mix, and Market C uses mostly older devices operating on older versions of Android. Market A and B have a lot of saturation, while Market C is a growing market and could increase revenue significantly through rapid adoption. The end of this sprint is the first offering of the app to Market C outside of beta release. Knowing this, let's design our mobile device groups for this sprint:
Group 1: High Priority, Market C
- Android OS versions: Lollipop and Kitkat (~53.2% global Android market as of April 2017: Android Dashboards)
- Android devices: Samsung Galaxy Grand, S4, S5; Motorola Lumia 535
- iOS versions: iOS 8, 7
- iOS devices: iPhone 5, 4S, 4
Group 2: Medium Priority, Markets A & B
- Android OS versions: Nougat, Marshmallow (~34.9% global Android market as of April 2017: Android Dashboards)
- Android devices: Samsung Galaxy Note 5, Galaxy 6
- iOS versions: iOS 10, 9
- iOS devices: iPhone 7, 6
Once the product has been released to Market C and mobile analytics data has been gathered, it will make sense to adjust the groups based upon the overall number, types of devices, and software being used to access the app. Mobile device groupings a few sprints later would be adjusted to reflect the overall adoption of the application:
Group 1: High Priority
- Android OS versions: Nougat, Marshmallow
- Android devices: Samsung Galaxy Note 5, Galaxy S6
- iOS versions: 10, 9
- iOS devices: iPhone 7, 6S
Group 2: Medium Priority
- Android OS versions: Lollipop, Kitkat
- Android devices: Samsung Galaxy S5, S4 mini, Grand; Motorola Lumia 535
- iOS versions: 8, 7
- iOS devices: iPhone 5, 4S, 4
Market C has been moved to medium priority a few weeks after the launch because more customers are still in markets A and B (or the more valuable ones are), but the company is still nurturing and supporting a growing market. It might not want to devote as many resources and real devices to testing for that market just yet.
Another option for mobile device grouping
Another way to look at the data is to see what devices are currently supported. Apple, for example, maintains a providers page. Check each provider to see the oldest version of iPhone supported. For a small subscription fee, Consumer Reports offers a similar list of providers and options for Android. Bugfender maintains a constantly updated free list of mobile device popularity by operating system and device type.
Use the two lists to come up with a selection set of devices to test on. If that list is too big, consider rotating by sprint or clumping categories of devices together. Once the list is complete, test similar products or competing products on the oldest devices.
For example, if your application embeds YouTube videos as the primary form of content, your team probably does not need to support any devices too old to run the YouTube native application or view YouTube in a browser.
Once the list is reasonably narrowed, it’s time to obtain the devices to test with.
Obtaining devices to test on
Another problem for serious mobile application developers is obtaining and maintaining devices for test. You have the option to buy your own devices and build a device lab yourself, or rent those devices from a cloud testing service.
Hosted device labs offer services where you can run tests against real or emulated/simulated mobile devices without actually needing to purchase the devices. Automated tests can be run as well, and some services support Appium, Calabash, and other popular mobile functional test automation frameworks. Hosted services can be expensive but might work well for companies that don’t have the resources to maintain their own device labs for testing.
Before choosing a hosted device-lab service, analyze how many devices you will ultimately need to reach your desired market and how frequently you anticipate running tests. Make sure that the device cloud service you choose offers all the options that you need, such as real-device testing or testing in parallel.
Companies that are willing to put in the time and money up front can use several DIY options to create their own device lab for tests. Some factors to consider when creating a device lab in-house include:
- Cost of devices
- Level of maintenance required to keep the lab up to date with new devices and operating system updates
- Technical expertise required to build a device farm that can perform tests in parallel and integrate painlessly into the development and testing workflows
For companies with lower budgets, building a DIY lab might mean using more emulators/simulators in your mobile testing and obtaining a select handful of your customer’s most-used devices. Preferably, you want to to cover 80% to 90% of your apps users with real-device testing, which could be cheap if 90% of your customers use 10 to 20 different devices. But if your customer devices are more diverse than that, you’ll have to make compromises.
Device fragmentation: Confront it head on
Businesses that want to see successful, widespread adoption of mobile apps in a varied market need to encourage robust testing solutions in spite of the diverse availability of mobile devices. Emulators simply aren’t enough. You need to use real-device testing on as many devices as your budget will allow. That’s why it’s crucial to use user analytics tools to get a ranked list of the most common devices your customers use to run your apps.
To enable developers and testers to work and test realistically, businesses can carefully define device groups of interest, invest in device labs (via the cloud, or DIY), and define test plans that are strategic so that they can reach the broadest number of devices, scenarios, and concerns with the least amount of work or repeated effort.
Additional reading
As mobile devices continue to permeate the lives of people around the world, it is important for product managers, developers, and testers of these products to keep track of what is happening in the world of mobile development regarding fragmentation. Here are several links that can help you stay on top of the devices you need to test.
- Android Dashboards—Get information about Android devices, their shared characteristics, and their overall presence in the Google Play ecosystem to drive development and testing decisions.
- Apple Device Support—Keep an eye on how many users are using various versions of iOS.
- AppBrain’s top devices by country—If you are focusing your product on a certain region, it is important to know what devices are made and distributed there.