Guidelines for Perfecting your Mobile Test Coverage

One of the biggest challenges mobile teams face for many years now, is the ability to nearly “scientifically” determine the right test coverage for their mobile apps (native).

While the mobile space is divided “only” into 2 families – iOS and Android, these families are highly fragmented and continuously introduce new devices, operating systems (OS) versions and challenging changes to the OS functionalities. These constant changes together with the large size of the mobile market are a huge pain.

Being successful in solving that challenge depends on various factors:

  • Knowing the market and target audience for your apps
  • Leveraging market data and app under test analytics
  • Building a Device lab that is based on risk and data-driven decisions
  • Keeping up monitoring the market to continuously calibrate the lab

Mobile Market Overview

Fig1: iOS Platforms Market Share (Source: Apple App Store)

If we investigate the iOS market share today, based on the Apple app store (Fig 1), there are 3 OS families. This is not 100% right, since iOS12 is already in its public beta 3. A beta version that is public means that any customer has the freedom to use it, and once using it, he expects it to work. With that in mind, testing even from a smoke perspective this platform is highly important – so we are considering 4 OS families and not 3. Testing on the latest IOS12 beta is also a great way to identify test automation code compatibility issues as well as early defects with your app, so it is a win-win situation to consider testing this version today.

In addition, when looking at the second and from a market share perspective the large OS platform that is Android, things are even more complex, since there are few more OS families and much more device models.

Fig 2: Android OS Market Share (Source: Google’s Android OS Dashboard)

Figure 2 above shows that ~30% of the Android devices are running android 7.x, 25.5% are on Android 6.x, 21% of the devices are on Android 5.x. While Android 8.x is only ~5.5% of the market, it is a must to cover as it’s the latest Android GA version. Regarding Android 4.x (KitKat) that holds a bit more than 10% of the market, it depends on your specific app, the geography in which your end users operate in, and of course the usage and analytics that your app/web is showing regarding this version. Similarly, to iOS, Android is also in mature stages of its next big release that is Android P, hence, need to have that version also included in the test coverage considerations.

Leveraging Market Data and Analytics

Each geography has its own mobile usage patterns, and popular devices. For that I release almost each 1-2 quarters the Factors magazine, to guide the coverage across 17 different countries – this is based on a lot of analytics coming from large amount of mobile transactions happening in these geographies. Globally, these are the most popular smartphones and tablets to include in your test lab.

Fig3: Mobile Platform Coverage Recommendations (Source: Perfecto)

Build a Device lab based on risk and data-driven decision

Leveraging market data from the app store, google play, and research firms as mentioned above is great, however each team has unique project requirements, budget considerations, and schedule constrains that based on risk, and quality patterns from previous releases should be baked into the overall coverage decisions.

To build the right device lab that considers all the above, teams need to look at their pipeline activities, testing suites size, automation coverage and based on risks define the top smartphones and tablets that would best serve their quality objectives. With that in mind, looking at the above recommended table as a baseline from which to start can be great since it offers both device variety that is global, and multiple OS versions including Beta.

Keeping up monitoring the market to continuously calibrate the lab

Solving the coverage complexity only serves team in a given time, however, and with the market continuously evolving, there needs to be a process in place to ensure that what is currently configured in the test lab still “holds” and provides relevant quality feedback to the developers. For that, it is important to continuously track the market, and prepare or leverage from the above mentioned Factors magazine a mobile market calendar, such that will point in advance key market events to allow teams to prepare and tune their labs accordingly.

Summary

In this blog, I covered some guidelines into mobile test coverage. While your teams know the app flows and functionalities, the app eventually gets installed (hopefully) multiple times on thousands of devices that operate in various locations and network conditions in the world, that’s why it is very important for dev and test teams to follow the above trends, and validate that their lab is fully aligned with the market from both devices and OS versions.