Test on Mobile Devices That Matter to Your Business
June 15, 2015

Amir Rozenberg
Perfecto Mobile

Choosing the right smartphones and tablets to test your apps against is just as important as the tests themselves. After all, if your app isn’t working at the right place (your customer’s device) at the right time (all the time), angry customer tweets and two-star app reviews will follow and the consequences for your business will be severe.
 
In some cases, your app won’t even make it into the app store. Twenty percent of apps are rejected from the Apple store because they’re buggy, crash-prone or have an inferior user interface. And when a new version of Android arrives, apps that worked fine on the previous OS version start to act like rejects themselves.
 
On the hardware side of the fence, new devices arrive every quarter, and device usage varies by geography. Windows Phones may have 3.8% market share in the US but in Europe it has an average market share of close to 9%, according to mobile market research firm, ComTech. Basing your testing of Windows Phones solely on US market data when your app also has a presence in, say, France (12%) and Italy (13.3%), would be a costly mistake.
 
The bottom line is that mobile devices and operating systems are always in flux. You need to keep a close eye on market changes so you’ll be testing your apps and responsive websites against the devices that represent your target users. To that end, we created a Mobile Test Coverage Index report, a quarterly release that aggregates data from over 4,000 device profiles and 360,000 hours of customer usage data, as well as mobile market share data and future device release timelines, to create a list of the 32 most relevant mobile devices.
 
The device selections address five key sectors: device model, screen size, GA date, hardware system and recommended operating systems. The number of devices in each test bracket may vary from quarter to quarter to reflect the restless mobile market. The report will be continuously updated to meet market changes.

While the report will not hand-deliver an answer for what devices and operating systems the company should specifically test against, it can be used as a benchmark from which it can customize their test coverage according to their own business and customer needs.
 
Some noteworthy insights from the research:


■ Almost a third (30 percent) of the market can be covered with 10-16 devices – including popular devices such as the Samsung Galaxy S5, iPhone 6 and HTC One M8 – based on market adoption, market device leaders, reference devices and device characteristics.

■ Regarding operating systems: iOS 7 still accounts for some 17 percent of total users out there, and should be included at the same level of iOS 8 test plans. Additionally, Android 5.0 adoption rates are low in both the US and Europe with Android KitKat (4.4.x) and JellyBean (4.2.x, 4.3.x) rating as the most commonly used versions.

■ Geographically speaking, iPhones hold twice as much share in the US than in Europe, with iPhone devices capturing 35% of the U.S. market, as opposed to 16% in EU5.

We can’t overstate how important device coverage is to a mobile app testing strategy, for both business and technical executives alike. A relevant device and operating system mix will get an organization much closer to the true end-user experience. However, it’s a challenge to keep your test coverage relevant given that mobile devices have a shelf life of around nine months. Benchmarks built from data like this report will help, but in an ever-changing market it’s essential to keep reviewing continually updated research.

Amir Rozenberg is Director of Product Management for Perfecto Mobile.

The Latest

March 23, 2017

Mature development organizations ensure automated security is woven into their DevOps practice, early, everywhere, and at scale, according to Sonatype's 2017 DevSecOps Community Survey ...

March 21, 2017

When it comes to food, we all know what's considered "good" and what's "bad". We can all understand this simple rule when eating. But for many, when it comes to software development, simple rules and advice from nutritional labels aren't always there for us ...

March 20, 2017

Monitoring and understanding what software is really doing, and maintaining good levels of software quality is increasingly important to software vendors today. Even a minor bug is capable of shutting down whole systems, and there is a real risk that development cycle pressure competes with quality assurance best practices ...

March 16, 2017

More than half (54 percent) of IT professionals surveyed indicate they have no access to self-service infrastructure, according to a new DevOps survey of 2,000 IT industry executives by Quali.This means that more than half of respondents take a ticket-based approach to infrastructure delivery, impacting productivity and increasing time to market ...

March 15, 2017

Driven by the adoption of cloud and modernization of application architectures, DevOps practices are fast gaining ground in companies that are interested in moving fast – with software eating everything - between "write code and throw it across the wall" to creating more pragmatic mechanisms that induce and maintain operational rigor. The intent behind DevOps (and DevSecOps) is quite noble and excellent in theory. Where it breaks down is in practice ...

March 13, 2017

There might be many people across organizations who claim that they’re using a DevOps approach, but often times, the “best practices” they’re using don’t align with DevOps methodologies. They can say what they do is “DevOps”, but what we’ve found is that many are actually not following basic agile methodology principles, and that’s not DevOps ...

March 09, 2017

The velocity and complexity of software delivery continues to increase as businesses adapt to new economic conditions. Optimizing and automating your deployment pipelines will dramatically reduce your lead times and enable you to deliver software faster and with better quality. Here are three more most common areas that generate the longest lead times ...

March 08, 2017

Every enterprise IT organization is unique in that it will have different bottlenecks and constraints in its deployment pipelines. With that being said, there are some common problem areas that typically produce the longest lead times in your software delivery process. Here are the most common areas that generate the longest lead times ...

March 06, 2017

The findings of an independent survey of IT leaders, application developers and database administrators, conducted by IDG Research for Datical, indicate that database administrators are unable to keep up with the pace and frequency of database changes caused by the accelerated pace of application releases, thus creating a bottleneck and delaying digital transformation initiatives. An overwhelming number of databases administrators (91 percent) and application development managers (90 percent) cited database updates as the cause for application release delays ...

March 02, 2017

A "Boost Caboose" is a secondary engine pulled on a trailer for the explicit purpose of increasing the output of the primary engine. In many ways, DevOps is its own form of Boost Caboose for application of agile methodologies within the modern software factory and SDLC/ADLC processes ...

Share this