Test on Mobile Devices That Matter to Your Business
June 15, 2015

Amir Rozenberg
Perfecto

Choosing the right smartphones and tablets to test your apps against is just as important as the tests themselves. After all, if your app isn’t working at the right place (your customer’s device) at the right time (all the time), angry customer tweets and two-star app reviews will follow and the consequences for your business will be severe.
 
In some cases, your app won’t even make it into the app store. Twenty percent of apps are rejected from the Apple store because they’re buggy, crash-prone or have an inferior user interface. And when a new version of Android arrives, apps that worked fine on the previous OS version start to act like rejects themselves.
 
On the hardware side of the fence, new devices arrive every quarter, and device usage varies by geography. Windows Phones may have 3.8% market share in the US but in Europe it has an average market share of close to 9%, according to mobile market research firm, ComTech. Basing your testing of Windows Phones solely on US market data when your app also has a presence in, say, France (12%) and Italy (13.3%), would be a costly mistake.
 
The bottom line is that mobile devices and operating systems are always in flux. You need to keep a close eye on market changes so you’ll be testing your apps and responsive websites against the devices that represent your target users. To that end, we created a Mobile Test Coverage Index report, a quarterly release that aggregates data from over 4,000 device profiles and 360,000 hours of customer usage data, as well as mobile market share data and future device release timelines, to create a list of the 32 most relevant mobile devices.
 
The device selections address five key sectors: device model, screen size, GA date, hardware system and recommended operating systems. The number of devices in each test bracket may vary from quarter to quarter to reflect the restless mobile market. The report will be continuously updated to meet market changes.

While the report will not hand-deliver an answer for what devices and operating systems the company should specifically test against, it can be used as a benchmark from which it can customize their test coverage according to their own business and customer needs.
 
Some noteworthy insights from the research:


■ Almost a third (30 percent) of the market can be covered with 10-16 devices – including popular devices such as the Samsung Galaxy S5, iPhone 6 and HTC One M8 – based on market adoption, market device leaders, reference devices and device characteristics.

■ Regarding operating systems: iOS 7 still accounts for some 17 percent of total users out there, and should be included at the same level of iOS 8 test plans. Additionally, Android 5.0 adoption rates are low in both the US and Europe with Android KitKat (4.4.x) and JellyBean (4.2.x, 4.3.x) rating as the most commonly used versions.

■ Geographically speaking, iPhones hold twice as much share in the US than in Europe, with iPhone devices capturing 35% of the U.S. market, as opposed to 16% in EU5.

We can’t overstate how important device coverage is to a mobile app testing strategy, for both business and technical executives alike. A relevant device and operating system mix will get an organization much closer to the true end-user experience. However, it’s a challenge to keep your test coverage relevant given that mobile devices have a shelf life of around nine months. Benchmarks built from data like this report will help, but in an ever-changing market it’s essential to keep reviewing continually updated research.

Amir Rozenberg is Director of Product Management at Perfecto

The Latest

May 17, 2018

The top two business priorities for CIOs of midsize enterprises (MSEs) in 2018 are growth and digital transformation. However, 57 per cent of MSEs are not yet delivering digital initiatives, according to findings from Gartner Inc.'s 2018 CIO Agenda Survey ...

May 15, 2018

Almost every company is facing the challenge of digital transformation today. This means rethinking and retooling your company to compete and succeed in an increasingly digital world. While digital transformation is not only about technology, the right tools can help. To find out what these right tools are, APMdigest asked experts from across the IT industry for their opinions on the essential tools to support digital transformation ...

May 08, 2018

With data breaches consistently being in the news over the last several years, it is no wonder why data privacy has become such a hot topic and why the European Union (EU) has put in place General Data Protection Regulation (GDPR) which will become enforceable on May 25, 2018, which is less than a month away ...

May 03, 2018

The prospect of increased workloads, combined with shrinking mainframe skillsets, has huge implications for mainframe DevOps. The only way for organizations to solve this skills gap crisis is by optimizing developer productivity. Drilling down a level further, what does this all mean for mainframe DevOps? ...

May 02, 2018

When it comes to operations and development, DevOps has changed the traditional compartmentalized style of development by eliminating silos. But what about the security team? Security is largely still siloed from operations and development. No doubt, many DevOps teams have some security controls baked into their automation processes, but a recent survey shows there are still alarming gaps ...

April 30, 2018

According to the 2018 Global Security Trends in the Cloud report, 93 percent of respondents faced challenges when deploying their current on-premises security tools in the cloud, and 97 percent lacked the tools, cross-functional collaboration and resources to gain proper insight into security across the organization. These numbers indicate a big problem in DevSecOps that needs to be addressed ...

April 26, 2018

Moving more workloads to the cloud is a top IT priority, so eventually it will be time to consider how to make those critical legacy applications cloud ready. In Part 1 of this blog, I outlined the first four of eight steps to chart your cloud journey. In addition, consider the next four steps below ...

April 25, 2018

Clearly, moving applications to the cloud delivers significant advantages. So what's standing in the way of full cloud adoption? For many companies it's those burdensome (but critically important) legacy applications. Moving more workloads to the cloud is a top IT priority. So, eventually it will be time to consider how to make those critical legacy applications cloud ready. Consider the following eight steps to chart your cloud journey ...

April 24, 2018

Developers and engineering teams are under increasing pressure to release higher quality software faster. Continuous testing has proven to be central to these efforts as it helps eliminate bottlenecks and ensures that automated testing is a constant throughout the development process, not an exercise relegated to the "last mile." The value of automated testing is more evident than ever before, with nearly half the respondents reporting that management is fully committed to automated testing and with plans to increase spending, according to the recent Sauce Labs Testing Trends for 2018 report ...

April 19, 2018

As development speed has become a competitive advantage, the DevOps team has sought to enable continuous integration and continuous delivery (CI/CD). For the CI/CD process to be successful, it must be fast and efficient. Any potential roadblocks that delay any part of the process increase cycle times and slow down delivery ...

Share this