For several years, we’ve been providing the most diverse and (by far) the largest mobile device farm for our users. Bitbar Testing has been incrementally growing year by year and more device runs have been executed quarter by quarter. Furthermore, every now and then it’s great to deep dive into all this data, gather and analyze the most interesting aspects of how people use their mobile test automation environments and frameworks.

Test Automation and Unlimited Concurrency are The Enablers

First of all, obtaining an enormous variety of taking a look at runs on device farm isn't potential while not test automation. Take a look at automation is that the enabler that creates coincidental use of mobile devices potential and permits users to run their app and tests at the same time on any variety of devices. this {can be} conjointly one among our sweet spots as we have a tendency to haven’t ever restricted the number of devices that users can use at the same time for his or her tests.

It’s conjointly nice to examine that native app and game developers use identical device roll – affirmative, slightly otherwise and take a look at scripts/used frameworks are totally different – however identical foundation serves each user teams extraordinarily well.

Especially throughout the past 2 years, take a look at automation has inherited play and helped mobile game developers to quickly and simply see however their games work across all potential device variants and obtain that important debugging data on that things ought to be fastened (before releasing). This has made plenty of nice knowledge on however games are optimized for end-user devices.

Overview of taking a look at Runs in Bitbar Testing

During 2016, we have automated mobile testing services to provide the foremost various (devices all around the world, with OS combos, etc.) and also the largest mobile device farm for humanoid and iOS apps, games and internet developers. Among this cluster of users, there are Fortune five hundred firms, a dozen of prime twenty mobile game developer firms, yet as millions of SMBs and indie developers.

Their mobile takes a look at runs to beat thousands of our devices a day, made a huge quantity of information, results, screenshots, recorded videos out of devices runs, and performance statistics on however their mobile apps/games work on these totally different handsets.

The total variety of taking a look at runs exceeded 184 million distinctive device runs, a majority of that was dead on humanoid devices, however with a big growing trend with iOS devices (quarter by quarter). This yields that ‘fragmentation’ is way away to be solved on each major mobile platform – and testing should take its place before prime firms wish to push their apps and games for shoppers.

From the information, about eighty-four of taking a look at runs were done on humanoid devices (and but Sixteen Personality Factor Questionnaire on iOS devices) that provided simply a small amount higher failure rate (17.8%) than on iOS devices (15.5%). this type of finding states that there are millions of problems conjointly on iOS devices, with AN increasing trend. Mainly, this can be owing to new iOS versions and a few new type factors with iOS devices (plus new genus Apis, new notification system, etc).

What’s extremely impressive to examine is that concurrency has been mounting quarter by quarter. The common variety of used mobile devices per take a look at run is currently over ten devices per a test run. 

However, there are still millions of outliers and variation here as many users still use one-device-at-time and a few take a look at automation specialists use tens (or generally even hundreds) of devices at the same time for his or her test runs.

The average take a look at a time conjointly varies loads. As an example, fast and quick coffee tests usually complete quickly and a few in-depth, thorough Appium take a look at runs could take 10x time. Naturally, it’s all concerning the scope of software testing and usually, a lot of logic is enclosed in those Appium tests (that Appium runs cowl purposeful aspects higher than unit-like tests with Espresso).

Most in style take a look at Automation Frameworks

Appium has been the foremost in style framework for purposeful testing of mobile apps, games and, to an exact extent, mobile internet yet (we didn’t embody atomic number 34 in these numbers as a majority of those atomic number 34 tweaks are more or less Appium-based). 

There are millions of sensible reasons why folks are victimization Appium (cross-platform, support for virtually any artificial language, nice coverage with API, etc) and admittedly it's worked extraordinarily well till recent changes with iOS UI Automation.

On Android, coffee and UIAutomator are very talked-about yet. And there also are sensible reasons why folks use/love these frameworks. Coffee provides in no time take a look at execution and UIAutomator provides a light-weight API that's straightforward to adapt and use your native apps. 

Each of those frameworks, however, are somewhat restricted to native apps solely. Again, the bulk of game developers either use Appium or some internal grown/developed framework.

Another (promising) new framework that has to inherit play is that the automaton Framework and it quickly took over the Robotium, which accustomed be one among the thought frameworks within the period of humanoid take a look at automation.

On iOS Appium and Calabash, each being cross-platform frameworks, are in style decisions for iOS take a look at automation. As several game developers are victimization Appium it’s conjointly obvious from the information that usage has been on each major platform, humanoid and iOS.

Another extremely used framework on iOS has been UI Automation (until the half-moon of 2016). Since that there has been (and can be) a transparent replacement movement to another framework and I’ll cowl this with a small amount a lot of details within the next chapter.

On each platform, there also are millions of alternative frameworks that didn’t get listed here. a number of those are internal, proprietary or simply niche frameworks that their users happen to use. As stated, there's no wrong alternative for a framework if that will the duty with efficiency and provides precise results on however apps, games and internet stuff work on real mobile devices.

New Trends with Mobile take a look at Automation Frameworks

Okay, let’s speculate a small amount.

Currently, we have a tendency to don’t see any major changes happening in humanoid take a look at automation. Appium, Calabash, and coffee are going to be positively in high use and possibly even grow stronger as Google tends to favor coffee and there are millions of Appium fans out there yet. Calabash hasn’t provided a lot of new the system late however one among its strengths is that “if you speak English, you'll write a mobile test”.

The most important changes are occurring in the iOS aspect. As UI Automation got deprecated by Apple, XCTest and XCUITest got a wonderful begin to be ensuing huge frameworks in iOS take a look at automation. However, the amendment by Apple hit Appium, because it was wishing on UI Automation as its foundation.