What is the performance of your users devices?

Performance

The first advice I give when I talk about measuring performance is to test your app on a lower end device.

The question is: how low end should you go? What is the performance of the devices your users are using?

By cross-referencing:

  • analytics data on some of our wide-audience apps

  • Geekbench device benchmarks

we can establish a baseline!

Attributing a performance score to a device

This is what Geekbench does for us. It runs some benchmarks on phones and attributes a CPU performance score. For instance, here's the result for an iPhone 13

iPhone 13 benchmarks

Geekbench 5 scores are calibrated against a baseline score of 1000 (which is the score of an Intel Core i3-8100). Higher scores are better, with double the score indicating double the performance.

Single-Core vs Multi-Core

When running, an app will spawn multiple threads. For instance, an Android React Native will likely spawn:

  • the JS thread, called mqt_js

  • threads starting with Fresco to display images (Fresco is the RN lib used by default to display images)

  • an OkHttp thread to make API calls

Below is an example of threads using the most CPU in my app, measured with this Flipper plugin image

Single-Core score correlates to the processing power of one CPU core only, basically how fast you can run one single thread. Multi-core score by opposition is how well can you run multiple processes in parallel.

For mobile apps, Single-Core is particularly important, as some threads can induce heavy calculation and can't be parallelized easily:

  • the UI thread, since it's the one responsible to render your app's frames at 60fps

  • the JS thread for React Native apps

For instance, the Samsung Galaxy A21s, one on the most sold phones, scores 170 in its single core benchmark

image

This is roughly 10 times less than the iPhone 13!
This means that the A21s will run JavaScript 10 times slower than the iPhone 13, which is why it is crucial to test on lower end devices.

Analyzing device performance distribution in our apps

Let's check the results for some of our apps.

1st app

User count: 963k on Android / 571k on iOS
Market: the world, but mostly Europe
Data: from March 2022, includes phones and tablets

Single CoreSingle core score distribution

How to read this graph? Well for instance:

  • we have ~210k Android devices scoring between 100 and 150

  • we have ~40k Android devices and ~20k iOS devices scoring between 500 and 550

Some more stats: 🤓

  Android iOS All
Mean 318 1228 655
Median 271 1311 507
33th Percentile 154 1103 288
25th Percentile 136 914 173
10th Percentile 98 724 119

  • 1/3 of Android users have a score below 154

  • 1/4 of all users have a score below 173

This is actually lower than what I expected! Which means a lot of users with low end devices are using the app, so we should ensure their experience is good.

Also, it's safe to say there's a big disparity between Android and iOS! 😅
On average, iOS devices run 5 times faster!

This is why React Native apps tend to run much better on iOS (and also the reason why I tend to focus on Android when measuring performance).

Android insightS

Taking Android alone, 74.8% of users have a score below 400.

Single core score distribution

This is because a lot of lower end devices on Android are quite popular! Here's a breakdown of the top 5 Android phones for this app:

Name User count Score Release year
Samsung Galaxy A51 16.3k 311 2020
Samsung Galaxy A12 10.7k 158 2020
Samsung Galaxy A21s 10.6k 170 2020
Samsung Galaxy A10s 9.3k 135 2019
Samsung Galaxy A10s 9.3k 135 2019
Redmi Note 8 Pro 8.8k 478 2019

Multi-Core

We actually get similar results in multi-core:Multi core score distribution

  Android iOS All
Mean 1135 2908 1793
Median 1079 3075 1433
33th Percentile 486 2131 892
25th Percentile 305 1299 386
10th Percentile 776 2213 1079

(if you're wondering, the very powerful device on the far-right is an iPad Pro)

iOS is still the winner here, even though this time, it's only 3 times faster on average. This is because even low-end Android phones tend to have a lot of cores (6 or 8) compared to iOS phones.

2nd app

Let's compare these results with another app:
User count: 1.21M on Android
Market: France
Data: from March 2022, includes phones and tablets

Single core score distribution

The results are very similar:

  App 1 App 2
Mean 318 325
Median 271 271
33th Percentile 154 169
25th Percentile 136 143
10th Percentile 98 105

The most sold Android devices differ only slightly:

Name User count Score Release year
Samsung Galaxy A20e 33.5k 231 2019
Samsung Galaxy A21s 25.3k 170 2020
Samsung Galaxy A12 22.3k 158 2020
Samsung Galaxy A51 19.7k 311 2020
Samsung Galaxy A10 17.6k 200 2019

 

Conclusion

Based on this evidence, it is crucial to use a low-end Android device to measure performance. If you just test your app on an iPhone 13, you will miss most of the issues that your users can have on the widely sold A21s, which runs 10 times slower.

At BAM, we chose to go for a Samsung J3 2017, scoring 107, better than ~15% of our Android users devices.