Smartphone benchmarks have always been important to everyone even if someone does not care about benchmarks, they will at least take a peek at smartphone benchmarks.
The real question here is, Do the Benchmarks matter or not?
The answer to this question is not a straight ‘Yes’ or ‘No’. Benchmarks have their pros and cons.
Smartphone benchmarks have evolved so much by now that we have benchmarks for every type of user demand.
First, we need to understand what a benchmark is? Only after that will we be able to say whether it matters or not.
What are Benchmarks?
Benchmark is a software that runs a simulation to replicate the real-life usage of a human. The real-life usage of a smartphone varies from user to user. Some users like to play games, some users watch Netflix and Youtube, some users make a lot of calls and SMS.
As you can see, there is a lot of variety in real-life usage. That being said, one benchmark software can not simulate all of these tasks. Most of the Benchmark software is made to perform just one task. This task could be CPU (Central Processing Unit or Processor) intensive, GPU (Graphics Processing Unit) intensive, browser-based or latency tests.
Some benchmarks like Antutu and Velamo will try to perform all of these tasks and give you a result. But the problem with these benchmarks is, they are not very comprehensive or detailed.
Therefore, it is better to run different benchmarks software for different tasks.
Watch the video below if you want to get a second perspective from a renowned Youtuber.
The most detailed benchmark based on CPU intensive tasks is GeekBench. For the GPU, it is 3D Mark. For normal smartphone usage, it is PC Mark, etc, etc. We will discuss all types of benchmarks below.
Due to the various real-life usage of humans, running just one benchmark is not a good idea. Instead, there are different benchmarks for each user.
Do Benchmarks Matter
Now to the million dollar question, do the benchmarks even matter?
One line answer, ‘Yes’.
If the benchmarks do not matter then why is their popularity being increased each day?
I will try to explain with an example why benchmarks matter.
Assume you are planning a party and you are rehearsing it one day before the party. You would try to do all the things, perform all the events originally planned, and plan for anything that might go wrong. But you can not include and plan for all the factors and things that would go wrong on the party day. All you can do is reduce the variability of anything going wrong.
This is why you have a rehearsal day before the party day so that you can minimize the things that could go wrong. Still, on the party day, a lot of things would be different than the rehearsal day. Something might go wrong, something that you have not even thought of because humans do not have any pattern.
The benchmark is like the rehearsal day. The scores on the benchmarks can not ever reflect the real-life performance of a smartphone because of human variability. But it gives you a pretty good idea of what the real-life performance would be. But you need to be using the right benchmark softwares.
Simply put, a benchmark is a simulation of real-life usage of a smartphone. In a simulation, various factors that might change the results in real life are not considered. A benchmark is trying to imitate a perfect human who only uses his/her smartphone linearly.
If you are using the right benchmark software depending on your real-life usage, you can get a pretty good idea if a specific smartphone would be good for your needs.
Why Benchmarks are Flawed?
While benchmarks do matter but the way most people use them is not really the right way to use them.
Normally, we run a benchmark and it completes all of its tests in about 3 minutes depending on the benchmark software. So basically, benchmarks put a lot of load on the smartphone for a few minutes which is rarely the case in real-life usage.
The benchmark, in this case, is basically a stress test. It determines what is the peak performance of a smartphone.
In real life, we sometimes put the smartphone under load for a long time even hours (Gaming) and sometimes for only a few minutes (Checking Notifications). So running a software that only simulates our real-life tasks for only 3 minutes or so can never give us a proper estimate of real-life performance.
Almost all the benchmarks provide only peak performance and not sustained performance. Sustained performance is the performance that a smartphone maintains after a long time of full load.
Normally, a smartphone would throttle its performance down to 60-70% after applying a 100% load for a long time. This is the reason why smartphones with the same processor have different performance because of the lower sustained performance.
Another problem, we often rely on benchmark softwares that do not perform the tasks that we do. For example, a person who uses his/her smartphone only for casual usage. Casual usage includes checking emails, calls, messages, consuming media. If this specific person is trying to get an idea of a smartphone performance by running Antutu Benchmark, he can not.
Antutu benchmark is made for power users, not for casual users. Antutu benchmarks put stress on CPU, GPU and memory. Common thinking would be, testing these three components would provide a pretty good estimate of how a smartphone will perform for a casual user which is definitely not the case.
Another case, running a 3DMark benchmark when you do not even use your smartphone for gaming is not right for you.
Similarly, running an Antutu benchmark and comparing the scores to see which one is the fastest, no please no…
It does not work like that. Antutu benchmarks perform CPU, GPU and memory tests. If one smartphone has a higher score on the Antutu benchmark does not mean it is faster than the others.
It means that in the tasks thrown at the smartphone by Antutu benchmarks, that smartphone is the best at running those tasks. This does not mean that the smartphone with higher Antutu would beat the other smartphones in every task or set of tasks.
Even if you just compare the Antutu CPU score of a specific smartphone with the other smartphones Antutu CPU scores, you can not be certain that the higher CPU score smartphone will perform better in every CPU intensive task.
This is why I like to use different benchmark softwares for CPU, GPU, Browser and normal usage.
When a benchmark is just based on one type of task, it gives better results. The benchmark focused on just one area like CPU will perform a lot of CPU related tasks compared to a benchmark like Antutu.
The proper way to use Benchmarks
The benchmarks may be flawed and may not reflect the real-life usage of a smartphone. Still, there are some ways you can use to utilize them.
But first, you need to look at your real-life usage and select a benchmark or benchmarks that fits your usage style. Once you have selected a benchmark software, follow the steps below.
As I have mentioned above, benchmarks only give us the peak performance of a smartphone. We are not interested in peak performance at all.
In order to replicate real-life usage, sustained performance is a lot more important than peak performance.
So how do we get sustained performance when most benchmarks do not even offer sustained performance.
One way to do this is to run a benchmark test again and again until the performance of the smartphone decreases. Smartphone performance would keep reducing.
At a certain point, the smartphone would stop reducing its performance. At this point, no matter how many more times you run a benchmark, the smartphone performance would not reduce.
Congratulations! This is the sustained performance of your smartphone. In most cases, it would be about 60-70% of the peak performance but could be less than that.
How to Benchmark a Device Properly?
The device should be formatted and factory reset. There should be no data or applications on the smartphone except your Gmail or Google Account. Google account is required to install the applications.
Device should be restarted before performing the tests.
Connect the device to a Wifi connection and turn on the Airplane mode to avoid getting any notifications during benchmarks.
Perform the benchmarks at least three times to get the peak performance for a particular benchmark.
To get sustained performance, you need to repeat the benchmarks again and again for 30 minutes at least.
Other ways to get sustained performance are the CPU throttling Test and the GFXBench Battery Test.
The CPU throttling test is to get the CPU sustained performance and you are supposed to run for 1 hour to get an accurate score.
The GFXBench Battery Test is used to get GPU sustained performance.
This is a battery test as well as GPU sustained performance test.
Charge the device to 100% and start the test. Normally the sustained GPU performance is about 60-70% of the peak performance. If it is lower than 60%, the smartphone is not good for long time gaming.
I used CPU throttling test on three devices and the results were surprising.
LG G6 and Google Pixel XL 2016 both have Snapdragon 821 processor. The LG G6 CPU clock speed is higher than Google Pixel XL 2016. Still, Google Pixel XL 2016 sustained performance blew away the LG G6. To be honest, LG G6 sustained CPU performance is disappointing.
The tests was performed for only 15 minutes on both devices.
LG G6 CPU throttled to 58% average throughout the test.
Google Pixel XL 2016 CPU throttled to 75% average throughout the test.
I also tested this on LG V30. The performance on LG V30 was nothing short of impressive.
I ran the test twice, one for 15 minutes and second for 30 minutes.
In the first 15 minutes test, LG V30 CPU throttled down to 86% average throughout the test.
In the second 30 minutes test, LG V30 CPU throttled down to 70% average throughout the test.
The results on the LG V30 are really good. Most smartphones are at 60% performance after the 30 minutes test.
This is another proof that a smartphone performance reduces significantly when it put under load for a long time.
Types of Benchmarks for different Users:
I hope you understand by now that there is no one magic benchmark that would fit every user’s real-life usage. Due to this, each user should use a benchmark or benchmarks that would replicate their real-life usage and not one benchmark rules all (Antutu benchmark).
Therefore, we would like to look organize all the benchmarks available based on various users around the world. So, next time you want to check a smartphone, you know what benchmarks you need to run on it.
Benchmarks for Power users:
Power users are the users who use their smartphones a lot throughout the day. Power users do not just use the phone but their usage involves putting the smartphone under constant load for a long period of time.
Power user’s usage involves putting stress on the CPU, GPU, Memory and pushes the thermal limit of the cooling solution. The main thing power users should be looking at is the sustained performance of CPU and GPU.
For example, a technology YouTuber that uses his/her smartphone for video recording and then uses the same smartphone for rendering and editing the video. The smartphone is under constant stress in video recording and rendering. This type of usage requires sustained and high CPU performance and a good heating solution.
Another example of a power user, A gamer who streams on twitch or YouTube using his smartphone. Even after the stream, the gamer plays a lot of games throughout the day. This type of usage requires good sustained GPU performance and a good thermal solution.
The following are the benchmarks that simulate the real-life usage of a power user.
- GeekBench 5 for CPU
- 3DMark For GPU
- PCMark for everyday tasks
- GFXBench Battery Test
- CPU Throttling test for CPU sustained performance
- CPDT benchmark for internal storage and RAM benchmark
You need to run all of these benchmarks at least three times to get accurate results.
Run the GFXBench battery test for sustained GPU performance.
I did not forget about the Antutu benchmark. Antutu benchmark is for beginners who do not understand benchmarks and not for power users who understand benchmarks.
Run the CPU throttling for 30 minutes to get an accurate sustained performance.
Benchmarks for Casual Users:
Casual users consist of a variety of users. Most of the casual users use their smartphone for calls, SMS, emails, social media, web browsing, video playback, music playback, Light games or 2D games, and reading.
Almost 70-80% of smartphone consumers around the word are casual users. This is why most of the tasks included in this area are also used by other types of users.
I think a smartphone that performs well in this segment can be used by any type of user. The reason is most manufacturers optimize their smartphones for the gamer’s or power users because these users understand the technical jargon. Also, big numbers are good for marketing.
As I explained previously, these marketing numbers only represent peak performance and not sustained performance which is more important.
A casual user normally does not want to understand the technical terms. So, they just look at the benchmark numbers to check the performance of the smartphone.
Now, there is nothing wrong with looking at the benchmark numbers but you should look at the benchmark numbers produced by a benchmark software that replicates your real-life usage.
There is not just one software that you could run to get a feel of smartphone performance for a casual user. Actually, there are multiple benchmark softwares you need to run:
- Passmark Performance Test is an all-purpose benchmark.
- PCMark is to test everyday use which is essential for casual users.
- GFXBench Battery Test
- AndroBench for storage benchmark
- Antutu Benchmark for CPU and GPU
The tests for casual are focused on more than one thing. But you need to run the CPU and GPU tests for at least 30 mins to get a good idea real sustained performance of your smartphone.
Run the GFXBench battery test for sustained GPU performance.
The most important benchmark for casual users is probably PCMark and the most important component to check for performance is storage speeds. So you need to run each and every test in PCMark.
Benchmarks for Gamers:
Mobile Gaming trend is rising day by day. Some games like PUBG, Fortnite, Clash of Clans, Clash Royale have taken mobile gaming to the next level.
It is now more important than ever to test a smartphone for gaming.
Just like checking a smartphone for a power user requires a lot of patience and effort. It is the same with gaming performance.
The real nuisance in a smartphone is the heating when the smartphone is subjected to a load for a prolonged time. As the smartphone heats, the thermal restrictions set by the manufacturer require the smartphone to reduce the CPU or GPU performance to maintain the smartphone temperatures.
This is why we need to know the sustained GPU and CPU performance of smartphone instead of just peak performance.
The benchmarks to use to test a smartphone for gaming are:
Run all the tests in 3DMark for OpenGL and Vulkan API at least three times.
GFXBench consists of a lot of small benchmarks. Run the GFXBench battery test for sustained GPU performance.
Benchmarks for Non-Techies:
This section is for the people who do not know what is exactly their real-life usage is and also do not want to spend a lot of time going through boring benchmark stuff.
If you are a newbie and do not understand a thing about benchmarks, this section is for you.
First, download and install Geekbench on your smartphone. Run the benchmark and compare the results with the Geekbench website.
I want you to download and run the PCMark benchmark on your smartphone.
You should see the left picture when you install and open PCMark10. If you click on ‘SWIPE FOR MORE TESTS’, you would get to the benchmark types which is shown in the right picture.
You need to install and run Work 2.0 and storage tests. You need to run these benchmarks at least 3 times.
This test is used to check the general performance of smartphones. General performance includes web browsing, video playback, reading and writing a PDF file, etc.
The next part is we need to check the 3D performance. Download and install 3Dmark from the play store. Run the 3DMark ‘Sling Shot Extreme’ for both OpenGL and Vulkan API at least three times.
Once you are done, it is time to check if your smartphone is performing well or not. If your smartphone is giving almost the same as the website, your benchmark went well and now you can get an idea of what your smartphone is capable of.
Go to Google.com and search ‘Your phone model UL Benchmarks’ as shown in the picture below.
Open the first result and then compare the scores.
For 3Dmark slingshot extreme. If the FPS for the 2nd and third are the same as the website, then it is good. If the FPS is too low on your device, there may be a problem.
For PCMark Android work 2.0. Compare this score with any other smartphone on the same website.
For Storage benchmark of PCMark. Look at the random read and random write. If at least one of them is above 10MB/s, your smartphone storage is good.
Conclusion:
Benchmarks do matter when you are using the right benchmark software depending on your use and you are benchmarking the right way.
Benchmarks do not perfectly reflect the real-life usage but they can come close if you benchmark a smartphone the right way.
Benchmarks that have a short duration of 5 minutes or less provides peak performance and not sustained performance. Sustained performance is way more important than peak performance.
Every time of smartphone user has to run different benchmark tests depending on his real-life usage.
Great content! Super high-quality! Keep it up! 🙂