Here we go again. Samsung has apparently been accused by ARS Technica of cheating on benchmarks again. This time it’s with the Note 3. Here’s the deal. ARS Technica was able figure out how it was done and went into a lot of detail to prove it:
After a good bit of sleuthing, we can confidently say that Samsung appears to be artificially boosting the US Note 3’s benchmark scores with a special, high-power CPU mode that kicks in when the device runs a large number of popular benchmarking apps. Samsung did something similar with the international Galaxy S 4’s GPU, but this is the first time we’ve seen the boost on a US device. We also found a way to disable this special CPU mode, so for the first time we can see just how much Samsung’s benchmark optimizations affect benchmark scores.
The smoking gun here is CPU idle speeds, which can be viewed with a system monitor app while using the phone. The above picture shows how differently the CPU treats a benchmarking app from a normal app. Normally, while the Note 3 is idling, three of the four cores shut off to conserve power; the remaining core drops down to a low-power 300MHz mode. However, if you load up just about any popular CPU benchmarking app, the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off. Stopping the CPU from idling shouldn’t in and of itself affect the benchmark scores a whole lot, so this was our first sign that something was wrong. Benchmarks exist to measure the performance of a phone during normal usage, and a device should never treat a benchmark app differently than a normal app.
While it’s difficult to determine every bit of special programming that affects the CPU while a benchmark is running, one sure-fire way to see what’s going on is to trick the phone into not entering a special “benchmark mode” during a benchmark. If we could defeat this behavior, we could have before and after benchmark numbers and thus see just how deep the rabbit hole goes. A bit of testing showed that the device’s boosted benchmark mode is triggered by the package names of the most popular benchmarking apps—loading Geekbench, for example, starts this mode. So we slapped together “Stealthbench,” a renamed version of Geekbench 3. By disassembling the benchmarking app, changing only the package name, and reassembling it, we could run the app without the CPUknowing we were running a benchmark app. The Note 3 should treat our benchmark like any other app and give a true representation of the phone’s performance relative to other devices.
The end result? They determined that this slight of hand boosted benchmark results by about 20%. This did not go unnoticed by Apple. Phil Schiller who is Apple’s marketing chief had this to say via Twitter:
It will be interesting to hear how Samsung defends themselves as this seems pretty clear cut to me. Though you always want to hear the other side of the story.
Like this:
Like Loading...
Related
This entry was posted on October 1, 2013 at 2:01 pm and is filed under Commentary with tags Samsung. You can follow any responses to this entry through the RSS 2.0 feed.
You can leave a response, or trackback from your own site.
Samsung Accused Of Cheating On Benchmarks….Again
Here we go again. Samsung has apparently been accused by ARS Technica of cheating on benchmarks again. This time it’s with the Note 3. Here’s the deal. ARS Technica was able figure out how it was done and went into a lot of detail to prove it:
After a good bit of sleuthing, we can confidently say that Samsung appears to be artificially boosting the US Note 3’s benchmark scores with a special, high-power CPU mode that kicks in when the device runs a large number of popular benchmarking apps. Samsung did something similar with the international Galaxy S 4’s GPU, but this is the first time we’ve seen the boost on a US device. We also found a way to disable this special CPU mode, so for the first time we can see just how much Samsung’s benchmark optimizations affect benchmark scores.
The smoking gun here is CPU idle speeds, which can be viewed with a system monitor app while using the phone. The above picture shows how differently the CPU treats a benchmarking app from a normal app. Normally, while the Note 3 is idling, three of the four cores shut off to conserve power; the remaining core drops down to a low-power 300MHz mode. However, if you load up just about any popular CPU benchmarking app, the Note 3 CPU locks into 2.3GHz mode, the fastest speed possible, and none of the cores ever shut off. Stopping the CPU from idling shouldn’t in and of itself affect the benchmark scores a whole lot, so this was our first sign that something was wrong. Benchmarks exist to measure the performance of a phone during normal usage, and a device should never treat a benchmark app differently than a normal app.
While it’s difficult to determine every bit of special programming that affects the CPU while a benchmark is running, one sure-fire way to see what’s going on is to trick the phone into not entering a special “benchmark mode” during a benchmark. If we could defeat this behavior, we could have before and after benchmark numbers and thus see just how deep the rabbit hole goes. A bit of testing showed that the device’s boosted benchmark mode is triggered by the package names of the most popular benchmarking apps—loading Geekbench, for example, starts this mode. So we slapped together “Stealthbench,” a renamed version of Geekbench 3. By disassembling the benchmarking app, changing only the package name, and reassembling it, we could run the app without the CPUknowing we were running a benchmark app. The Note 3 should treat our benchmark like any other app and give a true representation of the phone’s performance relative to other devices.
The end result? They determined that this slight of hand boosted benchmark results by about 20%. This did not go unnoticed by Apple. Phil Schiller who is Apple’s marketing chief had this to say via Twitter:
It will be interesting to hear how Samsung defends themselves as this seems pretty clear cut to me. Though you always want to hear the other side of the story.
Share this:
Like this:
Related
This entry was posted on October 1, 2013 at 2:01 pm and is filed under Commentary with tags Samsung. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.