Global Nerdy

They’re (Almost) All Dirty: The State of Cheating in Android Benchmarks is a pretty damning headline, but that’s only the start. In the article with that title, Anand Lai Shimpi (AnandTech’s namesake) and Brian Klug take a look at how various OEMs try to make themselves look better by detecting the execution of well-known benchmarking applications and in response tweaking the CPUs to run at full speed and to not drop into power-saving mode. Yes, you could run all the cores on a mobile device CPU at full speed constantly for all apps, but that isn’t done because doing so is such a power drain that you’d be recharging a half-dozen times a day. This practice of running all cores at full speed when a benchmarking app is detected is at best the equivalent of pretending to look really busy when the boss is walking by; at worst, it’s lying to the consumer.

In their article, they posted a table showing a number of current Android Devices and the benchmark tests for which they cheat. I love the title of their table, which I’ve copied and pasted below: I Can’t Believe I Have to Make This Table:

They also write (the emphasis is mine):

We started piecing this data together back in July, and even had conversations with both silicon vendors and OEMs about getting it to stop.

With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization.

It’s possible that older Motorola devices might’ve done the same thing, but none of the newer devices we have on hand exhibited the behavior. It’s a systemic problem that seems to have surfaced over the last two years, and one that extends far beyond Samsung.

Kudos to Apple and Motorola for not playing this game, and to the other OEMs: cut it out. Even if take away the issue of right versus wrong, there’s still a coldly Macchiavellian, pragmatic reason to not do so. The performance gains you get from rigging benchmark tests in this fashion are below 10%, and they’re not worth the bad press you’ll have to live with for the next little while.

Ladies Learning Code, who describe themselves as “a Toronto-based not-for-profit organization that runs workshops for women (and men) who want to learn beginner-friendly computer programming and other technical skills in a social and collaborative way” are celebrating their second birthday tomorrow night, Thursday, October 3rd at 7:30 p.m. at Hotel Ocho (195 Spadina Avenue, just over a block north of Queen). They’re throwing a fundraising party, and if previous get-togethers of theirs are any indication, this one should be a blast. If I weren’t Tampa, I’d be going to the party.

Tickets are $25.00 for early birds (you’ve got until about 11:00 p.m. on Wednesday, October 2nd to grab those) or $35.00 for not-so-early birds — they’re available at the event’s EventBrite page.

Want to know more about what Ladies Learning Code does? Check out this recent video by XDA Developers:

Reg Braithwaite and his book, JavaScript Allongé.

Reg Braithwaite’s book, JavaScript Allongé, which shows you how to make the most of JavaScript’s objects, functions, combinators, decorators, and quirks, is now free in both the copying and costs senses of the word. He’s put it under a Creative Commons Attribution-ShareAlike 3.0 Unported Licence, which means you can share, remix, and even make commercial use of it. He’s also saved you twenty dollars and made it available for free, and you can read it online as well as get the Markdown source.

He writes:

The book is barely a blip on the publishing scene when we judge JavaScript Allongé against titles like “Get your dream job as a Web Developer in 21 Days!” Those kinds of books sell bythe container-load. But nevertheless, people read it and tell me that it levels them up, and AFAIC that is a success. And as a bonus, I got my dream job, and now it’s time for me to walk the freedom talk. I’m now working for a company that is trying its dambdest to help every single programmer on the planet write more software and better software. And we give something away, crossing our fingers that more software and better software will be a good thing and that we’ll make enough to keep doing what we do.

I believe in this. That’s why I write, and that’s why I work at GitHub. And you helped me get here.

If you want to take your JavaScript to the next level, JavaScript Allongé is the book for you. Reg does a lot of deep thinking about how to write better code, and there’s a lot of that thinking in his book. Do yourself a favour — especially since it’s free now — and read it!

I like CollegeHumor’s graphic of the Seven Deadly Sins, with each sin rendered using the logo of the a suitably-matched site:

This article also appears in The Adventures of Accordion Guy in the 21st Century.

Here’s the current voting status at NextMicrosoftCEO.com, a site where you can throw in your two cents and vote for who you think will be the next leader of The Empire:

As of this writing, Valve’s Gabe Newell is in the lead, followed by Richard Stallman, then Linus Torvalds. No sign of former-Microsoftie-turned-Nokia CEO-turned back to Microsoftie Stephen Elop or that guy from Ford.

When the reviewers at Ars Technica noticed that Samsung’s Galaxy Note 3 “phablet” scored strikingly better than LG’s G2, despite having the exact same processor — a 2.3 GHz Snapdragon 800 processor — they decided to dig a little deeper. Their conclusion:

After a good bit of sleuthing, we can confidently say that Samsung appears to be artificially boosting the US Note 3’s benchmark scores with a special, high-power CPU mode that kicks in when the device runs a large number of popular benchmarking apps.

This isn’t the first time that Samsung has been caught artificially boosting benchmark results, but Ars Technica reports that this is the first time it’s been observed in a Samsung device shipped to the U.S.. Samsung defends the practice by saying that they boost performance for other apps as well.

In their investigation, Ars Technica did a fair bit of software disassembly and found the code that was actually boosting the speed, shown below. Remember that names and labels often get mangled in the compilation process, and that in the decompilation process, one often has to specify new names and labels, so while the logic of the code will be the same as the original source, the variable and constant names might not be:

{
BOARD_PLATFORM = SystemProperties.get("ro.board.platform");
mToken = 0;
PACKAGES_FOR_LCD_FRAME_RATE_ADJUSTMENT = new PackageInfo[0];
isEngBinary = "eng".equals(Build.TYPE);
PackageInfo[] arrayOfPackageInfo = new PackageInfo[26];
arrayOfPackageInfo[0] = new PackageInfo("com.aurorasoftworks.quadrant.ui.standard", false);
arrayOfPackageInfo[1] = new PackageInfo("com.aurorasoftworks.quadrant.ui.advanced", false);
arrayOfPackageInfo[2] = new PackageInfo("com.aurorasoftworks.quadrant.ui.professional", false);
arrayOfPackageInfo[3] = new PackageInfo("com.redlicense.benchmark.sqlite", false);
arrayOfPackageInfo[4] = new PackageInfo("com.antutu.ABenchMark", false);
arrayOfPackageInfo[5] = new PackageInfo("com.greenecomputing.linpack", false);
arrayOfPackageInfo[6] = new PackageInfo("com.greenecomputing.linpackpro", false);
arrayOfPackageInfo[7] = new PackageInfo("com.glbenchmark.glbenchmark27", false);
arrayOfPackageInfo[8] = new PackageInfo("com.glbenchmark.glbenchmark25", false);
arrayOfPackageInfo[9] = new PackageInfo("com.glbenchmark.glbenchmark21", false);
arrayOfPackageInfo[10] = new PackageInfo("ca.primatelabs.geekbench2", false);
arrayOfPackageInfo[11] = new PackageInfo("com.eembc.coremark", false);
arrayOfPackageInfo[12] = new PackageInfo("com.flexycore.caffeinemark", false);
arrayOfPackageInfo[13] = new PackageInfo("eu.chainfire.cfbench", false);
arrayOfPackageInfo[14] = new PackageInfo("gr.androiddev.BenchmarkPi", false);
arrayOfPackageInfo[15] = new PackageInfo("com.smartbench.twelve", false);
arrayOfPackageInfo[16] = new PackageInfo("com.passmark.pt_mobile", false);
arrayOfPackageInfo[17] = new PackageInfo("se.nena.nenamark2", false);
arrayOfPackageInfo[18] = new PackageInfo("com.samsung.benchmarks", false);
arrayOfPackageInfo[19] = new PackageInfo("com.samsung.benchmarks:db", false);
arrayOfPackageInfo[20] = new PackageInfo("com.samsung.benchmarks:es1", false);
arrayOfPackageInfo[21] = new PackageInfo("com.samsung.benchmarks:es2", false);
arrayOfPackageInfo[22] = new PackageInfo("com.samsung.benchmarks:g2d", false);
arrayOfPackageInfo[23] = new PackageInfo("com.samsung.benchmarks:fs", false);
arrayOfPackageInfo[24] = new PackageInfo("com.samsung.benchmarks:ks", false);
arrayOfPackageInfo[25] = new PackageInfo("com.samsung.benchmarks:cpu", false);
PACKAGES_FOR_BOOST_ALL_ADJUSTMENT = arrayOfPackageInfo;
mCameraCPUBooster = null;
mCameraCPUCoreNumBooster = null;
mCPUFrequencyTable = null;
mCPUCoreTable = null;
mRotationCPUCoreNumBooster = null;
mRotationGPUBooster = null;
}

The contents of the array arrayOfPackageInfo are the identifiers of well-known and often-used apps for benchmarking the performance of Android devices. The end result of this check is that it’s used by other code to alter CPU behaviour so that:

• If an app that has any one of these identifiers is running, the Note 3 stays in a mode with all 4 of its CPU cores active and running at full-speed 2.3 GHz mode. This special behaviour yields excellent benchmark results.
• If no such app is running, the Note 3 follows its default behaviour, which is to switch to a one-core-only energy-saving mode, with that single core running at 300 MHz. This normal behaviour extends battery life to something reasonable.

With such artificial boosting, the Note 3 gets impressive results. But by compiling Geekbench under a different name and thereby bypassing identification by the code above, Ars Technica were able to get more honest results:

Ars Technica astutely note:

The ironic thing is that even with the benchmark booster disabled, the Note 3 still comes out faster than the G2 in this test. If the intent behind the boosting was simply to ensure that the Note 3 came out ahead in the benchmark race, it doesn’t appear to have been necessary in the first place.

It’s a little crass, it’s a little desperate, and it’s…well, so, so Samgsung-y. That’s a real shame, because they do produce good stuff (I myself count a Galaxy S3 and a number of Samsung monitors among my devices).

Here’s something interesting for all you iPhone users and developers: a video featuring a side-by-side comparison of stock, freshly-restored, non jailbroken iPhones ranging from the newest and most powerful 5S, the also-new 5C, and every iPhone dating back to the “2G” (the colloquial name for the original iPhone released back in 2007):

One valuable take-away from this video: you don’t have to upgrade every time Apple release a new iPhone, as the difference isn’t all that dramatic from one version to the next. Unless you absolutely need a feature in the latest model, you may want to follow my strategy of upgrading every other or even every third model — that’s when you’ll see the dramatic differences.