Qualcomm let SemiAccurate benchmark their upcoming Snapdragon 820 SoC and it held one big surprise. The numbers lined up about where everyone expected them to, the surprise was how it felt.
Benchmarking has become a bit of an anomaly in the modern tech world, both highly accurate and also somewhat irrelevant. This means it is very easy to get numbers from a given program or test that compare directly to the same tests on other hardware. That is the easy part, A:B, 53FPS:41FPS, big number:small number, all comparable. Unless the tests don’t actually measure what you think they do, something that is far more common than anyone believes but this is all an argument for another time.
What is more problematic is the bit I am calling somewhat irrelevant, what those numbers mean. How much better does a score of 72K vs 11K on 3DScore2016R3.1c make a phone feel to the user? Will they notice the difference if an app is coded for a single core A9 era device with a GPU to match? In short how much of these numerical gains translate into anything the user sees or does? For the most part the bottlenecks in a modern phone/mobile device are on the network which no benchmarks really test and those that do are not really reliable or repeatable. In short do these numbers mean a damn thing?
With that in mind let me present you with the numbers, you can do with them what you wish, then we will get on to the part about mattering. Or not mattering. A couple of weeks ago, Qualcomm let SemiAccurate loose on an 820 development platform, in this case an MDP/S (Mobile Development Platform/Smartphone) from Intrinsyc. It looked like a smartphone, acted like a smartphone, and ran all the expected tests in the expected manner. There were a few minor far-pre-release system bugs but that is expected and was nothing unusual, basically the systems would misbehave occasionally and need rebooting. They looked like this.
Snapdragon 820 MDP/S platform with an LG G2
The specs on the system were fairly well-known, a 1.6GHz quad-core Kryo CPU, Adreno 530 GPU running at between 133-624MHz, and the full complement of ISPs, DSPs, and the like. There was 3GB of LPDDR4 at an unknown frequency, and 64GB of UFS flash. One interesting bit that stands out is the GPU is OpenGL ES3.1 + AEP which was actually taken advantage of by the Kishonti GFXBench – Car Chase sub-test. It looked purty, especially on the MDP/S’s 2560*1600 screen. Can you imagine if modern laptops had screen like this available to the mainstream like phones? Yes that was a dig at a few someones….
So here we go, in no particular order, and not in a pretty format either. If you need slick graphs and animation, please stare at the screen until the page finishes loading, it will happen, trust us. Wait… The rest of you can laugh along with us. Some commentary inserted where needed to break up the monotony of the results.
————<Begin *YAWN* part>————
The excitement starts with Antutu v6.0 Beta, ironic because it had been released in some markets at the time of testing but not in the US so technically it was still Beta. The MDP/S scored 132304 overall, 53233 on the 3D subtest, 40492 on UX, 30143 on CPU, and 8436 on RAM. Those who thought it would hit at least 40500 on UX just lost your bets. That is what we mean by commentary to break up the monotony, it is just as bad to write this stuff as it is to read.
Geekbench 3 brought the excitement back to testing with a single core score of 2301 and a multi-core score of 5459. For sub-tests it hit 2107 and 5758 on integer, single and multi-core respectively, 2107 and 5758 on FP for S/M once again, 3437 and 3917 on memory S/M, and if you care it was v3.3.2. Because of the abbreviations used for single and multi-core, we will limit snide commentary on this section, it is a family site after all. If you want the numbers for the raw sub-tests, feel free to email and ask.
Kishonti’s GFXBench was much more exciting because it has the below picture worthy sub-test, Car Chase. It is picture worthy because it includes OpenGL ES 3.1+AES support, and that like the cars pictured is new and shiny. It is the future of GPU standards and the 820’s Adreno 530 GPU supports it now. You should be excited, it is really a solid step forward for mobile graphics and for the moment, Car Chase is the only thing that can test it. [Editor’s note: If you are reading this on a typical laptop screen it will not do justice to the image from the phone screen below.]
Car Chase in full OpenGL ES 3.1+AES glory
On to the dry as twice toasted bread stuff, the numbers. They are all listed as frames on-screen, in this case a reported 2434*1600 and then off-screen 1920*1200 with a slash in the middle. The scores for Car Chase were 558.0/1046, Manhattan ES 3.1 892/1873, Manhattan ES3.0 1617/2948, and T-Rex 3108, 5074. For the sub-tests we had Tessellation at 1318/3606, ALU2 1300/4743, Diver Overhead 2 298.1/901.4, Texturing 7703/7789, and finally Render Quality 2512/3625. As you can see from the Diver results, the digit after the decimal point is used in GFXBench but we are not sure why we are making a big deal of it.
For the tests that really don’t matter but people seem to quote and whine about anyway we have three more for you. Those are an Octane score of 11440, 3DMark Ice Storm Unlimited at 24552, and Sling Shot ES 3.1 scored 2248. Lastly and least-ly, if you are wondering we do mean to deprecate this test, we have Kraken in picture form. It isn’t worth transcribing the fiddly numbers that mean nothing to the real world so we won’t. Enjoy?
Not sure why but Kraken numbers
So those are the numbers, both exciting, informative, and other adjectives that some people find enthralling. To us they were just numbers that do show how well the 820 SoC compares to other SoCs when in a similarly thermally constraining form factor. Please note that last bit when certain unethical vendors inevitably compare much higher wattage form factors to their competition later on, it will happen again this time, trust us. So now you have a set of numbers for the 820 you can do with what you want and draw conclusions on. Let the endless forum debates begin!
————</Begin *YAWN* part>————
Now for the less deterministic part, what do these numbers mean? How does that relate to your experience and end-user feel? In short do these bigger and better numbers actually mean anything to someone buying a Snapdragon 820 based device? That is the several hundred million if not billion dollar question, do these advances actually matter enough to buy an 820 based device over something else, or more likely is it worth the money to upgrade? That is what SemiAccurate set out to test with the 820, and cliché as it sounds, we were surprised by the results.
First the obligatory blurb about our testing methodology which was somewhat limited by our inability to test the modem in the 820. We were on fairly decent speed Wi-Fi for the test, and we replicated all of them on our current device, an LG G2 with a Qualcomm Snapdragon 801 on the same wireless network. What did we do? Basically the test consisted of going to all of the sites that I go to in a normal day that are anything but trivial. You probably have a few places you surf to regularly that bring your phone to its knees and make you wish you were on a laptop, essentially my version of those.
For us these sites consisted of the SemiAccurate back-end, a game site with over 2.5K thumbnails all in one big table, a few sites with video/flash heavily on them, and some news sites that aren’t coded as well as they might think. All were the non-mobile versions of the sites in question, where possible, because the web is dumb enough in grown-up form already. All of these sites took the 801 in the G2 to its knees even on a cached reload. Some were I/O bound, most are compute bound, and some are both. They all are painful to use in the real world but for one reason or other, I have to use all of them regularly and trust me they are annoying. None of these were apps, and all were tested starting with a clear cache and then several reloads.
If the mobile browsers had more control over cache sizes we would have tried a few more low-level tricks but like the mobile web, mobile browsers are made for normal user -30 IQ points. We loaded the pages on one phone, watched how it loaded, when control was returned to the user we started to scroll up and down, and noted how responsive things were during the process. The page was then reloaded, and this was repeated on both phones with the built-in Android browsers, downloaded Chrome, and Firefox mobile in some cases too.
If this sounds completely unscientific you are right, it is and that is the point. It is not a numerical benchmark, not comparable to anything else, and you can do it on your own just like we did. That said it directly answers the question of, “Do the numbers tested actually matter?” which to SemiAccurate is the one result that you will probably find useful. For us it was quite revealing, much more so than any numbers we got from the tests.
So how did the Snapdragon 820/Kyro/MDP/S do in this part? Really well actually, to save you some reading, it is a very worthwhile upgrade from an 801/A15 class device. Why do we say this? It was something we first noticed on the aforementioned game site, a single page consisting of almost 28MB of data and 2605 requests according to Firebug. On a dual Broadwell system running Firefox 42.0 on Linux Mint 17.2, it took 40-44 seconds to reload after the first rendering and took the system to its knees when Firebug was enabled and fully logging.
On the G2/801, it was, and still is, pretty painful to watch the same site load, it took a visibly long time for the table framework outlining the thumbnails to pop up and a seeming eternity for control to be returned to the user. By that we mean the page began to scroll again, all told it was a good 10-15 seconds of ‘lock’ before anything ‘worked’ again, even if the result was skeletal and unreadable for many more seconds. Browser limits did not let us load the whole page at once, it had progressive loading, but even what was visible was enough to hobble the G2. The other sites tested were annoying in different ways, slow to render and freezing the browser for may seconds while you stared at the screen.
On the 820 MPD/S the results were very different, noticeably so. The same pages popped up visibly faster and more importantly they were ‘usable’ or scrollable almost instantly. Even the ‘page of death’ above was quick enough not to be annoying and comparable to that of the Broadwell laptop albeit with different browsers, OSes, rezes, and just about everything else. Another side note is that the 820 platform did have a Qualcomm optimized browser but since that is offered to all 820 customers, we don’t think this is an unfair advantage to test with. We downloaded Firefox and tested with it specifically because it wasn’t optimized for the 820 and the results were similar, it was notably quicker than the 801 on the sites we tested.
And that brings us to the conclusion that we stated earlier. You have the raw numbers and you have SemiAccurate’s subjective and quite non-scientific results. The raw numbers speak for themselves, the subjective ones say to us that an 820 platform is actually a worthwhile upgrade if you are carrying around an A15-class device or… *GASP*… an even older one. If you care about exactly why the 820 is faster, feel free to peel apart the raw numbers and compare them to your phone. If you just want to know if it is an upgrade that you will notice, for us the answer is yes but do try it yourself when Snapdragon 820 phones hit the market in a few months.S|A