Is Qualcomm cheating in Vellamo?
days ago we were happy to report about the first set of leaked HTC One X scores on an American device powered by Qualcomm’s new 28nm S4 chip.
The scores look quite good for Qualcomm, but since we got hold of an Asus Transformer Prime we decided to dig a bid deeper. Both Nvidia and Qualcomm do not want to get involved in official fight but we were quite surprised with what we learned. We also talked to some engineers that want to remain unnamed and we came up with a few interesting things.
The fastest CPU in Quadrant is Hummingbird S5PC110 from Nexus S as tested here
Since we have tested this phone you can easily tell that this benchmark doesn't really use two or four cores properly. In this test S4 scores 4920 where the Tegra 3 based Transformer Prime scores 3954 and Tegra 2 scores 2154 on a custom ROM Optimus 2X.
The second test
is the Vellamo benchmark was a bit more disturbing as once you run it you see Qualcomm ads all over it. This didn't give us much confidence and it turns out that Qualcomm has a lot of power over this particular benchmark.
Our engineering friends are telling us that Velamo disables some hardware acceleration in compositing Deap Sea Canvas and See the sun canvas subtest. Honeycomb and ICS support hardware acceleration by default and disabling this probably hurt the general score. One can argue that it hurts S4 scores as well, but it definitely hurts Tegra 3 more. The benchmark isn't flushing commands in the Pixel Blender subtest and there is a suspicion that this might help Qualcomm S4 to gain a better score.
The most important issue is the fact that it is unclear how Vellamo scores sub test scores. In Third party benchmarks such as Sun Spider and Google V8 it turns out that Vellamo penalizes high Google V8 scores and if your score gets too high in V8, the general score gets lower. There are a lot of benchmarks out there and some of the ones that like more cores include Antutu, CF benchmark and Moonbat.
French enthusiasts managed to run Antutu here
, and Tegra 3 phone scores 10597, while the S4 based HTC One S scores 6458. This doesn't look so good for the phone that is based on S4 cores that should go after ARM’s upcoming A15 and it looks like that it cannot really beat the A9-based Tegra 3. In the real world, as long as the application is aware of four cores, there is a good chance that Tegra 3 will end up faster than the S4.
Naturally if you are reading this from the US and you really like your LTE from AT&T or Verison, I guess that you won’t have much choice and you will get the HTC One X with Qualcomm S4 as this chip also supports LTE. In the US it’s all about LTE and in Europe despite the fact that countries like Austria have LTE for more than a year now, even at €50 for 40GB there is almost no interest whatsoever. Networks need to put more advertising money and make 4G cool and it might happen. Of course, the lack of 4G devices is also an issue, but technology has a way of catching up.
The story gets even better when you know that there are lot of former ATI employees who take care of Adreno graphics and if natural selection and theory of evolution have taught us anything, it is that Nvidia and ATI are sworn enemies in any universe. Some readers might see a pinch of poetic justice in all of this, as Nvidia was accused of tweaking its GPU drivers to score more in PC benchmarks years ago.
Tudor Brown, the president of ARM that we meet a few years back at GlobalFoundries Dresden fab, once said that ARM does not want to get involved in GHz fight and this is now exactly what is going on, as punters are using benchmarks to prove of A is faster than B. Frankly I would be more concerned about battery life that I can get from a brand new phone as we got from five day battery life on feature phones to a day or so of battery life, and with LTE and heavy use, even this can go down to a few hours at best. If I found myself in Qualcomm’s shoes, this is what I would emphasize, the new 28nm process and energy efficiency, not skewed benchmarks.
Phones should be about the overall user experience, but how can you benchmark experience? It's a very subjective category and can hardly be expressed with cold numbers and statistics. We are not talking about PC components and the race for more performance, smartphones should be viewed as a complete package, battery life, build quality, UI and design come into play. Sheer performance is just one aspect and for many users it is still not the deciding factor and we believe it should not be.
In any case, watching the phone market will definitely be fun in the months to come.