The non-profit group said it would stop publishing SPEC CPU 2017 results for Chipzilla PCs running a dodgy version of the Chipzilla compiler.

The decision comes from worries over what the group calls a sneaky optimisation for a specific workload. In other words, SPEC accuses Chipzilla of fiddling with its products for better benchmark results.

SPEC CPU 2017 is a popular benchmark for top-of-the-range servers, data centres and workstations, testing performance across 43 benchmarks. The row is about the Chipzilla oneAPI DPC++/C++ Compiler, which has been allegedly tweaked for the 523.xalancbmk_r / 623.xalancbmk_s benchmarks.

A warning is now stuck to more than 2,600 results: “The compiler used for this result was doing a compilation that especially boosts the performance of the 523.xalancbmk_r / 623.xalancbmk_s benchmarks using insider knowledge,” thus making the results rubbish.

Although benchmark figures are generally not the same as real-world conditions, testing is meant to be as realistic as possible, so optimisations to suit specific benchmarks can lead to bent (and daft) results.

According to a Phoronix report, 2022.0 to 2023.0 models are hit, leaving models on either side of this date range untouched by the warning notices. The article says, “The SPEC CPU specific optimisation could result in a nine per cent overall SPECint speed boost and around four per cent for the SPECint rate.”