Most savvy PC buyers use benchmarking software results to help determine which processor will best suit their overall computing needs. However, charges of bias and manipulation over the last few days against one of the most recognised programs have brought to light the discrepancies in benchmarking software and its effectiveness to users.
Sysmark 2002 in the dock, but PC Advisor says don't judge PC purely on test results
AMD is launching an assault against the Sysmark 2002 benchmark, charging it has been altered to favour processors from Intel, according to Hal Speed, the aptly named US marketing manager for AMD.
Sysmark is distributed by a consortium called Bapco, or the Business Application Performance Corporation. Members of the group include Intel, Dell, IBM, HP, Microsoft and AMD, which recently joined the consortium. The group incorporates several industry publications, including InfoWorld, a division of IDG, parent company of PC Advisor.
AMD's processors outperformed Intel's using the Sysmark 2001 benchmark. AMD says that certain application tests were removed from that version and tests that favour Pentium 4 processors from Intel were repeated several times in the latest edition of the benchmarking software.
The company contends that the presence of Intel as the only major microprocessor vendor in Bapco has caused the benchmarking software to drift toward Intel's philosophy of real-world performance, without a balancing viewpoint from AMD.
Specifically, AMD charges that of 13 filters measuring the performance of Adobe's Photoshop software in Sysmark 2001, eight that favoured the Athlon processor were replaced with new filters more disposed to the Pentium 4 in the 2002 version. The latest Sysmark also uses multiple instances of three filters biased towards Intel and adds three filters that accommodate the Pentium 4, Speed said.
The portion of the benchmark that measures Microsoft Excel performance is heavily tilted toward sorting, a procedure that does not represent the most common uses of Excel, Speed also said. The Pentium 4 sorts data in Microsoft Excel faster than AMD's Athlon chips, according to the 2001 benchmarking results.
Also, tasks that favoured AMD for Microsoft's Access database were almost completely removed, according to AMD. The Access results from the 2001 benchmark benefitted Athlon processors, but the 2002 version draws much less of the overall score from Access results.
Intel would not comment on the specific allegations and our calls to Bapco went unanswered.
"Our whole product strategy is based on having objective benchmarks. We are [now] working with Bapco to ensure the industry has credible application-based benchmarks," Speed said.
The controversy over the Sysmark benchmark calls into question just how reliable benchmarks are in general.
"A good benchmark does mean something; their purpose in life is to distil the variables that affect real-world systems performance. The difficulty comes from the judgment call in what you test to measure real-world performance," said Dean McCarron, principal analyst at Mercury Research.
PC Advisor benchmarks PCs with WorldBench 4, developed by our sister magazine PC World in its San Francisco-based labs. Our tests are based on everyday tasks carried out by 11 different applications, to give an indication of how the PC will perform in the real world. WorldBench 4 is complemented by video encoding and 3D graphics tests developed in our own Test Centre.
While benchmarks are a useful measure of performance, our Reviews team always advises that you look at what the PC as a whole has to offer, rather than rely solely on test scores to determine which model is best for you needs.
Other factors come into play when we place PCs in our Top 10 charts — build quality, the monitor, warranty, peripherals, software, value for money and so on. More information on WorldBench 4 and how we score PCs can be found here.