jgreen@alliant.alliant.com (John C. Green jr) (01/09/91)
Public display of meaningful benchmarks improves competition which in turn improves products to the benefit of consumers. In 1974 Data General announced a new supermini, the Eclipse S200. Soon they ran a benchmark of the British government, the Whetstone, and were given a chart that listed the performance of every computer that had ever run the benchmark. DG's UK people gave a copy to the US headquarters. DG US took out full page ads in the trade press showing the top of the chart with the Eclipse S200 well above the DEC PDP-11/45. DG UK was angry because they had been given the results under non-disclosure conditions. My guess at the time as to why the British government did not want it disclosed was that the top of the chart had many high-end American scientific machines (now called supercomputers) followed by many high-end American mainframes followed by the fastest British mainframe followed by more American mainframes and some minicomputers followed by the second fastest British mainframe. Once the benchmark was public many companies improved their floating point hardware, compilers, and added caches. By 1988 the benchmark could fit in most machine's caches, many compilers did inlining, and many machines had vector registers so the benchmark lost much of its utility; it no longer provided a good prediction of the floating point intensive scientific application performance of a system. However, during its 14 year public useful life, it did cause numerous vendors to improve their scientific computer products. I strongly support the public display of meaningful benchmarks. I believe SPECmark and SPECthruput in the short term allow consumers to compare systems on a fair playing field and in the long term cause vendors to make meaningful improvements to their Unix systems to the benefit of all customers. On the other hand a benchmark such as bc offered neither benefit and died quickly. Public benchmarks will come and go. The good ones will spur competition and have long lives. jgreen@alliant.com (John Green) (508)486-1302 Alliant CSC