ciemo@bananapc.wpd.sgi.com (Dave Ciemiewicz) (06/01/89)
When I saw the article "A Set of Benchmarks for Evaluating Engineering Workstations", May, 1989, issue of IEEE Computer Graphics and Applications, I was initially furious. The data is two years old (gathered from "January 1987 to April 1987"). Some of the hardware is not available on the market today. For instance, SGI no longer makes the IRIS 4D/60G or IRIS 2400T. I may be wrong but I don't think you can buy GE Graphicon boards anymore. Not only is the data presented out of date, but I question the authors' results. First, Figure 1, "CPU benchmark performance" shows the authors' CPU benchmark took 64 seconds to run on an IRIS 2400T. By the authors' numbers, the IRIS 2400T should be 47% of a VAX 8600 yet their graph shows the IRIS 2400T at 30%. The plots for the HP350SRX and HP320SRX are at 15% and 10% yet the numbers should be 9% and 7%, respectively. This is sloppy both on the authors' part and on the part of the peer review and editorial staff. Such errors should never reach the readers. Also, the authors quote price/performance ratios but they don't give any configuration information about the systems involved. Factors like the amount of memory and number, types, and capacities of disk drives will affect the list price of a machine. Without this information, their price/performance data is useless. If the authors are so sloppy in the presentation of their data, I wonder how sloppy they were in the collection of their data. The authors seem to think that their benchmark is more useful than other benchmarks refered to in their article. Maybe it is more useful for their particular application. However, their particular benchmark may be useless for other computer graphics applications. The best way you can tell how well your application is going to run a given architecture is to run your application on that architecture. This seems to be the gist of IEEE CG&A editor John Standhammer's note at the beginning of the article. Maybe Mr. Standhammer waited approximately two years to publish the article so the description of the benchmarking process would be the only useful part of the article. This was a wise (or lucky) move on his part. I only wish he hadn't lured people into reading the article by plastering the first page with the banner "Workstation Benchmarks". In his note, Mr. Standhammer encourages "others to share their evaluation procedures and results." I encourage those people doing systems evaluations to do so accurately and fairly. I encourage the CG&A editorial staff and peer reviewers to subject systems evaluation articles to a very high degree of scrutiny. Lastly, I encourage readers of CG&A to demand this kind of quality. These opinions are my own and do not reflect those of Silicon Graphics, Inc. -- David M. Ciemiewicz Member of the Technical Staff WorkGroup Products Division Silicon Graphics, Inc.