klein@gravity.UUCP (06/02/87)
In article <4667@fritz.UUCP> martin@felix.UUCP (Martin McKendry) writes: >Gentlemen: > >I have read with interest the discussion on benchmarking. I am >sure that a benchmarking bake-off would be a good thing >for prospective customers. But for the engineers in the >audience, I would like to point out some of the social problems >associated with touting your company's wares. > >Any bake-off will have one winner and many losers. As an engineer >working for a particular vendor, you may have run tests that >convince you that you will be a winner. But be assured that in >some dimension you will be a loser. And if you proposed that >your company enter, you will be the one "responsible" for the >exposure of "weakness". I am happy that someone finally pointed out these issues. As engineers, we all want to know the "truth" about performance and limitations. While it would be fantastically valuable for the progress of our industry to understand the true values of architectural and implementation decisions, which a good set of benchmarks may help with, it is pretty much inconceivable to me that this can occur under the auspices of a trade show or conference. Marketing computers and computer systems is a very complex job with great risks and rewards. A poor sales/marketing job, irrespective of the technical merits, can sink a product faster than almost anything else (except maybe poor quality). Submitting a product to an open benchmarking when it is known in advance that the product will not be the "best" is poor marketing. Submitting it to an open benchmarking when it is not known in advance how the product will perform is also poor marketing. No marketer, salesperson, or any other high-level manager worth his/her salt will consent to a public demonstration of their product's inferiority. As a representative of your company, if you benchmark your system and it performs poorly, and you therefore cause the company to lose sales, you have just done yourself and your coworkers a major disservice. Maybe this isn't how it should be, but it is the reality of the marketplace. I am speaking in very general terms here, and this should not reflect on, or imply any actions of, my employer or anyone else's. Most companies in such a technical field try to present as much detailed technical information as possible for their products. For the meantime, we will need to rely on open, informal communication such as we've had from AMD and National, for example. I commend them very highly for this. But I don't expect to see the AMD booth or the National booth running the same benchmarks, and I understand the reasons why. -- Mike Klein klein@Sun.COM Sun Microsystems, Inc. {ucbvax,hplabs,ihnp4,seismo}!sun!klein Mountain View, CA