aglew@mcdurb.Urbana.Gould.COM (05/13/89)
>When the goal is measurement, why shouldn't the test conditions and >predicted outputs be made publicly available? > >Jonathan Krueger Well, sure... when I publish internal performance reports I try to specify the conditions, and although I haven't yet put out a "MIPS Performance Report" for Motorola MCD, when and if I'm asked to do so I will specify as much about the conditions as possible. But - have you ever tried to completely describe the configuration of a computer system that you are benchmarking? There's a lot of detail there. It's easy to get 4 times as much configuration text as it is numbers. "Just write down the important things" you say - and certainly I'll try. But even that can be a lot. And even then, it is easy to be bitten by configuration parameters that you didn't know about. Recently, for example, I was bitten by a difference between two apparently identical boards - same copper, same firmware. But a PAL change, to increase reliability, that had a "negligible effect on performance" (so nobody bothered to tell me). Well, in a real customer system the effect would probably be negligible -- but in the particular aspect of system performance I was looking at it made a big difference. Bottom line: yes, a responsible performance evaluator publishes his measurement conditions, and uses widely available benchmarks so that you can try to reproduce the results. But please don't flame me when you can't reproduce my results exactly. I take any benchmarks from people that are not at the factory door for their respective manufacturer with a large grain of salt. Oh, I'll believe the numbers - but there is probably a large variance. Take numbers for large computers with a larger grain of salt. Crays, Convexes, Goulds, etc. have a lot of people messing around with them - field service engineers changing a jumper that "has no effect on performance" but does. Fortunately, it's easier to maintain configuration control of small systems, like Motorola DELTA boxes, SUNs, MIPS' boxes. When I came into this role at Motorola MCD I got a very good piece of advice from someone who had done the same thing at Intel: get 100% control of the hardware you are going to be making measurements on. Shoot anyone else who comes near it. I can do this because I am more worried with evaluating OS performance than hardware (does that tweak to the disk driver make things run faster?), so I can keep hardware relatively constant while varying OS code. But I have nothing but pity for the poor sods who have to evaluate new hardware as it comes out of the shop - constantly managing a mish-mash of prototypes, latest revs, etc. Ouch! Last word: performance evaluation requires good statistical techniques more and moreto handle variations. I haven't seen such analysis in the published benchmark reports yet, but I expect to soon -- and, after all, benchmark reports are only a small part of performance evaluation.
dfields@mcdurb.Urbana.Gould.COM (05/19/89)
>/* Written 9:25 am May 16, 1989 by mpogue@dg.dg.com in mcdurb:comp.arch */ >/* ---------- "Re: Criteria ... [really: are N des" ---------- */ >In article <104996@sun.Eng.Sun.COM> khb@sun.UUCP (Keith Bierman - SPD Languages Marketing -- MTS) writes: >>> >>> My preference is to have a third party do the numbers (no fudging allowed!). >> >>Typically no understanding of the machine either. In former >>incarnations I was on the user side. It was (and still is) that most >>third parties don't know how to run the machine right. So as a >>customer I got each vendor to run my code, and explain what they did. >>directives could increase performance fourfold .... > > Yes, I agree that it is important to get somebody knowledgable to run >the benchmarks. Just any third party will not do (sorry I didn't make this >clearer). If a third party can't figure out how to make a simple benchmark run fast on your system, how do you expect your customers to be able port an aplication which takes advantage of your high performance system? Dave Fields // Motorola MCD dfields@urbana.mcd.mot.com uunet!uiucdcs!mcdurb!dfields
khb%chiba@Sun.COM (Keith Bierman - SPD Languages Marketing -- MTS) (05/20/89)
In article <28200317@mcdurb> dfields@mcdurb.Urbana.Gould.COM writes: > >If a third party can't figure out how to make a simple benchmark >run fast on your system, how do you expect your customers to be able >port an aplication which takes advantage of your high performance system? > a) Who said anything about simple ? b) Those really interested in max performance are typically willing to go to some efforts. Consider your (albeit few) NP1/2 users; all of Cray's; many of MIPS's etc. Complex systems (which includes many RISC set ups) require some complex usage. Keith H. Bierman |*My thoughts are my own. Only my work belongs to Sun* It's Not My Fault | Marketing Technical Specialist ! kbierman@sun.com I Voted for Bill & | Languages and Performance Tools. Opus (* strange as it may seem, I do more engineering now *)
aglew@mcdurb.Urbana.Gould.COM (05/21/89)
>/* Written 8:12 pm May 19, 1989 by khb%chiba@Sun.COM in mcdurb:comp.arch */ >In article <28200317@mcdurb> dfields@mcdurb.Urbana.Gould.COM writes: >> > >>If a third party can't figure out how to make a simple benchmark >>run fast on your system, how do you expect your customers to be able >>port an aplication which takes advantage of your high performance system? >> > >a) Who said anything about simple ? >b) Those really interested in max performance are typically > willing to go to some efforts. Consider your (albeit few) NP1/2 > users; all of Cray's; many of MIPS's etc. > >Complex systems (which includes many RISC set ups) require some >complex usage. (1) First off, to set things straight: Dave Fields, myself (Andy 'Krazy' Glew) and a few others that you see on the net are now happy to be MOTOROLA Microcomputer Division employees. Gould sold the Urbana Development Center [*] to Motorola just before Nippon Mining bought Gould, just before Nippon Mining sold Gould CSD to Encore, etc. Our mail paths will appear as ...Gould.COM for a while, until the various bureaucracies that control ARPAnet naming can digest us. But, already, if your mailer supports MX records, you can address us as *@urbana.mcd.mot.com. (2) Fewer people than you think are interested in intensive performance tuning. That was one of the things I learned at Gould, Cray is going to learn from the Japanese supercomputer manufacturers, and something that MIPS has already learned. Being willing to tune is one thing; having to tune is another. BTW, Keith, I'll be in your area next week. Got any glaciers to climb? Andy "Krazy" Glew aglew@urbana.mcd.mot.com uunet!uiucdcs!mcdurb!aglew Motorola Microcomputer Division, Champaign-Urbana Design Center 1101 E. University, Urbana, Illinois 61801, USA. My opinions are my own, and are not the opinions of my employer, or any other organisation. I indicate my company only so that the reader may account for any possible bias I may have towards our products.