abe@mace.cc.purdue.edu (Vic Abell) (12/19/90)
I have been asked if I have heard that several vendors have had to resubmit their PERFECT club reports. I haven't heard anything, and I don't even know if the rumor suggests that previously submitted reports were erroneous. Has anyone on the net heard anything about this? Vic Abell <abe@mace.cc.purdue.edu>
conte@crest.crhc.uiuc.edu (Tom Conte) (12/20/90)
In article <6439@mace.cc.purdue.edu>, abe@mace.cc.purdue.edu (Vic Abell) writes: > I have been asked if I have heard that several vendors have had to > resubmit their PERFECT club reports. I haven't heard anything, and > I don't even know if the rumor suggests that previously submitted > reports were erroneous. > Although I haven't heard about this, I want to point out it is difficult to `violate' the run rules for the PERFECT club. Unlike SPEC, the PERFECT club allows modification of the code for vectorization, parallelization, etc. (including hand modification). It's part of the philosophy: monkey around with the code as much as you wish to make it run well on your machine, just make sure the results are the same. For me, this makes PERFECT club numbers rather hard to interpret. ------ Tom Conte Center for Reliable and High-Performance Computing conte@uiuc.edu University of Illinois, Urbana-Champaign, Illinois I worry about insects: I bet the dinosaurs had `mammal exterminators' DISCLAIMER: (ug) opinions expressed are my own and not associated in any way with the University, CSRD, or the PERFECT club people
gc@sp12.csrd.uiuc.edu (George Cybenko) (12/21/90)
conte@crest.crhc.uiuc.edu (Tom Conte) writes: >In article <6439@mace.cc.purdue.edu>, abe@mace.cc.purdue.edu (Vic Abell) writes: >> I have been asked if I have heard that several vendors have had to >> resubmit their PERFECT club reports. I haven't heard anything, and >> I don't even know if the rumor suggests that previously submitted >> reports were erroneous. >> >Although I haven't heard about this, I want to point out it is difficult >to `violate' the run rules for the PERFECT club. Unlike SPEC, the PERFECT >club allows modification of the code for vectorization, parallelization, >etc. (including hand modification). It's part of the philosophy: monkey >around with the code as much as you wish to make it run well on your machine, >just make sure the results are the same. For me, this makes PERFECT club >numbers rather hard to interpret. There seems to be some confusion about the philosophy and methodology of the PERFECT Benchmarks. This may clarify the above remarks: a. PERFECT methodology allows baseline (compiler optimizations only) and optimized (hand tuning) results. Both are published in the PERFECT reports; b. Each of the 13 programs is presented in a single file and at the last meeting of the group, it was agreed that baseline means the programs should be compiled as single files, not broken out into individual subroutines, etc so that different compiler options could be applied to different program parts. This was an unanticipated twist that had not been previously discussed and everyone was asked to check their results in light of this stronger notion of baseline. Another aspect of baseline is that all system components must be released products (no beta versions, etc.). I hope this explains some of the comments of Abell and Conte. As far as we know, all results published to date are correct....new data that has NOT been published is being verified now. The PERFECT Benchmarks are 13 FORTRAN applications codes (and datasets) drawn from science and engineering totaling about 60,000 lines. The original goals (the effort was started in 1987) were to evaluate supercomputer performance on full applications, not kernels or algorithms. Unlike the SPEC codes which rehash Linpack and NAS Kernels, the PERFECT codes really do fluid mechanics, seismic migration, molecular dynamics, etc. Because they are relatively easy to port now and represent codes more typical of what you would find scientists and engineers using, they are used a lot by compiler writers and architecture researchers in simulations and experiments. The performance numbers are just one aspect of the effort. I could go on, but if you are interested in getting some papers and reports with data, send email to Cathy Warmbier (warmbier@csrd.uiuc.edu) asking for the PERFECT papers and reports. Requests for the codes should also go to Cathy. I've asked her not to send things out until the new data is final, so don't expect anything until early January. George Cybenko Center for Supercomputing Research and Development University of Illinois at Urbana gc@csrd.uiuc.edu (217) 244-4145
mash@mips.COM (John Mashey) (12/22/90)
In article <1990Dec21.071228.12282@csrd.uiuc.edu> gc@sp12.csrd.uiuc.edu (George Cybenko) writes: >conte@crest.crhc.uiuc.edu (Tom Conte) writes: > >>In article <6439@mace.cc.purdue.edu>, abe@mace.cc.purdue.edu (Vic Abell) writes: > >>Although I haven't heard about this, I want to point out it is difficult >>to `violate' the run rules for the PERFECT club. Unlike SPEC, the PERFECT >>club allows modification of the code for vectorization, parallelization, >>etc. (including hand modification). It's part of the philosophy: monkey >>around with the code as much as you wish to make it run well on your machine, >>just make sure the results are the same. For me, this makes PERFECT club >>numbers rather hard to interpret. By the way, in support of PERFECT, note that SPEC and PERFECT club have legitimately different goals, leading to the different measurement philosophies. Due to the desire for SPEC to offer better metrics for vendors to fight over, we avoid changing the code, to avoid gimmicks in reporting. On the other hand, I think it is GOOD that Perfect allows such changes, as it can be very instructive to see: a) How close the default and tuned versions are b) How much work it takes to get from one to the other c) What one can learn for hardware and software design In addition, it probably makes a lot more sense in the super-computer world or near-super-computer world, where people: a) Often have their own code b) Often have access to source, at least c) Can often gain get major gains by tuning, given the nature of these algorithms which is different from the workstation/supermini world, where more often one is using more third-party software, often not having access to the source, and where the most common programs do not offer as much scope for tuning. (There is always room for tuning, but not the orders-of-magnitude effects sometimes gainable in the vector, parallel, vector-parallel world.) -- -john mashey DISCLAIMER: <generic disclaimer, I speak for me only, etc> UUCP: mash@mips.com OR {ames,decwrl,prls,pyramid}!mips!mash DDD: 408-524-7015, 524-8253 or (main number) 408-720-1700 USPS: MIPS Computer Systems, 930 E. Arques, Sunnyvale, CA 94086
eugene@eos.arc.nasa.gov (Eugene Miya) (12/22/90)
It is noted SPEC and PERFECT have communicated.
Note: PERFECT has one of our programs (ARC2D). SPEC has the NAS Kernels,
both have several routines in common. The point of a kernel is to
have some representativeness.
Added note: the PEREFECT suite used to have ARC3D. The UIUC was informed
that that this program is on the Export Control List, and its distribution
outside the US is prohibited.
--e.n. miya, NASA Ames Research Center, eugene@eos.arc.nasa.gov
{uunet,mailrus,most gateways}!ames!eugene
AMERICA: CHANGE IT OR LOSE IT.davidsen@sixhub.UUCP (Wm E. Davidsen Jr) (12/24/90)
In article <1990Dec19.190758.8285@ux1.cso.uiuc.edu> conte@crest.crhc.uiuc.edu (Tom Conte) writes: | etc. (including hand modification). It's part of the philosophy: monkey | around with the code as much as you wish to make it run well on your machine, | just make sure the results are the same. For me, this makes PERFECT club | numbers rather hard to interpret. I think that both the idea of running the unchnaged code on various machines and running the code tuned any old way which still produces the right answer are both valid. By running unchanged code you get a good idea of how well the compiler and CPU work in solving problews coded in the obvious way. Since this is the lowest cost in programming time, most people would like to get good results without hand coding. Some so-called FORTRAN programs we run on the Cray and Convex systems are about as portable as assembler! By tuning the code to get the most from the machine, you get an idea of the hardware limits of the system. If you are pushing the limits of what can be computed within the limts of your budget or lifespan, you are interested in this. Finally the amount of improvement caused by tuning the source tells you how well the compiler works. If you get a big improvement, it usually means the compiler is not doing a good job (note usually). I find all this information useful, and the baseline vs tuned is certainly a useful data set to me. I would like to see a third set, from untouched source which has been broken into separate files to allow various compilation option to be used as needed, but this is certainly not needed, just interesting. -- bill davidsen - davidsen@sixhub.uucp (uunet!crdgw1!sixhub!davidsen) sysop *IX BBS and Public Access UNIX moderator of comp.binaries.ibm.pc and 80386 mailing list "Stupidity, like virtue, is its own reward" -me