crowl@rochester.UUCP (11/02/87)
In article <864@tut.cis.ohio-state.edu> manson@tut.cis.ohio-state.edu (Bob Manson) writes: >It means nothing to try to compare my computer with my dumb compiler to a >well-developed VAX machine with a real compiler. So why run benchmarks in >compiled languages??? People run real code in compiled languages, so it is the performance of the compiled languages that has any meaning. >It's easier that way, you don't have to write individual programs for each >machine that might actually show off their abilities and improved instructions. You also don't introduce the programmer for each machine as another variable. The competence of the programmer, on the particular machine, can make a large difference in the resulting speed. The high level language approach also prevents the assembly programmer from "cheating" with algorithmic short cuts. >I'll admit that it does mean something for compiled language users but not >really for performance comparisons-you're comparing apples and oranges, or >really the efficiency of the compilers on the machines. The effectiveness of compiling a high level language into the particular architecture affects the performance of the architecture. After all, people use high level langauges. There are two approaches to fair benchmarks. 1) Use whatever _publicly_ available compilers you want on the target machine. If one compiler is better than another, tough. The available compilers define the available compute power of the machine. 2) Attempt to equalize the effect of the compiler. This allows you to get at the architectural performance better. Note that "equivalence" of compilers is dependent on their size, their development time, and their execution time. The first approach is best for customers choosing a machine, while the second approach is best for designers evaluating an architecture. In both cases, the use of a high level language is appropriate. -- Lawrence Crowl 716-275-9499 University of Rochester crowl@cs.rochester.edu Computer Science Department ...!{allegra,decvax,rutgers}!rochester!crowl Rochester, New York, 14627
eugene@pioneer.arpa (Eugene Miya N.) (11/03/87)
Subject: Re: Towards A Meaningful Performance Measure In article <3806@sol.ARPA> crowl@cs.rochester.edu (Lawrence Crowl) writes: >Of course, 780's are becoming scarce. We may have to pick another machine just >to keep the base machine readily available. Suggestions? Yes, the Cray X-MP. 1) Good resolution clock Important like the need for Atomic clocks and conventional time keeping. The Machine as a whole has a performance all computers will eventually strive. [Do you believe in Dan Ingall's Cray on a wrist? ;-)] 2) The Hardware Performance Monitor The requirement for non-intrusive counting is important. 3) The software for intrusive measurement (FLOTRACE) is not too bad either. The current OS's don't page (a plus or minus depending where you are coming from). The problem with the X-MP is that it is a soon to be discontinued line. It's quite reliable and has architectural features now emerging in micros. Another problem is its memory is unlike most other machines [performance due to interleaving is distinct], but this not a serious problem. The Y-MP and the Cray-3 should have this type of hardware designed into it [not to say it does, users have to push for it]. But, I have not heard. From the Rock of Ages Home for Retired Hackers: --eugene miya, NASA Ames Research Center, eugene@ames-aurora.ARPA "You trust the `reply' command with all those different mailers out there?" "Send mail, avoid follow-ups. If enough, I'll summarize." {hplabs,hao,ihnp4,decwrl,allegra,tektronix}!ames!aurora!eugene
tim@amdcad.AMD.COM (Tim Olson) (11/03/87)
In article <3285@ames.arpa> eugene@pioneer.UUCP (Eugene Miya N.) writes: | In article <3806@sol.ARPA> crowl@cs.rochester.edu (Lawrence Crowl) writes: | >Of course, 780's are becoming scarce. We may have to pick another machine just | >to keep the base machine readily available. Suggestions? | | Yes, the Cray X-MP. | | The problem with the X-MP is that it is a soon to be discontinued line. | It's quite reliable and has architectural features now emerging in | micros. Another problem is its memory is unlike most other machines | [performance due to interleaving is distinct], but this not a serious | problem. The Y-MP and the Cray-3 should have this type of hardware | designed into it [not to say it does, users have to push for it]. | But, I have not heard. Another problem: it is not a very common computer. Nearly everyone who is doing some serious benchmarking at least has access to a VAX. If it weren't for the memory restrictions and porting problems, I'd vote for the IBM PC with specified compilers -- common as dirt (also *runs* as slow as dirt, but that's a different matter ;-) -- Tim Olson Advanced Micro Devices (tim@amdcad.amd.com)
pmk@hall.cray.com (Peter Klausler) (11/04/87)
In article <18966@amdcad.AMD.COM>, tim@amdcad.AMD.COM (Tim Olson) writes: | In article <3285@ames.arpa> eugene@pioneer.UUCP (Eugene Miya N.) writes: | | In article <3806@sol.ARPA> crowl@cs.rochester.edu (Lawrence Crowl) writes: | | >Of course, 780's are becoming scarce. We may have to pick another machine just | | >to keep the base machine readily available. Suggestions? | | Yes, the Cray X-MP. | Another problem: it is not a very common computer. Nearly everyone who | is doing some serious benchmarking at least has access to a VAX. Beware of using "X-MP" as a GENERIC unit of performance. Even for a single-processor benchmark using only instructions common to all X-MPs, significant performance differences arise between different machines due to: 1) Different clock rates. The clock period on an X-MP is adjustable (within a fairly narrow range), and current X-MPs are configured with a clock period 1.0ns faster than original 1983 vintage X-MPs. 2) Different memory parts (MOS or bipolar). 3) Different memory bank configurations. If your benchmark code can be multiprocessed, or use the special instructions found only on some X-MPs, then you'll get even more variation between machines. As with any other computer available is more than one configuration, please provide complete information about your X-MP when publishing benchmark results. Statements like "Fortran on a Cray X-MP: 34.3 sec" just aren't sufficient; please specify whether you're using an X-MP/14se or a full-blown X-MP/416. (After reviewing this before posting, it occurs to me that most comp.arch readers are probably aware of what I've brought up. Forgive me if I'm being redundant.) Peter Klausler pmk@hall.cray.com Compiler Development ...!{ihnp4,sun!tundra}!hall!pmk Cray Research, Inc. (612) 681-5828 Mendota Heights, Minnesota I like languages with P's in their names.