jack@linus.claremont.edu (01/04/91)
Hi, I just ran some interesting Mathematica benchmarks on a NeXTstation, a Mac IIci (with Math coprocessor), and a DECStation model 3100. The results are a little surprising. The Mathematica benchmarks are a suite of 41 different tests from a number of different sources. The test took a total of anywhere from about 2.5 minutes for the NeXT to 22.5 minutes for the Mac IIci. A summary of the tests follows (the tests are too long to post here). NeXTStation vs. DECStation 3100 Slope of Best Fit of time for each test 1.35 (NeXT as x, DEC as y) Correlation Coefficent 0.98 Mean of ratio's of each test 1.20 Standard Deviation of ratio's 0.3 NeXTStation vs. Mac IIci with coprocessor Slope of Best Fit of time for each test 9.51 (NeXT as x, Mac as y) Correlation Coefficent 0.97 Mean of ratio's of each test 8.94 Standard Deviation of ratio's 3.52 The NeXTStation was about 20-30% faster on this group of benchmarks than the DECStation 3100. This is surprising since the DECStation is rated and 13.9 Mips and the NeXTstation benches at 15.0 Mips. You would expect from the Mips rating only about a 10% increase. I wonder if it has anything to do with RISC vs CISC architectures ?(The NeXTstation is CISC --- The DECStation is RISC.) The Mac IIci was surprisingly slow. A 68030 NeXT (same CPU as the IIci ) is only roughly 3 times slower that the NeXTstation. I wonder if the performance difference might have to compiler optimization or perhaps hardware (caching?)? The complete tests and the Mathematica benchmark are available via anonymous FTP from FENRIS.CLAREMONT.EDU. I will try to benchmark a Mac IIfx over the next few weeks (I think there is one on campus with Mathematica). If anyone has Mathematica running on a different architecture please feel free to grab the Benchmark and run it. If you do so, please put the results back into /pub/submission on FENRIS. I am really interested in seeing the results vs a SPARC, IBM PowerStation, and a DECStation 5000. The Mathematica benchmark is by no means perfect. It does not (nor is intended to) replace LINPACK, SPEC, Mips or any other widely used benchmark. There is a fair deal of variation of the results depending on the specific test. But I do like the test because it is fairly quick and portable (assuming the demo system has Mathematica). I also like the fact that it is a benchmark that uses a "real life" application. System vs. System performance tends to vary with the application. I also like the fact that, as far as I know, no manufacturer has written a compiler to optimize this benchmark. Comments? ---Jack Jack Stewart Jack@Hmcvax (Bitnet) User Support Coordinator, jack@hmcvax.claremont.edu (Internet) Harvey Mudd College, jack@fozzie.claremont.edu (NeXT-Mail) Claremont, Ca. 91711 714-621-8006
jclee@cory.Berkeley.EDU (James C. Lee) (01/04/91)
> The NeXTStation was about 20-30% faster on this group of benchmarks > than the DECStation 3100. This is surprising since the DECStation is > rated and 13.9 Mips and the NeXTstation benches at 15.0 Mips. You > would expect from the Mips rating only about a 10% increase. or maybe it's because the 15.0 MIPS is a conservative estimate. I know that some manufacturers using 68040 have rated their computers at 17.0 MIPS or higher.... James
eachus@linus.mitre.org (Robert I. Eachus) (01/04/91)
This sounds about right. On any memory intensive benchmark performance of the Mac II's is throttled by memory access to less than 2 MIPS (maximum memory access rate = 5 Mbytes/sec = 1.25 Million words/sec). The IIfx is also memory bandwidth limited--the memory system is much faster but so is the processor, so performance is throttled to about 5 MIPS. This is also true of the NeXT, but to a much smaller extent in the 68040 version. A 25 MHz 68030 which never has to wait for memory access can approach 8 MIPS, and some Amiga's get quite close to this when set up correctly (use SetCPU to turn on the data cache and copy the ROMs into fast memory. On a A3000 with 80ns SCRAMs this tests at about 7.7 Dhrystone 2.1 C MIPS, and shows little or no degradation on larger benchmarks). A 25 MHz 68040 can reach 25 MIPS, but as far as I know no one is yet shipping a design that does this as it can make main memory VERY expensive, and ordinary (~100 ns) memory parts don't cause as much degradation as on the 68030 due to the larger caches. -- Robert I. Eachus with STANDARD_DISCLAIMER; use STANDARD_DISCLAIMER; function MESSAGE (TEXT: in CLEVER_IDEAS) return BETTER_IDEAS is...
ulrich@jargon.whoi.edu (01/05/91)
In article <1991Jan3.214940.1@linus.claremont.edu> jack@linus.claremont.edu writes: >Hi, > >I just ran some interesting Mathematica benchmarks on a NeXTstation, a >Mac IIci (with Math coprocessor), and a DECStation model 3100. The >results are a little surprising. The Mathematica benchmarks are a >suite of 41 different tests from a number of different sources. The >test took a total of anywhere from about 2.5 minutes for the NeXT to >22.5 minutes for the Mac IIci. > >NeXTStation vs. Mac IIci with coprocessor > >Slope of Best Fit of time for each test 9.51 > (NeXT as x, Mac as y) >Correlation Coefficent 0.97 > >Mean of ratio's of each test 8.94 >Standard Deviation of ratio's 3.52 > >The Mac IIci was surprisingly slow. A 68030 NeXT (same CPU as the >IIci ) is only roughly 3 times slower that the NeXTstation. I wonder >if the performance difference might have to compiler optimization or >perhaps hardware (caching?)? > >Jack Stewart Jack@Hmcvax (Bitnet) >User Support Coordinator, jack@hmcvax.claremont.edu (Internet) >Harvey Mudd College, jack@fozzie.claremont.edu (NeXT-Mail) >Claremont, Ca. 91711 714-621-8006 > I ran some of the benchmarks on my Mac IIci with a Daystar Cache Card installed, and put the results on fenris.claremont.edu in pub/submission for Jack to statistically analyze....I don't have time to run the numbers myself but it looks like the IIci is as much as twice as fast with the cache card ($289 well spent, it seems). Nathan Ulrich Deep Submergence Laboratory Woods Hole Oceanographic Institution ulrich@jargon.whoi.edu
shwake@raysnec.UUCP (Ray Shwake) (01/06/91)
jack@linus.claremont.edu writes: >The NeXTStation was about 20-30% faster on this group of benchmarks >than the DECStation 3100. This is surprising since the DECStation is >rated and 13.9 Mips and the NeXTstation benches at 15.0 Mips. You >would expect from the Mips rating only about a 10% increase. I wonder >if it has anything to do with RISC vs CISC architectures ?(The >NeXTstation is CISC --- The DECStation is RISC.) More likely, you should be looking at relative floating point potential, reflected in the MFLOPS, rather than the MIPS figures. The '040 has demonstrated real strengths in this area. >The Mac IIci was surprisingly slow. A 68030 NeXT (same CPU as the >IIci ) is only roughly 3 times slower that the NeXTstation. I wonder >if the performance difference might have to compiler optimization or >perhaps hardware (caching?)? I seem to recall figures suggesting the '040 has roughly 10 times the floating point performance of the '030 with FPP. Thus your figures don't come as a surprise.
lee@wang.com (Lee Story) (01/08/91)
jack@linus.claremont.edu writes: >Hi, >I just ran some interesting Mathematica benchmarks on a NeXTstation, a >Mac IIci (with Math coprocessor), and a DECStation model 3100. The >results are a little surprising. The Mathematica benchmarks are a ... >The Mathematica benchmark is by no means perfect. It does not (nor is >intended to) replace LINPACK, SPEC, Mips or any other widely used >benchmark. There is a fair deal of variation of the results depending >on the specific test. But I do like the test because it is fairly >quick and portable (assuming the demo system has Mathematica). I also >like the fact that it is a benchmark that uses a "real life" >application. System vs. System performance tends to vary with the >application. I also like the fact that, as far as I know, no >manufacturer has written a compiler to optimize this benchmark. >Comments? I can't resist a response here. Is this part of the continuing attempt to post the most worthless pseudo-benchmark proposals?? (ref. the extended discussion of 'bc' as a benchmark) I've used Mathematica frequently since the first version, in its Mac, PC (dos), and NeXt manifestations, and like many others have bemoaned the poor performance on the Macintosh. How the Wolfram folks (esp. Mr. Gray) could make the Mac the target machine for such a lovely front-end, while crippling the performance of the engine, not allowing for use of the kernel on the Mac from another (Mac's) front-end, not using Multifinder to implement RunThrough[], etc. etc., I just can't understand. Change flames here. As a benchmark, Mathematica is wholly bogus. It has been reorganized (somewhat ineptly, it appears) to run with the Mac's segmented code resource scheme, with PharLap on dos, etc. The compilers used are unstated in its documentation, and the source is not available for recompilation. Thus it cannot be used to benchmark raw processor/memory speed or processor/memory/compiler speed. What's left? Lee -- ------------------------------------------------------------------------ "Had the man looked through another peephole he might have said, ...." Lee Story (lee@wang.com) Wang Laboratories, Inc. ------------------------------------------------------------------------
madler@pooh.caltech.edu (Mark Adler) (01/08/91)
Lee Story (lee@wang.com) notes that:
>> As a benchmark, Mathematica is wholly bogus.
after pointing out how it is crippled on the Mac.
The purpose of benchmarks is to help someone make an educated guess about
how their more commonly used applications will perform on various machines.
Of course, none of the "general purpose" benchmarks will tell you how your
application will run precisely--a guess is as good as it gets. The best
benchmark, by far, is to test your applications on the machines you might
be using, renting, or buying.
Therefore, if you use Mathematica most of the time, then Mathematica is
the perfect benchmark program. Bar none. In fact, the very problems Lee
uses to argue that Mathematica is no good as a general purpose benchmark,
illustrate how useless the general purpose benchmarks are at telling you
how your program will run (if your program is Mathematica, in this case).
Mark Adler
madler@pooh.caltech.edunerd@percy.rain.com (Michael Galassi) (01/08/91)
>>I just ran some interesting Mathematica benchmarks on a NeXTstation, a >>Mac IIci (with Math coprocessor), and a DECStation model 3100. The >>results are a little surprising. The Mathematica benchmarks are a ... >As a benchmark, Mathematica is wholly bogus. It has been reorganized >(somewhat ineptly, it appears) to run with the Mac's segmented code >resource scheme, with PharLap on dos, etc. The compilers used are >unstated in its documentation, and the source is not available for >recompilation. Thus it cannot be used to benchmark raw processor/memory >speed or processor/memory/compiler speed. What's left? Two things: Mathematica is a GREAT benchmark if it is the principal intended application for a machine. I did a litle soapbox act a few days ago about how vile MIPS is as a figure of merit unless two processors of the same architectural family are being compared. Doug DeJulio & Erik Buck both pointed out to me that most people actualy mean VAX MIPS equivalents (is this what DEC calls a VUP?) when they say MIPS. While I don't like VAX MIPS as a way of comparing systems I was out of line in my objection. -- Michael Galassi | nerd@percy.rain.com MS-DOS: The ultimate PC virus. | ...!tektronix!percy!nerd
dan@gacvx2.gac.edu (01/08/91)
In article <aywmrc.2dq@wang.com>, lee@wang.com (Lee Story) writes: ... much deleted ... > As a benchmark, Mathematica is wholly bogus. It has been reorganized > (somewhat ineptly, it appears) to run with the Mac's segmented code > resource scheme, with PharLap on dos, etc. The compilers used are > unstated in its documentation, and the source is not available for > recompilation. Thus it cannot be used to benchmark raw processor/memory > speed or processor/memory/compiler speed. What's left? I challenge the assumption that it is a bogus benchmark. Many of us still buy computers for the applications they run, and not because of the raw processor speed a measured by standard benchmarks. We want to buy the hardware that will run the software we want to use as fast as we can afford. Mathematica is of great interest to the educational community. One of the professors here used grant money to purchase a SGI personal Iris ($16,000) so he could do his research, using Mathematica. The SGI did not do as well as the NeXTstation ($4,000) in the Mathematica benchmark. You supply the evidence for much of my arguement in your post. Mathematica on the Mac is not able to match the performance of Mathematica on other platforms, because of arcitecture limitations, and possibly less than optimal porting. If the MacOS and the Mac's hardware it performs less than it would under ideal benchmarks. I would not rank the Mathematica benchmarks in the same catagory as the standard benchmarks (ideal/theoretical) they have their value, however I would not rank it as a bogus benchmark, I prefer to rank it as a practical or real life benchmark. We have to bechmark more than just the machine, we also need to take into account the software we use on that hardware. -- Dan Boehlke Internet: dan@gac.edu Campus Network Manager BITNET: dan@gacvax1.bitnet Gustavus Adolphus College St. Peter, MN 56082 USA Phone: (507)933-7596
smithw@hamblin.math.byu.edu (Dr. William V. Smith) (01/08/91)
Mark Adler (madler@piglet.caltech.edu) writes: [in response to some drivel about how real programs are useless benchmarks] >Therefore, if you use Mathematica most of the time, then Mathematica is >the perfect benchmark program. Bar none. In fact, the very problems Lee >uses to argue that Mathematica is no good as a general purpose benchmark, >illustrate how useless the general purpose benchmarks are at telling you >how your program will run (if your program is Mathematica, in this case) RIGHT! RIGHT! RIGHT! RIGHT! -- __________________Prof. William V. Smith____________________ EMail: smithw@hamblin.math.byu.edu or uunet!hamblin.math.byu.edu!smithw SMail: Math Dept. -- 314 TMCB; BYU; Provo, UT 84602 (USA) NeXTmail: smithw@mathnx.math.byu.edu Phone: +1 801 378 2061 FAX: +1 801 378 2800
melling@cs.psu.edu (Michael D Mellinger) (01/09/91)
Does the NeXT contain a large off-chip cache? I don't recall seeing it mentioned anywhere. -Mike