painter@lll-crg.llnl.gov (Jeff Painter) (09/01/89)
I am looking for Gabriel or other benchmarks that would be useful for
comparing Lisp performance of the following machines, preferably with
compilers from the same company (e.g. Allegro Common Lisp or Lucid
CL). If you have such information, or can point the way to it, please
E-mail a reply to painter@lll-crg.llnl.gov. I will post a summary, of
course. Thanks in advance!
MIPS machines, especially DecStation 3100
various Explorers, Lambda, and other Lisp machines
Suns, especially SparcStations
Vaxes
Jeffrey F. Painter painter@lll-lcc.llnl.gov
L-316, Lawrence Livermore Laboratory painter@lll-crg.llnl.gov
Livermore CA 94550 USA 415-422-0675, FTS 532-0675
415-423-7613, FTS 543-7613painter@lll-crg.llnl.gov (Jeff Painter) (10/03/89)
About a month ago, I posted the following. - I am looking for Gabriel or other benchmarks that would be useful for - comparing Lisp performance of the following machines, preferably with - compilers from the same company (e.g. Allegro Common Lisp or Lucid - CL). If you have such information, or can point the way to it, please - E-mail a reply to painter@lll-crg.llnl.gov. I will post a summary, of - course. Thanks in advance! I apologize to anyone who has been waiting for the summary. I decided to postphone it until I had access to an interesting machine to run some benchmarks myself. These results are below. Most of you who read this know at least as well as I to take these benchmarks with a grain of salt; there isn't enough information here to make reasonably good comparisons. Nevertheless I find them interesting. ________________________________________________________________ From shiffman@Sun.COM Thu Aug 31 18:56:46 1989 (Hank Shiffman) I can provide the numbers for Sun Common Lisp 3.0 (Lucid). The numbers I have are user run time figures and are based on full type declarations and full optimization. BTW, these figures are the official Sun results for these benchmarks. ---------------------------- Okay. Here are the figures for a Sparcstation 1 vs. a 3/60. Keep in mind that these are user run time results in seconds, so the amount of physical memory and disk performance would have little effect on the results. Ephemeral GC was turned off and swap space was large enough to allow each benchmark to run to completion without a dynamic GC. I don't have results for a 3/50. Figure roughly half the CPU performance of a 3/60. Actual clock time results would be significantly worse, given the small amount of memory on a stock 3/50. SS1 3/60 8MB 20MB Tak .07 .36 Stak .59 1.82 Ctak .59 1.58 Takl .45 1.64 Takr .11 .50 Boyer* 2.85 8.92 Browse* 3.65 15.56 Destru .38 1.38 Trav-Init* .95 3.24 Trav-Run* 6.04 26.90 Deriv .75 3.10 Dderiv .98 3.88 Div2-Iter .39 .94 Div2-Recur 1.69 1.38 FFT* .56 3.68 Puzzle* 1.24 4.06 Triang* 17.86 113.84 Fprint .24 1.02 Fread .33 1.88 Tprint .12 .88 Frpoly* R2-10 .52 1.46 R3-10 .21 .94 R-15 .75 2.88 R3-15 1.42 6.70 R2-15 4.47 12.14 Advantage over 3/60 All benchmarks 4.10 1.00 *Selected set 4.54 1.00 -- Hank Shiffman (415) 336-4658 Marketing Technical Specialist Software Engineering Technologies ...!sun!shiffman Sun Microsystems, Inc. shiffman@Sun.com ________________________________________________________________ From mohammad@hpclmp.hp.com Tue Sep 5 16:39:03 1989 (Mohammad Pourheidari) The following benchmarks are for Lucid 3.0 running on an HP 850. We got these results from Lucid. [3-run averages; individual run figures deleted- jfp] EGC incapable MT off ctak Average: 0.55 stak Average: 0.57 tak Average: 0.11 takl Average: 0.44 takr Average: 0.16 boyer Average: 2.79 browse Average: 4.24 dderiv Average: 1.10 deriv Average: 0.84 destru Average: 0.40 iterative Average: 0.32 recursive Average: 0.52 fft Average: 0.88 fprint Average: 0.34 fread Average: 0.38 frpoly r 2 Average: 0.00 frpoly r2 2 Average: 0.01 frpoly r3 2 Average: 0.00 frpoly r 5 Average: 0.01 frpoly r2 5 Average: 0.07 frpoly r3 5 Average: 0.02 frpoly r 10 Average: 0.13 frpoly r2 10 Average: 0.61 frpoly r3 10 Average: 0.23 frpoly r 15 Average: 0.88 frpoly r2 15 Average: 7.57 frpoly r3 15 Average: 1.50 puzzle Average: 1.95 tprint Average: 0.25 traverse-init Average: 1.12 traverse-run Average: 11.15 triang Average: 32.02 EGC incapable MT on ctak Average: 0.56 stak Average: 0.58 tak Average: 0.11 takl Average: 0.45 takr Average: 0.16 boyer Average: 2.81 browse Average: 4.20 dderiv Average: 1.09 deriv Average: 0.84 destru Average: 0.39 iterative Average: 0.32 recursive Average: 0.51 fft Average: 0.89 fprint Average: 0.31 fread Average: 0.38 frpoly r 2 Average: 0.00 frpoly r2 2 Average: 0.01 frpoly r3 2 Average: 0.00 frpoly r 5 Average: 0.01 frpoly r2 5 Average: 0.07 frpoly r3 5 Average: 0.02 frpoly r 10 Average: 0.13 frpoly r2 10 Average: 0.61 frpoly r3 10 Average: 0.22 frpoly r 15 Average: 0.86 frpoly r2 15 Average: 7.59 frpoly r3 15 Average: 1.51 puzzle Average: 1.75 tprint Average: 0.25 traverse-init Average: 1.12 traverse-run Average: 11.17 triang Average: 31.99 EGC capable EGC off MT off ctak Average: 0.55 stak Average: 0.63 tak Average: 0.11 takl Average: 0.44 takr Average: 0.16 boyer Average: 2.81 browse Average: 4.27 dderiv Average: 1.10 deriv Average: 0.86 destru Average: 0.40 iterative Average: 0.33 recursive Average: 0.51 fft Average: 0.88 fprint Average: 0.34 fread Average: 0.40 frpoly r 2 Average: 0.00 frpoly r2 2 Average: 0.01 frpoly r3 2 Average: 0.00 frpoly r 5 Average: 0.01 frpoly r2 5 Average: 0.07 frpoly r3 5 Average: 0.02 frpoly r 10 Average: 0.13 frpoly r2 10 Average: 0.61 frpoly r3 10 Average: 0.23 frpoly r 15 Average: 0.88 frpoly r2 15 Average: 7.59 frpoly r3 15 Average: 1.52 puzzle Average: 1.75 tprint Average: 0.26 traverse-init Average: 1.12 traverse-run Average: 11.16 triang Average: 32.00 EGC capable EGC off MT on ctak Average: 0.55 stak Average: 0.63 tak Average: 0.11 takl Average: 0.44 takr Average: 0.16 boyer Average: 2.81 browse Average: 4.27 dderiv Average: 1.10 deriv Average: 0.86 destru Average: 0.40 iterative Average: 0.33 recursive Average: 0.51 fft Average: 0.88 fprint Average: 0.34 fread Average: 0.40 frpoly r 2 Average: 0.00 frpoly r2 2 Average: 0.01 frpoly r3 2 Average: 0.00 frpoly r 5 Average: 0.01 frpoly r2 5 Average: 0.07 frpoly r3 5 Average: 0.02 frpoly r 10 Average: 0.13 frpoly r2 10 Average: 0.61 frpoly r3 10 Average: 0.23 frpoly r 15 Average: 0.88 frpoly r2 15 Average: 7.59 frpoly r3 15 Average: 1.52 puzzle Average: 1.75 tprint Average: 0.26 traverse-init Average: 1.12 traverse-run Average: 11.16 triang Average: 32.00 EGC on MT off ctak Average: 0.57 stak Average: 0.70 tak Average: 0.11 takl Average: 0.44 takr Average: 0.16 boyer Average: 4.35 browse Average: 4.73 dderiv Average: 1.23 deriv Average: 0.96 destru Average: 0.44 iterative Average: 0.39 recursive Average: 0.59 fft Average: 0.88 fprint Average: 0.32 fread Average: 0.38 frpoly r 2 Average: 0.00 frpoly r2 2 Average: 0.01 frpoly r3 2 Average: 0.00 frpoly r 5 Average: 0.01 frpoly r2 5 Average: 0.07 frpoly r3 5 Average: 0.02 frpoly r 10 Average: 0.14 frpoly r2 10 Average: 0.61 frpoly r3 10 Average: 0.22 frpoly r 15 Average: 0.98 frpoly r2 15 Average: 8.23 frpoly r3 15 Average: 1.83 puzzle Average: 1.75 tprint Average: 0.26 traverse-init Average: 2.20 traverse-run Average: 10.91 triang Average: 32.02 EGC on MT on ctak Average: 0.56 stak Average: 0.70 tak Average: 0.11 takl Average: 0.44 takr Average: 0.16 boyer Average: 4.35 browse Average: 4.74 dderiv Average: 1.22 deriv Average: 0.96 destru Average: 0.44 iterative Average: 0.39 recursive Average: 0.58 fft Average: 0.88 fprint Average: 0.34 fread Average: 0.39 frpoly r 2 Average: 0.00 frpoly r2 2 Average: 0.01 frpoly r3 2 Average: 0.00 frpoly r 5 Average: 0.01 frpoly r2 5 Average: 0.07 frpoly r3 5 Average: 0.02 frpoly r 10 Average: 0.13 frpoly r2 10 Average: 0.61 frpoly r3 10 Average: 0.23 frpoly r 15 Average: 0.99 frpoly r2 15 Average: 8.27 frpoly r3 15 Average: 1.83 puzzle Average: 1.76 tprint Average: 0.27 traverse-init Average: 2.18 traverse-run Average: 10.95 triang Average: 32.05 DHRYSTONES (FL C = Foreign Loaded C. Lisp EGC incapable with Multitasking off.) C FL C Lisp - ---- ---- Microseconds for one run through Dhrystone: 50.4 55.0 192.0 Dhrystones per Second: 19841.3 18181.8 5208.3 C to Lisp: 3.81 FL C to Lisp: 3.49 > >Thank you for the data. I presume that EGC refers to garbage >collection, and MT to multitasking. > Egc refers to "Ephemeral Garbage Collection". One other thing I forgot to tell you on those benchmarks. They were run in multiuser mode. I will be looking forward to seeing your results. ________________ Below are the Lisp 3.0 Gabriel Benchmarks which were run on a series 835 system which was running HPUX version 7.0. Environment: S800 835 with 32 Mb memory, 132 Mb swap space Ephemeral Garbage Collection and Multitasking on Compiler settings: compilation-speed = 0 speed = 3 safety = 0 Results are user run times in seconds. The average of three consecutive runs is given, along with the individual run times. [individual times deleted- jfp] Benchmark Average Time --------- ------------ ctak .51 stak .64 tak .10 takr .15 boyer 4.10 browse 4.90 dderiv 1.26 deriv 1.01 destru .38 div2 iterative .39 div2 recursive .56 fft .42 fprint .30 fread .38 frpoly r 10 .20 frpoly r2 10 .58 frpoly r3 10 .22 frpoly r 15 .90 frpoly r2 15 7.27 frpoly r3 15 2.66 puzzle 1.60 tprint .22 triang 29.35 ________________________________________________________________ I obtained a version of the Gabriel benchmarks from gatekeeper.dec.com, and ran them on an LMI Lambda Lisp machine, with its Release 3 software (operating system, compiler, etc.). Following are the total running times. I was unable to get other kinds of timing on this machine; indeed, some thins like distinguishing between user and system time contradict the spirit of a Lisp machine. These times are not comparable to most benchmark data because they include garbage collection, swapping memory to disk, and other such system and non-CPU time. For some benchmarks most of the time is devoted to such things, for others none of it. However, no unusual unrelated tasks were being run when I ran these benchmarks. These times are the averages of three runs, in seconds. Run-to-run variation suggests that only the first one or two digits are worth looking at. The FRPOLY benchmarks are a bit different from those above. It would seem that a FRPOLY-4 time is the sum of the R-15, R2-15, and R3-15 FRPOLY times, and similarly 3 corresponds to R-10,... and 2 to R-5,... But I must emphasize that the times here aren't comparable to the times shown for other machines. BOYER -------- 54.981 BROWSE ------- 51.779 CTAK ---------- 5.220 DDERIV ------- 12.394 DERIV --------- 9.584 DESTRU -------- 6.710 DIV2-ITER ----- 7.752 DIV2-RECUR ---- 9.876 FFT ---------- 33.231 FPRINT ------- 16.566 FREAD -------- 19.893 FRPOLY-1 ------ 0.044 FRPOLY-2 ------ 0.629 FRPOLY-3 ------ 7.500 FRPOLY-4 ----- 57.995 PUZZLE ------- 40.769 STAK ---------- 8.947 TAK ---------- 21.046 TAKL --------- 22.455 TAKR ---------- 2.102 TPRINT ------- 12.589 TRAVERSE-INIT 32.675 TRAVERSE-RUN 173.256 TRIANG ------ 902.500 ________________________________________________________________ The following timings are from a DecStation 3100 running a beta test version of Franz Allegro Common Lisp. These benchmark programs were obtained from gatekeeper.dec.com in Sept. 1989. Documentation for the Lisp states that the first regular release will be significantly faster. The compiler settings are; Speed 3 Safety 0 (fastest setting). Times are in seconds, and are the average over 3 runs. Run-to-run variation is usually .01 seconds or less for cpu time, and anywhere from 0 to 50% for total real running time. The first time below is the non-gc user cpu time and the second time is the total elapsed time. Relevent to the elapsed time figures is that the DS3100 was running in two-user mode, without any other than the usual user processes running. It had 16MB memory. Its virtual memory was in an RZ55 disk drive. (In my opinion, it is not worthwhile to run nontrivial Lisp programs on a diskless workstation). The results with Speed 1 Safety 1 (the default compiler setting) were about 20% slower for the cases I looked at. BOYER ------ 1.816 4.039 BROWSE ----- 2.376 4.519 CTAK -------- .296 .536 DDERIV ------ .656 1.161 DERIV ------- .520 .942 DESTRU ------ .420 .698 DIV2-ITER --- .123 .236 DIV2-RECUR -- .236 .511 FFT --------- .333 .606 FPRINT ------ .153 .416 FREAD ------- .543 .956 FRPOLY-1 ---- .000 .090 FRPOLY-2 ---- .053 .091 FRPOLY-3 ---- .743 1.243 FRPOLY-4 --- 7.086 13.108 PUZZLE ------ .903 1.648 STAK -------- .643 1.093 TAK --------- .686 1.148 TAKL -------- .376 .629 TAKR -------- .093 .155 TPRINT ------ .216 1.952 TRAVERSE-INIT .513 1.142 TRAVERSE-RUN 4.870 8.196 TRIANG ---- 18.176 30.327 ________________________________________________________________ What I was really interested in when posting my original query was to predict whether a large Macsyma application now running on a Lisp machine would run any faster on any of the newer high-end workstations. We don't have that program ported to any such workstation yet. However, Leo Harten has ported DOE-Macsyma to a DecStation 3100 with Allegro Common Lisp, and a few of the Macsyma demo programs ran about 3.5 to 5.5 times as fast as they do on an LMI Lambda Lisp machine. This is consistent with the results of the FRPOLY benchmarks above. Jeffrey F. Painter painter@lll-lcc.llnl.gov L-316, Lawrence Livermore Laboratory painter@lll-crg.llnl.gov Livermore CA 94550 USA 415-422-0675, FTS 532-0675 415-423-7613, FTS 543-7613
sandell@batcomputer.tn.cornell.edu (Gregory Sandell) (10/03/89)
Having seen the results for a SPARC station benchmark test, I am interested in seeing how a Mac IIx performs in Allegro Common LISP. I work in an evironment where we use both. Has anybody checked this out? Greg Sandell
ledford@snowbird.rtp.dg.com (01/11/90)
I am interested in benchmarks for Common Lisp. Particularly, name, where available, and your impressions of each one. Thanks in advance. Bruce Ledford ledford@dg-rtp.dg.com Industry Standard Applications Data General Corp.