[comp.sys.mac] Benchmarking C Compilers

mouser@Portia.Stanford.EDU (Michael Wang) (07/21/88)

In article <76000259@p.cs.uiuc.edu> gillies@p.cs.uiuc.edu writes:
> Recently, Comp.arch listed the results of a Dhrystone test of
> different C compilers.  The results included LightspeedC, at 2467
> Dhrystones, and also Sun 3/50's, at as much as 4000+ Dhrystones on a
> 16.67Mhz 68020 (gcc compiler).  So I don't think that LightspeedC is
> nearly as fast as possible.  A better compiler for the same CPU (at
> 1Mhz more clocks) runs almost 8/5 as fast!  Oh well, some day we will
> have extremely efficient compilers for the macintosh.... I hope.

Later, in article <4990@husc6.harvard.edu> singer@endor.harvard.edu
(Rich Siegel) writes:
>       I agree that LightspeedC isn't as efficient as possible,  but to
> compare itts execution on a Mac II to GCC's execution on a Sun 3 is unfair,
> since the two machines have fundamentally different hardware designs.
>
>        If you want to  make a valid comparison, run gcc on a Mac II running
> A/UX. There's more to a comparison than just saying "The sun 3 has the same
>CPU as the Mac II, and the same clockk speed (+/- 1MHz), but LightspeedC runs
> slower on the Mac II, so it must be inferior."
> 
>               --Rich

Well, if you believe vendor literature, the gcc compiler, developed by
Green Hills Software (not GNU CC), running on a Macintosh II under A/UX hits
6600 Dhrystones compared to 2950 Dhrystones for the standard cc under A/UX.

What this means, if you believe benchmarks, is either that the Macintosh II
version of the gcc compiler is much better than the Sun 3/50 version, or that
a Macintosh running A/UX is faster than a Sun 3/50 running BSD UNIX.  I tend
to believe the later case :-).

Unfortunately, on the purely Macintosh level, Lightspeed C, at 2500-2900
Dhrystones, has quite a way to go before it matches an established optimizing
compiler like gcc (sorry Rich).  Of course you can always argue that the
Mac OS is quite a bit different than A/UX.  On the more positive side,
Lightspeed C does do about as well as the standard AT&T cc compiler (which I
assume was running in its optimizing mode).

-Michael Wang

+---------------+-----------------------------------------------------------+
| Michael Wang  |  Stanford University, Stanford, CA  94305                 |
|---------------+-----------------------------------------------------------|
| ARPAnet, CSNET, BITNET, Internet :  mouser@portia.stanford.edu            |
| UUCP:  ...decwrl!portia.stanford.edu!mouser                               |
| AppleLink:  ST0064                                                        |
+---------------------------------------------------------------------------+

shane@chianti.cc.umich.edu (Shane Looker) (07/22/88)

[So much discussion deleted you should thank me.]


>Unfortunately, on the purely Macintosh level, Lightspeed C, at 2500-2900
>Dhrystones, has quite a way to go before it matches an established optimizing
>compiler like gcc (sorry Rich).  Of course you can always argue that the
>Mac OS is quite a bit different than A/UX.  On the more positive side,
>Lightspeed C does do about as well as the standard AT&T cc compiler (which I
>assume was running in its optimizing mode).
>
>-Michael Wang


What all of you seem to be forgetting (or not realizing, or ignoring, or...)
is that Lightspeed C is optimized for fast COMPILE.  In a compiler you 
can have fast compilation or highly optimized code generation.  Not both.
LSC does not do peephole optimization either.  (I've seen a trivial case 
in LSC which could have been 6 instrucions, instead it was 10.  The code 
kept loading a register with the same value.  Acerbic comments on C's lack
of 'with' style construct inserted here.)

MPW is a slow compiler which has tighter optimization.  I know that.  I use
LSC because of the development time difference.

All we need to do is get Mike Kahl to put a separate optimization switch in
so you can have a slow final compile which has tight code :-)  (I'd actually
love this, but it would probably mean writing an entire second compiler.)

Shane Looker
Looker@um.cc.umich.edu

singer@endor.harvard.edu (Rich Siegel) (07/22/88)

In article <583@mailrus.cc.umich.edu> shane@um.cc.umich.edu (Shane Looker) writes:
>
>[So much discussion deleted you should thank me.]
>
	Thank you!!! :-)

>What all of you seem to be forgetting (or not realizing, or ignoring, or...)
>is that Lightspeed C is optimized for fast COMPILE.  In a compiler you 
>can have fast compilation or highly optimized code generation.  Not both.

	Since it's been announced, I can talk about it now... The upcoming
Lightspeed Pascal 2.0 generates *the* best code of the Mac-based Pascal
compilers, AND it compiles faster than version 1.0.

	There is this pervasive idea that to have good code you have to have
a slow compiler. That may be true if you're DEC and want to do 10 passes
and optimize out loop invariants and all that, but for simply "good" code
(efficient use of registers, for example), compile time is NOT
necessarily inversely related to code quality.

	LightspeedC 3.0 doesn't have a new code generator because there
wasn't time. However, I suspect that Mike will let no grass grow under 
his feet...

		--Rich




Rich Siegel
Quality Assurance Technician
THINK Technologies Division, Symantec Corp.
Internet: singer@endor.harvard.edu
UUCP: ..harvard!endor!singer
Phone: (617) 275-4800 x305

gillies@p.cs.uiuc.edu (07/22/88)

I would be very interested in seeing how fast Apple's MPW C runs on a
Mac II under Multifinder.  If some kind soul on the net would run this
benchmark, I could bathe him/her in glowing *praise*.  There's another
16.67Mhz 68020 computer that attains 4000+ dhrystones -- It's made by
Harris.

I really doubt that Green Hills marketing hype (6900+ Dhrystone for C
under A/UX).  Perhaps the compiler is tracking down & removing dead
code!  That isn't fair -- the entire benchmark is supposed to be "live
code".  Also, is it the new VERSION 2.n of the Dhrystone benchmark?
It was a lot easier to find dead code in the old version 1.0 benchmark.

Model		Proc	Clock	O/S	Compiler/Options	noReg	Reg
---------------------------------------------------------------------------
Harris MCX-3/40 68020	16.67	HS/UX	HS/UX C v3.0B 		4178	4215
Sun 3/260	68020	25.0	SunOS4.5 gnucc 1.17		6993	7012
Sun 3/160	68020	16.67	SunOS4.5 gnucc 1.17		3681	3681

I agree that a fast turnaround-compiler shouldn't produce wonderful
code, but I was bemoaning the lack of ANY efficient compilers for the Mac
II.  LSC claims to outperform some other compilers (Aztec, Turbo), so
I used it for comparison.

As far as adding LSC switches (speaking off-the-cuff): register
allocation is perhaps something that can be done well & quickly on a
per-basic-block basis.  Peephole optimization might be hard, since
LSC probably emits binary opcodes, not assembly code.  Too bad.
Peephole optimizers are sometimes both trivial & effective.  As Rich
might say -- I want to have my cake & eat it too!

Maybe we should move this discussion to comp.sys.mac.programmer.

Don Gillies, Dept. of Computer Science, University of Illinois
1304 W. Springfield, Urbana, Ill 61801      
ARPA: gillies@cs.uiuc.edu   UUCP: {uunet,ihnp4,harvard}!uiucdcs!gillies

awd@dbase.UUCP (Alastair Dallas) (07/22/88)

I happen to know one of the authors of gcc.  My impression of his skill is
such that I believe the ridiculously high numbers quoted for its performance.
As for LightspeedC, it's obvious from the documentation that Think's main
claim for their compiler is speed of compilation, not execution.  I have 
always assumed that LSC's output was slightly larger and slightly slower
than optimal.  Time-critical stuff should be in assembler anyway.

Programming on a Mac for a Mac is such a difficult proposition, I think the
ideal development environment would use LSC's approach: optimize for a very
fast compile-link-run-edit cycle so that the programmer is constantly in
touch with the "feel" of the running program.  If you stick to standard,
portable C, you can easily (so say those who have done it) port to MPW,
which generates somewhat tighter code (again, so I'm told).

By the way, I *still* haven't gotten 3.0.  I called Symantec and they said
there's no mistake; it'll be here "soon."  Soon means the day after the 
retail channel is completely filled, I guess.  <grumble, grumble>.
Calm down, Rich--I _like_ LightspeedC; I'd just _like_ to have it in hand.

/alastair/

kaufman@polya.Stanford.EDU (Marc T. Kaufman) (07/23/88)

In article <76000275@p.cs.uiuc.edu> gillies@p.cs.uiuc.edu writes:

>I really doubt that Green Hills marketing hype (6900+ Dhrystone for C
>under A/UX).  Perhaps the compiler is tracking down & removing dead
>code!  That isn't fair -- the entire benchmark is supposed to be "live
>code"...

au contraire, my boy, removing dead code is EXACTLY what is wanted from an
optimizing compiler.  Along with loop strength reduction, address merging,
register marking, common subexpression elimination, etc., etc.

Marc Kaufman (kaufman@polya.stanford.edu)

suitti@haddock.ISC.COM (Steve Uitti) (07/26/88)

In article <422@dbase.UUCP> awd@dbase.UUCP (Alastair Dallas) writes:
>
>I happen to know one of the authors of gcc.  My impression of his skill is
>such that I believe the ridiculously high numbers quoted for its performance.
>As for LightspeedC, it's obvious from the documentation that Think's main
>claim for their compiler is speed of compilation, not execution.  I have 
>always assumed that LSC's output was slightly larger and slightly slower
>than optimal.  Time-critical stuff should be in assembler anyway.
	Note however that a slow compiler does not guarantee better
(or even good) code.  Otherwise bright people often buy Microsoft C
over Turbo C since it costs five times as much, has a three to four
times slower compile speed, etc.  So far, Turbo C has always produced
code that is smaller, and that runs within 5 percent of usoft C (half
of the time it is quicker).  My comparison between LSC and Sun's "cc"
was very favorable, considering that a Sun III has a quicker CPU than
a Mac II.  I'd really like to see a quick compiler for a UNIX box.

>Programming on a Mac for a Mac is such a difficult proposition
	Not as bad as on a Commodore 64.  C Power does pretty well here.
>
>By the way, I *still* haven't gotten 3.0.
>/alastair/
	I receive LSC 3.0 while on vacation, in the last couple of
weeks.  I put my request in the mail the day after Rich posted the
announcement.  I reran my floating point benchmarks, and obtained a
factor of ten improvement on the Mac II.  Good.  My PC clone (at 4.77
MHz, but with an 8087) is no longer quicker than my Mac II for
floating point using a real language.

	Stephen Uitti