[comp.arch] Benchmarking ...really Compiler flames

sorgatz@ttidca.TTI.COM ( Avatar) (06/05/87)

Summary:people go to school for 4 years to code like this?

Keywords:compiler design, compiler designers


In article <7261@amdahl.amdahl.com> chuck@amdahl.UUCP (Charles Simmons) writes:
>
>The holistic approach suggests that you need to simulate the application
>environment on each system that you are considering buying.  This
>obviously has a number of difficulties associated with it.  Perhaps
>the biggest difficulty is accurately determining what the application
>environment will actually look like.

This presumes that the applications designer should be given free reign to
do silly stuff. Far better would be to determine the performance of a system
based on the "raw" power of the machines attributes directly.

>
>The reductionistic approach suggests that system performance can
>be estimated by examining a number of pieces of the system.  In
>particular, I would suggest the following pieces:
>
{{many exellent categories}}

>3)  context switching speeds:  how long does it take for the system
>to process an interrupt?  how long does it take for the system to
>switch between user programs?

 This is mostly a measure of how directly the operating system can react to
the real-world events or interrupts, not something that UNIX is famous for
doing in an intellegent fashion.

>4)  compiler performance:  how well does the compiler optimize?
>how slow is the compiler?  how large are the binaries produced by
>the compiler?

 This is the central point: most compilers are dumb. They appear to have been
written by someone scarcely familar with the hardware environment. The classic
examples of 'jump-to-the-next-instruction' or 'save-all-registers/do-something-
non register-related/restore-all-registers' are the biggest wastes of MIPS
although some optimizers will nail these => 50% of the time.

>
>Any other reductionists want to help me out with my argument?
>
>-- Chuck

 Consider this to be my contribution: Wakeup compiler experts! Read the !@#$%^&
chip spec! Don't assume that your 'meta-pass-intermediate code generators
are going to produce efficient code; they dont!
-- 
-Avatar-> (aka: Erik K. Sorgatz) KB6LUY
Citicorp(+)TTI
3100 Ocean Park Blvd.   (213) 450-9111, ext. 2973
Santa Monica, CA  90405 {csun,philabs,randvax,trwrb}!ttidca!ttidcb!sorgatz

crowl@rochester.arpa (Lawrence Crowl) (06/05/87)

In article <787@ttidca.TTI.COM> sorgatz@ttidcb.UUCP
(Erik Sorgatz - Avatar) writes:
>This is the central point: most compilers are dumb. They appear to have been
>written by someone scarcely familar with the hardware environment.

No.  Compilers are written by people who generally do not have the time to
integrate all the details of the hardware into the compiler.  There are two
major requirements on a compiler. One, it MUST generate code which semantically
matches the source.  Two, it SHOULD generate code which is competitive in
efficiency with other compilers.  If the compiler does not meet requirement
one, it does not matter if it meets requirement two.  Now, judging when
requirement two is satisfied is a management decision.  Most managers do not
want to double their investment in a compiler to get 25% faster code.

>Consider this to be my contribution: Wakeup compiler experts! Read the !@#$%^&
>chip spec! Don't assume that your 'meta-pass-intermediate code generators are
>going to produce efficient code; they dont!

My first task is to get correct code.  "Meta-pass-intermediate code generators"
achieve this goal with minimal development effort.  Now, if you want me to
handle the case when instruction A is faster than instruction B, unless B
follows C, except in kernel mode, etc. then you must justify to managment
the SUBSTANTIAL additional cost.

If you want a cheap, efficient compiler, you must give me a simple, orthogonal
architecture.

-- 
  Lawrence Crowl		716-275-5766	University of Rochester
			crowl@rochester.arpa	Computer Science Department
 ...!{allegra,decvax,seismo}!rochester!crowl	Rochester, New York,  14627

shebs@utah-cs.UUCP (Stanley Shebs) (06/05/87)

In article <787@ttidca.TTI.COM> sorgatz@ttidcb.UUCP (Erik Sorgatz - Avatar) writes:

> This is the central point: most compilers are dumb. They appear to have been
>written by someone scarcely familar with the hardware environment. The classic
>examples of 'jump-to-the-next-instruction' or 'save-all-registers/do-something-
>non register-related/restore-all-registers' are the biggest wastes of MIPS
>although some optimizers will nail these => 50% of the time.

Only the dumbest of compilers will miss these.  Jumps to the next instruction
and extraneous stack ops can even be handled using 'sed' on the assembly
language output (I've done it).

>Consider this to be my contribution: Wakeup compiler experts! Read the !@#$%^&
>chip spec! Don't assume that your 'meta-pass-intermediate code generators
>are going to produce efficient code; they dont!

\begin{flame}
I can never finish reading the chip spec because I get sick and have to go
home halfway through!  So-called features get included because some hardware
engineers figure that *their* assembly-language programs are representative
of programming as a whole, and so they put in perverted instructions/timings/
details.  Or the hardware has some "optimization" that makes it totally
impossible for a compiler writer to figure the performance without actually
running the code to see what happens!
\end{flame}

There are very good compilers in the world that do take advantages of
specific machine features.  However, they tend to be 1-2 orders of magnitude
more expensive than your public-domain Pascals and BSD C compilers, and so
most places don't have them.  Good code generation can be expected to take
many person-years of effort, per machine per language.  The whole
motivation for various automated techniques for building code generators
is to bring this cost down.  Unfortunately, the techniques are only as
good as our models of hardware behavior, and the days are long gone when
program performance could be predicted by adding together the timings
of each instruction.  When the accurate machine simulator is a 40,000
line C program, it's going to be very very hard for automated code generator
constructors (or humans, for that matter) to figure out where the good
performance *really* is, as opposed to where the hardware designers say
it is!

Likewise, the motivation for intermediate languages is that for M languages
and N machines, use of an intermediate language means M+N pieces of compiler
instead of MN whole compilers.  If we could all standardize on one language
or one machine, the problem would pretty much go away, but there's not much
chance of that.

The compiler experts are already awake, but there are limits to how many
hardware-imposed rings they can jump through...

>-Avatar-> (aka: Erik K. Sorgatz) KB6LUY

							stan shebs
							shebs@cs.utah.edu

henry@utzoo.UUCP (Henry Spencer) (06/09/87)

> Consider this to be my contribution: Wakeup compiler experts! Read the !@#$%^&
> chip spec! Don't assume that your 'meta-pass-intermediate code generators
> are going to produce efficient code; they dont!

Well then, give us your !@#$%^&ing chip specs enough in advance so that we
can do good work, and give us chips that actually *MEET* the !@#$%^&ing
spec so we don't have to spend all our time doing your debugging for you!!
Oh yeah, and convince the !@#$%^&ing marketing people to promise realistic
dates for compiler delivery.  Then maybe we'll have a chance to do it right
for once!   :-) :-) :-)
-- 
"There is only one spacefaring        Henry Spencer @ U of Toronto Zoology
nation on Earth today, comrade."   {allegra,ihnp4,decvax,pyramid}!utzoo!henry