[comp.arch] Application specific benchmarking

ravi@crackle.amd.com (Ravi Chitturi) (09/26/89)

	I would like to start a discussion on the topic of "Benchmarking  RISC/CISC machines for
 embedded control". Those of you who attended 1989 Microprocessor forum would agree that
 currently available benchmarks provided by most chip vendors are very ambiguous at the least
 and are of little practical use to the customers per se. As things stand, it is hard to 
 distinguish the truth from marketing hype.
	
	I would suggest creation of application specific benchmarks e.g.,benchmarking programs for
 embedded control. Currently, all chip makers can pick and choose only the programs that run
 well on their machines. This makes every chip look equally good, but with different set of
 benchmarking programs, and thus confuses the customer. It would be nice to see standard
 benchmarking programs for embedded controllers. We may have to create 2 or 3 sets of benchmarking
 programs to cover the entire embedded control spectrum adequately.


							......Ravi 

 P.S. Disclaimer : Above mentioned comments and opinions are solely mine and not AMD's.

scarter@gryphon.COM (Scott Carter) (09/29/89)

In article <27483@amdcad.AMD.COM> ravi@crackle.amd.com (Ravi Chitturi) writes:
>
>	I would like to start a discussion on the topic of "Benchmarking  RISC/CISC machines for
> embedded control". Those of you who attended 1989 Microprocessor forum would agree that
> currently available benchmarks provided by most chip vendors are very ambiguous at the least
> and are of little practical use to the customers per se. As things stand, it is hard to 
> distinguish the truth from marketing hype.
>	
>	I would suggest creation of application specific benchmarks e.g.,benchmarking programs for
> embedded control. Currently, all chip makers can pick and choose only the programs that run
> well on their machines. This makes every chip look equally good, but with different set of
> benchmarking programs, and thus confuses the customer. It would be nice to see standard
> benchmarking programs for embedded controllers. We may have to create 2 or 3 sets of benchmarking
> programs to cover the entire embedded control spectrum adequately.
>
We've been battling this, primary application areas avionics, weapon control,
missile guidance, etc.  Not so much your high-volume application, but still
respectable $$ :).

One major problem is that any non-trivial benchmark typically takes weeks
to simulate.  This assumes that you can manage to find a non-trivial
benchmark with no (or removable) I/O or OS dependencies.  Trying to simulate
the I/O has proved an insurmountable obstacle so far (given budget constraints,
etc.).  I'm not sure that one can make up an embedded system benchmark that
is necessarily any better a predictor than some seemingly vanilla benchmark,
e.g. our missile guidance benchmarks correlate as well with Doduc as with
each other.

In another case, a binning and clustering benchmark originally written for
the 68020 (Green Hills) made heavy use of shorts in a context where ints would
not have been a major memory waster.  This wound up hitting a pathology of
a certain major RISC vendor's compiler, with a 20+% performance hit.  What
do you do with that kind of data? (besides bitch to the vendor ...)

I'd still love to see something in this area, but I'm not sure that a
specific benchmark suite will be that attainable.

ravi@crackle.amd.com (Ravi Chitturi) (09/30/89)

 	Few netusers have asked me to post the same article
 in a more readable format, so here it is.

	I would like to start a discussion on the topic
 of "Benchmarking RISC/CISC machines for embedded control".
 Those of you who attended 1989 microprocessor forum
 would agree that currently available benchmarks provided
 by most chip vendors are very ambiguous at the least and
 are of little practical use to the customers per se. As 
 things stand, it is hard to distinguish the truth from
 marketing hype.

	I would suggest creation of application specific
 benchmarks e.g.,benchmarking programs for embedded control.
 Currently all chip makers can pick and choose only the
 programs that run well on their machines. This makes
 every chip look equally good, but with different sets of
 benchmarking programs, and this confuses the customer.
 It would be nice to see standard benchmarking programs
 for embedded controllers. We may have to create 2 or 3
 sets of benchmarking programs to cover the entire
 embedded control spectrum adequately.

						Ravi

DISCLAIMER : Above mentioned comments and opinions are
             solely mine and not AMD's.