blaine@nmtvax.UUCP (04/11/85)
in article <> wfmans@ihuxb.UUCP (w. mansfield)
MIPs are loosely related to IDC numbers, and in many respects IDC numbers
are more interesting. MIPs tend to be linear with IDC#s.
I know of no benchmark for MIPs which is
standardized between vendors. I have worked for several computer vendors
in the past(non IBM compatible) in competitive performance. All that we could
find out was that each vendor had his own way of measuring MIPs. (i.e. load
the mix up with "typical jobs" and use a hardware monitor to count
instructions).
Among non IBM compatible vendors the problems get worse because not only is
there no common workload, but no common operating system, and vastly different
instruction sets.
IDC numbers are loosely related to MIPs. Most people get their MIPs and IDC#
directly or indirectly from Computerworld. Computerworld is a weekly
publication with a very large subscription base.
What used to happen was when a new computer was announced, Computerworld(CW)
would call up vendors and ask for a MIPs rating. The vendor would complain
that it is not really a meaningfull metric, but in the end would end up
providing a MIPs rating (the last thing you would want is for them to guess
or ask your competitor for a number).
Eventually CW (as I remember) started publishing "relative performance"
numbers. These were cited as having come from the IDC corporation, the sister
company to CW.
A friend of mine, Steve Howard, decided to find out what IDC numbers mean.
Initially, all we could find out was that everyone agrees that the starting
point is that an IBM 370/158-3 =45. Steve went right to the source and
eventually talked with Will Zachman, V.P. of R&D for International Data
Corporation. He told Steve that the IDC number is an approximate measure
of performance; in fact, Will said that the numbers do not attempt to predict
throughput, CPU power or any other specific measureable quantity. He did claim
that the numbers can be used to rank systems by "performance" and indicate
approximately (again no definition) the ratios between them.
IDC usually does not rate non-IBM architectures, but is often asked to do
so by CW(they are sister companies). When you see "CW estimate" in
in Computerworld, it means that IDC has refused to give them numbers for that
system. God knows how CW comes up with their estimates when IDC refuses to
give them an estimate, but its probably by the same process that IDC uses.
IDC does no measurements. The main ingredients of the numbers are
vendor-supplied statements comparing 2 systems (i.e. The 3033 is n.n times
as powerful as the 4341-2). If the number produced by this method seems
unreasonable to IDC or is objected to by users or vendors, "adjustments"
may be made. IDC is very willing to accept informantion which is claimed
to increase the accuracy of its numbers.
The result of this is that almost no work or thought (opinion) is put into
establishing the numbers except to react to complaints.
WHEN THE SQUEAKING STOPS, THE WHEEL IS CONSIDERED OILED.
Given this sorry state of affairs concerning their creation, we decided to
find out what we could by looking at the numbers themselves. Why? because
management asks, Why? because marketing asks, Why? because customers ask,
Why? because the consumer is always right (if not correct).
We decided to look at how IDC#s relate to CPU power. To find out, we turned
to CPU power informantion published by the Institute for Software Engineering
(ISE). ISE used instruction kernals to measure CPU power for three workload
classes: Moves and compares, Engineering Calculations, and Decimal Arithmetic.
We plotted the IDC number vs. ISE power for each workload type. The systems
considered were IBM 4331-1, CDC 480-II, IBM 4341-2, IBM 3032, NAS/7000,
IBM 3033N, IBM 3033U, and AHMDAL V8. We observed a very linear relationship
with all three Power measures.
This showed us that the published IDC numbers could be used (at some risk) as
a measure of raw CPU power (not throughput) in spite of the haphazard method
used in making them. One can obtain "accurate" IDC numbers for non-IBM arch
systems by using architecture-independent processor power metrics on your
systems and a couple IBM systems with known IDC#s, and after assuring yourself
that there are the proper linear relationships, develop and apply the linear
translation.
Note that IDC#s and MIPs imply a certain combination of power delivered
to workloads (three cups COBOL, two cups FORTRAN). So it is really most
meaningful (if I dare use that word) on business type systems.
I only use IDC#s or MIPs when it is the only data available on a competitive
system. Loose MIPs sink ships!
--
Blaine Gaither ucbvax!unmvax!nmtvax!blaine
Computer Science Department blaine@nmtdavet@oakhill.UUCP (Dave Trissel) (04/16/85)
In article <408@nmtvax.UUCP> blaine@nmtvax.UUCP ( Gaither) writes: > >IDC usually does not rate non-IBM architectures, but is often asked to do >so by CW(they are sister companies). When you see "CW estimate" in >in Computerworld, it means that IDC has refused to give them numbers for that >system. God knows how CW comes up with their estimates when IDC refuses to >give them an estimate, but its probably by the same process that IDC uses. > Indeed, during the 1983 CW listing of superminis I noticed that two systems were MC68000 based running at the same clock frequency and yet one had twice the MIPS rating as the other. It turns out one system had tandom MC68000s for fault-tolerant capability, yet CW figured that meant that it had double the performance. Of course, both probably had equal or near equal performance. Motorola Semiconductor Inc. Dave Trissel Austin, Texas {seismo,ihnp4,gatech}!ut-sally!oakhill