hamachi@ucbkim (02/25/85)
From: hamachi@ucbkim (Gordon Hamachi)
Attention would-be compiler buyers. Here are the results from my
informal survey of C compilers for the Macintosh. At the Macworld Expo
this weekend I went from booth to booth, running the same benchmark
through each vendor's compiler.
The benchmark was the sieve program as published in Byte magazine,
August 1983. The benchmark did not use register variables. The "Size"
entry in the table refers the the size in bytes of the a.out resulting from
compiling the sieve program. The "Small" entry shows the size of the a.out
for the trivial program main(){printf("Hello, world.\n");}. This gives a
measure of the incremental size of the code produced by the compiler.
Vendor Memory Disk Compile Link Load+Run Run Size Small
===========================================================================
Megamax 512K Floppy 7.9 71 13.0 6.6 5484 5276
Megamax 512K Corvus Omni 2.7 23 9.2 6.6 5484 5276
Softworks 128K Floppy 77.4 111.9 15.8 8.8 34560 26368
Consulair 512K HyperDrive 19.2 14.5 15.2 8.0 10496 10240
Hippo 2 512K Ram Disk 11.6 8.8 14.1 --- 13864 ---
Note that no two vendors had the same hardware configuration. It is
hard to make comparisons because of these hardware differences. I assume
that a RAM disk is faster than the HyperDrive, which is faster than the Corvus
Omni, which is faster than the floppy. How much faster? I've written
to the companies to see if they will send me software for further analysis.
Run times were measured with the stopwatch on my digital wrist watch. I'd
like to redo this using the tickcount() function from the Mac toolbox.
Manx Aztec C was not in attendance. Too bad.
Megamax looks pretty good in every category except linking time. It compiles
like the proverbial bat out of hell. I don't see linking time
as a problem as you win big in the size of the resulting a.out file.
Theirs is the only linker that throws out routines it doesn't need. If only
you could selectively turn that optimization on and off. Softworks uses
the Apple (Consulair) linker. I forget, maybe Hippo does too.
They go directly from source to object, with no intermediate assembly step.
It is fast, but you can't look at asm code except by disassembling.
On the other hand, I found a couple of bugs in the Megamax compiler.
Remember when you read this that I spent more time with this compiler than
any of the others, so don't jump to any conclusions.
The Softworks compiler seemed a bit more complicated to use. It has built-in
optimization that you can't turn off. On the other hand, unlike the Megamax
compiler, you can look at an intermediate assembly language file and even
munge around with it. Some quirks of Softworks: You MUST have at least one
global variable, and that global variable must be initialized at compile
time. Every line must be terminated by a carriage return. It is a compiler
error if the last line ends in a '}'. My impression is one of killing a
gnat with an atom bomb. It is a port of the Whitesmith C compiler. Does this
mean that it is a "real" compiler, capable of gracefully handling mammoth
programs?
Consulair has the most Mac-like user interface. It seemed slick and the
easiest to use. Better performance may be on the way--code optimization and
floating point are planned for mid April. Originally this release was planned
to be much earlier. Bill DuVall's wife said that Bill has been tied up putting
the finishing touches on the Apple MDS package. Their booth seemed to get a
LOT of attention. It was hardest to get near this booth, let alone run a
benchmark.
Hippo had a unix-like interface. The prompt was even a percent sign. Many
of the expected commands like "ls" and "rm" were there. You compile by typing
"cc" followed by command line arguments. You exit to the finder by typing
"logout". There's a certain appeal to this kind of environment. I'd like
to use it for a few days and then decide which is better. Let's face it,
the easiest to use systems make you select a file from a scrolling dialog
box, but if the file you want isn't visible in the window, it is clumsy and
annoying to scroll the window in search of the file.
All the vendors were helpful and friendly. The folks at Softworks and at
Hippo seemed more distracted (burned-out?) than the others. Consulair and
Megamax seemed the most lively and helpful with my evaluation project.
Not to leave anyone out, I talked to a guy who swears by the Aztec C compiler
from Manx.
Please use these numbers for comparison only. Your mileage will differ,
and may be less. Some weaknesses with my evaluation: Non-uniform
hardware. Only one small program, no "real sized" ones. Timing was a bit
sloppy, and tended to include the time to load a program. Thus, compilers
generating big a.out files tend to run slower by my way of measuring.
My best advice is to try as many of these compilers as possible and draw your
own conclusions. Or, hope that all vendors send me evaluation copies, and
watch this space for future announcements.
If anyone knows of a Macintosh C compiler I've left out, please tell me
their name and address.
--Gordon Hamachisjl@amdahl.UUCP (Steve Langdon) (03/05/85)
Gordon Hamachi's comparison of C compilers at the Macworld Expo was certain
to influence the buying decisions of many readers of net.micro.mac. While
he was careful to disclaim his tests as a definitive comparison, people are
likely to use it to guide their buying decisions. As someone who attended
the Expo, I realize that it was difficult to perform any tests. Gordon is
to be congratulated for getting any comparative results. However, when
posting this type of commentary, care is needed. Gordon noted the differences
in hardware configuration, but did not provide some other important information.
He has already posted a note on the differences in default int size between
Megamax and Mac C (from Consulair). There are other differences that will
affect the times he quoted. For example, "Load+Run" time can be significantly
affected by the version of the Finder used. Anyone who has used one of the
2.xX series Finders knows that they launch applications quicker than 1.1g.
I own a copy of FastFinder (TM) which launches much faster than any standard
finder. Other important speed differences that can occur (particularly on
floppies) relate to file layout on disk. If a file is fragmented it can
substantially increase the time spent on disk I/O.
All of my comments above are a long-winded way of saying that you should be
very cautious about using a single benchmark to select an important tool
like a compiler. Many other factors need considered, for example, support
may be important to you (it is to me). As someone who has used Mac C since
last October I would recommend it to other buyers even if the code was
substantially slower than an alternative. I would do this because Consulair
have been consistently helpful when I ran into problems. No compiler is
free from bugs. One just has to read net.unix.whatever to read about PCC
problems. Fixes and reasonable means of avoiding existing bugs are important.
Those who are reading comparisons of compilers should be careful to consider
what they are buying when comparing compiler prices. I cannot describe the
contents of packages other than Mac C. With Mac C you currently get the C
compiler itself, a pre-release version of the Apple MDS system, and the standard
libraries in relocatable form. If you buy the Mac C Toolkit you receive some
additional useful library routines in both source and relocatable form, you
also get the source of the standard libraries. Mac C is available without copy
protection for an extra $25 when you sign a further license agreement. In
either case a reasonable amount of telephone support is included. The policy
of other companies is likely to differ, so check what they charge for library
source and support.
By now you are probably bored with reading advice on how to buy a compiler so
I will present some entertainment in the form of further benchmark information.
As people insist on using the sieve, I will go along with the crowd. However,
we should agree on a common version of the code to insure that results are
comparable. I have therefore included (below) source that I propose we use for
future comparisons of C compilers for the Mac. Changes from the original are
timing based on the Mac's tick counter, pauses before and after the run, and
provision for register variables and different integer sizes.
Results on my Mac using a version of Mac C that supports register variables
(semantically as well as syntactically) are as follows:
*** WARNING THIS VERSION IS NOT AND NEVER WILL BE RELEASED ***
*** IT IS AN INTERIM VERSION OF THE RELEASE SCHEDULED FOR ***
*** APRIL 15. CONSULAIR'S CURRENT DEVELOPMENT VERSION IS ***
*** FASTER. ***
32 bit integers 16 bit integers
vanilla 7.75 6.2
regvars 4.73 3.92
regvars + regpointer 4.45 3.67
One interesting discovery that I made while generating these numbers was that
Bill Duval's Hyperdrive equipped Mac is a lot faster than a standard Mac. When
I was picking up the compiler used to generate the results above I saw him run
the sieve on his machine. When I ran the same program on my machine it was
about 25% slower. He has verified the speed difference on his machines. It
appears that there are more reasons to buy a Hyperdrive than just the I/O
improvements. More on this subject when GCC provides additional information.
Standard disclaimer. I have no connection with Consulair other than as a
satisfied customer (Who paid full price for his compiler).
#include <stdio.h>
#define regvars
#define regpointer
#define int16bit
#define SIZE 8190
#define FALSE 0
#define TRUE 1
#define NTIMES 10
#define tickCount *((long *)0x16A)
#ifdef int16bit
#Options -I
/*Mac C way to specify 16 bit ints. Replace as needed for other compilers*/
#endif /* int16bit */
char flags[SIZE+1];
main() /* compute primes using sieve of Eratosthenes */
{
long ticks; /* added to allow timing */
#ifdef regvars
register int i, prime, k, count, iter;
#ifdef regpointer
register char *flag;
flag = flags;
#else /* regpointer */
#define flag flags
#endif /* regpointer */
#else /* regvars */
#define flag flags
int i, prime, k, count, iter;
#endif /* regvars */
getchar(); /* added to allow disk activity to cease */
printf("10 iterations: ");
ticks = -tickCount;
for(iter = 1; iter <= NTIMES; iter++) {
count = 0;
for(i=0; i<=SIZE; i++)
flag[i]=TRUE;
for(i=0; i<=SIZE; i++) {
if(flag[i]) {
prime = i + i + 3;
/*printf("\n%d",prime);*/
for(k = i + prime; k <= SIZE; k += prime)
flag[k]=FALSE; /* discard multiples */
count++;
}
}
}
ticks += tickCount;
printf("%d primes in %d.%d Seconds", count, ticks/60,
(((ticks % 60) * 100)+30)/60);
getchar(); /* added to allow output to be read */
exit(0);
}
For those without the original posting at hand, Gordon's results are presented
below. He used default int size, and no register variables. I do not have
the actual source used.
>
> Vendor Memory Disk Compile Link Load+Run Run Size Small
> ===========================================================================
> Megamax 512K Floppy 7.9 71 13.0 6.6 5484 5276
> Megamax 512K Corvus Omni 2.7 23 9.2 6.6 5484 5276
>
> Softworks 128K Floppy 77.4 111.9 15.8 8.8 34560 26368
>
> Consulair 512K HyperDrive 19.2 14.5 15.2 8.0 10496 10240
>
> Hippo 2 512K Ram Disk 11.6 8.8 14.1 --- 13864 ---
--
Stephen J. Langdon ...!{ihnp4,hplabs,sun,nsc}!amdahl!sjl
[ The article above is not an official statement from any organization
in the known universe. ]