FELLOWS@UNB.CA ("David M. Fellows") (07/04/90)
I have just finished an evaluation of the TopSpeed and
QuickMod compilers as tools for use in an introductory
programming course. My recommendation is to use QuickMod. We
shall see next fall how it performs under load.
The information obtained may be of interest to this list. A
brief summary is given here, an electronic copy of the complete
report can be requested by email to
fellows@unb.ca
and giving a return email address. Ask for report TR90-01 "An
evaluation of PC-based Modula-2 Compilers".
A printed copy of the report can be obtained by writing
(regular mail) to me at the address below.
EXCERPTS FROM THE REPORT:
Fast compilation, good error diagnostics and robustness,
and convenient editing were the prime factors considered in this
evaluation. Debugger performance was played down because I think
that students learn more by doing their own diagnostic output.
The products evaluated were Stony Brook QuickMod and
TopSpeed Modula 2 (versions 1.14 and 2.0). The shareware
product, FST Modula, was given early consideration but was
rejected because run-time errors produce only a hex dump of
the machine state.
QuickMod
The minimum hardware requirements are 512 K RAM and 2-360 K
floppy diskdrives. 640 K RAM is required to work with programs
with many modules or large arrays. 640 K is also needed to use
the debugger from within the environment. 720 K floppy disks are
likely to be much more "comfortable" than 360 K disks. QuickMod
is usable from 2-360 K floppies or 1-720 K floppy. Individual
users will be "happier" with a hard disk. DOS 3.0 is apparently
the earliest version of DOS that the system reliably works with.
The QuickMod runtime library includes a fairly close
implementation of the library described in Wirth's report. It
also includes a number of modules specific to MS-DOS and the IBM-
PC. These were not tested, but appear to provide useful
functions.
TopSpeed Modula-2. Versions 1 and 2.
Version 1 is now available only to the education market.
Version 2 is the current commercial product. Version 2 has more
glitz an improved interface to other DOS programming tools and is
a lot bigger, slower and more expensive than version 1.
TopSpeed is an optimizing compiler. The design criteria
were evidently fast execution and compatibility with other DOS
programming tools generally. It has an air of "being all things
to all people". Version 1's stated minimum hardware requirements
are 2-360K floppy diskettes. This configuration was not tested
since my other tests indicate that a hard disk would be required
for satisfactory operation on an individual machine. 640K RAM is
required in practice. DOS 3.0 or later is required. Version 2
requires a hard disk.
The TopSpeed runtime library is quite different from the
"standard" one. This produced some problems in running textbook
problems "off the shelf". An implementation of Wirth's InOut
module (and others) is provided in source form.
3. Evaluation
Evaluation was done in two phases. The first was an
informal evaluation of the usability of the system. This was
done by entering, debugging and running a total of 4 programs and
noting problems, features, etc. The second was a performance
test of the systems installed on a JANET network.
3.1 Usability
Typical student-sized programs were liberally seeded with errors:
typing, deliberate syntax, undeliberate logic. The utility of
the systems in diagnosing them was assessed. TopSpeed Modula-2,
Version 2 was not given a full test, but it appears to be
essentially the same as Version 1 for our purposes.
TopSpeed generally has very good syntax error messages.
QuickMod has even better ones. Both systems place the user back
in the editor with the cursor positioned at the first error at
the end of a compile which fails. Subsequent errors are located
by pressing the PF8 key. TopSpeed allows you to skip backwards
as well as forwards, QuickMod does not.
Both systems use "Wordstar-like" editors. I don't
particularly like either, but both are adequate for the job. The
TopSpeed editor has more flexibility with respect to windows - up
to 4, arbitrarily sized and placed. QuickMod will retain as many
files as will fit in RAM. The windows can be switched to display
any file in memory very quickly.
Both compilers have options for generating code to check
for subscripts out of range, use of nil pointer, etc. These are
specified using menu selection and, for TopSpeed, by using
"magic" comments. TopSpeed required a magic comment in each
module to turn on pointer checking. This was disaster prone.
My overall impression, gained while debugging the test
programs, was that QuickMod provided a more robust working
environment. I crashed the machine several times, requiring a
power-off re-boot, while using TopSpeed; never while using
QuickMod. I attribute this in part to the better pointer
checking in QuickMod.
My overall impression was that the TopSpeed menu and
editing interface is somewhat better than the QuickMod interface.
The difference is not large, there are good and bad points to
each.
3.2 Performance
The primary benchmark used for performance evaluation
was the following sequence of actions applied to a single
module program.
1. Load system and program (Timed)
2. Modify the program with a null-effect modification
(Not timed)
3. Run the program ("make", compile, link and execute)
(Timed to start of execution)
4. Run the program again (Timed to start of execution).
5. Edit the program to induce errors. ("VAR" changed
to "VAOR") (Not timed)
6. Run the resulting program. (Timed to placement of
cursor in file for error correction).
7. Exit system (Timed)
This benchmark was run on these configurations:
1. A true IBM PC-XT
2. A diskless IBM-PC (8088 processor) served by the
JANET E-cluster file server.
3. An IBM PS-2 Model 25 (8086 processor) served by the
JANET E-cluster file server.
For the tests of configurations 2 and 3 the network was otherwise
idle. The file server is an AT-class machine with 90 megabyte
disk drives.
Each compiler was setup to include the most protection,
diagnostic, and safety features. Optimization was inhibited for
the TopSpeed compilers.
The results of this test are summarized in Table 3.2.1
Table 3.2.1
Compilation Benchmarks
Load Run Run Compile Exit
System Again time System
error
____________________________________________________________
PC-XT
QuickMod 10 6.3 1.0 5.6 NM
TopSpeed V1.14 9.4 31 1.2 14 NM
8088/JANET
QuickMod 18 19 1.0 6.6 13.5
TopSpeed V1.14 18 82 2.5 8 8
TopSpeed V2 41 200 10 NM 14
8086/JANET
QuickMod 14.5 15 0.5 5.5 11
TopSpeed V1.14 13.5 73 4.0 46 6
TopSpeed V2 28 139 9 28 8
Notes: 1. All times in seconds
2. NM - Not Measured
A secondary benchmark was run to determine program execution
speeds. This consisted of timing the execution of a selection
sort procedure sorting from 100 to 500 random long integers. The
measured portion of this benchmark does no I/O.
The results are summarized in table 3.2.3. Some
measurements of WATFOR77 executing the same algorithm coded in
Fortran are also shown for comparison. In each case full
diagnostics, no optimization was specified.
Table 3.2.3
Execution Speed Benchmark Times (seconds)
Number of elements Top Speed QuickMod WATFOR77
sorted V1.14
________________________________________________________
100 0.7 0.9 2.5
200 1.8 2.4 8.5
250 2.9 3.8 NM
400 7.1 9.2 NM
500 10.8 14.2 NM
The TopSpeed compiler is much slower at compiling than
the QuickMod compiler. Most of the difference can be attributed
to disk activity. This disk activity will put a severe strain on
the JANET network and file server.
As Table 3.2.3 shows, the compiled code from TopSpeed is
about 30% faster than QuickMod. In our application this
difference is not nearly as significant as the compile times.
3.3 Summary
Table 3.3.1 attempts to quantify and summarize my
impressions of the two systems. Version 2 of Top Speed is not
considered because it is noticeably poorer in performance than
its cheaper predecessor. The scale is 0-10, high numbers are
better.
Table 3.3.1
Summary Comparison
Top Speed QuickMod
V1.14
______________________________________________________
editor 7 7
menus, selection sequences 8 6
libraries 6 9
error messages 8 10
robustness 5 8
debugger 8 7
"making" speed 3 8
hardware resources required 5 7
per student disk space 7 5
______________________________________________________
Total (90 max) 57 67
4.0 BUSINESS CONSIDERATIONS
The following information is paraphrased from the vendors
advertisments and promotional literature. It is obvoiusly
subject to change. License fees are to students or educational
institutions.
TopSpeed Modula-2 pricing is as follows (US$):
V1.17 (incl. debugger) $60 direct to students, incl.
shipping
Site license: $500 first 10
$30 each additional
100 stations $3200.00
V2.0 $130 direct to students including shipping
Site license: $995 first 10
$50 each additional
100 stations $5495.00
Stony Brook QuickMod - Pricing (US$):
$40 each on orders of 20 or more for students
Site license: $ 250 up to 10
$1000 up to 50
$2500 unlimited (on university's
machines)
(10 copies of documentation)
100 stations $2500.00
APPENDIX
Supplier Addresses
QuickMod:
Stony Brook Software
187 E. Wilbur Rd., Suite 9
Thousand Oaks, CA 91360
U.S.A.
(805) 496-5837
TopSpeed:
Jenson & Partners International
1101 San Antonio Road, Ste. 301
Mountain View, CA 94043
U.S.A.
USA 1-800-543-5205
Canada 1-800-543-8452
Europe (01) 253-4333
David M. Fellows
Professor,
Faculty of Computer Science
University of New Brunswick
P. O. Box 4400
Fredericton, N. B., Canada
E3B 5A3
Telephone: 506 453 4566