laser-lovers@uw-beaver (01/13/84)
From DRF@SU-SCORE.ARPA Fri Jan 13 00:05:37 1984 Please forward this to Laser-Lovers, if it seems reasonable. -david I know all benchmarks are wrong, but just to get a feel for things, I tried the following experiment: Create a file with 100 paragraphs, each with ten sentences that say "This is lots and lots of text." Run it on an unladen 20 with Scribe and then TeX (add "\bye" at the end for TeX). The results I get (using Control-T) are that TeX takes an average of 12.5 CPU seconds, while Scribe takes an average of 11.5 CPU seconds. Individual measurements varied almost up to 1 second. This seems to indicate that TeX and Scribe run at about the same speed. In Scribe's favor, it did make a Press file directly, whereas TeX needs an extra step to do that, though that program typically takes about a tenth of the total time of the TeX run. Also, no hyphenation was going on (they're all short words in the test), so the fact that Tex has a better, but slower, hyphenater didn't get measured. One interesting thing was that Scribe produced 6 and a fraction pages of output, while TeX produced 5 and a fraction. Does that mean that TeX runs `slower' than Scribe? Of course not; rather than `seconds per page' perhaps we should measure `microseconds per character'. Another unfair part of this experiment was that it doesn't measure the speed of any complicated formatting capabilities. While these costs are pretty standard from one Scribe user to the next, they vary more among TeX users because folks use different macros, the efficiency of which tend to vary greatly depending on the author. On the other other hand, note that TeX is written in Pascal, while Scribe (I am told) is written in Bliss. The Pascal compiler used on the 20 has many good features, but `quality, high-speed code' is not among them. The DEC 20 Bliss (I am told) served as a study in implementing optimizing compilers. Thus, we might expect to find different results on, say, Vax/VMS, where the Pascal compiler is quite good. Indeed, on a Vax 782 (yes, 782) running VMS, I timed a run of the same TeX and Scribe jobs (using SET CONTROL=T). TeX took 20 CPU seconds, while Scribe took 40. Scribe was now outputting for the Imagen, and this time it produced 8 and a fraction pages, so things aren't directly comparable. Finally, I moved the same input files to a Vax 780 running 4.? Berkeley Unix. This time Scribe chose to output in Press format again, but for some reason it produced 8+ pages. (TeX, of course, produced the identical 5+ page DVI file on each machine). According to the "time" command in /bin, TeX took "41.0 Real 30.9 User 2.1 Sys" while Scribe took "42.0 Real 35.8 User 2.3 Sys" time. Interesting. Disclaimers: Of course, I only did a few timings on each machine, and while it was late at night, there were other things going on that may have skewed the results. Benchmarks never measure what you want them to. I work for Prof. Knuth's TeX project. Plug: The TeXbook is now available at a bookstore near you. Only $14.95!! -david -------
laser-lovers@uw-beaver (laser-lovers) (01/13/84)
From DRF@SU-SCORE.ARPA Fri Jan 13 10:42:45 1984 Please forward this to Laser-Lovers, if it seems reasonable. -david I know all benchmarks are wrong, but just to get a feel for things, I tried the following experiment: Create a file with 100 paragraphs, each with ten sentences that say "This is lots and lots of text." Run it on an unladen 20 with Scribe and then TeX (add "\bye" at the end for TeX). The results I get (using Control-T) are that TeX takes an average of 12.5 CPU seconds, while Scribe takes an average of 11.5 CPU seconds. Individual measurements varied almost up to 1 second. This seems to indicate that TeX and Scribe run at about the same speed. In Scribe's favor, it did make a Press file directly, whereas TeX needs an extra step to do that, though that program typically takes about a tenth of the total time of the TeX run. Also, no hyphenation was going on (they're all short words in the test), so the fact that Tex has a better, but slower, hyphenater didn't get measured. One interesting thing was that Scribe produced 6 and a fraction pages of output, while TeX produced 5 and a fraction. Does that mean that TeX runs `slower' than Scribe? Of course not; rather than `seconds per page' perhaps we should measure `microseconds per character'. Another unfair part of this experiment was that it doesn't measure the speed of any complicated formatting capabilities. While these costs are pretty standard from one Scribe user to the next, they vary more among TeX users because folks use different macros, the efficiency of which tend to vary greatly depending on the author. On the other other hand, note that TeX is written in Pascal, while Scribe (I am told) is written in Bliss. The Pascal compiler used on the 20 has many good features, but `quality, high-speed code' is not among them. The DEC 20 Bliss (I am told) served as a study in implementing optimizing compilers. Thus, we might expect to find different results on, say, Vax/VMS, where the Pascal compiler is quite good. Indeed, on a Vax 782 (yes, 782) running VMS, I timed a run of the same TeX and Scribe jobs (using SET CONTROL=T). TeX took 20 CPU seconds, while Scribe took 40. Scribe was now outputting for the Imagen, and this time it produced 8 and a fraction pages, so things aren't directly comparable. Finally, I moved the same input files to a Vax 780 running 4.? Berkeley Unix. This time Scribe chose to output in Press format again, but for some reason it produced 8+ pages. (TeX, of course, produced the identical 5+ page DVI file on each machine). According to the "time" command in /bin, TeX took "41.0 Real 30.9 User 2.1 Sys" while Scribe took "42.0 Real 35.8 User 2.3 Sys" time. Interesting. Disclaimers: Of course, I only did a few timings on each machine, and while it was late at night, there were other things going on that may have skewed the results. Benchmarks never measure what you want them to. I work for Prof. Knuth's TeX project. Plug: The TeXbook is now available at a bookstore near you. Only $14.95!! -david -------
laser-lovers@uw-beaver (laser-lovers) (01/13/84)
From furuta@uw-june Fri Jan 13 11:09:11 1984 I found David Fuchs' timings to be of interest so I repeated some of them on a lightly loaded VAX 780 running Berkeley Unix, 4.1bsd. Once again, it appears that Scribe and TeX run at fairly comparable speeds for this simple benchmark (although, in both cases, it took more time than David's run on his Unix 780). Scribe, producing output for the GSI device took 1:12 real, 58.5 user, and 3.1 system Producing output for a file device took 0:47 real, 41.6 user, and 2.6 system Comparatively, TeX took 1:03 real, 48.6 user, and 2.2 system I then wondered where troff fit into this scheme. I modified the input file to put a ".br" between the paragraphs. Troff ran without any macro packages and without any of its preprocessors. Hence, I would suspect that the measurements would be highly biased towards troff. However, the values weren't that different. Running a troff -t with the output sent to /dev/null took 0:45 real, 35.4 user, and 1.4 system Running a nroff with the output going to a file took 0:29 real, 21.6 user, and 1.2 system One surprising difference, though, are the relative amounts of memory needed for the runs. Interestingly, TeX was smaller than Scribe although both were pretty big---Scribe averaged somewhere around 205k bytes of program memory space and somewhere between 172k and 220k bytes of data memory space. TeX took 137k of program memory space and 149k of data memory space. Troff was substantially smaller---25k program and between 23k and 25k data. TeX's performance is particularly surprising to me because it uses the Berkeley pc compiler *without* optimization although the i/o routines have been replaced with C-language routines, speeding them up substantially. I believe that the Unix Scribe is machine translated from the Bliss to C. Of course troff was written directly in C. Echoing David's comments; this benchmark is probably too simple to provide realistic results. I think it'd be quite interesting if someone tried running comparisons using some more realistic files. Any takers? --Rick