forbus@uiucdcsp.CS.UIUC.EDU (10/23/85)
Your best source for raw performance data is "Performance and Evaluation of Lisp Systems" by Richard P. Gabriel, $25 from MIT press This is the book form of Gabriel's DARPA-funded study. One of the amazing regularities is that performance of specialized lisp machines is vastly better than any mainframe run as a STAND-ALONE machine, much less time-shared. The only exception is running a Cray stand-alone; no measurements are available for a time-shared Cray. The second aspect concerns programming environments, and that's much harder to quantify. The basic argument against Unix tools as being relevant to Lisp is that they are basically string-oriented. For operating-system type things, for supporting C programs and the like they certainly are important. But they provide no interesting help for writing, editing, debugging, and maintaining large Lisp programs. The best source of information on environments is to call up the sales reps of the various lisp machine companies (Symbolics, TI, Xerox, LMI) and ask them for some technical specs. Obviously they are more than happy to provide such information, and have people who make their living discussing these issues.
shebs@utah-cs.UUCP (Stanley Shebs) (10/31/85)
In article <3500007@uiucdcsp> forbus@uiucdcsp.CS.UIUC.EDU writes: >Your best source for raw performance data is > "Performance and Evaluation of Lisp Systems" > by Richard P. Gabriel, $25 from MIT press > >This is the book form of Gabriel's DARPA-funded study. One of the amazing >regularities is that performance of specialized lisp machines is vastly >better than any mainframe run as a STAND-ALONE machine, much less >time-shared. The only exception is running a Cray stand-alone; no >measurements are available for a time-shared Cray. Are we looking at the same study? The numbers on my crummy old xerox copies of the results don't show that at all. Admittedly they are incomplete compared to the book (which I've not yet purchased a copy of), but I don't believe the numbers have changed any. On the average, the 3600 does the best of any lisp machine. The 3600 gets 12.0 secs for the boyer benchmark, while PSL on a DEC-20 (a mainframe presumably) gets 11.96 secs. Not "vast" in my book. PSL on a 68K (HP200) gets 25.52 secs, which is half as fast as the 3600. I suppose that might be considered "vast", but then the 68K *is* a microprocessor... For other benchmarks, the 3600 (and the other lisp machines) fare worse. Running "browse", the 3600 gets 32.1 secs against the HP200's 39.84; a 20 running PSL gets 23.81 secs. Running "puzzle", the 3600 gets 14.21 secs, and the HP200 zips along at 10.85 secs! That's the best result; the worst puts the 68K PSL about 6 times slower. The mainframes range all over from substantially better to substantially worse. My reading overall is that the 3600 is on the average comparable to a VAX 780 running PSL, and faster than a 68K Unix box, but not impressively so. The true moral of the study is that no Lisp can be judged "best" or even "fastest" on the basis of benchmarks; each implementation has its strengths and weaknesses. In fact, the proper audience of this book is the implementors themselves; we now have a basis for deciding what aspects of our Lisps need work, and which implementations we ought to look at to improve those aspects. stan shebs
forbus@uiucdcsp.CS.UIUC.EDU (11/03/85)
Ahem. How many people who buy vax 780's give them to a single user? Worse still, many people who buy Suns seem to run them with multiple users, and often swapping over a network to a shared disk. The latter is a crazy idea for AI programs; while the ethernet may or may not be a bottleneck, sharing a disk between several processors certainly is! And of course, none of the Gabriel benchmarks really test paging performance, which is where non-trivial AI programs spend most of their time. Some of my students have tried to bring up programs written for Symbolics machines on Suns, and vice-versa. Factors of 15 performance improvement are not uncommon. The Suns involved were NOT Sun-3's, and the dialect was Franz, so these figures shouldn't surprise anyone. I'm quite curious to see what will happen with Lucid's CommonLisp on Sun-3's when we can try serious-sized programs. Which, thankfully, with CommonLisp we can do with a minimum of pain. Remember, however, the original question refered to lisp machines versus standard time-sharing environments. If a stand-alone 20 with PSL performs in the neigborhood of a Symbolics, then how will it do with 20-60 users? Answer: very badly! I think it is safe to say that there is NO computer on the market which runs Common Lisp (other lisps are simply not contenders at this stage of the game) which will provide for several users at once the same performance they can get if they are sitting at stand-alone workstations (be they Symbolics, Sun, TI, or Xerox).
tgd@orstcs.UUCP (tgd) (11/06/85)
The discussion thus far has centered on the raw speed of lisp machines versus time-shared machines. It was my impression that the original question referred more to the programming environment. Lisp machines generally have a much better integrated programming environment than main frames. I don't have any hard evidence concerning the relative productivity of programmers on mainframes and in dedicated programming environments, but I can supply some anectodal evidence comparing the Interlisp-D environment and the unix/franz/emacs environment on vaxen. Interlisp-D is far superior to franz. The display-based structure editor is extremely powerful, the system tracks all changes to files and permits you to save "in-core" changes back out into source files. It is smart enough to recompile only those functions that have changed. It supports the construction and incremental maintenance of masterscope databases (a wonderful interactive cross-refernce facility). It provides a large set of tools for implementing graphical browsers and displays to user datatypes. It includes a nice WYSIWIG text editor and drawing tools. I estimate that I am twice as effective at solving simple problems In Interlisp-D as in franz. I would never even attempt constructing a large system in franz. There isn't even a simple way (to my knowledge) in frnz to find out what the arguments to a function are, let alone finding a list of the currently defined functions. Lisp machines provide these features of course (in Interlisp-D, the ?= command does this). I would be very interest6ed to see some objective studies of programmer behavior comparing systems such as these. I would be particularly interested in comparisons between the Interlisp-D and SYmbolics programming environments. There have been many gripes on the net about the difficulty of learning "lisp machines", but I suspect that these apply mostly to Symbolics/LMI/TI machines rather than to Interlisp-D. --Tom Dietterich