neeraj@demon.siemens.com (Neeraj Bhatnagar) (03/16/89)
I have recently been using Medley 1.0 release of the
COMMON LISP environment from the ENVOS corporation. So far I have
been very disappointed by the performance of the system.
I tried the following function as a benchmark on ENVOS Common Lisp
running on Sun 3/60 with 8Mbyte of RAM and on the CORAL Common Lisp
(from the CORAL corpn. Cambridge, MA) on a MAC SE/30 with 4Mbyte of
RAM:
(defun counter()
(setq x nil) ;list of 0 length
(loop ;loop for ever
(setq x (cons nil x)) ;increase length by 1
(if (eq (mod (length x 100)) 0) (print (length x)))))
Here are the results:
Envos Coral
----- -----
200 1200
400 1800
600 2500
900 3300
1000 3600
1200 4400
1400 4800
1800 5900
2200 7100
2600 8700
3000 9400
3300 10300
4000 12400
4500 13900
5000 15300
5500 16600
(stopped)
Above numbers should be read as follows: when Envos was printing 200/400/600/../5500
CORAL was printing 1200/1800/2500/../16600. I started both executions simultaneously.
CORAL running on half the memory was thrice as fast as ENVOS. Both the MAC and the
SUN were dedicated to the LISP programs and did not run any other program at that time.
In the past I have seen that CORAL on MAC SE (older MAC) also outperforms
ENVOS on SUN 3/60.
I shall appreciate the reactions of the other users by Email or
newsgroup.
Neeraj
neeraj@demon.siemens.com
welch@giza.cis.ohio-state.edu (Arun Welch) (03/17/89)
In article <7237@siemens.UUCP>, neeraj@demon.siemens.com (Neeraj Bhatnagar) writes: > > I have recently been using Medley 1.0 release of the > COMMON LISP environment from the ENVOS corporation. So far I have > been very disappointed by the performance of the system. > Well, for starters, you really ought to use some real benchmarks. While lisp has changed some since they first came out, out, the Gabriels are still the standard for lisp systems. On another front, the performance on the sun can be affected by a lot of things, which dont affect the Mac. For example, How many other processes were going on on the Sun? Did it have a local disk, or was it swapping across the net? Was the emulator and the vmem file on the local disk, or off on the net somewhere? How fast was the disk, if local? There are a bunch of other reasons why you're comparing apples and oranges, like how much time was spend in GC by the Sun (does Coral have a GC?), the fact that Coral runs only in core, while the Sun was using a virtual memory, etc. Also, realise that the Envos product is not running 'native' on the sun, but rather running an emulator of the Xerox dmachine microcode. Yup, this is gonna result in a bit of a performance hit. Numbers I'be heard are that a Sun 3/60 with 8M is ~ the speed of a dandelion, but there are so many variables that the number is pretty meaningless. I can come up with the timings on a Dove, but not from your benchmark (not having an SE with Coral to time it against). The other major difference, which can't be measured by any benchmarks, is the difference in environments. Unfortunately, there's no way to quantise a Lisp programming environment other than the fact that programmer A can produce more code on environment B than on environment C. What those of us working on the Envos platform enjoy is the rich environment, providing lots of tools for writing, debuging, and so on. I'll also admit that I'm biased. I happen to *like* the Envos environment. ...arun ---------------------------------------------------------------------------- Arun Welch Lisp Systems Programmer, Lab for AI Research, Ohio State University welch@tut.cis.ohio-state.edu
fischer@arisia.Xerox.COM (Ronald A. Fischer) (03/17/89)
You were comparing the *performance* of one lisp function run in both images, and then comparing the total size of each image. This doesn't seem quite the right way to do it. How about also looking at the size of the compiled function itself in both Lisps? As well as the memory usage. Speed versus space. Let's take a look at the bigger picture. ENVOS Lisp has taken a commercially difficult speed/space tradeoff. Its always easier to say you're faster than the competition on a few tightly constrained benchmarks, but what has the cost of this been? Ultimately it has caused the Lisp vendors to emphasize speed over all else. The result has been incredibly memory hungry implementations. Of course, hardware vendors love these sort of tendencies. By using a compiler that generates bytecodes at the bottm layer of the system ENVOS has a highly portable, but rather slow Lisp. Code size for any function is small, and remember that one of Lisp's major failings is its space use. ENVOS Lisp also uses CDR coding for lists, another speed slowdown that shrinks the size of the resulting memory image.r For some of the highly optimized native Lisps the code and data memory use is enormous. This affects swapping. For a large lexical analysis application that runs at Xerox PARC an experiment was performed: run the application in ENVOS Lisp and "another popular brand." The other Lisp compiled native machine code for the Sun 4. ENVOS Lisp was initially slower, but as the run continued for the three days that it normally takes to complete, the other lisp swapped itself to death (it crashed) and the ENVOS Lisp image ran to completion. As regards the size of the ENVOS Lisp image, again, bear in mind what's really in there. Most lisps are really just a runtime, an interpreter, a compiler, and a few trivial debugging tools. Lisp's vaunted prototyping ability has been more based on the language's traditional inclusion of an interpreter and its dynamic typing than any particular set of tools that are always provided. A few of the other lisps may have been around long enough to get an editor interface and maybe some good hooks into the operating system's window toolkit. ENVOS Lisp includes in its basic image such things as a window based structure editor that dynamically prettyprints code depending on window size, various proramming utilities including a system for tracking residential definitions, a window system, an operating system, a complete toolkit for building applications including a very clever grapher, remote debugging facility a la what the C people currently call a debugger, a *real* debugger with window interface, etc. There's alot of stuff in there which drives the size of the image up, but not because the compiled functions are enormous, rather because there's alot of functionality in there. You may also optionally load in our document editor (TEdit), Bitmap editor, structured graphics editor (Sketch) or applications like NoteCards or Rooms. ALL of these have programmer's interfaces as well so that they can be used to decreases application develpment time. ALL of these are also integrated tightly with one another using "imageobjects" a concept similar to what HP is now selling as "New Wave" and which Apple and Microsoft may someday get into their systems. All of this stuff originally came out of the clever folks at Xerox PARC, and its an amazing and very useful set of tools. I've also not touched on the basic problem that many highly optimized Lisps will now produce "unsafe" code, i.e. it will crash your Lisp image or worse operating system if simple errors occur. This has traditionally been another major Lisp advantage, one whose importance has been de-emphasized in the race for improved speed. ENVOS Lisp always produces "safe" code. Most other Lisps can too with appropriate compiler settings, but compare the resulting code size and speed under these settings too. Many checks are built into the ENVOS bytecode that will need to be explicitly coded otherwise. So, please be careful in how you benchmark. Consider size of the resulting function as well as its speed. Consider how "safe" the highly optimized code is. Consider the amont of memory needed to store your data. Consider how much shorter your development time might be with a better set of development tools or application toolkit. And finally, as a fellow Lisp developer, I wish you the best of luck in this volatile and small marketplace. (ron) Ron Fischer ENVOS Corp. Project Manager PS- These views are my own but are likely to be shared by others, I hope.
welty@steinmetz.ge.com (richard welty) (03/18/89)
In article <654@arisia.Xerox.COM>, Ronald A. Fischer writes:
*For a large lexical analysis application that runs at Xerox PARC an
*experiment was performed: run the application in ENVOS Lisp and
*"another popular brand." The other Lisp compiled native machine code
*for the Sun 4. ENVOS Lisp was initially slower, but as the run
*continued for the three days that it normally takes to complete, the
*other lisp swapped itself to death (it crashed) and the ENVOS Lisp
*image ran to completion.
while i have no special knowledge of the ENVOS System or of its
Xerox predecessors, i can confirm that oftimes benchmark-oriented
Common Lisps for stock hardware give the appearance of speed
on small jobs, and loose totally on large ones. We've been doing
some preliminary experiments with our model matcher, and have
discovered that while Lucid Common Lisp on the Sun appears to
be distinctly faster than our Symbolics 3600s at the start of
a job, when the jobs are very large Lucid runs into a brick
garbage collection wall, while in the specialized Symbolics
environment the job reaches completion without any special
fuss or bother (we are going to try and set up some carefully
controlled tests on price-comparable symbolics and sun systems
in the near future, following the philosophy that the best
benchmark is the program you are actually using on the data
you are actually using.) from Ron's description, i presume
that the situation is very similar for ENVOS.
as regards Ron's comments about environments, i agree totally.
we've found the tools provided with Lucid Common Lisp to be
inadequate for our purposes -- the lisp may be fast for small
jobs, but speed in execution isn't always an acceptable
substitute for quality tools. i'd much rather have a real
inspector, a good window debugger, and a tightly integrated editor
than 50% more execution speed, to tell you the truth. i get
more work done that way.
*PS- These views are my own but are likely to be shared by others, I
*hope.
by at least one other, i think.
richard
--
richard welty 518-387-6346, GE R&D, K1-5C39, Niskayuna, New York
welty@steinmetz.ge.com welty@crd.ge.com uunet!steinmetz!welty
``It is not GE Company Policy that P = NP'' -- Bob Mattheysesplogan@mntgfx.mentor.com (Patrick Logan) (03/18/89)
In article <7237@siemens.UUCP> neeraj@demon.siemens.com (Neeraj Bhatnagar) writes:
=> Here are the results:
=>
=> Envos Coral
=> ----- -----
=>
=> 200 1200
=> 400 1800
=> 600 2500
=> 900 3300
=> 1000 3600
=> 1200 4400
=> 1400 4800
=> 1800 5900
=> 2200 7100
=> 2600 8700
=> 3000 9400
=> 3300 10300
=> 4000 12400
=> 4500 13900
=> 5000 15300
=> 5500 16600
=>
=> I shall appreciate the reactions of the other users by Email or
=> newsgroup.
6.0 4.5 4.1666665 3.6666667 3.6 3.6666667 3.4285715 3.2777777 3.2272727 3.3461537 3.1333334 3.121212 3.1 3.088889 3.06 3.0181818
Initial Reaction: Envos is gaining on Coral. Keep iterating.
--
Patrick Logan ...!{decwrl,sequent,tessi}!mntgfx!plogan
Mentor Graphics Corporation
Beaverton, Oregonthorn@godot.radonc.unc.edu (Jesse Thorn) (03/22/89)
In article <654@arisia.Xerox.COM> Ronald A. Fischer writes about considering Lisp development environments when comparing/benchmarking different Lisp systems. >Consider how much shorter your development time >might be with a better set of development tools or application >toolkit. While I have never worked with ENVOS I did spend 4 years doing product development in InterLisp on Xerox D machines - the environment sounds identical. I can not overemphasize how valuable the development environment was to the development effort. These tools included the structure editor DEDIT, the object inspector, the D window system, the debugger package, SPY, MASTERSCOPE, CLISP, DWIM, LOOPS, etc, etc. The environment really helped to reduce development time and it allowed people to do rapid prototyping e.g. make massive exploratory code changes quickly. ( Refer to "Power Tools for Programmers" (?) by Beau Sheils onetime president of Xerox AIS (?). This magazine article describes the benefits of a rich development environment i.e. the D machine. It has appeared in _Datamation_ and several books on programming environments and user interfaces. ) I have since done product development in VAXLisp under VMS and Ultrix, PSL under Ultrix, and Golden Common Lisp under MS-DOS. They produce very fast code but when I last used them they had just about NIL in the way of development tools - vi/edt/emacs and what was called a "debugger". When moving to these Lisps it seemed that the first thing developers who were familiar with D machines or Symbolics boxes did was to write some basic tools that were missing in these environments (extensions to the debugger, xref utilities, performance timing utilities, a file package manager, etc). Not what you wanted to do when faced with deadlines. Project development slowed considerably due to the effort put into developing and maintaining the support tools as well as the relative lack of a supportive development environment. The environment IS the Lisp at least for development. In the best of worlds you would do development on your D machine equivalent and then port the finished code to your "fast" production machine. >I've also not touched on the basic problem that many highly optimized >Lisps will now produce "unsafe" code, i.e. it will crash your Lisp >image or worse operating system if simple errors occur. This has... This makes debugging extremely difficult. I have worked with "fast" Lisps that have had this problem. At one time a popular MS-DOS Lisp had a compiler bug that would produce code that trashed memory and initiated a perpetual GC. It took weeks to find the cause, and weeks to convince the vendor that there was a problem and get a patch. This all happened a few months before the product was to be shipped! A nostalgia for my old 1108 and that wonderful software from Xerox Parc has prompted this message. I now code in C on Unix systems - surely the Goths have invaded the temple... P.S Anyone from Bachman or Apex out there? Drop me a line. Jesse "It could be worse - It could be DOS" Thorn North Carolina Memorial Hospital Chapel Hill, NC UUCP: ...!mcnc!godot!thorn, INTERNET: thorn@godot.radonc.unc.edu
roberts@studguppy.lanl.gov (Doug &) (03/22/89)
The comments made regarding the sparseness of non-lispm development environments could have come from any of those of us in my group here at Los Alamos who in the last couple of years have switched from Symbolics' to Suns and Lucid lisp. We _REALLY_ missed the frame-based debugger & inspector, as well as the powerfull ZMACS editor. While Lucid 3.0 is an improvement over 2.1 it still has a long way to go before the development environment will compare to a Symbolics. The editor has greatly improved in 3.0, but the inspector and debugger still reek of the early 70's :-(... The sad fact is that most current-day lisp programmers have never worked on a lispm, and so they don't know what they're missing. --Doug -- =============================================================== Douglas Roberts Los Alamos National Laboratory Box 1663, MS F-602 Los Alamos, New Mexico 87545 (505)667-4569 dzzr@lanl.gov ===============================================================
mikel@apple.com (mikel evins) (03/22/89)
In article <654@arisia.Xerox.COM> fischer@arisia.Xerox.COM (Ronald A. Fischer) writes: > Consider how much shorter your development time > might be with a better set of development tools or application > toolkit. I won't claim that Allegro CL on the Mac has the richness of a Symbolics or Xerox Lisp machine; I've had my hands on a Symbolics and on TI machines (though, regrettably, not a Xerox), and I know something of what I'm missing. I will say, though, that, for a personal computer, or microcomputer, Allegro is a very nice programming environment. Yes, it has gc, it has a nice windowing debugger with a view of the execution stack, and inspectors for all kinds of variables, including system variables not visible in most Mac programming environments; it is very well integrated with the Mac interface toolbox. In short, for a personal computer (as opposed to workstation) implementation, it's pretty nice, and real fast, and quite robust, so far as I've seen so far, though not yet a match for Lisp machines. Just thought I'd put in a good word for the underdog. (I hope the identity of my employer won't bias readers with regard to my integrity; I liked Allegro CL a long time before I had any idea Apple was going to buy Coral).
mkent@dewey.soe.berkeley.edu (Marty Kent) (03/22/89)
In article <1003@internal.Apple.COM> mikel@apple.com (mikel evins) writes: >missing. I will say, though, that, for a personal computer, or >microcomputer, Allegro is a very nice programming environment. Yes, it has >gc, it has a nice windowing debugger with a view of the execution stack, >and inspectors for all kinds of variables, including system variables not >visible in most Mac programming environments; it is very well integrated >with the Mac interface toolbox. First, I'm speaking from the perspective of a long-time Interlisp programmer (5 years or so). I've been programming with Allegro for about a year. I sure would like to see something like an end to all this BULLSHIT about what a wonderful programming environment Allegro provides. It has a half-baked set of buggy tools. So the debugger has -windows-! It has a lot less power and flexibility than the debugger from Franz Lisp circa 1981. The inspector doesn't allow you to set the values of variables, only to look at them, and it crashes the machine quite regularly with system divide-by-zero errors. The stepper bombs if you try to step into a CASE form. There's no profiler. There are no lexical analysis tools. The editor is a text editor, not a structure editor (of course some folks will argue in favor of this, but I've found structure editors superior for modifying (if not "typing in") lisp code). There's a reason I'm more than a little ticked off about all these lacks in the Allegro environment: about a year ago I spoke with the president of Coral and gave him a little pitch about developing some of these missing links and he told me "this is -LISP- Why should we pay you to develop these tools when the user community will develop them for free?" Who was it that used to say "The quality goes in before the name goes on"... Marty Kent Sixth Sense Research and Development 415/642 0288 415/548 9129 MKent@dewey.soe.berkeley.edu {uwvax, decvax, inhp4}!ucbvax!mkent%dewey.soe.berkeley.edu Kent's heuristic: Look for it first where you'd most like to find it.