apollo@ecf.toronto.edu (Vince Pugliese) (02/15/88)
i would first like to thank those who took the time to
respond to my request for information regarding prolog
implementations for apollo. aside from a considerable
number of public-domain prologs we were able to discover that
quintus has a version for the apollo as well.
while most of the members of our research group recognize that
often the working environment is more important than just blinding
speed it would still be nice to be able to compare different implementations
on a quantitative basis i.e. performance. it is for this reason
that i am asking, yet again, for some assistance from my fellow
netters if anyone knows of a standard benchmark programs for prolog.
if i'm not mistaken, lisp has the gabriel suite and i'm curious
to know if such a similar situation exists for prolog.
either e-mail or post though i believe posting would be of
greater benefit to the community.
vince pugliese
apollo@ecf.toronto.eduok@quintus.UUCP (Richard A. O'Keefe) (02/19/88)
In article <470@mv03.ecf.toronto.edu>, apollo@ecf.toronto.edu (Vince Pugliese) writes: > > While most of the members of our research group recognize that > often the working environment is more important than just blinding > speed it would still be nice to be able to compare different implementations > on a quantitative basis i.e. performance. It is for this reason > that I am asking, yet again, for some assistance from my fellow > netters if anyone knows of a standard benchmark programs for Prolog. > If i'm not mistaken, Lisp has the Gabriel suite and I'm curious > to know if such a similar situation exists for prolog. Evan Tick has translated the Gabriel suite into Prolog, to the extent that it made sense to do so. You should check AI Expert for (June 87??); the benchmarks used there are available. There are in fact a great many benchmark suites around for Prolog. There was a Prolog Benchmarking Workshop held at Aerospace Corp in LA a couple of weeks ago, the aim of which was to put together a **realistic** Prolog benchmark suite. The single biggest lesson we learned from it was that it was a lot harder than anyone expected. The attitude at Quintus is that anyone who wants to know what the performance of Quintus Prolog is is welcome to get an evaluation copy of Quintus Prolog and make his own measurements. There is currently no benchmark suite which will be as informative as running your own programs. An extremely important thing to bear in mind is that small benchmarks provide almost no information about the performance of a SmallTalk/Lisp/Prolog system on real programs. (In fact, some of the existing Prolog benchmark suites provide almost no information about anything.) Don't forget to test your real programs with real data, on the actual machines you intend to use (beware of the effect of physical memory size, of load on the file/paging server, of a simultaneously present editor, ...). I am in deadly earnest when I warn you that NO benchmark suite you can get at the moment is likely to provide you with reliable guidance about how YOUR programs will behave. Not least whether your programs will run at all! There is simply no substitute for measuring your own programs. An extremely important reason for using your own programs for such measurements is that the single most important thing about a Prolog system is that it should give you the right answers: if you run a program that the vendors have never seen you stand a much better chance of uncovering bugs... Especially with "free" systems, this is a real problem. But what should you do if converting your code to a different dialect would be a lot of work? Then that is a major factor in itself. One of the valuable things which will come out of the Aerospace workshop is an idea of what you have to do to adapt a realistic Prolog program to different systems. (This is the kind of thing the BSI group *should* have done, but instead they decided to invent their own language.) If you look at small benchmarks, you will seriously underestimate the difficulties. Despite the above, if you get an evaluation copy of Quintus Prolog, we do provide several of the existing benchmarks. We just don't endorse any of them as meaning much. If you want to know which horse is going to come first, you don't time them over a 1-foot course!