oz@yunexus.yorku.ca (Ozan Yigit) (05/03/91)
jinx@zurich.ai.mit.edu writes: >The numbers don't tell the whole story. When analyzing benchmarks you >have to be very careful. [detailed analysis of a toy benchmark omitted] Bill's point is well taken, but it had been made at a grander scale by Richard Gabriel quite sometime ago, which brings us to the point: We seem to have very little by the way of benchmark code for Scheme implementations, some (most?) of Gabriel stuff is not quite up-to-date for latest n, R^nRS. The Repository does contain some code that can be used as benchmarks (see mixfib, conform[1], peval etc.) but these probably require some agreement, and some careful analysis a la' Gabriel (and a la' PI of Jinx). Now that we have an adequate standard for the language, maybe it is about time to come up with a standard-compliant benchmark suite. We have all learned that benchmarks results should be consumed with some care to avoid indigestion, but not having a consistent suite makes matters inedible. oz --- [Scheme versions of Gabriel benchmarks circa 1987/88 are placed in the repository. Also note prop.scm and proplist.scm] --- In seeking the unattainable, simplicity | Internet: oz@nexus.yorku.ca only gets in the way. -- Alan J. Perlis | Uucp: utai/utzoo!yunexus!oz
Robert Hieb <hieb@heston.cs.indiana.edu> (05/03/91)
oz@yunexus.yorku.ca (Ozan Yigit) writes: >Now that we have an adequate standard for the language, maybe it is about >time to come up with a standard-compliant benchmark suite. We have all >learned that benchmarks results should be consumed with some care to avoid >indigestion, but not having a consistent suite makes matters inedible. For whomever undertakes this task: Please do not just transliterate C or Lisp code into Scheme. Unfortunately, it is easy to do so, since Scheme has assignments, loops, etc. Instead a translation is needed, using the idioms of Scheme. For instance, in the pi benchmark code previously posted the treatment of the "q" parameter is unnatural in Scheme. Similar examples can be found in the Scheme versions of the Gabriel benchmarks. It would be nice to have Scheme benchmarks that used closures and continuations in a natural way.
oz@yunexus.yorku.ca (Ozan Yigit) (05/05/91)
hieb@heston.cs.indiana.edu (Robert Hieb) writes: >Please do not just transliterate C or Lisp code into Scheme. >Unfortunately, it is easy to do so, since Scheme has assignments, >loops, etc. I cannot think of any reason to want to transliterate C into scheme for benchmarking purposes, though there is much lisp code that admittedly look attractive for this purpose. >Instead a translation is needed, using the idioms of Scheme. I think I know what you mean. Last year I spent two days annotating the entire ops5 source listing (yes, on paper!) for scheme conversion, and the result turned out to be mostly descriptive of a re-write (using scheme idioms) than pseudo-mechanical transliteration. I suspect that most schemers would not feel good about any code resulting from simple transliterations. As OK says[1], Elegance is NOT optional. >It would be nice to have Scheme benchmarks that used closures and >continuations in a natural way. True. I expect that any agreed-upon set of benchmarks would have to be properly representative of the language, in its current standard and practice, with its idioms and all. oz --- [1] O'Keefe, Richard A. "The Craft of Prolog", MIT Press, Cambridge, Mass. 1990 --- A pencil is so far the most effective software | internet: oz@nexus.yorku.ca design tool I have found... -- Amanda Walker | uucp: utai/utzoo!yunexus!oz