rcd@ico.ISC.COM (Dick Dunn) (08/30/89)
Cautious disclaimer: I'm speaking as an individual, not for ISC.
The August 21, 1989 issue of _UNIX_Today!_ has an article reviewing the
Intel (nee Bell Tech) MPE--a 386-based "workstation". The discussion
itself is OK, but the comparison to other machines is a mess. The MPE
with Bell Tech's UNIX is compared for I/O performance to other hardware/
software pairs which are in a different league. The selection is so badly
biased that the MPE could hardly help but win out. Now PLEASE NOTE that I
am NOT saying that the bias was deliberate. I am making NO attempt to
suggest that the author of the article played favorites. It may well be
that he just made a truly inept set of choices, or tested with whatever
old hardware and software he had at hand. It doesn't matter; the results
are bad and they don't present other vendors fairly. I'll illustrate the
problems by comparing the MPE to the box used to test 386/ix.
The Intel/Bell Tech machine is a 25 MHz 386 with 64K cache. 386/ix was
tested on a 16 MHz 386 "Inboard" (a 386 add-in board) which has no cache.
This probably represents about a 2:1 CPU speed difference.
The MPE had a large CDC ESDI drive--which is presumably 10 Mb/s, 1:1
interleave, and under 20 ms average seek. The 386/ix machine had a Maxtor
XT-1085 on the standard old WD controller--with 5 Mb/s, 2:1 interleave, and
27 ms seek. It's hard to give an accurate value for the cumulative effect
of these factors, but it's probably in the range of 2:1 to 3:1.
The MPE was tested with V.3.2; the 386/ix version used was 1.0.6, which is
a V.3.0 and was the release before ISC's Fast File System was incorporated.
Why, when ISC was first out of the gate with V.3.2, is _UNIX_Today!_ doing
a test with our V.3.0 and Bell Tech's V.3.2??? The 1.0.6 release is *last
year's* system, and it's particularly annoying that they used such an old
system for an I/O benchmark when the current system has a major I/O
improvement. (The Microport system used for comparison is 2.2, which is
also pretty old.)
Why is the MPE compared to old software and older, cheaper hardware?
What's the point in publishing such a lopsided comparison? I'd love to
hear anyone else's insights into this matter.
--
Dick Dunn rcd@ico.isc.com uucp: {ncar,nbires}!ico!rcd (303)449-2870
...Are you making this up as you go along?rcd@ico.ISC.COM (Dick Dunn) (08/30/89)
In article <16054@vail.ICO.ISC.COM>, rcd@ico.ISC.COM (that's me) complained about a _UNIX_Today!_ article. At one point I said: > The Intel/Bell Tech machine is a 25 MHz 386 with 64K cache. 386/ix was > tested on a 16 MHz 386 "Inboard" (a 386 add-in board) which has no cache. > This probably represents about a 2:1 CPU speed difference. I was wrong; the Inboard DOES have a cache and the CPU speed difference (based on other cached 386-25's) is more like 1.6 - 1.7. This is still significant, and doesn't affect the remainder of what I wrote since the more significant differences are in the disk subsystems. -- Dick Dunn rcd@ico.isc.com uucp: {ncar,nbires}!ico!rcd (303)449-2870 ...Are you making this up as you go along?
plocher%sally@Sun.COM (John Plocher) (08/31/89)
I too was astonished by what seemed to be a blatent ad for Bell Tech Unix (BTU) It seemed that the "reviewer" must have had old copies of ISC and Microport sitting around and used them for comparison with a brand new copy of Bell Tech. On one hand, if the goal of the review had been "PC Workstations" (as the title suggests), then he should have used BTU on ALL his hardware platforms. This would have given the reader a feeling about how hardware changes affect performance. On the other hand, looking at "Unix on PC Workstations", he should have installed and used the different versions of Unix on the same hardware. This would have given the reader a way to evaluate the products supplied by the different vendors. Of course, the reviewer should have gotten the latest versions of Unix from all the vendors - Microport 3.0e, ISC V/386 3.2, and Bell Tech Unix 3.2, as well as Everix ESIX/386 3.2C, SCO Unix 3.2, ATT 3.2, and Dell Computer's 3.2 version. On the third hand, if the thrust had been "Complete Workstation Packages" (Those coming complete with both Hardware and Unix Software), he should have compared some of the other 386 based "workstations" in this price range (many of the Apollo systems, the Sun 386i, Convergent, ATT WGS systems, and others). Instead of this, the reviewer used Sun3s (68020) and a couple of bigger Vaxen (VAX/785 & VAX/8600)! As Dick said, ISC 1.0.6 is a year old! Microport 2.2 is almost TWO years old! The different hardware platforms and the old versions combine to make a complete joke of the "review". As an overview of 3.2 Unix and the Bell Tech/Intel product line it is digestable, but to class this as a review goes a bit too far over the line. The real laugh came when reading the author's background: Bob Morein is a software author and is well-known for his versatile benchmark- ing tools. He is a frequent contribu- ter to computer publications. Morein can be reached at bob@utoday. Versatile Benchmarks? With a methodology like he used in this article? Yes, I see. His benchmarks can prove anything he wants them to! :-) -John Plocher In article <16054@vail.ICO.ISC.COM> rcd@ico.ISC.COM (Dick Dunn) writes: >Cautious disclaimer: I'm speaking as an individual, not for ISC. > >The August 21, 1989 issue of _UNIX_Today!_ has an article reviewing the >Intel (nee Bell Tech) MPE--a 386-based "workstation". The discussion >itself is OK, but the comparison to other machines is a mess.
jerry@xroads.UUCP (Jerry M. Denman) (08/31/89)
There are lies, there are damn lies; then there are benchmarks. (author unknown) Nuff said
karl@ddsw1.MCS.COM (Karl Denninger) (09/01/89)
>Response 2 of 2 (193) by plocher%sally at Sun.COM on Thu 31 Aug 89 01:39 >[John Plocher] >(53 lines) > >I too was astonished by what seemed to be a blatent ad for Bell Tech >Unix (BTU) It seemed that the "reviewer" must have had old copies of >ISC and Microport sitting around and used them for comparison with a >brand new copy of Bell Tech. .... >The real laugh came when reading the author's background: > > Bob Morein is a software author and is well-known for his > versatile benchmark- ing tools. He is a frequent contribu- ter > to computer publications. Morein can be reached at > bob@utoday. > >Versatile Benchmarks? With a methodology like he used in this >article? Yes, I see. His benchmarks can prove anything he wants them >to! :-) I too was disgusted with this so-called "Benchmark" or "review". It is at times like this when one wonders just how interested in objective reporting these trade magazines really are. The damage done by "reviews" like this is incalculable. Sure, you can tell those people who call you why the review is bogus, and that the writer is an idiot. But those who _don't_ call, simply because they took the advice in the article and bought what was "recommended", are lost forever. It does every vendor who was so maligned a great amount of damage. The question I have for Unix Today! is "how and why did you let this get by?" If this was not deliberate, and I have to assume it wasn't without some evidence to the contrary, then it simply smacks of not caring about the accuracy of the information reported. Other trade publications have done this kind of thing before. I expected better from Unix Today. I guess they're just another pulp magazine like the rest of them. So much for their credibility; I guess you can't trust them either. Oh well. I am forwarding this entire thread to bob@utoday. I suggest those of you who are similarly outraged to the same; vendors who were maligned should demand a retest and retraction, as well as a public apology from both the author and magazine. IMHO it's the least they owe you. -- Karl Denninger (karl@ddsw1.MCS.COM, <well-connected>!ddsw1!karl) Public Access Data Line: [+1 312 566-8911], Voice: [+1 312 566-8910] Macro Computer Solutions, Inc. "Quality Solutions at a Fair Price"
jhood@biar.UUCP (John Hood) (09/01/89)
In article <123997@sun.Eng.Sun.COM> plocher@sun.UUCP (John Plocher) writes: >I too was astonished by what seemed to be a blatent ad for Bell Tech >Unix (BTU) It seemed that the "reviewer" must have had old copies of >ISC and Microport sitting around and used them for comparison with a >brand new copy of Bell Tech. Seems to me that the numbers for the other Unixes come from a comparative review that I saw some while back in some other magazine. I suspect that rather than rerun the benchmarks, the author pulled some old numbers out of a dusty file. I'm not impressed either. No matter how you slice it, the numbers are useless for comparing Bell Tech's hardware or software against anything else. --jh (bias note: I'm not too impressed with either Bell Tech or UToday...) -- John Hood, Biar Games snail: 10 Spruce Lane, Ithaca NY 14850 domain: jhood@biar.uu.net bang: anywhere!uunet!biar!jhood Looking for a OS hacking programming job in the Ithaca area...
greenber@utoday.UUCP (Ross M. Greenberg) (09/02/89)
This is me in an official capacity for UNIX Today!. First, in the future, please make sure to advise me of an ongoing discussion regarding UNIX Today: our objective is to get better at what we do. Sometimes we do a damned fine job. Sometimes we don't. Based on what I'm reading here, I guess we blew the benchmarks on the MPE. Main reason: we're small. We don't have all the hardware and software we need in-house to get all the benchmarking we'd like. We're addressing that most rapidly with some stuff I can't go into until the deal is signed. Suffice it to say that I hereby promise you that screw-ups as you believe happened will never happen again. But...how did it happen, you might wonder? Well, those benchmarks *were* done honestly. Nothing phoney baloney about it. They were not done to make the MPE look good -- or bad. Those are the real numbers that came out. It is difficult to benchmark a new piece of hardware. Frankly, I don't hold much faith in the <generic>stone tests, prefering an application simulation myself. But, getting your hands on a piece of hardware and then figuring the best benchmarks to run, and what to run them against is a tough thing to do on a two week production cycle. Not a good enough excuse, I know. Now, we use a lot of freelancers. Bob Morien, the author of the MPE review, happens to be one of them. Freelancers work *directly* under my supervision. No review or techie stuff gets out into the paper without my personal approval. I am the one ultimately responsible for the errors in judgement in letting those particular benchmarks get printed. Due to some heavy deadline pressures, *I* didn't review them as thouroughly as I normally would have. I blew it. I screwed up. I should have gone back to Bob and asked him to rerun the benchmarks with more current and comparable stuff. Bob's actually a very good benchmarker. He simply didn;t have all the hardware and software on-hand at the time to re-run the benchmarks. He screwed up a little. I screwed up in a major way. If it's any consolation to you, I noticed the problem when we got back the proofs. It was too late at that point to do anything about it as the paper was to be printed about two hours later. We're working hard to make the paper better. We need your help. If we screw up, send the mail directly to me, or advise me immediately what newsgroup to join to take part in the discussion. Anyway...we'll be reviewing quite a bit of hardware in the very near future. You can be sure that the benchmarks for that hardware will show a comparison with the MPE benchmark. If it's a dog, the readers will know. If it's a shining light, they'll know, too. Benchmarks are always a problem spot. In two or three issues, you'll see what we're doing about it. I'll bet you all a drink at the next UseNix that you'll be more than satisfied. But you have to promise me that, once I drink all those drinks I win, somebody helps carry me back to my room, ok? Ross M. Greenberg UNIX TODAY! 594 Third Avenue New York New York 10016 Review Editor Voice:(212)-889-6431 BBS:(212)-889-6438 uunet!utoday!greenber BIX: greenber MCI: greenber CIS: 72461,3212 To subscribe, send mail to circ@utoday with "Subject: Request"
randy@chinet.chi.il.us (Randy Suess) (09/02/89)
In article <966@utoday.UUCP> greenber@utoday.UUCP (Ross M. Greenberg) writes: >This is me in an official capacity for UNIX Today!. >Based on what I'm reading here, I guess we blew the benchmarks on the >MPE. >I blew it. I screwed up. >I screwed up in a major way. >Ross M. Greenberg >UNIX TODAY! 594 Third Avenue New York New York 10016 >Review Editor Voice:(212)-889-6431 BBS:(212)-889-6438 I like this guy! Now, if I could only get a subscription.... -- Randy Suess randy@chinet.chi.il.us
larry@nstar.UUCP (Larrty Snyder) (09/03/89)
In article <9468@chinet.chi.il.us>, randy@chinet.chi.il.us (Randy Suess) writes:
Randy - I was one of your CBBS sysops many years ago back
when my CompuPro S100 machine (Z80) was state of the art -
anyhow - I am looking for a BBS to run under 386/ix - are
you aware of such a beast (for either free or a reasonable
amount) ?