mis@Seiden.com (Mark Seiden) (04/19/91)
does anyone have already an ftp-able (uucp-able) implementation of tpc-b, or some other sql benchmark, preferably for sybase? my intentions are to start with tpc, add domain-specific transactions, but start with a substrate that times TPS and cranks up and down number of simultaneous users... i have so far been unsuccessful in persuading sun or sybase to part with their tpc-b implementation, though they seem very happy to quote numbers in press releases and semi-technical quasi-marketing literature. pity it isn't currently possible to replicate the benchmarks on other configurations than they ran it on. (or am i misstating the case?) mark seiden, mis@seiden.com, 1-(203) 329 2722 (voice), 1-(203) 322 1566 (fax)
tim@ohday.sybase.com (Tim Wood) (04/21/91)
In article <3632@dagobah.UUCP> mis@Seiden.com (Mark Seiden) writes: > >i have so far been unsuccessful in persuading sun or sybase to part with >their tpc-b implementation, though they seem very happy to quote numbers >in press releases and semi-technical quasi-marketing literature. > >pity it isn't currently possible to replicate the benchmarks on other >configurations than they ran it on. (or am i misstating the case?) A couple of inaccuracies: the TPC-B is a documented benchmark. Anyone is free to create the schema, emulate the workload on the desired configuration and measure the results. The whole idea of a fixed configuration is to control that variable in order to make a meaningful comparison between products for a standard workload. The technology we use to emulate the workload is probably proprietary; after all, it saves labor to know how to model accurately x00 terminals firing transactions at a database. How would it be in a vendor's interest to tell everyone (ie competitors) how that's done? Moreover, if a vendor publishes an implementation, it incurs a support obligation. That can put it in the position of supporting the benchmarking efforts of those with contrary interests, who may not exercise complete impartiality before making performance claims about the product. It is then very difficult for the vendor to evaluate the measurer's methods. (Mini-disclaimer: this statement is completely hypothetical, it does not apply to any particular vendor or individual.) The inflexibility you perceive is greater elsewhere in the industry; there are companies that prevent, as a condition of the software licence, those who run their own benchmark implementations from disclosing their measurements. There are two kinds of useful benchmark: 1) audited ones of a standard workload on a constant configuration, used to comparison-shop among vendors and produce a short list; 2) the prospective customer's workload, or a model of it. (2) is the more important, because it predicts better how the DBMS will perform on one's application; this is the only important question, ultimately. -TW --- Sybase, Inc. / 6475 Christie Ave. / Emeryville, CA / 94608 415-596-3500 WORK:tim@sybase.com {pacbell,pyramid,sun,{uunet,ucbvax}!mtxinu}!sybase!tim PLAY:axolotl!tim@toad.com {sun,uunet}!hoptoad!axolotl!tim Dis claim er dat claim, what's da difference? I'm da one doin da talkin' hea.
gupta@cai.com (04/23/91)
In article <3632@dagobah.UUCP>, mis@Seiden.com (Mark Seiden) writes: > [...] > pity it isn't currently possible to replicate the benchmarks on other > configurations than they ran it on. (or am i misstating the case?) > > mark seiden, mis@seiden.com, 1-(203) 329 2722 (voice), 1-(203) 322 1566 (fax) Mark, TPC benchmarks (TPC-A and TPC-B) require a full disclosure, including sample source code of the driver, etc. The benchmarks themselves are specified in detail. To get a copy of the benchmark definitions, contact Shanley Public Relations 777 N. First Street, Suite 600 San Jose, CA 95112-6311 USA Ph: (408) 295-8894 FAX (408) 295-2613 Email: shanley@cup.portal.com All vendors are required to publish full disclosure reports as part of the agreement that allows them to claim TPC-A or TPC-B numbers. You should be able to get a copy of the same from Sybase. Yogesh Gupta Computer Associates
dhepner@hpcuhc.cup.hp.com (Dan Hepner) (04/23/91)
From: mis@Seiden.com (Mark Seiden) >i have so far been unsuccessful in persuading sun or sybase to part with >their tpc-b implementation, though they seem very happy to quote numbers >in press releases and semi-technical quasi-marketing literature. > >pity it isn't currently possible to replicate the benchmarks on other >configurations than they ran it on. (or am i misstating the case?) Released TPC-B numbers are required to be accompanied by a "full disclosure statement". All of the ones I've seen have included a source listing. The TPC standard itself contains a sample 1/2 page of SQL statements which implements: /* ** DoOne - Exceutes a single TPC BM transaction */ void DoOne() { .... It should be noted that the code is an example of what needs to be done, not the benchmark itself. You can expect however that any vendor who has released such numbers has probably brought more resources to bear on tuning the test than you are likely to, and it may be difficult to actually reproduce the number without the assistance of the vendor. Dan Hepner
dhepner@hpcuhc.cup.hp.com (Dan Hepner) (04/24/91)
From: tim@ohday.sybase.com (Tim Wood) >A couple of inaccuracies: the TPC-B is a documented benchmark. Anyone >is free to create the schema, emulate the workload on the desired >configuration and measure the results. > [...] >The technology we use to emulate the workload is probably proprietary; >after all, it saves labor to know how to model accurately x00 terminals >firing transactions at a database. TPC-B does not require emulating terminals. How would it be in a vendor's interest >to tell everyone (ie competitors) how that's done? TPC-A, on the other hand, does. It also requires disclosure as to how this was done, so as to allow both competitors and customers to judge whether or not the emulation was realistic. Moreover, if a vendor >publishes an implementation, it incurs a support obligation. That can >put it in the position of supporting the benchmarking efforts of those >with contrary interests, who may not exercise complete impartiality >before making performance claims about the product. TPC specifically bars publishing results without the cooperation of all relevant vendors involved. It was the direct intent of TPC however to require that such results be reproducible (again, by either customers or competitors) based on reading the full disclosure, although in reality this goal may have proven elusive, in part due to the acceptability of software not (yet) generally available. Dan Hepner