jain@manage.enet.dec.com (Raj Jain, DEC, 550 King st, Littleton, MA, 01460, 508-486-7642, Fax:486-5279) (05/05/91)
The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling By Raj Jain John Wiley & Sons, New York, ISBN 0471-50336-3, 720 pages, April 1991. This book emphasizes simple-to-use techniques that can be applied by computer system designers, managers, marketing professionals, buyers, analysts, and others who need to compare alternative systems (computers, processors, devices, networks, databases, operating systems, languages, algorithms, or applications) or manage such projects. The 720-page volume covers virtually every aspect of systems evaluation from performance specification, capacity planning, and monitoring systems in use, to summarizing measured data, designing experiments, simulating future designs, and modeling the effect of proposed changes. The book reveals how to avoid common mistakes, and shows how to protect oneself from misleading analyses presented by others. Given a set of performance data on two or more systems, it is sometimes possible to manipulate the data, benchmarks, analysis, metrics, or data presentation such that either system can be shown to outperform the other! More than 150 examples and case studies cover topics such as evaluation of microprocessors, RISC processors, remote procedure calls, the UNIX operating system, garbage collection algorithms, interconnection networks, local area networks, text-formatting programs, cache and scheduler design issues and more. It has been selected as the *main* selection by the NewBridge Library of Computer and Information Sciences (LCIS) bookclub. EXPERTS' OPINIONS... "At last, a welcome and needed text for computer professionals who require practical, ready-to-apply techniques for performance analysis. Highly recommended!" --- Professor Leonard Kleinrock, Univ of California at Los Angeles "The Art of Computer Systems Performance Analysis is an extraordinary book. It has a practical, problem-oriented style which appeals immediately to engineers engaged in real-world design and analysis." --- Vint Cerf, Chairman, ACM SIGCOMM "An entirely refreshing text which has just the right mixture of theory and real world practice. The book is ideal for both classroom instruction and self study." --- Professor Raymond L. Pickholtz, President, IEEE Communications Society "An extraordinarily comprehensive treatment of both theoretical and practical issues." --- Dr. Jeffrey P. Buzen, Internationally known performance expert "It is the most thorough book available to date." --- Professor Erol Gelenbe, Universite Rene Descartes, Paris "This is an unusual object, a textbook that one wants to sit down and peruse. The prose is clear and fluent, but more important, it is witty." --- Allison Mankin, in Simulation "The frequent use of case studies was effective and sometimes entertaining." --- Craig Partridge, Editor, Computer Communication Review "It is extremely comprehensive -- I am hard pressed to think of a performance evaluation technique that isn't discussed here. The emphasis is on practical techniques that could be used without a great deal of mathematical sophistication." ---Professor David Finkel, Performance Evaluation Review "I found that the pace of the material was very even, which makes it a particularly suitable text from which to teach." --- Professor Jon Crowcroft, Univ College London, in Computer Communications A LIST OF INTERESTING "DON'T MISS" TOPICS o How to show a better performance for your system without any changes (Beware of ratio games played by your competitors) (Performance games people play) [p 130-131, 146, 165-174] o Four rat holes to avoid in performance presentations (Four easy ways to stall anybody's performance presentation) [p 162] o Twenty six ways to stall your competitor's presentation (How to kill your competitors' performance presentations) [p 161-162] o Six mistakes to avoid in preparing charts for presentations [p 144-146] o Six tricks to watch out for in your competitors' presentation graphics [p 146-150] o Twenty two mistakes to avoid in your performance analyses [p 14-22] o Ten problems in computer systems capacity planning [p 125-127] o Twelve common mistakes in benchmarking computer systems [p 127-130] o Seven benchmarking games to watchout [p 130-131] o Thirty four ways to improve your programs' performance [p 114-116] o Twenty six guidelines for good graphics in your presentations [p 141-143] o Eleven mistakes to avoid in empirical modeling [p 266-269] o Six common mistakes in experimental design [p 278-279] o Eight mistakes that may lead to incorrect simulation results [p 394-395] o Seven reasons why most simulations fail [p 395-397] o Six guidelines for selecting seeds for random number generators [p 453-455] o Six myths about random-number generators [p 455-458] o Thirteen system behaviors that are difficult to analyze using queueing models [p 620-622] OTHER TOPICS DISCUSSED I AN OVERVIEW OF PERFORMANCE EVALUATION o When to measure, simulate, or model? o How to select the right performance criteria? o How to specify performance requirements? II MEASUREMENT TECHNIQUES AND TOOLS o Which benchmarks are commonly used in the industry? o How to design the right workload for your system? o How to monitor your distributed systems? o How to use accounting logs to determine the workload of your system? o How to plan and manage capacity required for your computer installation? III PROBABILITY THEORY AND STATISTICS o How to summarize measurements with a single number? o How to report variability? o How much confidence can you put on data with a large variability? o How many measurements do you need to compare two systems? o How to compare systems using multiple benchmarks? IV EXPERIMENTAL DESIGN AND ANALYSIS o How many experiments do you really need? o How to get the most information with the minimum number of experiments? o Is one system really better than another? Or is it the effect of very different benchmarks? o How to isolate measurement errors? o How to check if a model is adequate? Part V: SIMULATION o What language should you use for a simulation? o How to verify and validate a simulation model? o How long to run a simulation? o How to generate random numbers and how to select seeds? o What probability distributions should you use? Part VI: QUEUEING MODELS o How to use simple queueing models to quickly answer common questions? o How to obtain response time, queue lengths, and device utilizations? o How to obtain bounds, variance, and other statistics on system performance? o How to subdivide a large queueing network model and solve it? FOR INSTRUCTORS AND STUDENTS Designed to be the main textbook for a first course on performance evaluation, it provides a comprehensive treatment of all aspects of performance analysis. You no longer need separate books for measurement, statistics, experimental design, simulation, and queueing theory. If you teach a course on computer systems such as architecture, engineering, networks, or databases, you can use it as a supporting textbook to cover performance issues. The chapters are organized so that each one can be presented in 45 minutes, with time left over to discuss the exercises and solutions in a typical 55-minute class. This makes the text ideally suited for a one- or two-semester course. Practice exercises at the end of each chapter can be used for homework. Solutions to the exercises appear at the end of the book. TABLE OF CONTENTS The book consists of 36 chapters divided into six parts. The chapter titles are listed below. 1. AN OVERVIEW OF PERFORMANCE EVALUATION: Introduction. Common Mistakes and How to Avoid Them. Selection of Techniques and Metrics. 2. MEASUREMENT TECHNIQUES AND TOOLS: Types of Workloads. The Art of Workload Selection. Workload Characterization Techniques. Monitors. Program Execution Monitors and Accounting Logs. Capacity Planning and Benchmarking. The Art of Data Presentation. Ratio Games. 3. PROBABILITY THEORY AND STATISTICS: Summarizing Measured Data. Comparing Systems Using Sample Data. Simple Linear Regression Models. Other Regression Models. 4. EXPERIMENTAL DESIGN AND ANALYSIS: Introduction to Experimental Design. 2**k Factorial Designs. (2**k)r Factorial Designs with Replications. 2**(k-p) Fractional Factorial Designs. One-Factor Experiments. Two-Factor Full Factorial Design without Replications. Two-Factor Full Factorial Design with Replications. General Full Factorial Designs with k Factors. 5. SIMULATION: Introduction to Simulation. Analysis of Simulation Results. Random-Number Generation. Testing Random-Number Generators. Random-Variate Generation. Commonly Used Distributions. 6. QUEUEING MODELS: Introduction to Queueing Theory. Analysis of a Single Queue. Queueing Networks. Operational Laws. Mean-Value Analysis and Related Techniques. Convolution Algorithm. Hierarchical Decomposition of Large Queueing Networks. ABOUT THE AUTHOR With over sixteen years of experience in the field of computer systems performance, Raj Jain is a Senior Consulting Engineer at Digital Equipment Corporation. He received the Ph.D. degree in Computer Science from Harvard University and has taught courses on performance analysis at Massachusetts Institute of Technology. Author's Address: Raj Jain, 137 Dutton Road, Sudbury, MA 01776-2804, USA. Internet: Jain@Erlang.enet.DEC.Com BOOK REVIEWS o Simulation, Vol. 56, No. 1, January 1991, p. 60. o Computer Communications Review, Vol. 21, No. 1, January 1991, p. 13. o Performance Evaluation Review, Vol. 18, No. 3, November 1990, p. 21-22. o ConneXions, Vol. 5, No. 2, February 1991, p. 22. o Computer Communications, Vol. 14, No. 4, May 1991. o Computer Marketing Newsletter, Vol. XIV, No. 12, May 1991, p. 4. The book (ISBN 0471-50336-3, LCCN: QA76.9E94J32 1991, 720 pp., April 1991, $52.95, Hardcover) is published by John Wiley & Sons, 1 Wiley Drive, Somerset, NJ 08875, Phone: (908)-469-4400, (800)-225-5945 Ext 2499, Fax: (908)-302-2300. It is available NOW at most technical bookstores in the United States. It can also be obtained directly from the publisher.