rs@uunet.UU.NET (Rich Salz) (09/10/87)
Submitted-by: "Mark A. Ardis" <maa@sei.cmu.edu>
Posting-number: Volume 11, Issue 38
Archive-name: test.el/Part03
#! /bin/sh
# This is a shell archive, meaning:
# 1. Remove everything above the #! /bin/sh line.
# 2. Save the resulting text in a file.
# 3. Execute the file with /bin/sh (not csh) to create:
# MANIFEST
# test.texinfo
export PATH; PATH=/bin:/usr/bin:$PATH
echo shar: "extracting 'MANIFEST'" '(1149 characters)'
if test -f 'MANIFEST'
then
echo shar: "will not over-write existing file 'MANIFEST'"
else
sed 's/^X//' << \SHAR_EOF > 'MANIFEST'
XFile Archive Description
X---- ------- -----------
XMANIFEST 1 This file
Xbox.script 2 Example test script
Xexample.texinfo 2 Appendix part of user manual
Xhooks.el 2 Example hooks for customization
Xtest.el 2 Root of test package
Xtest.texinfo 1 Main part of user manual
Xtst-achieve.el 2 Achieve-state routines
Xtst-analyze.el 2 Analyze test results
Xtst-annotate.el 2 Annotate package with results
Xtst-capture.el 2 Capture-state routines
Xtst-display.el 3 Display test results
Xtst-equal.el 3 Test states for equality
Xtst-inequal.el 3 Modifications to notion of equality
Xtst-instrument.el 3 Instrument package for testing
Xtst-utilities.el 2 Miscellaneous utilities
Xuser.texinfo.el 2 Wrapper for hard-copy user manual
SHAR_EOF
if test 1149 -ne "`wc -c < 'MANIFEST'`"
then
echo shar: "error transmitting 'MANIFEST'" '(should have been 1149 characters)'
fi
fi
echo shar: "extracting 'test.texinfo'" '(60887 characters)'
if test -f 'test.texinfo'
then
echo shar: "will not over-write existing file 'test.texinfo'"
else
sed 's/^X//' << \SHAR_EOF > 'test.texinfo'
X@setfilename test
X@node top, introduction, , (dir)
X@ifinfo
XThere should be more nodes than listed below, but at least the chapters are
Xisolated.
X@menu
X* introduction:: Purpose of the package.
X* starting:: How to get started.
X* testing:: Testing a package.
X* instrumenting:: Instrumenting to measure testing effectiveness.
X* analyzing:: Analyzing the results of testing.
X* displaying:: Displaying the results of analysis.
X* running:: Running a testscript in batch.
X* epilogue:: Final words.
X* example:: An example package to test.
X@end menu
X@end ifinfo
X@node introduction, starting, , top
X
X@chapter Introduction
X
XThis document explains how to use the Test package to test GNU Emacs
Xpackages. It is an extension of Chapter 22 [Compiling and Testing
XPrograms] of the GNU Emacs Reference Manual.
X
X@section Purpose of the Package
X
XThis package provides tools to aid in developing regression tests for Emacs
Xpackages and to assess the efficacy of those tests. GNU Emacs packages,
Xwritten in a dialect of Lisp, provide extensions to the basic text editing
Xfeatures of Emacs. Most of these packages perform state-changing
Xoperations. For example, a package may create new buffers, modify text,
Xchange the locations of the cursor and various marks, write files or modify
Xanything else accessible to Emacs (which is just about everything in Unix).
XTraditional Lisp debugging strategies are not sufficient to adequately test
Xa package that performs extensive state-manipulation. The Test package
Xprovides a strategy that is.
X
X@section Test Approach
X
XThe Test system is designed to support structural (or ``white-box'')
Xtesting. (Another commonly used test approach is functional, or
X``black-box'' testing.) It automates the process of running a number of
Xtests cases designed to analyze ``coverage'' of the package (that is, find
Xparts of the code that are not exercised) and ``expression'' (that is, what
Xvalues are returned by each expression, and which expressions always
Xreturned the same values).
X
X@section Testing Method
X
XTest provides mechanisms to ease the creation and execution of sets of
Xtests. Some of the features are designed to simplify testing activity,
Xwhile others are designed to aid the analysis of testing.
X
XTest supports coverage testing. That is, it provides instrumentation
Xmechanisms designed to check for inadequacies of testing of the following
Xtwo forms:
X@itemize @bullet
X@item
XFailure to execute an expression.
X@item
XFailure to produce more than one value for an expression.
X@end itemize
XIn each case, the fault may be in the tests (i.e., not enough testing was
Xperformed), or the fault may be in the program (i.e., an ``unnecessarily
Xcomplex'' program has been written). (There is also the possibility that
Xno fault exists, such as the use of ``constant'' functions for readability
Xor style. In practice these cases are easy to identify and ignore.)
X
XYou should use the following strategy in creating tests with this
Xsystem:
X@enumerate
X@item
XCreate a simple set of tests that appear to be necessary to test the basic
Xfunctionality of the package.
X@item
XAdd additional tests to ensure that every expression is executed, and that
Xeach expression produces at least two values over the entire set of tests.
X@item
XRun the set of tests with instrumentation to check for inadequacies.
X@item
XIf there are reported inadequacies, then correct the code or improve the
Xtesting, and go back to the previous step.
X@end enumerate
XThere is no guarantee that a set of tests that passes these coverage checks
Xwill find all errors. But, such a set of tests has the property that every
Xexpression in the code has been exercised in a minimally significant way
X(i.e., every expression has produced at least two different values). In
Xpractice, a great number of errors can be detected with this method.
X
XThe importance of coverage testing increases as a package is modified.
XChanges to one part of a package may have unexpected effects on other
X(unconsidered) parts. Coverage testing ensures that all parts of a package
Xare tested, at least minimally. Coverage tests also serve as a
Xspecification of the functionality of a package.
X
XFigure 1 illustrates the overall flow of events using this test
Xapproach.
X
X@group
X@example
X +------------+ +--------------+
X | Instrument | | Generate |
X | o Control | | o Test Cases |
X | o Data | | o Script(s) |
X +------------+ +--------------+
X \ / ^
X \ / |
X \ / |
X \ / |
X \ / |
X \ / |
X +-------------+ |
X | Execute | |
X | the | |
X | Test(s) | |
X +-------------+ |
X | |
X | |
X +-------------+ |
X | Analyze, | |
X | Display | |
X | Results | |
X +-------------+ |
X | |
X | |
X ^ |
X / \ |
X / \ No |
X < Done? >-------------+
X \ /
X \ /
X \ /
X v
X | Yes
X |(Test Goals Achieved)
X V
X
X
X
X@end example
X@center Figure 1. General Test Algorithm
X@end group
X
X@section Overview
X
XThe Test product consists of four major subsystems:
X@itemize @bullet
X@item
XRegression Subsystem -- a set of functions to support regression testing of
XEmacs packages.
X@item
XCoverage Subsystem -- a set of functions to support coverage testing of
XEmacs packages.
X@item
XAnnotation Subsystem -- a set of functions to provide support for creating,
Xusing and displaying information about Emacs Lisp code.
X@item
XEquality Subsystem -- a set of functions used by the testing
Xsubsystems that provide different interpretations of equality.
X@end itemize
X
XThe following sections explain the use of these functions to support
Xthe steps in the testing process:
X@itemize @bullet
X@item
XGetting Started
X@item
XInstrumenting a package
X@item
XTesting a Package
X@item
XAnalyzing Test Results
X@item
XDisplaying Test Results
X@end itemize
X
XThe system is designed to test Emacs-Lisp programs, and is implemented as a
Xminor mode of Emacs-Lisp mode (See Section 28.1 [Minor Modes] of the GNU
XEmacs Reference Manual).
X
X@node starting, testing, introduction, top
X@chapter Getting Started
X
XThis section describes how to begin using the Test package to test
XEmacs packages.
X
X@section Loading and Initialization
X
XTo start testing you probably want a fresh Emacs session with no extraneous
Xbuffers or processes hanging around. Non-essential extras will just slow
Xdown the state capture and achieve functions. If you have a busy
X@file{.emacs} file you may want to keep a trimmed back copy just for test
Xsessions.
X
XOnce in Emacs visit the file containing Emacs-lisp code to be tested,
Xthen enter:
X@example
XM-x load-file
Xtest.el
XM-x test-mode
X@end example
XThe mode line at the bottom of your window should show ``(Emacs-lisp
XTest)'' in the mode field. You are now in Test mode.
X
XProbably the first thing to do at this point is to use @samp{C-c n} to
Xgenerate a new buffer containing a test script template. This is a good
Xstarting point for creating a new test script; the template is shown in
Xsection 2.3 of this document. The task now confronting you is to develop a
Xtest suite that will exercise your package throughly, not only to test it
Xnow but also to have an automated regression test. The remainder of this
Xdocument will guide you through this task.
X
XFigure 2 illustrates the overall flow of data in the Test system.
X
X@group
X@example
X +-------------+
X |Source Text |
X +-------------+
X \
X \
X \
X Instrument
X /
X /
X /
X +-------------+ +-------------+
X |Instrumented | | Test Cases |
X | Text | +-------------+
X +-------------+ /
X \ /
X \ /
X \ /
X \ /
X Execute
X \
X \
X \
X \
X +-------------+
X |Test Results |
X +-------------+
X / ^
X / |
X / |
X / v
X Display Analyze
X
X
X
X@end example
X@center Figure 2. Test Data Flow
X@end group
X
X@section Test Mode
X
XThe package `@samp{test.el}' provides several functions to the user:
X@table @samp
X@item C-c n
XStart a new test script
X@code{ (tst-new-script-buffer)}
X@item C-c i
X(Re-)instrument a test program buffer @code{ (tst-instrument)}.
X@item C-c x
XTest the program, using the test script @code{ (tst-execute)}.
X@item C-c c
XRecord a GNU Emacs state for later use
X@code{ (tst-capture-state)}.
X@item C-c a
XAchieve a previously saved GNU Emacs state
X@code{ (tst-achieve-state)}.
X@item C-c C-a
XAchieve a state previously saved-to-file
X@code{ (tst-achieve-state-from-file)}.
X@item C-c C-c
XCapture current emacs state to file
X@code{ (tst-capture-state-to-file)}.
X@item C-c C-r
XRead a previously saved-to-file state into a variable
X@code{ (tst-read-state-from-file)}.
X@item C-c C-w
XWrite a state previously captured in a variable to a file
X@code{ (tst-write-state-to-file)}.
X@item C-c f
XAnalyze (filter) the results of running the test(s) @code{ (tst-analyze)}.
X@item C-c d
XDisplay the annotations about the test @code{ (tst-display-mode)}.
X@item C-c q
XExit test mode @code{ (test-mode)}.
X@end table
X
X@section Contents of the *test script* Buffer
X
X
X@example
X(defun test-script ()
X (interactive)
X ; Local Variables
X (let (pre post actual)
X ; Body
X (switch-to-buffer foo)
X (erase-buffer)
X (insert-file foo)
X (goto-char (point-min))
X (emacs-lisp-mode)
X (test-mode)
X (tst-instrument)
X ; 1st test run
X (tst-read-state-from-file 'post expected-state-file1)
X (tst-achieve-state-from-file 'pre initial-state-file1)
X (foo args)
X (tst-capture-state 'actual nil nil)
X;;; If you have a lot of tests, consider a while loop
X ) ; let
X) ; defun
X@end example
X
X@section The Example -- box.el
X
XThe package `@samp{box.el}' provides two functions to the user:
X@code{box-region} and @code{unbox}.
XThe first function encloses the current region with a box:
X
X@group
X@example
X/********************************************\
X* A boxed region *
X* may have lines of text of varying lengths. *
X* A side effect of boxing is to normalize *
X* the length of lines. *
X\********************************************/
X@end example
X@end group
X
XThe second function undoes the effect of the first by removing any boxes
Xwithin the region. This simple package will be used in this document as an
Xexample of a package to be tested using the Test package.
X
X@node testing, instrumenting, starting, top
X@chapter Testing Your Package
X
XAs mentioned at the beginning of this document, Emacs packages are
Xgenerally state changing operations. You invoke the package with a session
Xthat is in some arbitrary @i{initial state}. The package modifies
Xselected aspects of this @i{initial state} in a defined manner and leaves
Xthe session in some @i{final state}. The @dfn{state} of a GNU Emacs
Xsession consists of information on buffers, windows, processes, and other
Xglobal information. The components of the state that are used in the Test
Xpackage are listed later in this section.
X
X@dfn{Regression Testing} is a technique for verifying that changes in a
Xpackage do not unintentionally degrade the quality of the code. Any
Xchanges in one part of the package should not alter the functionality of
Xother parts of the package. State information plays two roles in
Xregression testing of Emacs packages. These are:
X@enumerate
X@item
Xinitiating the run of a package in a consistent and
Xpre-determined @dfn{initial state}, and
X@item
Xcomparing the @dfn{actual final state} after the completion of a
Xpackage to a @dfn{desired final state}.
X@end enumerate
XThe regression testing components of @samp{test.el} allow you to
X@dfn{capture} session states for later use, @dfn{achieve} these states to
Xset up a test run, and test the @dfn{equality} of states for package
Xverification.
X
XThe following example demonstrates these steps for regressions testing an
Xarbitrary package. Assume that a working version of some package exists.
XNow suppose you wish to add some new function that requires some
Xmodifications in the original code. After adding the new function you need
Xto verify that the original functionality of the package has not changed.
XIn other words, you need to verify that the code has not @dfn{regressed}
Xfrom its original functionality. To perform this testing, you
X@dfn{capture} the initial state of a session, run the original package, and
X@dfn{capture} the final state. Following this, you @dfn{achieve} the
Xinitial state that you previously captured, run the new version of the
Xpackage, and perform an @dfn{equality} test to compare the new final state
Xto the one that was captured after the run of the original package. The
Xstates should be equivalent if the test is successful.
X
XTwo extensions to this state comparison paradigm are:
X@enumerate
X@item
XTesting for partial equivalence - Each Emacs package is supposed to change
Xcertain aspects of a session state, but leave others unchanged. For
Xexample, a package may change the text of the current buffer but should
Xleave the window configuration of the session intact. Partial equivalence
Xtesting consists of the following steps:
X@enumerate
X@item
XCapture the initial state.
X@item
XRun the function to be tested.
X@item
XCompare the final state to the initial state. The states should be
Xequivalent in all areas except those that the test function is supposed to
Xchange.
X@end enumerate
X@item
XTesting functions that undo the effects of other functions - Some emacs
Xpackages have a function that perform some state change and a corresponding
Xfunction that undoes that change. Testing these types of packages consists
Xof the following steps:
X@enumerate
X@item
XCapture the initial state.
X@item
XRun the function that performs the state change.
X@item
XRun the function that undoes the state change.
X@item
XCompare the final state to the initial state. They should be equivalent.
X@end enumerate
X@end enumerate
XThe remainder of this section describes the elements of the Test
Xpackage that implement these aspects of regression testing.
XIn summary, these are:
X@itemize @bullet
X@item
X@code{capture} - saves the current state of the Emacs session
Xin either a bound variable or UNIX file.
X@item
X@code{partial capture} - save part of the current state of the
XEmacs session. Some or all global variables may be excluded, and a subset
Xof buffers may be selected.
X@item
X@code{achieve} - sets the state of the current session to one
Xthat is either in a bound variable or a UNIX file.
X@item
X@code{equality} - compares two emacs sessions states for
Xequivalence.
X@item
X@code{partial equality} - compares selected aspects of two emacs
Xsession states for equivalence.
X@item
X@code{inequality} - using user defined hooks, compares two
Xsession states within certain degrees of inequality (e.g. compares buffer
Xcontents ignoring whitespace).
X@end itemize
X
X@section Capturing and Achieving Emacs States
X
X@subsection GNU Emacs State Definition
X
XThe @dfn{capture} and @dfn{achieve} functions of @samp{test.el} represent
Xan Emacs state as the hierarchy of data shown below.
X@itemize @bullet
X@item
X@code{Global Bound Symbols} - Most of the information about an
XEmacs session is contained in the values of symbols global to the session.
XIn conventional programming language terms, this session information can be
Xviewed as the values of global variables. Some examples of common global
Xbound symbols are:
X@itemize @bullet
X@item
X@samp{global-abbrev-table} - the list of abbreviations global to
Xthe session and their meanings.
X@item
X@samp{global-map} - a vector describing the mappings of all keys
Xto Emacs commands.
X@item
X@samp{debug-on-error} - a flag that indicates that the Emacs
Xdebugger should be called on an error (if non-nil).
X@end itemize
X@item
X@code{Buffer Information} - Each Emacs session contains one or
Xmore buffers. The capture functions store the following information for
Xeach active buffer:
X@itemize @bullet
X@item
Xbuffer name - a unique name that identifies the buffer.
X@item
Xfile name - if the buffer is associated with a UNIX file, the
Xname of that file.
X@item
Xpoint - the value of the cursor point in the buffer.
X@item
Xmark - the value of the buffer's mark.
X@item
Xcontents - the contents of the buffer represented as a text string.
X@item
Xmodified - a flag indicating if the buffer has been modified
Xsince last saved.
X@item
Xlocal map - the local key map for this buffer.
X@item
Xlocal bound symbols - symbols (variables) that are local to this
Xbuffer (e.g., the @samp{current-mode} of the buffer).
X@end itemize
X@item
X@code{Window Information} - At any time an Emacs session has
Xone or more windows visible. The capture functions store the following
Xinformation for each window:
X@itemize @bullet
X@item
Xedges - the upper left and lower right corner of the window in
Xscreen coordinates.
X@item
Xbuffer - the buffer occupying this window.
X@item
Xstart - the integer position in the buffer where display starts
Xin the respective window.
X@item
Xpoint - the integer position in the buffer where the point for
Xthis window sits.
X@item
Xcurrent flag - a flag that is non-nil if this is the window in
Xwhich the screen cursor currently is located.
X@end itemize
X@item
X@code{Process Information} - An Emacs session may have invoked
Xseveral UNIX processes during its lifetime. These processes may have
Xcompleted or may still be running. Two common processes that run during an
XEmacs sessions are the @samp{shell} and the process that displays the time,
Xdate and system load on the @dfn{mode line}. The @samp{Test} capture
Xfunctions store the follow information for each process:
X@itemize @bullet
X@item
Xbuffer - the name of the buffer to which the process is attached
X(nil if the process is not related to a buffer, e.g., the display time
Xprocess).
X@item
Xprocess mark - the marker for the end of the last output from
Xthe process.
X@item
Xcommand - the UNIX command that invoked the process.
X@item
Xexit status - if the process has completed, the UNIX return code
Xfrom that process.
X@item
Xfilter - the name of an Emacs lisp function that receives all
Xoutput of the process.
X@item
Xname - a unique name of the program invoked in the process.
X@item
Xsentinel - the name of an Emacs lisp function that is called
Xwhen the process changes state (e.g. completes).
X@item
Xstatus - indicates whether the process is running, complete,
Xetc.
X@end itemize
X@end itemize
X
XA significant item that is excluded from this state information is the
Xcurrent definition of bound functions. Recall that an Emacs user can
Xdefine any new function with a @samp{defun} and load that function.
XFurthermore, the user can redefine any existing function (e.g.
X@code{forward-char} !) in this manner. Saving the current definition of
Xall bound functions would result in a huge state vector, and is therefore
Xnot done.
X
X@subsection Capturing an Emacs Session State
XWhen regression testing an Emacs package, you will mainly use the state
Xcapture functions of Test in two ways:
X@enumerate
X@item
Xto save a known @dfn{initial state} to use as the base state for
Xa run of an Emacs package, and
X@item
Xto save a @dfn{final state} to later compare to the state of the
Xsession after the run of a modified package.
X@end enumerate
X
X@subsubsection Commands for Capturing State
X
X@table @samp
X@item tst-capture-state-to-file
XWrites the current state of the Emacs session to a file. You are prompted
Xfor the name of the file, the @dfn{exclude-variables} and the
X@dfn{buffer-list}. Refer to the @code{Usage Notes} below for details on
Xthese last two items. If the file you specify already exists it is
Xoverwritten.
X@item tst-write-state-to-file
XWrites a state that is saved in a bound variable to a file. You are
Xprompted for the state variable name (which was created with
X@samp{tst-capture-state}) and the name of the file. If the file already
Xexists it is overwritten.
X@item tst-capture-state
XSaves the state of the Emacs session in a state variable. You are prompted
Xfor the name of the file, the @dfn{exclude-variables} and the
X@dfn{buffer-list}. Refer to the @code{Usage Notes} below for more details
Xon these last two items. If the bound variable you specify already exists,
Xit is overwritten.
X@end table
X
X@subsubsection Usage Notes
X
XThe amount of information saved by @samp{tst-capture-state-to-file} and
X@samp{tst-capture-state} is controlled by your responses to the command
Xprompts. These are:
X@table @samp
X@item List of buffers to capture:
XThere are three possible responses to this prompt:
X@enumerate
X@item
X@samp{nil} to indicate that state information on all buffers should be
Xsaved (the default).
X@item
XA list of of buffer names indicating for which buffers state information
Xshould be saved. For example, to save only the states of buffers
X@samp{bar} and @samp{foo} you enter:
X@example
X ("bar" "foo")
X@end example
X@item
XA singleton list that is the name of a non-existent buffer if you do not
Xwish to save any buffer state information. For example, if no buffer
X@samp{xxx} exists you could enter
X@example
X ("xxx")
X@end example
Xto specify that capture not save any buffer state information.
X@end enumerate
X@item List global vars to exclude:
XThere are three possible responses to this prompt:
X@enumerate
X@item
X@samp{nil} to capture all global variables.
X@item
X@samp{all} to exclude all global variables from the captured
Xstate.
X@item
XA list of global variable names to exclude from the captured state. For
Xexample, if you enter
X@example
X ("obarray" "values")
X@end example
Xthe two global variables, @samp{obarray} and @samp{values}, are not
Xcaptured. This value is the default for the prompt since these two
Xvariables have very large values that are rarely of use in testing.
X@end enumerate
X@end table
X
X@subsection Achieving an Emacs State
X
XWhen using Test you will mainly use the achieve state functions to return
Xyour session to a known @dfn{initial state}. With these functions you can
Xmake repeated test runs by simply achieving the @dfn{initial state} between
Xeach run.
X
X@subsubsection Commands for State Achieve
X@table @samp
X@item tst-read-state-from-file
XReads a session state from a file into a bound symbol. You are prompted
Xfor the name of the file and the name of the bound symbol. The file should
Xhave been created by @samp{capture-state-to-file} or
X@samp{write-state-to-file}. If the bound symbol already exists, it will be
Xoverwritten.
X@item tst-achieve-state-from-file
XSets the state of the emacs session to that saved in a file. You are
Xprompted for the name of the file. The file should have been created by
X@samp{capture-state-to-file} or @samp{write-state-to-file}.
X@item tst-achieve-state
XSets the state of the emacs session to that saved in a bound symbol. You
Xare prompted for the name of the bound symbol. The symbol must have been
Xcreated by @samp{tst-capture-state}.
X@end table
X
X@subsubsection Usage Notes
X
XAchieving the state of a previous Emacs session is a non-trivial task that
Xcan have many unexpected side-effects. These are listed below:
X@enumerate
X@item
XIf you want @samp{achieve} to preserve all existing buffers that were not
Xpart of the state you are achieving, you may set the variable
X@samp{tst-achieve-buffers-nondestructively} non nil.
X@item
XAchieving a previous state completely resets your Emacs session. Any
Xinformation in your current state will be overwritten by the achieved
Xstate. For example, any changes in buffers will not exist after the
Xachieve process. You should, therefore, make sure that you save any
Xchanges to disk before running achieve.
X@item
XIf you are not careful you may find that previous versions of disk files
Xoverwrite current changed versions. The following scenario demonstrates
Xthis problem.
X@enumerate
X@item
XSuppose you have a file @samp{mytest} that is in a buffer of an emacs
Xsession at the time you run @code{tst-capture-state-to-file}.
X@item
XNow suppose you make changes to this file over a period of a few days, and
Xthen run @code{tst-achieve-state-from-file}.
X@item
XYour emacs session will now contain the old version of the file.
X@item
XIf written to disk, this old version will overwrite the current version of
X@samp{mytest}.
X@end enumerate
X
XYou can avoid this problem by testing packages only on dummy test files, or
Xusing a special test directory with copies of files that are permanently
Xstored in other directories, or by not capturing buffer @samp{mytest} and
Xsetting @code{tst-achieve-buffers-nondestructively} to @code{t}.
X
X@item
XThe state of an emacs session is partially dependent on the state of your
Xentire UNIX session. Therefore, the achieve state functions may not be
Xable to restore your state to the exact one that was captured. For
Xexample, a process can not be started if the run image for the program no
Xlonger exists on disk. When testing and comparing states, it is best to
Xrely only on the more common aspects of an Emacs session (e.g. buffers,
Xwindows).
X@item
XAs explained earlier, the capture functions do not save all aspects of the
Xsession. A significant example is the current definition of all bound
Xfunctions. Therefore, the achieve functions will not restore these aspects
Xof the session state.
X@end enumerate
X
X@section Testing for equality
X
XThe equality package of @b{Test} compares two Emacs states for equivalence
Xby invoking special equality functions on each of the data components
Xcaptured in the state. Each of these equality functions compares only a
Xsmall part of the entire state. The results of these comparisons are
Xwritten into the buffer @samp{*equal-log*}. If two states are not equal
Xthen you can determine where components differ by looking at this buffer.
X
XThe section @b{Equality Functions} provides a list of all of the
Xequivalence testing functions. The section @b{Partial Equality} explains
Xhow you can accomplish testing a subset of a state. This is useful if you
Xare only interested in differences between certain aspects of state, such
Xas only the contents of buffers, but you do not care to hear about
Xdifferences in other components, such as windows. The section
X@b{Inequality} explains how to add hooks to the equality functions. These
Xhooks are called by the equality functions when two components are not
Xequal but before a result is returned. A hook that you write can be
Xinserted at this point and change a ``not equal'' result into an ``equal.''
X
X@subsection Equality Functions
X
XThe functions to compare states for equality are divided into two
Xcategories. One set of functions compares complete states, the second set
Xof functions only uses the buffer components.
X
XThe following state equality functions are defined for interactive use.
X@table @code
X@item tst-equ-state
XCompares two states for equality. Each component in the first state is
Xcompared to its corresponding component in the second state.
X@item tst-equ-sessions
XCompares the @b{sessions} component of two states. The sessions components
Xcontains all global variables. These includes those that define key-maps
Xand syntax tables.
X@item tst-equ-buffers
XCompares the @b{buffers} components of two states. Compares all of the
Xbuffers in one state with those of a second state.
X@item tst-equ-windows
XCompares the @b{windows} components of two states.
X@item tst-equ-process
XCompares the @b{process} components of two states.
X@end table
X
XThe following functions operate on just the @b{buffer} components of
Xstates. Objects of this type can be extracted from state objects by the
Xfunction @code{tst-equ-find-buff-with-name}.
X@table @code
X@item tst-equ-buffer-state
XCompares two buffers for equality. The following components are checked for
Xequality: @b{point}, @b{mark}, @b{contents}, @b{modified}, @b{file},
X@b{local-variables}.
X@item tst-equ-point
XCompares the @b{point} component of two buffers.
X@item tst-equ-mark
XCompares the @b{mark} component of two buffers.
X@item tst-equ-file
XCompares the @b{file} component of two buffers.
X@item tst-equ-modified
XCompares the @b{modified} component of two buffers.
X@item tst-equ-contents
XCompares the @b{contents} component of two buffers. Contents are compared
Xas two single strings.
X@item tst-equ-contents-line
XCompares the @b{contents} component of two buffers. Contents are viewed as
Xcomposed of lines of text, and compared line-by-line.
X@item tst-equ-contents-region
XCompares the @b{contents} component of two buffers. Only the contents
Xwithin a region are compared. Region is delimited by point and mark.
X@end table
X
X@subsection Partial Equality
X
XIf you are interested in only checking the equality of a subset of an Emacs
Xstate, then there are four methods you can use.
X
X@table @code
X@item tst-capture-state
XYou can use this function to capture only a subset of state. Once this has
Xbeen done, equality testing will only be performed on those parts of the
Xstate that where captured.
X@item tst-equ-state-functions
XThis variable contains a list of equality functions to be invoked when
Xcomparing two states. You can change it to test only those components of
Xstate that interest you. As an example, you might want to remove
X@samp{tst-equ-windows} if you are not interested in differences in windows
Xbetween states.
X@item tst-equ-buff-state-functions
XThis variable contains a list of equality functions to be invoked when
Xcomparing two @dfn{buffer states}. You can change it to test only those
Xcomponents of @dfn{buffer state} that interest you. As an example, you
Xmight want to remove @samp{tst-equ-point} and @samp{tst-equ-mark} if you
Xare not interested in those parts of the states.
X@item tst-equ-find-buffer-with-name
XUse this function to obtain only the part of state that is associated with
Xa particular buffer. You can then pass the result of this function to
Xbuffer testing equality functions.
X@end table
X
X@subsection Inequality
X
XSometimes the comparison of two objects yields ``not equal'' when wish to
Xignore certain inequalities. Each of the equality functions will execute a
Xhook, if one is defined, and the comparison of the two objects yields ``not
Xequal.'' Within the hook you can write your own test for equality and
Xchange the result of the comparison if you so desire.
X
XFor each equality function, there exists a hook symbol that, when defined,
Xwill be run if the two objects are not equal. The name of the hook symbol
Xcan be found by adding @samp{-hook} to the equality function name. The
Xvalue of this hook should then be set to the name of the hook function you
Xwish to execute. From within the the hook the two objects being compared
Xcan be accessed by adding @samp{1} and @samp{2} to the equality function
Xname. The variable @code{tst-equ-result} should be set to @samp{t} or
X@samp{nil} from within your hook.
X
XThe following example shows a hook, called @code{ignore-zero-points}, that
Xis executed whenever the result of comparing the point components of two
Xbuffers is not equal. From within this function the two point components
Xare accessed as @samp{tst-equ-point1} and @samp{tst-equ-point2}. The hook
Xchanges the result of a point comparison from @samp{nil} to @samp{t} if one
Xof the points is at position zero.
X
X@example
X
X(setq tst-equ-point-hook 'ignore-zero-points)
X
X(defun ignore-zero-points ()
X "Equality point hook, changes result to t if one point
X is at position zero."
X
X (if (or (equal tst-equ-point1 0) (equal tst-equ-point2 0))
X (setq tst-equ-result t) ; return t if one is zero
X ;else
X (setq tst-equ-result nil); otherwise return nil
X )
X)
X
X@end example
X
XOf particular usefulness are the hooks that are associated with the
Xcontents of two buffers. The equality function @samp{tst-equ-bs-contents}
Xcompares the contents of two buffers by comparing strings that contain the
Xentire contents of each buffer. Should this comparison fail it is sometimes
Xuseful to compare the two strings after eliminating all white-space from
Xeach string. The following hook, provided with this package, does such a
Xcomparison.
X
X@example
X
X(setq tst-equ-contents-hook 'ignore-white-space)
X
Xdefun ignore-white-space()
X " Compares the contents of two buffers after removing all
X white-space from each."
X
X (setq tst-equ-result (string-equal-less-regexp "\\s "
X tst-equ-contents1 tst-equ-contents2))
X)
X@end example
X
XA second contents hook of interest is the one associated with the
X@samp{tst-equ-contents-line function}. This function compares the contents
Xof two functions on a line-by-line basis. The hook for this function
Xdiffers from all other hooks, in that it is called once per line, instead
Xof once per call to its associated function. The two lines are accessed as
X@samp{tst-equ-line1} and @samp{tst-equ-line2}.
X
X
X@section Testing example
X
XThe example that follows illustrates how to write an emacs-lisp function
Xthat uses the Test package to test another function you have written. In
Xthe example the two user supplied functions under test are:
X@samp{box-region} and @samp{unbox}.
X
X@example
X;;; test-box uses the Test package to test the functions box and unbox
X;;;
X test-box ()
X (interactive)
X ; Local Variables
X (let (initial-state ; to hold states
X boxed-state
X final-state
X capture-buffer-list
X )
X
X ; First create a buffer to use the functions on
X (get-buffer-create "box.junk")
X (set-buffer "box.junk")
X (erase-buffer)
X (insert-file "/project/gnutest/test/box.junk")
X
X ; Save the initial state
X ; Since we're only concerned with one buffer include
X ; include that in the capture list
X (setq capture-buffer-list '("box.junk"))
X (tst-capture-state 'initial-state capture-buffer-list nil)
X
X ; execute the box function
X (goto-char (point-min))
X (set-mark (point-max))
X (box-region nil)
X
X ; capture the boxed state
X (tst-capture-state 'boxed-state capture-buffer-list nil)
X
X ; unbox and capture again
X (unbox nil)
X (tst-capture-state 'final-state capture-buffer-list nil)
X
X ; compare the initial state to the final state
X ; they should be exactly equal
X (if (tst-equ-state initial-state final-state
X "Compare before box to after unbox")
X (message "Box/unbox test passed")
X (message "Box/unbox test failed"))
X
X ) ; let
X ) ; defun
X@end example
X
X@subsection Contents of *equal-log*
X
XEach of the equality functions writes the results of its comparison into a
Xbuffer named ``*equal-log*''. If this buffer does not exist, then it is
Xcreated. The contents of the buffer are never erased nor is the buffer
Xever deleted, these actions are left to the user.
X
XThe buffer is created in Outline Mode. This allows selective hiding of
Xcomparisons that are not of interest to the user.
X
XIf the result of a comparison is nil (i.e. not equal) then that entry in
Xthe log is flagged with a question mark (?). In most cases, the two
Xdiffering objects are also logged as in the following example.
X
X@example
X* ?State comparison: Compare before box to after unbox
X
X** ?Sessions state
X*** ? Global symbols
X data-bytes-free 5350948 5176868
X this-command nil kill-region
X data-bytes-used 383452 557532
X ?post not found in second state
X post nil nil
X ?file not found in second state
X file
X statevar state post
X
X
X** ?Buffers state
X
X** Comparison of buffers named: *scratch*
X*** point: "1" "1"
X*** mark: nil nil
X*** contents: contents equal
X*** modified: nil nil
X*** file: nil nil
X*** local-vars: local variables are equal
X
X** Comparison of buffers named: *Minibuf-0*
X*** point: "1" "1"
X*** mark: nil nil
X*** contents: contents equal
X*** modified: nil nil
X*** file: nil nil
X*** local-vars: local variables are equal
X
X** Comparison of buffers named: box.el
X*** point: "1" "1"
X*** mark: 4480 4480
X*** contents: contents equal
X*** modified: t t
X*** file: nil nil
X*** local-vars: local variables are equal
X
X** Comparison of buffers named: box.el-instrumented
X*** point: "5608" "5608"
X*** mark: 5608 5608
X*** contents: contents equal
X*** modified: t t
X*** file: nil nil
X*** local-vars: local variables are equal
X
X** ?Comparison of buffers named: box.junk
X*** ?point: "1" "234"
X*** ?mark: 235 1
X*** ?contents: contents not equal
X*** modified: t t
X*** file: nil nil
X*** local-vars: local variables are equal
X
X
X** Processes state
X
X
X** Window state
X
X*** window
X**** window-edges: (0 0 10 9) (0 0 10 9)
X**** window-buffer: "box.junk" "box.junk"
X**** window-start: 1 1
X**** window-point: 1 1
X**** current-window: t t
X
X
X@end example
X
X@node instrumenting, analyzing, testing, top
X@chapter Instrumenting a Package
X
XUse the @code{tst-instrument} command for instrumenting Lisp code.
XInstrumenting copies the contents of the current buffer to a new buffer
Xnamed for the current buffer concatenated with @samp{-instrumented}. The
Xnew buffer is set to emacs-lisp-mode. Instrumenting initializes the
Xannotation data base, adds the instrumentation to the copied code, and
Xfinally evaluates the entire copied buffer.
X
X@section Instrumentation Probes
X
XInstrumentation acts on certain lists that represent functions within a
X@code{defun} by adding an instrumentation probe. Candidate lists are
Xidentified and Lisp code is added around the list. A list representing a
Xfunction that has the form, @samp{(function arg1 arg2)}, after insertion of
Xthe instrumentation probe will have the form, @samp{(tst-cover #id
X(function arg1 arg2))}. Instrumentation assigns the line number of the
Xfunction within the buffer as the #id.
X
X@section Cover Function
X
XThe instrumentation probe consists of the invocation of the function
X@code{tst-cover} with two arguments, a unique identifier and the function
Xthat was instrumented. @code{Tst-cover} uses the identifier as a key into
Xthe annotation data base, stores the result of the instrumented function,
Xincrements an invocation counter, and returns the result of the
Xinstrumented function as its function value.
X
X
X@section Instrumentation Example
X
XOriginal Code:
X
X@example
X(defun factorial (n)
X (let
X (result)
X (if
X (< n 2)
X (setq result 1)
X (setq result
X (* n
X (factorial
X (1- n)))))
X result))
X@end example
X
XInstrumented Code:
X
X@example
X(defun factorial (n)
X (tst-cover 2 (let
X (result)
X (tst-cover 4 (if
X (tst-cover 5 (< n 2))
X (tst-cover 6 (setq result 1))
X (tst-cover 7 (setq result
X (tst-cover 8 (* n
X (tst-cover 9 (factorial
X (tst-cover 10 (1- n))))))))))
X result)))
X@end example
X
X@node analyzing, displaying, instrumenting, top
X@chapter Analyzing the Test Results
X
XThis section explains how to use the @code{analyze} function
X
X@section Analysis Functions
X
XThe Coverage Analysis package is designed to run after a series of tests
Xhas been performed using the instrumented version of the code under test.
XIt retrieves data from the annotation database and detects the following
Xtesting anomalies:
X
X@itemize @bullet
X@item
XFailure to execute an expression.
X@item
XFailure to produce more than one value for an expression.
X@end itemize
XWhen these conditions are detected additional attributes are stored in the
Xannotation database in a format suitable for extraction by the display
Xcomponent of Test.
X
XAnalysis is invoked by the command:
X
X@example
X tst-analyze
X@end example
X
XIf an expression is not executed, the attribute ``zero'' is inserted with a
Xconstant value. This constant value is defined by the following variable:
X@example
X tst-anl-zero-counts
X@end example
X
XIf an expression returns a constant value, the attribute ``constant'' is
Xinserted with the value of the constant. If an expression is only executed
Xonce, it is deemed to return a constant result.
X
XYou can re-run the analysis function without re-instrumenting the code
Xunder test. If, for example, an initial set of tests indicates a number of
Xunexecuted expressions, you can run additional tests and analyze the
Xcombined results without re-running the complete set of tests. Note
Xhowever, that if you re-instrument the code, the annotation database is
Xreinitialized and the results of previous tests, including any analysis
Xresults will be lost (unless you explicitly save the annotation database
Xusing tst-ann-get-db).
X
X@section Sample Analysis Results
X
XNormally you would view the results of an analysis using the display
Xpackage on-line. However, the following indicates the information produced
Xby the analysis.
X
X@example
X# Test analysis of box.el
X# (lines which were never evaluated during tests, indicated by
X# the string ``NEVER->>'', or which returned the same value
X# every time they were evaluated, including the value.)
X
Xbox.el:31== nil
Xbox.el:33== nil
Xbox.el:34== nil
Xbox.el:35== nil
Xbox.el:37== 1
Xbox.el:38== 0
Xbox.el:39== nil
Xbox.el:41== nil
Xbox.el:42== nil
Xbox.el:49== 1
Xbox.el:50== nil
X
X...
X
Xbox.el:94== nil
Xbox.el:95== nil
Xbox.el:109== #<marker at 1 in box.junk>
Xbox.el:115== nil
Xbox.el:116== t
Xbox.el:121== NEVER->>
Xbox.el:122== NEVER->>
Xbox.el:128== nil
Xbox.el:129== nil
Xbox.el:134== NEVER->>
Xbox.el:135== NEVER->>
Xbox.el:140== nil
X
X...
X
X@end example
X
X@node displaying, running, analyzing, top
X@chapter Displaying Test Results
X
XThe @dfn{coverage} component of the Test package is useful for measuring
Xhow thoroughly a test script exercises an emacs-lisp package. The
Xinteractive Test display mode lets you browse the annotated code. The
Xbatch display function creates a summary report which may be browsed later
Xusing the Emacs ``compilation'' mode.
X
X@section Display Mode---creating annotation windows
X
XEvaluating an instrumented buffer of emacs-lisp code creates a database of
X@dfn{annotations} for each instrumented line. The two data stored during
Xevaluation are: @samp{count}, the number of times each line has been
Xevaluated, and @samp{values}, a list of the resultant values. Analyzing
Xthe database adds two new annotations to various lines in the database:
X@samp{zero}, the line was never evaluated, and @samp{constant}, the value
Xreturned was the same every time it was evaluated. Note that a line
Xevaluated exactly once will be flagged as having a constant value.
X
XDisplay mode puts up these @dfn{annotations} beside the lisp code in one or
Xmore @dfn{annotation windows} which are linked to the @dfn{lisp window} so
Xthat all windows scroll together. The functions available for creating
Xannotation windows are as follows:
X
X@table @samp
X@item C-c c
XOpen an annotation window which highlights code which returned the
Xsame value during testing (@samp{tst-display-constant}).
X
X@item C-c z
XOpen an annotation window which highlights code which was never
Xevaluated during testing (@samp{tst-display-zero}).
X
X@item M-x tst-display-open-buffer
XCreate a buffer containing the database values for a particular attribute
Xover all lines of the lisp buffer. Valid attributes are @samp{count}, the
Xcount of how many times that line was evaluated, and @samp{values}, a list
Xwith each element the result of one evaluation of that line.
X
X@item M-x tst-display-open-window
XOpen an annotation window onto a buffer created by a prior evaluation
Xof @samp{tst-display-open-buffer}.
X
X@item C-c C-h
XOpen a help window showing the key bindings for display mode
X(@samp{tst-display-mode-help}).
X
X@item C-c q
XExit display mode (@samp{tst-display-mode-exit}).
X@end table
X
XTypically the first thing you will do after entering display mode is to
Xtype @samp{C-c c C-c z} to create two annotation windows which point out
Xany lines in the lisp buffer which either were never evaluated or always
Xreturned the same value. In the first case you probably need to add to
Xyour test script to either call an unreferenced function or exercise the
Xother side of a conditional expression. The second case may require some
Xinvestigation. One possibility is that the line is something like:
X@example
X(setq very-important-variable nil).
X@end example
XThis expression, of course, always returns the same value. Another
Xpossibility is that the line was only evaluated once, in which case you
Xmight want to add to your test script to hit it again. The third
Xpossibility is that you overlooked a parameter when composing your test
Xscript.
X
XIf you want to look at the other annotations in the database, you may do so
Xwith the commands @samp{M-x tst-display-open-buffer} and @samp{M-x
Xtst-display-open-window}. You must use them in that order, and each one
Xwill prompt for an ``attribute name'' which may be either of @samp{count}
Xor @samp{values}.
X
X@section Display Mode---moving within annotation windows
X
XThe annotation windows are kept in step with the lisp window through
Xseveral functions which take the place of the usual cursor movement
Xcommands while in display mode. These are described below:
X
X@table @samp
X@item C-n
XMove down one line vertically in the lisp window and all associated
Xannotation windows. On reaching the bottom of the window, scroll windows
Xtogether (@samp{tst-display-next-line}).
X
X@item C-p
XMove up one line vertically in the lisp window and all associated
Xannotation windows. On reaching the top of the window, scroll windows
Xtogether (@samp{tst-display-previous-line}).
X
X@item C-v
XScroll forward in the lisp window and any associated annotation windows
Xkeeping them aligned (@samp{tst-display-scroll-up}).
X
X@item M-v
XScroll backward in the lisp window and any associated annotation windows
Xkeeping them aligned (@samp{tst-display-scroll-down}).
X
X@item C-c l
XClear screen and redisplay, scrolling the lisp window and any associated
Xannotation windows together to center the line containing point
X(@samp{tst-display-redraw}).
X
X@end table
X
XThe usual cursor up and down keys are @samp{C-n} and @samp{C-p},
Xrespectively. In display mode these move point in all annotation windows
Xsimultaneously so that when the limits of the screen are reached all
Xwindows will scroll together. Similarly the scroll-up and scroll-down keys
X@samp{C-v} and @samp{M-v} are rebound in display mode to functions which
Xkeep track of what annotation windows are open and scroll them in step with
Xthe lisp buffer. It is still possible for the windows to get out of step
X(e.g. after using @samp{M-<}), so the @samp{C-c l} key will resynchronize
Xall annotation windows while recentering the line containing point just as
Xthe @samp{C-l} key does for a single window.
X
X@section Batch Display of Analysis
X
XQuite separate from the display mode is a display function that is designed
Xto give a concise summary of test coverage without your interaction. This
Xfunction can be run in batch or interactively. (It was originally intended
Xfor only batch operation, hence the name.)
X
X@table @samp
X@item M-x tst-display-batch
XGenerate a @dfn{compilation style} buffer containing @dfn{zero} and
X@dfn{constant} analyses from the database.
X@end table
X
XYou need not be in display mode to use @samp{tst-display-batch}. You are
Xmost likely to use it in a test script run from batch mode to dump out a
Xsummary of test coverage at the end of a run. However it is also useful
Xfrom interactive mode because it takes advantage of the Emacs function
X@samp{C-x`} (@samp{next-error}), often used to view compiler error
Xmessages. The first invocation of @samp{C-x`} parses the error messages in
Xthe buffer named @samp{*compilation*} then places point in one window on
Xthe line of code referenced by the error message shown at the top of the
Xother window. Successive @samp{C-x`} keystrokes advance to successive
Xerror messages and the corresponding lines in the code buffer.
X
XThe @samp{M-x tst-display-batch} command writes into the
X@samp{*compilation*} buffer a line for every line in the lisp buffer that
Xhas either a @samp{zero} or @samp{constant} annotation. Subsequent uses of
X@samp{C-x`} advance to the next error and the corresponding line in the
Xlisp buffer.
X
X@section The Example
X
XThe following is the output of the evaluation of @samp{tst-display-batch}
Xin our example test script. Notice that some deficiencies in the test
Xscript are pointed out: error conditions were not exercised (see lines
X121-122 and 134-135 in Appendix A); and not enough different input texts
Xwere ``boxed'' to really stress the package under test (e.g. line 40, input
Xtext always same width).
X
X@example
X# Test analysis of box.el
X# (lines which were never evaluated during tests or returned
X# the same value every time they were evaluated.)
Xbox.el:31== nil
Xbox.el:33== nil
Xbox.el:34== nil
Xbox.el:35== nil
Xbox.el:37== 1
Xbox.el:38== 0
Xbox.el:39== nil
Xbox.el:40== 41
Xbox.el:41== nil
Xbox.el:42== nil
Xbox.el:43== 42
Xbox.el:47== 44
Xbox.el:49== 1
Xbox.el:50== nil
Xbox.el:51== nil
Xbox.el:52== nil
Xbox.el:53== nil
Xbox.el:55== nil
Xbox.el:56== nil
Xbox.el:57== nil
Xbox.el:58== nil
Xbox.el:59== nil
Xbox.el:62== nil
Xbox.el:63== nil
Xbox.el:64== nil
Xbox.el:65== nil
Xbox.el:67== nil
Xbox.el:68== nil
Xbox.el:69== 1
Xbox.el:70== 12
Xbox.el:71== nil
Xbox.el:72== nil
Xbox.el:73== nil
Xbox.el:74== 12
Xbox.el:75== nil
Xbox.el:80== nil
Xbox.el:91== nil
Xbox.el:93== nil
Xbox.el:94== nil
Xbox.el:95== nil
Xbox.el:109== #<marker at 6 in box.junk>
Xbox.el:115== nil
Xbox.el:116== t
Xbox.el:121== NEVER->>
Xbox.el:122== NEVER->>
Xbox.el:128== nil
Xbox.el:129== nil
Xbox.el:134== NEVER->>
Xbox.el:135== NEVER->>
Xbox.el:140== nil
Xbox.el:141== nil
Xbox.el:143== nil
Xbox.el:144== t
Xbox.el:145== NEVER->>
Xbox.el:147== t
Xbox.el:149== nil
Xbox.el:150== nil
Xbox.el:151== nil
Xbox.el:153== nil
Xbox.el:154== nil
Xbox.el:157== nil
Xbox.el:159== nil
Xbox.el:160== #<marker at 6 in box.junk>
X
X@end example
X
X@node running, epilogue, displaying, top
X@chapter Running Test in Batch Mode
X
X@section Preparation
X
XBefore running a set of tests, you must prepare the following items:
X@itemize @bullet
X@item
XA package to test.
X@item
XA test script, usually written as a single function.
X@item
XA set of precondition states---one for each test (although many
Xtests may share the same precondition).
X@item
XA set of expected postcondition states---one for each test (some
Xsharing is possible).
X@end itemize
XTest scripts are discussed in the next section.
X
XYou will probably want to use the state-capturing functions (e.g.,
X``tst-capture-state'', ``tst-write-state-to-file'') to prepare and save
Xstates in files. In doing this, strive for minimality of states. That is,
Xavoid complicating the states with extra components, particularly buffers
Xand windows, that do not contribute to the functionality being tested.
XThis will make it easier to read and comprehend the testing states, and
Xwill result in more efficient test scripts. (The instrumentation and
Xanalysis components of Test are designed to assist you in discovering the
Xinadequacies of your test scripts. It is best to start with too little
Xtesting, and add additional tests as the analysis dictates.)
X
XBelow is an example scenario for preparing the states used in a testing
Xscript discussed later. Note that the functions would be invoked
Xinteractively, even though they are shown in ``program invocation'' style.
X@example
X(load "test.el")
X(load "box.el")
X(find-file "sample")
X... go to an appropriate location ...
X(set-mark)
X... go to an appropriate location ...
X(tst-capture-state-to-file "pre-box")
X(box-region)
X(tst-capture-state-to-file "exp-box-1")
X(unbox)
X(tst-capture-state-to-file "exp-unbox-1")
X(box-region t)
X(tst-capture-state-to-file "exp-box-2")
X(unbox t)
X(tst-capture-state-to-file "exp-unbox-2")
X@end example
XNote that in this scenario the expected result of applying a function is
Xcreated by applying the function to the precondition state. It is hard to
Ximagine how the test could fail, but it might for unexpected reasons. A
Xbetter method of creating the test states is to use an independent method,
Xsuch as repeated application of simpler functions (e.g., ``insert'',
X``next-line''). Still, ``safe'' tests, such as that shown above, are
Xuseful for regression testing.
X
X@section Batch Test Scripts
X
XA test run usually consists of the following steps:
X@enumerate
X@item
XRead testing states from files.
X@item
XInitialize the testing environment:
X@enumerate
X@item
XFind the object package (the code to be tested).
X@item
XInstrument the package.
X@end enumerate
X@item
XRun each test:
X@enumerate
X@item
XAchieve the precondition state of the test.
X@item
XExecute the function to be tested.
X@item
XCapture the postcondition state.
X@item
XCompare the postcondition state to the expected postcondition state.
X@end enumerate
X@item
XSave the results of testing in a file.
X@item
XAnalyze the testing (filter the collected data) and save the results.
X@end enumerate
XNote that a set of tests may reuse testing states, especially the
Xpreconditions of tests. That is why it is wise to read all of the needed
Xtesting states from files first. Also, it is good style to minimize state
Xchanges during testing (except for the execution of tested functions, of
Xcourse), so that the test script may be modified without unforeseen side
Xeffects.
X
XBelow is an example test script for testing functions ``box-region'' and
X``unbox'' in package ``box.el''. This script does not contain enough
Xtests, which the instrumentation should expose.
X
X@example
X(load "test.el")
X
X(defun script ()
X (let (post-state pre-box exp-box-1 exp-box-2
X pre-unbox exp-unbox-1 exp-unbox-2)
X ; Load states from files
X (tst-read-state-from-file 'pre-box "pre-box")
X (tst-read-state-from-file 'exp-box-1 "exp-box-1")
X (tst-read-state-from-file 'exp-box-2 "exp-box-2")
X (tst-read-state-from-file 'pre-unbox "pre-unbox")
X (tst-read-state-from-file 'exp-unbox-1 "exp-unbox-1")
X (tst-read-state-from-file 'exp-unbox-2 "exp-unbox-2")
X ; Instrument package
X (find-file "box.el")
X (tst-instrument)
X ; Initialize test environment
X (find-file "sample")
X ; Tests
X ; TEST box-1
X (tst-achieve-state pre-box)
X (box-region nil)
X (tst-capture-state 'post-state)
X (tst-equ-state post-state exp-box-1)
X ; TEST box-2
X (tst-achieve-state pre-box)
X (box-region t)
X (tst-capture-state 'post-state)
X (tst-equ-state post-state exp-box-2)
X ; TEST unbox-1
X (tst-achieve-state pre-unbox)
X (unbox nil)
X (tst-capture-state 'post-state)
X (tst-equ-state post-state exp-unbox-1)
X ; TEST unbox-2
X (tst-achieve-state pre-unbox)
X (unbox t)
X (tst-capture-state 'post-state)
X (tst-equ-state post-state exp-unbox-2)
X ; Results of tests
X (set-buffer "*equal-log*")
X (save-file "equal-log-sample")
X ; Analysis of testing
X (tst-analyze)
X (tst-display-batch)
X ) ; let
X) ; defun script
X@end example
X
X@section Batch Invocation of Test
X
XTo invoke the test script from batch, use the following Unix command:
X@example
X% gnuemacs -batch -l script.el -f script -kill &
X@end example
XNote that the test script explicitly loads ``test.el'' at the beginning
Xof ``script.el''.
X
X
XAn instrumented package runs considerably slower than an uninstrumented
Xpackage. It is a good idea to test your script without instrumentation
X(and probably without many of the tests). Once you are satisfied with the
Xscript, run it at lower priority and/or at non-peak hours to avoid
Xinconveniencing other users.
X
X@section Advanced Batch Use -- Filtering
X
XThe state information available for ``capture'ing and ``achieve''ing is
Xusually more than needed for any particular test. You can improve the
Xperformance of testing by filtering out the unneeded components of states.
XThere are mechanisms for doing this before and after testing:
X@itemize @bullet
X@item
XThe ``capture'' functions use global options to determine the
Xcomponents to capture.
X@item
XThe ``achieve'' functions are similarly parameterized.
X@item
XThe ``equality'' functions use global options to determine which components
Xto compare. Also, they can be weakened (i.e. return ``true'' more often)
Xthrough the use of hooks and replacement functions.
X@item
XThe ``analysis'' functions reduce the instrumentation data to two special
Xcases: lack of evaluation and ``constant value across testing''.
X@item
XThe ``*equal-log*'' buffer may be viewed in ``outline'' mode, with
Xuninteresting cases suppressed through elision.
X@item
XThe batch display function produces output that may be viewed with the
X``next-error'' function of ``compilation''.
X@end itemize
XIn using these filters you should be aware that too much filtering defeats
Xthe purpose of testing.
X
X@node epilogue, example, running, top
X@chapter The Epilogue
X
X@section How to Extend the Package
X
X@subsection Equality Tests
X
XIf you want to extend the equality package to compare other aspects of your
Xenvironment, simply write your own comparison function and add it to the
X'function-vector' for the appropriate area. For example, if you were
Xinterested in comparing the @code{foobar} attribute of buffers, first write
Xa function, @code{tst-equ-foobar}, and then add it to the list of functions
Xexecuted for a buffer, @code{tst-equ-buff-state-functions}.
X
XSimilarly, if you wanted to test the @code{foo} attribute of a state, add
Xthe new function to the list of functions executed for a state,
X@code{tst-equ-state-functions}.
X
XYou can also extend the equality tests by the use of hooks. The method is
Xdefined in more detail in the section on Inequality.
X
X@subsection Instrumentation
X
X@subsection Display Functions
X
XThe interactive display mode attempts to provide the fiction of annotation
Xwindows which are part of the code window. In retrospect it may be that
Xsimply inserting the text into a new buffer along with the code would be as
Xuseful. It also may be that the batch style report in conjunction with the
X``compilation'' error parsing is just as useful in practice.
X
XThere are certainly other types of annotation which could usefully be
Xdisplayed, like the number of times each line was executed, etc.
X
XIt might make sense to bind @samp{tst-display-batch} to a key in
X@code{test-mode}. It also might make sense to remove the binding for
X@code{tst-analyze} and call it only from within the display functions.
X
X@subsection Annotation Functions
X
XThe annotation database is accessed twice every time an instrumented line
Xof code is evaluated. This code contributed substantially to the slow-down
Xof instrumented code and time spent making it more efficient would be
Xrewarding.
X
X@subsection Analysis Functions
X
XIf you want to add further functions to the Analysis Package, you will have
Xto modify the main function, ``tst-analyze'', which invokes the individual
Xanalysis functions. It currently invokes ``tst-anl-zero-counts'' and
X``tst-anl-constant-values''. Your new function should use the same idiom
Xas these functions, i.e., use ``tst-ann-get-lines'' to retrieve the list of
Xlines for which annotation holds information and then use ``mapcar'' to
Xapply your single line analysis function to each line's data. You can then
Xuse ``tst-ann-put'' to store your results.
X
X@node example, , epilogue, top
X@appendix The Complete Example
X@include example
SHAR_EOF
if test 60887 -ne "`wc -c < 'test.texinfo'`"
then
echo shar: "error transmitting 'test.texinfo'" '(should have been 60887 characters)'
fi
fi
exit 0
# End of shell archive
--
Rich $alz
Cronus Project, BBN Labs rsalz@bbn.com
Moderator, comp.sources.unix sources@uunet.uu.net