Andrew Patrick <csr@calvin.doc.ca> (04/27/91)
Submitted-by: Andrew Patrick <csr@calvin.doc.ca>
Posting-number: Volume 1, Info 4
Archive-name: guide-rev
Comp.Sources.Reviewed
Guidelines for Reviewers
(Version 1.3)
This document is designed to provide guidelines and suggestions for use
while reviewing software for the news group "comp.sources.reviewed".
It also provides potential submitters with information about what the
reviewers will be looking for, and may be useful to others who are
asked to evaluate software.
These guidelines may change as we gain more experience in reviewing
software, and comments are welcome.
Be Timely
---------
If you cannot provide a review of a submission in a reasonable amount of
time (say 1-2 weeks), then please ask the moderator to select another
reviewer. If you decline to review a submission because you are too
busy, your name will be placed at the end of the reviewers queue. This
means that there may be some period of time until you are selected to
review another submission (depending on your areas of interest and
expertise), but that is probably what you need in this situation anyway.
If you refuse a package because you don't have the equipment or skills
to do the job (reviewers are chosen based on the equipment, skills, and
interests they have listed but there may be some mistakes in
assignments), you will be left at the head of the queue. This means you
should get the next package that matches your equipment and areas of
interest.
Record Your Work
----------------
A crucial part of the review process is recording what you do. You
must make complete and accurate notes of your time spent working on the
package. Don't rely on your memory to record the problems or
suggestions you come across, because you will forget quickly as you
move on to other things. So, the first thing to do when you get a
submission is to start documenting everything that happens. Since you
will probably be working on several machines, you may have to work with
pencil and paper (gasp).
Evaluating the Format of the Submission
---------------------------------------
Was it in the form of a "shar" file, or similar packaging appropriate
for the architecture?
Did in unpack correctly?
Did it unpack in the places you expected it to?
Evaluating the Description and Purpose
--------------------------------------
Is there a README file that contains:
- the purpose and value of the software (give details here)
- the types of systems it is intended for (e.g., BSD only,
Unix and DOS, etc.)
- any dependencies in the system (e.g., must have perl to
run)
- known limitations of the software
- the authors of the software (with e-mail addresses)
- the "patchlevel" of the software (see below)
- any copying or distribution conditions
Is there a "patchlevel.h" file (or equivalent) indicating the version
number of the software?
Building the Software
---------------------
Is there an Installation document explaining how to build the software?
Are the options and conditional parts of the package (e.g., do this for
DOS, this for System V, etc.) clearly documented?
Is there a proper Makefile? (Imakefile for X software?)
Did the make run correctly?
Are building and installation two separate steps?
Did the software build on each of the architectures you were able to
test? (Give details about your testing environment.)
Testing the Software
--------------------
Did the software run on each of the systems you tested?
Did the program perform the functions it was supposed to perform?
It is not your place to fix the software you are reviewing. However, if
you find a problem with an obvious solution, make sure you document it
so the authors can make use of it.
The Documentation
-----------------
The documentation is a very important part of a submission, so it
should be reviewed carefully. The documentation may be built-in to the
program and be in the form of document files and/or man pages. It is
difficult to give simple rules for documentation, and not all programs
require the same amount of documentation, but here are some things to
consider.
General Guidelines
------------------
Is the documentation written in a form that is clear, precise, and
easy to understand?
Is each feature or function of the program documented?
Is the documentation organized? Sections and sub-sections often
make documents easier to read and understand.
Documentation should be provided in "text only" form. An author may
choose to also provide formatted manuals that use PostScript or TeX,
but a readable form should be provided.
Guidelines for Unix Documentation
---------------------------------
Is there a man page? Man pages should contain the following
sections (at least):
NAME required
SYNOPSIS required
AVAILABILITY optional
DESCRIPTION required
OPTIONS required if there are any command-line options
ENVIRONMENT required if there is provision for environment
variables
FILES required if use is made of other files
SEE ALSO optional
DIAGNOSTICS required if the program can produce debugging output
NOTES optional
BUGS required if any are known
AUTHOR optional
These sections should be complete (e.g., all the OPTIONS must be
described).
Guidelines for VMS Documentation
--------------------------------
Is there a VMS HELP file? HELP files should contain the following
sub-topics (at least):
Parameters if there are any
Command_Qualifiers if there are any
Examples if appropriate
Is there additional documentation to supplement the VMS HELP file?
Guidelines for DOS Documentation
--------------------------------
There are no standards for DOS documentation, and this makes the
review process difficult. We suggest that the information and
features described above should also be present in DOS
documentation, although they may not be in the form described
above.
Functionality and Features
--------------------------
Does the software perform some function that is valuable?
Are there obvious features or additions that would improve the package?
Overall Evaluation
------------------
What is your overall evaluation -- would you recommend it to a friend?
Would you suggest that this submission:
- be accepted for posting as is
- be accepted after minor revisions (that you have detailed)
- be returned to the author with a recommendation to make
major changes and re-submit
- be rejected
Submitting a Summary
--------------------
The final step is to return a summary to the moderator. Please make
sure you quote the "reference name" in your summary, preferably in the
subject line.
DO NOT send your summary to the author. The moderator will collect
all the reviews, prepare a grand summary, and forward it to the author.
Your summary should provide as much detail as possible. Make sure you
give details on the kinds of machines you tested on. You may choose
to use this file as a template to record your review.
If you are recommending that a submission be posted, you should also
provide a short summary that would be appropriate for publication with
the software. This summary might mention why you found the package
useful, what you liked about it, what machines you tested it on, and
what limitations you did find. Your name will not be attached to this
review summary when it is returned to the authors or posted.
Contacting the Authors
----------------------
Your identity will not be revealed to the authors unless you choose to
do so. If you feel that you would like to contact the authors for
clarification, you may contact them directly or ask the moderator to
forward your question. You should only contact the author for
clarification or additional information needed to complete your
review. You should not comment on the submission at this time, but
instead save your comments for the report you send to the moderator.
You should also ensure that your interactions with the author are
conducted in a courteous and professional manner.
In addition, you should ensure that the other reviewers working on that
submission also receive the information you get back from the authors.
Further, you must do so in a manner that does not interfere with their
wishes not to identify themselves to the authors.
It is not a good idea to suggest that authors make patches to software
during the review process. If you find a problem that the authors are
able to fix easily, document the problem and suggest that minor
revisions are needed before the submission is posted. It is important
that you are not evaluating software that has been patched, while others
are working with the original submission.
You should not send the author any comments about, or evaluation of, the
submission. That information should be sent to the moderator in your
final summary, and he will collect the comments from all the reviewers
and forward them to the author if appropriate..
--
Andrew Patrick acting as Comp.Sources.Reviewed Moderator
Department of Communications, Ottawa, CANADA
csr@calvin.doc.CA