rsalz@uunet.uu.net (Rich Salz) (06/27/89)
Submitted-by: utzoo!henry
Posting-number: Volume 19, Issue 81
Archive-name: cnews2/part04
: ---CUT HERE---
echo 'notebook/newslock':
sed 's/^X//' >'notebook/newslock' <<'!'
X.DA "22 April 1989"
X.TL
XLocking in C News
X.AU
XHenry Spencer
X.AI
XDept. of Zoology
XUniversity of Toronto
X.LP
XSeveral parts of C News need some way of locking parts of the news
Xsubsystem against concurrent execution.
XVarious system-specific locking system calls exist, but none of them
Xis truly portable, and most of them provide far more functionality than
Xwe need.
X.LP
XC News locking uses the \fIlink\fR(2) system call and pre-agreed names.
X\fILink\fR has the necessary characteristic for safe locking:
Xit is an atomic test-and-set operation.
XFurthermore, it exists in all Unixes.
X.LP
XAll locks are created in the NEWSCTL directory
X(see \fIConfiguration Mechanisms in C News\fR for where this directory
Xis to be found and how programs can determine this)
Xand have names starting with `LOCK'.
XTo acquire a lock, first create a temporary file in NEWSCTL with a name
Xof the form `L.\fIn\fR', where \fIn\fR is your process id.
XYou are urged to also write your process id, in decimal ASCII, into this
Xfile.
XThen attempt to link the
Xtemporary
Xfile to `LOCK\fIx\fR', where \fIx\fR is chosen based
Xon what sort of locking you wish to do.
XExisting lock names are:
X.TS
Xcenter;
Xll.
XLOCK relaynews, modifications to control files
XLOCKinput input subsystem processing spooled input
XLOCKbatch batcher preparing batches
XLOCKexpire expire expiring articles
X.TE
XIf the link fails, sleep and try again.
XIf it succeeds, proceed.
XThe temporary file may be removed then or at the same time as the lock
Xis removed.
XPrograms are expected to make a determined effort to remove lock files
Xwhen they terminate, normally or as a result of signals.
X.LP
XShell programs have an additional problem in that System V has broken
X\fIln\fR(1) so that it removes a pre-existing destination file.
XC News therefore provides a pure, simple locking program under
Xthe name NEWSBIN/newslock (if the recommendations in
X\fIDirectory Layout and PATH in C News\fR are followed, this will
Xautomatically be in the search path of shell programs).
XUsage is `newslock\ tempfile\ lockfile';
Xexit status is 0 for success, 1 for failure, 2 for wrong number of arguments.
XNo messages are printed for normal failure, so no redirection of output
Xis needed.
X.LP
XA suitable locking procedure for a shell file using the standard
Xconfiguration facilities is:
X.DS
Xlock="$NEWSCTL/LOCKxxx" # modify name as appropriate
Xltemp="$NEWSCTL/L.$$"
Xecho $$ >$ltemp
Xtrap "rm \-f $ltemp ; exit 0" 0 1 2 15
Xwhile true
Xdo
X if newslock $ltemp $lock
X then
X trap "rm \-f $ltemp $lock ; exit 0" 0 1 2 15
X break
X fi
X sleep 30
Xdone
X.DE
XA template of this form can be found in the file \fInewslock.sh\fR.
X.LP
XAlthough there are various thorny questions associated with breaking
Xlocks by dead programs, reboot is a time when surviving locks are
Xdefinitely invalid.
X(Although there are problems even here if a networked group of systems
Xare not rebooted as a unit.)
XFor this and other reasons, a system running C News should execute
XNEWSCTL/bin/newsboot at reboot time (e.g. from \fI/etc/rc\fR).
!
echo 'notebook/config':
sed 's/^X//' >'notebook/config' <<'!'
X.DA "18 Feb 1989"
X.TL
XConfiguration Mechanisms in C News
X.AU
XHenry Spencer
X.AI
XDept. of Zoology
XUniversity of Toronto
X.SH
XIntro
X.LP
XThere is an overall problem with news stuff in that some pathnames
Xand such are site-specific.
XIn C News this hits both shell files and C programs, the former a
Xparticular inconvenience because they are not compiled and hence can't
Xeasily pick it up from a library at compile time.
XIt's easy to edit shell files, but there are too many of them for this
Xto be very convenient.
X.LP
XSeveral mechanisms are needed to solve the entire problem..
X.SH
XConfiguration Parameters
X.LP
XAlthough there are many things that theoretically could be site-specific,
Xthere are only a few that really crop up constantly,
Xare highly likely to change,
Xand have to be known to running code.
XHere's a tentative list, with internal names and common defaults:
X.TS
Xlll.
XNEWSARTS pathname of the article database /usr/spool/news
XNEWSCTL pathname of the control-file directory /usr/lib/news
XNEWSBIN pathname of the program directory /usr/lib/newsbin
XNEWSPATH shell PATH setting for news /bin:/usr/bin
XNEWSUMASK default umask for file creation 002
XNEWSMASTER where to send mail about trouble usenet
XNEWSCONFIG see "Subst" section /usr/lib/news/bin/config
X.TE
X.SH
XEnvironment Variables
X.LP
XAll C News programs
Xthat care about configuration parameters are expected to look at
Xenvironment variables with the same names as the parameters, and to
Xuse the environment values to override internal defaults.
XThis is primarily aimed at testing and special purposes rather than
Xproduction use, as there is a fundamental problem:
Xthe environment is insecure.
XThe environment variables are nevertheless very useful.
X.LP
XThere is a need for a canned interface to the environment variables,
Xto simplify their use.
XThere is also a requirement for centrally-set defaults for normal
Xproduction use.
XThese requirements are addressed differently for C and shell programs.
X.SH
XC Interface
X.LP
X\fILibcnews\fR provides a set of inquiry functions:
X.LP
X.RS
X.IP "char *fullartfile(char *name);"
XReturns a full pathname for the NEWSARTS-relative file \fIname\fR;
Xif \fIname\fR is NULL, returns the value of NEWSARTS.
X.IP "char *ctlfile(char *name);"
XReturns a full pathname for the NEWSCTL-relative file \fIname\fR;
Xif \fIname\fR is NULL, returns the value of NEWSCTL.
X.IP "char *binfile(char *name);"
XReturns a full pathname for the NEWSBIN-relative file \fIname\fR;
Xif \fIname\fR is NULL, returns the value of NEWSBIN.
X.IP "char *newspath(void);"
XReturns the value of the NEWSPATH parameter.
X.IP "int newsumask(void);"
XReturns the value of the NEWSUMASK parameter.
X.IP "char *newsmaster(void);"
XReturns the value of NEWSMASTER.
X.RE
X.LP
XFunctions which return string values return pointers to areas which they
Xmay later re-use, so the values should be copied if they are to persist
Xpast later calls.
XThere is also a function \fIartfile\fR which returns a relative pathname,
Xas a special-purpose optimization,
Xand a function \fIcd\fR which does a \fIchdir\fR and notes the location
Xfor future use by \fIartfile\fR.
X.LP
XThe first inquiry function to be invoked automatically initializes
Xtheir internal housekeeping.
XAgainst the possibility of malicious setting of environment variables,
Xif any environment variables are in fact present to override default
Xvalues, and at least one value in fact differs from the default,
Xthis initialization includes a call to a function the user must supply:
X.LP
X.RS
X.IP "void unprivileged(void);"
XDrop any special powers that should not be available to a normal user
Xprogram,
Xe.g. those obtained by set-uid bit.
X.RE
X.LP
XPrograms that do not use the set-uid bit can normally have a null
Ximplementation of \fIunprivileged\fR, but to encourage thought about
Xthe matter this is \fInot\fR provided as a default.
X.LP
XSuitable external declarations for all these functions can be found
Xin the include file \fIconfig.h\fR in the C News include directory.
X.SH
XShell interface
X.LP
XAll shell files that want to know configuration parameters should
Xexecute NEWSCTL/bin/config using the `.' command;
Xit sets shell variables with the same names as the parameters.
XNote that it does not export these variables, so subordinate shell
Xfiles need to pick up \fIconfig\fR themselves.
XThis is to preserve the property that subordinate programs see environment
Xvariables set only if the user set them.
X.SH
XSubst
X.LP
XThe alert mind will have noticed a circularity in the previous section:
Xit's not possible to find NEWSCTL/bin/config until one knows where NEWSCTL is.
XThere are also occasional annoyances in that documentation and such wants
Xto know the local defaults, and can't get them from either the C or shell
Xinterface.
XHence the use of \fIsubst\fR.
X.LP
X\fISubst\fR does substitutions into source files (including shell files)
Xin such a way that it is not necessary to maintain two versions of the file.
XIn the top-level directory of the C News sources are four files:
X\fIsubst\fR itself, the \fIsubstitutions\fR file defining the values of
Xthe configuration parameters, and \fIsubst.hs\fR and \fIsubst.gc\fR,
Xwhich are lists of files (relative to the top-level source directory)
Xthat need substitutions done.
X(The use of two `list' files reflects C News's dual authorship.)
XSee the \fIsubst\fR(1) manual page for the gory details of the syntax.
XThe only C program affected is the configuration-inquiry part of the
Xlibrary, but all shell files that need any of the configuration parameters
Xneed to be in one of the lists.
X.SH
XProgram startup
X.LP
XAs a result of all this, a C program which needs to know a configuration
Xparameter simply calls the appropriate inquiry function,
Xand is prepared for a return call to \fIunprivileged\fR.
XA shell program has to do a bit more work; it should start with something
Xon the order of:
X.DS
X#! /bin/sh
X# foobar \- does foo, bar, and bletch
X
X# =()<. ${NEWSCONFIG\-@<NEWSCONFIG>@}>()=
X. ${NEWSCONFIG\-/usr/lib/news/bin/config}
X
XPATH=$NEWSCTL/bin:$NEWSBIN/xxx:$NEWSBIN:$NEWSPATH ; export PATH
Xumask $NEWSUMASK
X.DE
X(A prototype for this is in conf/proto.sh.)
X(See an accompanying document for the logic of the PATH setting.)
XThe NEWSCONFIG configuration parameter specifies the location of the
Xconfigurer shell program, which is then executed to set up the rest
Xof the parameters.
XThe reason for doing it this way is to ensure that new parameters
Xcan be added without having to change every shell file.
!
echo 'notebook/ihave.pic':
sed 's/^X//' >'notebook/ihave.pic' <<'!'
X.lf 1 ihave.pic.in
X.LP
X... 0 0 601 509
X... 0.000i 5.082i 6.000i 0.000i
X.nr 00 \n(.u
X.nf
X.PS 5.082i 6.000i 6
X.lf 71
X\h'1.737i'\v'4.872i'\D'l3.374i -0.030i'
X.sp -1
X\h'5.012i'\v'4.868i'\D'l0.100i -0.026i'
X.sp -1
X\h'5.011i'\v'4.818i'\D'l0.100i 0.024i'
X.sp -1
X\h'3.275i'\v'4.762i'\v'.2m'\h'-\w'\f1\s10\&articles\f1\s0'u/2u'\f1\s10\&articles\f1\s0\h'-\w'\f1\s10\&articles\f1\s0'u/2u'
X.sp -1
X\h'5.042i'\v'2.166i'\v'.2m'\h'-\w'\f1\s10\&\f1\s0'u/2u'\f1\s10\&\f1\s0\h'-\w'\f1\s10\&\f1\s0'u/2u'
X.sp -1
X\h'5.431i'\v'0.519i'\v'.2m'\h'-\w'\f1\s10\&\f1\s0'u/2u'\f1\s10\&\f1\s0\h'-\w'\f1\s10\&\f1\s0'u/2u'
X.sp -1
X\h'3.314i'\v'0.100i'\v'.2m'\h'-\w'\f3\s14\&ihave/sendme processing diagram\f1\s0'u/2u'\f3\s14\&ihave/sendme processing diagram\f1\s0\h'-\w'\f3\s14\&ihave/sendme processing diagram\f1\s0'u/2u'
X.sp -1
X\h'1.348i'\v'0.399i'\v'.2m'\h'-\w'\f3\s10\&utzoo\f1\s0'u/2u'\f3\s10\&utzoo\f1\s0\h'-\w'\f3\s10\&utzoo\f1\s0'u/2u'
X.sp -1
X\h'4.183i'\v'1.927i'\D'e1.817i 0.399i'
X.sp -1
X\h'5.092i'\v'0.399i'\v'.2m'\h'-\w'\f3\s10\&utstat\f1\s0'u/2u'\f3\s10\&utstat\f1\s0\h'-\w'\f3\s10\&utstat\f1\s0'u/2u'
X.sp -1
X\h'0.210i'\v'0.689i'\v'.2m'\h'-\w'\f1\s10\&article\f1\s0'u/2u'\f1\s10\&article\f1\s0\h'-\w'\f1\s10\&article\f1\s0'u/2u'
X.sp -1
X\h'0.369i'\v'0.689i'\D'l0.459i 0.000i'
X.sp -1
X\h'0.729i'\v'0.714i'\D'l0.100i -0.025i'
X.sp -1
X\h'0.729i'\v'0.664i'\D'l0.100i 0.025i'
X.sp -1
X\h'1.288i'\v'0.639i'\v'.2m'\h'-\w'\f1\s10\&match #1 I\f1\s0'u/2u'\f1\s10\&match #1 I\f1\s0\h'-\w'\f1\s10\&match #1 I\f1\s0'u/2u'
X.sp -1
X\h'0.759i'\v'1.448i'\D'l0.000i -0.349i'
X.sp -1
X\h'0.759i'\v'1.098i'\D'l0.958i 0.000i'
X.sp -1
X\h'1.717i'\v'1.098i'\D'l0.000i 0.349i'
X.sp -1
X\h'1.717i'\v'1.448i'\D'l-0.958i 0.000i'
X.sp -1
X\h'1.288i'\v'1.208i'\v'.2m'\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'\f1\s10\&batcher on\f1\s0\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'
X.sp -1
X\h'1.288i'\v'1.318i'\v'.2m'\h'-\w'\f1\s10\&msgid file\f1\s0'u/2u'\f1\s10\&msgid file\f1\s0\h'-\w'\f1\s10\&msgid file\f1\s0'u/2u'
X.sp -1
X\h'4.542i'\v'4.203i'\D'~ -0.230i 0.000i -0.240i -0.319i -0.160i -0.240i -0.080i -0.080i'
X.sp -1
X\h'3.922i'\v'3.617i'\D'l-0.088i -0.053i'
X.sp -1
X\h'3.887i'\v'3.652i'\D'l-0.053i -0.088i'
X.sp -1
X\h'5.101i'\v'3.814i'\D'l0.080i 0.150i'
X.sp -1
X\h'5.112i'\v'3.887i'\D'l0.069i 0.076i'
X.sp -1
X\h'5.156i'\v'3.864i'\D'l0.025i 0.100i'
X.sp -1
X\h'1.917i'\v'2.686i'\D'~ 0.160i 0.000i 0.080i -0.399i 0.160i -0.240i 0.080i -0.080i'
X.sp -1
X\h'2.343i'\v'2.055i'\D'l0.053i -0.088i'
X.sp -1
X\h'2.308i'\v'2.020i'\D'l0.088i -0.053i'
X.sp -1
X\h'1.288i'\v'2.186i'\D'l-0.010i 0.260i'
X.sp -1
X\h'1.257i'\v'2.345i'\D'l0.021i 0.101i'
X.sp -1
X\h'1.307i'\v'2.347i'\D'l-0.029i 0.099i'
X.sp -1
X\h'1.238i'\v'1.448i'\D'l0.050i 0.300i'
X.sp -1
X\h'1.247i'\v'1.653i'\D'l0.041i 0.094i'
X.sp -1
X\h'1.296i'\v'1.645i'\D'l-0.008i 0.103i'
X.sp -1
X\h'1.318i'\v'1.597i'\v'.2m'\h'-\w'\f1\s10\&(deadlock avoidance)\f1\s0'u/2u'\f1\s10\&(deadlock avoidance)\f1\s0\h'-\w'\f1\s10\&(deadlock avoidance)\f1\s0'u/2u'
X.sp -1
X\h'0.349i'\v'3.564i'\D'e2.017i 0.479i'
X.sp -1
X\h'5.092i'\v'2.955i'\D'l0.010i 0.379i'
X.sp -1
X\h'5.074i'\v'3.235i'\D'l0.028i 0.099i'
X.sp -1
X\h'5.124i'\v'3.234i'\D'l-0.022i 0.100i'
X.sp -1
X\h'5.111i'\v'3.165i'\v'.2m'\h'-\w'\f1\s10\&(deadlock avoidance)\f1\s0'u/2u'\f1\s10\&(deadlock avoidance)\f1\s0\h'-\w'\f1\s10\&(deadlock avoidance)\f1\s0'u/2u'
X.sp -1
X\h'5.092i'\v'3.524i'\v'.2m'\h'-\w'\f1\s10\&match #4 F or \(es\f1\s0'u/2u'\f1\s10\&match #4 F or \(es\f1\s0\h'-\w'\f1\s10\&match #4 F or \(es\f1\s0'u/2u'
X.sp -1
X\h'5.111i'\v'4.123i'\v'.2m'\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'\f1\s10\&batcher on\f1\s0\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'
X.sp -1
X\h'5.191i'\v'4.283i'\v'.2m'\h'-\w'\f1\s10\&filenames (opt.)\f1\s0'u/2u'\f1\s10\&filenames (opt.)\f1\s0\h'-\w'\f1\s10\&filenames (opt.)\f1\s0'u/2u'
X.sp -1
X\h'2.077i'\v'1.977i'\D'l2.106i -0.050i'
X.sp -1
X\h'4.084i'\v'1.954i'\D'l0.099i -0.027i'
X.sp -1
X\h'4.083i'\v'1.904i'\D'l0.100i 0.023i'
X.sp -1
X\h'5.092i'\v'2.126i'\D'l0.000i 0.499i'
X.sp -1
X\h'5.067i'\v'2.526i'\D'l0.025i 0.100i'
X.sp -1
X\h'5.116i'\v'2.526i'\D'l-0.025i 0.100i'
X.sp -1
X\h'3.215i'\v'1.867i'\v'.2m'\h'-\w'\f1\s10\&ihave control\f1\s0'u/2u'\f1\s10\&ihave control\f1\s0\h'-\w'\f1\s10\&ihave control\f1\s0'u/2u'
X.sp -1
X\h'3.195i'\v'3.484i'\v'.2m'\h'-\w'\f1\s10\&sendme control\f1\s0'u/2u'\f1\s10\&sendme control\f1\s0\h'-\w'\f1\s10\&sendme control\f1\s0'u/2u'
X.sp -1
X\h'5.101i'\v'3.654i'\v'.2m'\h'-\w'\f1\s10\&inews to.utzoo\f1\s0'u/2u'\f1\s10\&inews to.utzoo\f1\s0\h'-\w'\f1\s10\&inews to.utzoo\f1\s0'u/2u'
X.sp -1
X\h'1.288i'\v'1.937i'\v'.2m'\h'-\w'\f1\s10\&match #2 F or \(es\f1\s0'u/2u'\f1\s10\&match #2 F or \(es\f1\s0\h'-\w'\f1\s10\&match #2 F or \(es\f1\s0'u/2u'
X.sp -1
X\h'1.288i'\v'2.057i'\v'.2m'\h'-\w'\f1\s10\&inews to.utstat\f1\s0'u/2u'\f1\s10\&inews to.utstat\f1\s0\h'-\w'\f1\s10\&inews to.utstat\f1\s0'u/2u'
X.sp -1
X\h'3.215i'\v'1.078i'\v'.2m'\h'-\w'\f1\s10\&\f1\s0'u/2u'\f1\s10\&\f1\s0\h'-\w'\f1\s10\&\f1\s0'u/2u'
X.sp -1
X\h'1.242i'\v'0.975i'\D'l0.006i 0.103i'
X.sp -1
X\h'1.291i'\v'0.985i'\D'l-0.043i 0.093i'
X.sp -1
X\h'1.248i'\v'1.078i'\D'l0.050i -0.260i'
X.sp -1
X\h'0.809i'\v'5.082i'\D'l0.000i -0.399i'
X.sp -1
X\h'0.809i'\v'4.682i'\D'l0.908i 0.000i'
X.sp -1
X\h'1.717i'\v'4.682i'\D'l0.000i 0.399i'
X.sp -1
X\h'1.717i'\v'5.082i'\D'l-0.908i 0.000i'
X.sp -1
X\h'1.288i'\v'4.782i'\v'.2m'\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'\f1\s10\&batcher on\f1\s0\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'
X.sp -1
X\h'1.278i'\v'4.932i'\v'.2m'\h'-\w'\f1\s10\&filenames\f1\s0'u/2u'\f1\s10\&filenames\f1\s0\h'-\w'\f1\s10\&filenames\f1\s0'u/2u'
X.sp -1
X\h'1.278i'\v'3.804i'\D'l-0.020i 0.879i'
X.sp -1
X\h'1.235i'\v'4.582i'\D'l0.023i 0.100i'
X.sp -1
X\h'1.285i'\v'4.583i'\D'l-0.027i 0.099i'
X.sp -1
X\h'4.213i'\v'3.574i'\D'l-1.847i -0.010i'
X.sp -1
X\h'2.466i'\v'3.540i'\D'l-0.100i 0.024i'
X.sp -1
X\h'2.466i'\v'3.590i'\D'l-0.100i -0.026i'
X.sp -1
X\h'1.358i'\v'3.484i'\v'.2m'\h'-\w'\f1\s10\&match #5 F\f1\s0'u/2u'\f1\s10\&match #5 F\f1\s0\h'-\w'\f1\s10\&match #5 F\f1\s0'u/2u'
X.sp -1
X\h'4.213i'\v'3.574i'\D'e1.777i 0.479i'
X.sp -1
X\h'1.358i'\v'3.644i'\v'.2m'\h'-\w'\f1\s10\&rnews to.utstat/sendme\f1\s0'u/2u'\f1\s10\&rnews to.utstat/sendme\f1\s0\h'-\w'\f1\s10\&rnews to.utstat/sendme\f1\s0'u/2u'
X.sp -1
X\h'4.642i'\v'2.955i'\D'l0.000i -0.329i'
X.sp -1
X\h'4.642i'\v'2.626i'\D'l0.908i 0.000i'
X.sp -1
X\h'5.551i'\v'2.626i'\D'l0.000i 0.329i'
X.sp -1
X\h'5.551i'\v'2.955i'\D'l-0.908i 0.000i'
X.sp -1
X\h'5.042i'\v'2.735i'\v'.2m'\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'\f1\s10\&batcher on\f1\s0\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'
X.sp -1
X\h'5.042i'\v'2.845i'\v'.2m'\h'-\w'\f1\s10\&msgid file\f1\s0'u/2u'\f1\s10\&msgid file\f1\s0\h'-\w'\f1\s10\&msgid file\f1\s0'u/2u'
X.sp -1
X\h'1.228i'\v'2.626i'\v'.2m'\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'\f1\s10\&batcher on\f1\s0\h'-\w'\f1\s10\&batcher on\f1\s0'u/2u'
X.sp -1
X\h'1.288i'\v'2.735i'\v'.2m'\h'-\w'\f1\s10\&filenames (opt.)\f1\s0'u/2u'\f1\s10\&filenames (opt.)\f1\s0\h'-\w'\f1\s10\&filenames (opt.)\f1\s0'u/2u'
X.sp -1
X\h'4.542i'\v'4.443i'\D'l0.000i -0.479i'
X.sp -1
X\h'4.542i'\v'3.963i'\D'l1.278i 0.000i'
X.sp -1
X\h'5.820i'\v'3.963i'\D'l0.000i 0.479i'
X.sp -1
X\h'5.820i'\v'4.443i'\D'l-1.278i 0.000i'
X.sp -1
X\h'0.639i'\v'2.925i'\D'l0.000i -0.479i'
X.sp -1
X\h'0.639i'\v'2.446i'\D'l1.278i 0.000i'
X.sp -1
X\h'1.917i'\v'2.446i'\D'l0.000i 0.479i'
X.sp -1
X\h'1.917i'\v'2.925i'\D'l-1.278i 0.000i'
X.sp -1
X\h'1.298i'\v'0.729i'\v'.2m'\h'-\w'\f1\s10\&rnews\f1\s0'u/2u'\f1\s10\&rnews\f1\s0\h'-\w'\f1\s10\&rnews\f1\s0'u/2u'
X.sp -1
X\h'0.839i'\v'0.669i'\D'e0.918i 0.300i'
X.sp -1
X\h'0.499i'\v'1.977i'\D'e1.577i 0.419i'
X.sp -1
X\h'5.062i'\v'1.857i'\v'.2m'\h'-\w'\f1\s10\&match #3 I\f1\s0'u/2u'\f1\s10\&match #3 I\f1\s0\h'-\w'\f1\s10\&match #3 I\f1\s0'u/2u'
X.sp -1
X\h'5.181i'\v'1.977i'\v'.2m'\h'-\w'\f1\s10\&rnews to.utzoo/ihave\f1\s0'u/2u'\f1\s10\&rnews to.utzoo/ihave\f1\s0\h'-\w'\f1\s10\&rnews to.utzoo/ihave\f1\s0'u/2u'
X.sp -1
X\h'5.111i'\v'1.647i'\v'.2m'\h'-\w'\f1\s10\&ihave processing\f1\s0'u/2u'\f1\s10\&ihave processing\f1\s0\h'-\w'\f1\s10\&ihave processing\f1\s0'u/2u'
X.sp -1
X\h'1.438i'\v'3.245i'\v'.2m'\h'-\w'\f1\s10\&sendme processing\f1\s0'u/2u'\f1\s10\&sendme processing\f1\s0\h'-\w'\f1\s10\&sendme processing\f1\s0'u/2u'
X.sp -1
X.sp 1+5.082i
X.PE
X.if \n(00 .fi
X.lf 71
!
echo 'notebook/ctlmsg':
sed 's/^X//' >'notebook/ctlmsg' <<'!'
X.TL
XControl Message Implementation in C News
X.AU
XGeoff Collyer
X.AI
XDepartment of Statistics
XUniversity of Toronto
X.SH
XIntroduction
X.PP
XNetnews
X.I "control messages"
Xare ordinary-looking netnews articles which contain the
Xspecial header
X.B "Control:" .
XSuch articles are filed in the pseudo-newsgroup
X.I control
Xand cause related actions by the local news system,
Xsuch as mailing a file to the poster of the control message.
X.SH
XBuilt-ins
X.PP
X.B ihave ,
X.B sendme ,
Xand
X.B cancel
Xare handled internally by
X.I relaynews ,
Xbecause processes cannot share open
X.I dbm (3)
Xdatabases,
Xthere is no standard way to close them,
Xand
Xthese control messages read or write the
X.I history
Xfiles,
Xincluding the
X.I dbm
Xfiles.
X.I Ihave
Xand
X.I sendme
Xare also permitted to have message-id arguments containing
X.B <
Xand
X.B > ,
Xboth of which are rejected by
X.I relaynews
Xin arguments to externally-implemented control messages,
Xsince they are shell metacharacters
X(\c
X.B /
Xand
X.B ..
Xare also bounced)
Xand could be indicative of an attempt to
Xdo something nasty.
X.SH
XNormal Control Messages
X.PP
XMost control messages are implemented by
X.I relaynews
Xby executing the command following
X.B Control: ,
Xwith a search path of
X.I $NEWSCTL/bin:$NEWSBIN/ctl
Xand with standard input set to
Xthe control message article.
XThe command inherits
X.I relaynews 's
Xuser and group ids,
Xtypically
X.I news ;
Xthis can be important
Xto gain access rights to control files.
XThe news system will be locked
X(by
X.I $NEWSCTL /lock)
Xwhile the command runs,
Xbecause
Xthis is often important
Xfor manipulating control files
Xfrom the command,
Xand
Xbecause
Xthe news system is locked
Xwhile
X.I relaynews
Xruns.
XIf that command returns non-zero exit status,
Xmail is sent to
X.I $NEWSMASTER
X(usually
X.B usenet ).
XStandard output and standard error
Xoften are redirected to
X.I $NEWSCTL /log
Xand
X.I $NEWSCTL /errlog
Xrespectively;
Xinvocations of
X.I relaynews
Xby
X.I inews
Xare exceptions.
!
echo 'notebook/ctlfiles':
sed 's/^X//' >'notebook/ctlfiles' <<'!'
X.DA "9 June 1989"
X.TL
XFiles in /usr/lib/news (aka NEWSCTL)
X.AU
XHenry Spencer
X.AI
XDept. of Zoology
XUniversity of Toronto
X.LP
XThe following files can appear in \fI/usr/lib/news\fR (or wherever the
Xsite's ``NEWSCTL''
Xdirectory\(emsee \fIDirectory Layout and PATH in C News\fR\(emis).
XThere may be others for the sake of local news readers, etc., but these
Xare the ones C News knows about.
X.IP "active" 15
XMajor control file:
Xlist of newsgroups recognized at this site, including current maximum
Xand minimum article numbers and moderation status.
XUpdated by \fIrelaynews\fR and \fIupact\fR.
X.IP "active.old"
XPrevious \fIactive\fR.
XCreated by \fIupact\fR.
X.IP "active.times"
XList of created newsgroups and when they were created, aimed at making it
Xpossible for news readers to be smarter about knowing when a group is new.
XNot yet supported by standard news readers.
X.IP "batchlog"
XLatest batcher log, created by \fIsendbatches\fR,
Xshowing backlogs.
X.IP "batchlog.o*"
XPrevious batcher logs.
X.IP "batchparms"
XControl file for \fIsendbatches\fR, specifying how to feed other sites.
X.IP "bin"
XMaster override directory for programs, searched before any other by
Xall C News software.
XNormally contains only the \fIconfig\fR shell file
Xspecifying where to find everything else.
X.IP "errlog"
XError log from \fIrelaynews\fR.
X.IP "errlog.o*"
XPrevious \fIerrlog\fRs.
X.IP "explist"
XControl file for \fIexpire\fR, specifying what gets expired, and when,
Xand what archiving is done.
X.IP "history"
XList of articles currently known at this site, with reception dates and
Xpathnames.
XUpdated by \fIrelaynews\fR and \fIexpire\fR.
X.IP "history.dir"
XPart of the \fIdbm\fR index for \fIhistory\fR.
X.IP "history.pag"
XOther part of the \fIdbm\fR index for \fIhistory\fR.
X.IP "history.o"
XPrevious \fIhistory\fR.
XCreated by \fIexpire\fR.
X.IP "localgroups"
XList of local groups and descriptions, for use by \fIcheckgroups\fR
Xcontrol-message handler (which uses it to decide what groups are
Xlegitimate even though the \fIcheckgroups\fR message did not mention them).
XEach line is a group name, a tab, and a terse description of the group.
XThe descriptions are read only by humans.
X.IP "log"
XLog file from \fIrelaynews\fR, reporting what was received and when.
X.IP "log.o*"
XPrevious \fIlog\fRs.
X.IP "mailname"
XName of the site for purposes of mail, typically a domainized name.
XUsed in building ``From:'' lines in newly-posted news.
XIf not present,
X``\fIhostname\fB.uucp\fR'' is assumed.
X.IP "mailpaths"
XMailing routes for submissions to moderated newsgroups.
X.IP "newsgroups"
XFile created by \fIcheckgroups\fR processing, with names and descriptions
Xof newsgroups.
XFor human reading only.
XFormat identical to that of \fIlocalgroups\fR.
X.IP "newsgroups.bac"
XPrevious \fInewsgroups\fR file.
XCreated by \fIcheckgroups\fR processing.
X.IP "organization"
XName of the organization,
Xfor \fIinews\fR's use in creating the \fIOrganization:\fR header
Xfor a posted article.
X.IP "postdefltdist"
XDefault distribution (if any; default ``world'') for \fIpostnews\fR.
X.IP "postdefltgroup"
XDefault newsgroup (if any; default is to insist on the user supplying one)
Xfor \fIpostnews\fR.
X.IP "replyusepath"
XThis file should exist if and only if the local ``mail'' command is
Xunable to handle `@' addresses,
Xmeaning that replies to control messages (etc.) must use the ``Path:''
Xline instead of ``From:''.
XThe contents are ignored.
X.IP "server"
XHost name of the news server, where all operations (posting, etc.)
Xshould be done.
XIf file does not exist, current host is assumed.
X.IP "setnewsids"
XSet-uid-root program to get user and group ids correct for \fIrelaynews\fR.
XNeeded only
Xif the system is old enough to refuse to do \fIsetuid(geteuid())\fR.
XIncluded in NEWSCTL rather than NEWSBIN because its presence is a significant
Xpolicy issue for the administrator, i.e. it is, sort of, a control file.
X.IP "sys"
XMajor control file specifying what groups are legitimate here, and what groups
Xare fed to other sites.
X.IP "watchtime"
XLast time \fInewswatch\fR was run.
X.IP "whoami"
XName of the system for news purposes, for \fInewshostname\fR.
XIf file does not exist,
Xother sources (\fIhostname\fR etc.) are consulted.
!
echo 'notebook/flow':
sed 's/^X//' >'notebook/flow' <<'!'
XThe flow of news articles through C news is as follows:
Xa batch is received by some network service and passed to rnews,
Xwhich invokes $NEWSBIN/input/newsspool to queue the batch in $NEWSARTS/in.coming;
Xeventually $NEWSBIN/input/newsrun will be run by cron,
Xit uncompresses any files in.coming and
Xeither hands the article to $NEWBIN/relay/relaynews locally or invokes
Xrelaynews over a network connection on its file server.
X
Xrelaynews writes the name of each spool file onto the ends of batch files
Xfor neighbours. Eventually cron runs $NEWSBIN/batch/sendbatches, which
Xbatches up the outgoing articles and transmits them over some network.
!
echo 'notebook/vms':
sed 's/^X//' >'notebook/vms' <<'!'
X.DA "27 May 1989"
X.TL
XC News vs. VMS
X.AU
XHenry Spencer
X.AI
XDept. of Zoology
XUniversity of Toronto
X.PP
XTo run C News at all, you need a fairly good emulation of Unix.
XThere are several such for VMS.
XThey have various minor imperfections.
XThe only one we specifically \fIknow\fR of that is a real problem for
XC News is the inability to make real links for cross-postings.
XThere is some half-hearted code in various places that tries to deal
Xwith this situation.
XIt has \fInot\fR been tested too thoroughly.
X.PP
X\fIRelaynews\fR normally files an article under its first group and then
Xmakes hard links into further groups.
XIf \fIrelaynews\fR finds itself unable to make a hard link,
Xit will try making a symbolic link instead.
XThe one situation where this will fall down is if a news article's header
Xis enormous, too big to fit in core.
XIn this case, \fIrelaynews\fR stores the article in a temporary file,
Xmakes links to it under all the appropriate names, and then unlinks the
Xtemporary name.
XThis obviously isn't going to work if the appropriate-name links are symbolic.
XWe believe this case essentially never happens on large-address-space
Xmachines, and is quite rare even for small address spaces.
X.PP
X\fIExpire\fR has a \fB\-l\fR option that tells it to consider the first
Xname of an article as the ``leader'', not to be deleted until all others
Xhave been deleted.
X.PP
XThe one place where extra work would be necessary would be \fImkhistory\fR,
Xwhich has no notion that some links are different from others.
X.PP
XSo far as we know, we don't get into any of the other trouble areas of
XUnix emulation on VMS, at least with the Eunice emulator.
XWe don't have a VMS handy for testing,
Xso we make no guarantees.
!
echo 'notebook/rfcerrata':
sed 's/^X//' >'notebook/rfcerrata' <<'!'
X.TL
XErrors in RFC 1036
X.AU
XGeoff Collyer
X.AI
XDepartment of Statistics
XUniversity of Toronto
X.SH
XIntroduction
X.PP
XRFC 1036,
Xthe standard
X.I "du jour"
Xfor the format of Usenet (netnews) messages
Xcontains significant errors,
Xenumerated below.
XReferences are made to
XRFC 850,
Xthe previous netnews message format standard.
X.SH
XHeader order insignificant
X.PP
XBetween
XRFC 850
Xand
XRFC 1036,
Xa sentence stating that the order of message headers is insignificant
Xhas fallen out of the standard.
XThis may be a reflection of the reality that
XB 2.10
Xnews did indeed care about the order
Xof
X.B From:
Xand
X.B Path: .
X.SH
X``Re:'' is only three characters
X.PP
XOne sees the contradiction
X``the four characters "Re:"''
Xrepeatedly;
Xthere should be a space after the colon.
X.SH
Xcmsg incorrectly described
X.PP
XSimilary,
XRFC 1036
Xclaims that a
X.B Subject:
Xprefix of
X``cmsg''
Xwill be interpreted as denoting a control message;
Xthe correct prefix is
X``cmsg\ ''
X(including a space).
X.SH
XXref is transmitted
X.PP
XRFC 1036
Xsays that
X.B Xref:
Xheaders should not be transmitted,
Xyet they are stored on disk as part of message headers,
Xso they will be transmitted by both B and C news.
XThe standard appears to be too strict.
X.SH
Xcancels should propagate always
X.PP
XRFC 1036
Xclaims that
X.I cancel
Xcontrol messages should stop propagating when their target messages
Xare not present;
Xit would improve the efficacy of
X.I cancel s
Xto propagate them anyway.
X.SH
Xihave/sendme not documented
X.PP
XThe description of the ihave/sendme protocol is so vague
Xas to be useless to an implementor.
XSee the
XC news
Xdocumentation for an adequate description of the protocol.
XThe description in
XRFC 1036
Xalso contains an error:
X.I remotesys
Xis not optional;
Xgiven that there may be multiple message-ids preceding it,
Xthere would be no way
X(other than ad-hocery)
Xto tell if the final argument were a message-id
Xor a
X.I remotesys .
!
echo 'notebook/README':
sed 's/^X//' >'notebook/README' <<'!'
XThis directory is the C News Implementor's Notebook, containing various
Xinternal documents describing what goes on inside. These should be read
Xcautiously, as some of them may not be 100% up to date, although we've
Xtried to keep them current.
X
XMost everything here is set up for the -ms macros, although very little
Xfancy formatting is used in general. The standard R, I, and B are the
Xonly fonts used. Conversion to another macro package should not be hard.
X
XIhave.pic.in is an illustration for ihave; ihave.pic is what you get after
Xyou run ihave.pic.in through pic (in case you have a troff that knows about
Xthe graphics commands but don't have pic); tmac.pic is extra macros for
Xuse in troffing pic stuff.
X
XConfig and newslock need tbl. Nothing here uses eqn.
!
echo 'notebook/ihave':
sed 's/^X//' >'notebook/ihave' <<'!'
X.TL
XSetting Up Netnews Feeds Using the Ihave/sendme Protocol
X.AU
XGeoff Collyer
X.AI
XDepartment of Statistics
XUniversity of Toronto
X.SH
XIntroduction
X.PP
XThe ihave/sendme protocol is a means of conserving network bandwidth
Xat the expense of some real-time delays in getting netnews.
XIt pre-dates the NetNews Transfer Protocol (NNTP)
Xfor the TCP/IP protocol stack by several years and attempts much
Xthe same optimisation,
Xbut ihave/sendme is batched,
Xunlike NNTP.
X.PP
XIhave/sendme
Xis sketched in extremely vague terms in
XARPA Internet RFC 1036 (nee 850),
Xbut the description therein is so lofty as
Xto be useless as a protocol specification.
X.SH
XInto the Breach
X.PP
XThe ihave/sendme strategy is for a site which has just received a new
Xnetnews article to send an
X.I ihave
Xnetnews control message
Xcontaining the Message-ID of the new article to each
Xof its ihave/sendme
Xneighbours;
Xthe neighbour consults its netnews
X.I history
Xfile and if it has not seen the article,
Xsends back a
X.I sendme
Xcontrol message containing the original Message-ID;
Xupon receipt of the
X.I sendme ,
Xthe first site will transmit the article via normal channels
Xto the requesting neighbour.
X.PP
XDue to the high volume of netnews,
Xsending individual articles is always a performance disaster
X(see B News,
XNNTP and unbatched ihave/sendme),
Xso the golden rule of netnews is
X``Thou shalt batch thine articles.''.
XThis makes the above sketch a little slower and more complicated:
Xnow individual articles are not sent,
Xbut batches of
X.I ihave ,
X.I sendme ,
Xand
Xnetnews messages are transmitted,
Xincurring further delay since netnews batchers are usually
Xrun only once per hour,
Xsay.
X.PP
XTo permit
X.I sys
Xfile entries to distinguish
X.I ihave s
Xfrom
X.I sendme s
Xfrom
Xordinary netnews traffic,
Xtwo distributions have been usurped for use by ihave/sendme links:
X.I ihave
Xand
X.I sendme .
XWhen
X.I relaynews
Xis processing a control message of the form
X.DS
XControl: keyword system
X.DE
Xwhere
X.I keyword
Xmay be
X.I ihave
Xor
X.I sendme ,
X.I relaynews
Xgenerates a reply
Xby finding the
X.I sys
Xfile line that will match a newsgroup of
X.B to. system
Xand a distribution of
X.B ihave
X(or
X.B sendme
Xas appropriate),
Xand sending its reply
X(a
X.I sendme
Xcontrol in the case of
X.I ihave,
Xor ordinary articles in the case of
X.I sendme )
Xto the matched channel.
X.PP
XWe will now walk through an example ihave/sendme set up between
Xtwo C news sites
X(\c
X.I utzoo
Xand
X.I utstat ),
Xwith reference to the following
X.I sys
Xfile fragments and flow diagram.
X.br
X.ne 2i
X.SH
Xutzoo's sys file
X.LP
X.DS L
X.ft B
X# for testing ihave/sendme control messages: an ihave/sendme link with utstat.
X# batcher turns batch file -> giant ihave control message & posts to to.utstat (#1)
Xutstat:rec.music.synth/all,!sendme,!ihave:I:utstat.ihave/togo
X# sends ihave & sendme messages via normal batching (or uux) (#2, #4)
Xutstat-ctl:to.utstat/all,!sendme,!ihave:f:utstat/togo
X# batcher turns batch file -> giant sendme control message & posts to to.utstat (#3)
Xutstat-send-ids:to.utstat/ihave:I:utstat.sendme/togo
X# final sending of the articles themselves (#5) (optional; can share with #2 & #4)
Xutstat-real:to.utstat/sendme:f:utstat/togo
X.ft R
X.DE
X.br
X.ne 2i
X.SH
Xutstat's sys file
X.LP
X.DS L
X.ft B
X# for testing ihave/sendme control messages: an ihave/sendme link with utzoo.
X# batcher turns batch file -> giant ihave control message & posts to to.utzoo (#1)
Xutzoo:rec.music.synth/all,!sendme,!ihave:I:utzoo.ihave/togo
X# sends ihave & sendme messages via normal batching (or uux) (#2, #4)
Xutzoo-ctl:to.utzoo/all,!sendme,!ihave:f:utzoo/togo
X# batcher turns batch file -> giant sendme control message & posts to to.utzoo (#3)
Xutzoo-send-ids:to.utzoo/ihave:I:utzoo.sendme/togo
X# final sending of the articles themselves (#5) (optional; can share with #2 & #4)
Xutzoo-real:to.utzoo/sendme:f:utzoo/togo
X.ft R
X.DE
X.so ihave.pic
X.SH
XAn Example
X.PP
XA new article arrives on
X.I utzoo
Xin newsgroup
X.I rec.music.synth
Xand is matched by
X.I sys
Xfile line
X.B "#1 on utzoo" ,
Xwhich writes the Message-ID of the article
Xonto a batch file
X(\c
X.I $NEWSARTS/out.going/utstat.ihave/togo ).
XThere are some subtleties here:
Xit would be essential to add
X.B ,!to
Xto #1's subscription list
Xif that list contained
X.B all
Xor
X.B to
Xto prevent leaking messages intended for the other
X.I sys
Xentries out #1;
Xand
Xit is always worthwhile to prevent leaking local
Xnewsgroups out one's full feeds,
Xso if #1's subscription list contained
X.B all
Xone would want to add something like
X.B ,!general
Xor
X.B ,!utstat .
XSome time later,
Xthe batcher runs on #1's batch file.
X.PP
XThe batcher on
X.I utzoo
Xprepares
Xa batch of Message-IDs as an
X.I ihave
Xcontrol message to
X.I utstat
Xand submits it to
X.I "inews -h" ,
Xwhich will match
X.I sys
Xfile line
X.B "#2 on utzoo" ,
Xwhich will typically write the file name of the
Xcontrol message
X(in the
X.I control
Xpseudo-newsgroup)
Xon the usual
X.I utstat
Xbatch file
X(\c
X.I $NEWSARTS/out.going/utstat/togo ).
X(For greater speed,
Xone can have the #2
X.I sys
Xfile line simply execute
X.I uux (1)
Xor
X.I mail (1)
Xbut this can result is a lot of short batches being sent,
Xwhich will increase
X.I rnews
Xprocessing overhead on
X.I utstat .)
X.PP
XSome time later,
Xthe batcher will run on
X.I utzoo
Xon the usual
X.I utstat
Xbatch file
Xand will produce batches,
Xincluding the
X.I ihave
Xcontrol message,
Xand transmit them to
X.I utstat
Xby means specified in
X.I $NEWSCTL/batchparms .
X.PP
XWhen the
X.I ihave
Xcontrol message arrives on
X.I utstat ,
Xit will match
X.I sys
Xfile line
X.B "#3 on utstat"
Xand have its Message-ID
Xwritten on the
X.I utzoo
X.I sendme
Xbatch file
X(\c
X.I $NEWSARTS/out.going/utzoo.sendme/togo ).
X.PP
XSome time later,
Xthe batcher will run on
X.I utstat
Xon that batch file,
Xwhich will produce a
X.I sendme
Xcontrol message
Xand submit it to
X.I "inews -h" ,
Xwhich will match
X.I sys
Xfile line
X.B "#4 on utstat" ,
Xwhich will write the name of the
X.I sendme
Xcontrol message in the
X.I control
Xpseudo-newsgroup on
Xthe usual
X.I utzoo
Xbatch file
X(\c
X.I $NEWSARTS/out.going/utzoo/togo)
X.PP
XSome time later,
Xthe batcher will run on
X.I utstat
Xon the usual
X.I utzoo
Xbatch file,
Xand will produce batches including
Xthe
X.I sendme
Xand send them to
X.I utzoo
Xby means specified in
X.I $NEWSCTL/batchparms .
X.PP
XWhen the
X.I sendme
Xcontrol message arrives on
X.I utzoo ,
Xit is matched by
X.I sys
Xfile line
X.B "#5 on utzoo" ,
Xwhich writes the file names of the articles
Xnamed by Message-ID in
Xthe
X.I sendme
Xcontrol message
Xon the usual
X.I utstat
Xbatch file.
X.PP
XSome time later,
Xthe batcher on
X.I utzoo
Xwill run and produce batches,
Xincluding the articles requested by the
X.I sendme ,
Xand will send them to
X.I utstat
Xby means specified in
X.I $NEWSCTL/batchparms .
X.SH
XComparison with B News
X.PP
XThere is a lot of activity involved in ihave/sendme,
Xand there are at least five distinct channels
Xand three transactions needed to send a batch of articles.
XB news ``simplifies'' the
X.I sys
Xfile by requiring certain
X.I sys
Xfile options (sic)
Xto be set to fixed values,
Xand
Xby overloading madly.
XThe resulting
X.I sys
Xfile
X.I is
Xshorter,
Xbut it is also much more difficult to decypher,
Xand if you should want to specify non-default options,
Xyou are out of luck.
!
echo 'notebook/ihave.pic.in':
sed 's/^X//' >'notebook/ihave.pic.in' <<'!'
X.LP
X.PS 6
Xscale=100
Xbox invis ht 509 wid 601 with .sw at 0,0
Xline -> from 174,21 to 512,24
X"\f1\s10\&articles\f1\s0" at 328,32
X"\f1\s10\&\f1\s0" at 505,292
X"\f1\s10\&\f1\s0" at 544,457
X"\f3\s14\&ihave/sendme processing diagram\f1\s0" at 332,499
X"\f3\s10\&utzoo\f1\s0" at 135,469
Xellipse ht 40 wid 182 at 510,316
X"\f3\s10\&utstat\f1\s0" at 510,469
X"\f1\s10\&article\f1\s0" at 21,440
Xline -> from 37,440 to 83,440
X"\f1\s10\&match #1 I\f1\s0" at 129,445
Xbox ht 35 wid 96 with .nw at 76,399
X"\f1\s10\&batcher on\f1\s0" at 129,388
X"\f1\s10\&msgid file\f1\s0" at 129,377
Xspline -> from 455,88\
Xto 432,88\
Xto 408,120\
Xto 392,144\
Xto 384,152
Xline -> from 511,127 to 519,112
Xspline -> from 192,240\
Xto 208,240\
Xto 216,280\
Xto 232,304\
Xto 240,312
Xline -> from 129,290 to 128,264
Xline -> from 124,364 to 129,334
X"\f1\s10\&(deadlock avoidance)\f1\s0" at 132,349
Xellipse ht 48 wid 202 at 136,152
Xline -> from 510,213 to 511,175
X"\f1\s10\&(deadlock avoidance)\f1\s0" at 512,192
X"\f1\s10\&match #4 F or \(es\f1\s0" at 510,156
X"\f1\s10\&batcher on\f1\s0" at 512,96
X"\f1\s10\&filenames (opt.)\f1\s0" at 520,80
Xline -> from 208,311 to 419,316
Xline -> from 510,296 to 510,246
X"\f1\s10\&ihave control\f1\s0" at 322,322
X"\f1\s10\&sendme control\f1\s0" at 320,160
X"\f1\s10\&inews to.utzoo\f1\s0" at 511,143
X"\f1\s10\&match #2 F or \(es\f1\s0" at 129,315
X"\f1\s10\&inews to.utstat\f1\s0" at 129,303
X"\f1\s10\&\f1\s0" at 322,401
Xline <- from 125,401 to 130,427
Xbox ht 40 wid 91 with .nw at 81,40
X"\f1\s10\&batcher on\f1\s0" at 129,30
X"\f1\s10\&filenames\f1\s0" at 128,15
Xline -> from 128,128 to 126,40
Xline -> from 422,151 to 237,152
X"\f1\s10\&match #5 F\f1\s0" at 136,160
Xellipse ht 48 wid 178 at 511,151
X"\f1\s10\&rnews to.utstat/sendme\f1\s0" at 136,144
Xbox ht 33 wid 91 with .nw at 465,246
X"\f1\s10\&batcher on\f1\s0" at 505,235
X"\f1\s10\&msgid file\f1\s0" at 505,224
X"\f1\s10\&batcher on\f1\s0" at 123,246
X"\f1\s10\&filenames (opt.)\f1\s0" at 129,235
Xbox ht 48 wid 128 with .nw at 455,112
Xbox ht 48 wid 128 with .nw at 64,264
X"\f1\s10\&rnews\f1\s0" at 130,436
Xellipse ht 30 wid 92 at 130,442
Xellipse ht 42 wid 158 at 129,311
X"\f1\s10\&match #3 I\f1\s0" at 507,323
X"\f1\s10\&rnews to.utzoo/ihave\f1\s0" at 519,311
X"\f1\s10\&ihave processing\f1\s0" at 512,344
X"\f1\s10\&sendme processing\f1\s0" at 144,184
X.PE
!
echo 'notebook/tmac.pic':
sed 's/^X//' >'notebook/tmac.pic' <<'!'
X.de PS \" start picture (bwk)
X. \" $1 is height, $2 is width in units
X.if t .sp .3
X.in (\\n(.lu-\\$2)/2u
X.ne \\$1
X..
X.de PE \" end of picture
X.in
X.if t .sp .6
X..
!
echo 'batch/README':
sed 's/^X//' >'batch/README' <<'!'
XThis is C News output batching.
X
XThe bencode stuff (including bdecode.c, which is over in ../input) may be
Xworth installing as separate programs in a public directory. crctab.c and
Xcoder.h are parts of it.
X
X"make r" runs a moderately thorough regression test.
!
echo 'batch/Makefile':
sed 's/^X//' >'batch/Makefile' <<'!'
X# Configuration stuff.
XCOPTS=-O
XCFLAGS=$(COPTS) -I../include
XLDFLAGS=
XLIBS=../libcnews.a
X# =()<NEWSARTS = @<NEWSARTS>@>()=
XNEWSARTS = /usr/spool/news
X# =()<NEWSBIN = @<NEWSBIN>@>()=
XNEWSBIN = /usr/lib/newsbin
X# =()<NEWSCTL = @<NEWSCTL>@>()=
XNEWSCTL = /usr/lib/news
X# workaround for System V make bug
XSHELL = /bin/sh
X
XOUTGOING = $(NEWSARTS)/out.going
XPGMDIR = $(NEWSBIN)/batch
X
XPGMS=batcher batchih batchsm batchsplit comp compcun nocomp viainews viauux \
X sendbatches compc7 c7encode viamail viapmail bencode compb
XDTR=Makefile batcher.c batchih batchsplit c7encode.c comp compc7 compcun \
X newsbatch.8 nocomp sendbatches viainews viauux viamail \
X bencode.c coder.h crctab.c compb viapmail
XALL = $(PGMS) batchparms
X
Xall: $(ALL)
X chmod +x $(PGMS)
X
Xbatcher: batcher.o $(LIBS)
X $(CC) $(CFLAGS) $(LDFLAGS) batcher.o $(LIBS) -o $@
X
Xc7encode: c7encode.o $(LIBS)
X $(CC) $(CFLAGS) $(LDFLAGS) c7encode.o $(LIBS) -o $@
X
Xbencode: bencode.o crctab.o $(LIBS)
X $(CC) $(CFLAGS) bencode.o crctab.o $(LIBS) -o $@
X
Xbencode.o: coder.h
X
Xbininstall: all
X -if test ! -d $(PGMDIR) ; then mkdir $(PGMDIR) ; fi
X cp $(PGMS) $(PGMDIR)
X
Xnewsinstall: batchparms
X -if test ! -d $(OUTGOING) ; then mkdir $(OUTGOING) ; fi
X -if test ! -r $(NEWSCTL)/batchparms ; then cp batchparms $(NEWSCTL) ; fi
X
Xbatchparms: Makefile
X echo '# 100KB, after compress, is 10 minutes at 1200 baud' >$@
X echo '# 20 batches is somewhat arbitrary, about 1MB per site' >>$@
X echo '# defaults: 2.11-compatible compression, transfer by uux' >>$@
X echo '#' >>$@
X echo '# site size queue builder muncher sender' >>$@
X echo '# ---- ---- ----- ------- ------- ------' >>$@
X echo '/default/ 100000 20 batcher compcun viauux' >>$@
X echo '#' >>$@
X echo '# sample ihave/sendme setup' >>$@
X echo 'louie.ihave 100000 20 batchih nocomp viainews' >>$@
X echo 'louie.sendme 100000 20 batchsm nocomp viainews' >>$@
X
Xbatchsm: batchih
X sed 's/ihave/sendme/g' batchih >$@
X
Xtest.1:
X echo 'a test' >$@
X
Xtest.2:
X echo 'another test' >$@
X
Xtest.3:
X cat batcher.c batcher.c batcher.c batcher.c batcher.c >$@
X
Xr: all test.1 test.2 test.3
X echo a 1 >togo
X echo b 1 >>togo
X echo c 1 >>togo
X echo d 1 >>togo
X echo e 1 >>togo
X echo f 1 >>togo
X echo g 1 >>togo
X echo h 3 >>togo
X echo i 20 >>togo
X echo j 999 >>togo
X rm -rf bin
X mkdir bin
X echo 'ln $$* >/dev/null 2>/dev/null' >bin/newslock
X echo 'cat' >bin/munch
X echo '( echo --- ; cat ) >>output' >bin/xmit
X echo 'echo 2 ; cat' >bin/munch2
X echo '( echo +++ ; cat ) >>output' >bin/xmit2
X echo 'echo 4' >bin/queuelen
X echo 'echo 99' >bin/spacefor
X ln batcher bin/batcher
X ln batchsplit bin/batchsplit
X chmod +x bin/*
X NEWSCTL=`pwd` ./batchsplit 1
X echo a 1 >test.good ; cmp test.good togo.1
X echo b 1 >test.good ; cmp test.good togo.2
X echo c 1 >test.good ; cmp test.good togo.3
X echo d 1 >test.good ; cmp test.good togo.4
X echo e 1 >test.good ; cmp test.good togo.5
X echo f 1 >test.good ; cmp test.good togo.6
X echo g 1 >test.good ; cmp test.good togo.7
X echo h 3 >test.good ; echo i 20 >>test.good ; cmp test.good togo.next
X echo j 999 >test.good ; cmp test.good togo
X rm togo.*
X echo `pwd`/test.1 >togo
X echo ./test.2 >>togo
X echo `pwd`/test.3 666 >>togo
X ./batcher togo >test.out
X echo "#! rnews `wc -c test.1 | awk '{print $$1}'`" >test.good
X cat test.1 >>test.good
X echo "#! rnews `wc -c test.2 | awk '{print $$1}'`" >>test.good
X cat test.2 >>test.good
X ( echo --- ; cat test.good ) >test.out1
X ( echo +++ ; echo 2 ; cat test.good ) >test.out2
X echo "#! rnews `wc -c test.3 | awk '{print $$1}'`" >test.temp
X cat test.3 >>test.temp
X cat test.temp >>test.good
X ( echo --- ; cat test.temp ) >>test.out1
X cmp test.good test.out
X ./batcher -d `pwd` togo >test.out
X cmp test.good test.out
X rm test.out togo
X rm -rf out.going batchparms test.out test.good
X mkdir out.going out.going/foo out.going/bar
X echo 'foo 500 10 batcher munch xmit' >batchparms
X NEWSARTS=`pwd` NEWSCTL=`pwd` ./sendbatches -d >test.out
X echo foo >test.good
X cmp test.out test.good
X echo '/default/ 10000 5 batcher munch2 xmit2' >>batchparms
X NEWSARTS=`pwd` NEWSCTL=`pwd` ./sendbatches -d | sort >test.out
X ( echo foo ; echo bar ) | sort >test.good
X cmp test.out test.good
X NEWSARTS=`pwd` NEWSCTL=`pwd` ./sendbatches -d bletch >test.out
X echo bletch >test.good
X cmp test.out test.good
X >batchlog
X >batchlog.o
X NEWSARTS=`pwd` NEWSCTL=`pwd` ./sendbatches
X test ! -s out.going/foo/output
X test ! -s out.going/bar/output
X echo `pwd`/test.1 200 >out.going/foo/togo
X echo `pwd`/test.2 200 >>out.going/foo/togo
X echo `pwd`/test.3 200 >>out.going/foo/togo
X echo `pwd`/test.1 4000 >out.going/bar/togo
X echo `pwd`/test.2 4000 >>out.going/bar/togo
X echo `pwd`/test.3 4000 >>out.going/bar/togo
X echo `pwd`/test.3 4000 >test.left
X NEWSARTS=`pwd` NEWSCTL=`pwd` ./sendbatches
X cmp out.going/foo/output test.out1
X cmp out.going/bar/output test.out2
X test ! -s out.going/foo/togo
X test ! -s out.going/bar/togo
X cmp test.left out.going/bar/togo.2
X sed 1d batchlog.o >test.log
X test ! -s test.log
X sed 1d batchlog >test.log
X echo 'bar backlog 1 (batches flowing)' >test.good
X cmp test.good test.log
X rm -rf test.* togo out.going bin batchparms batchlog batchlog.*
X
Xdtr: $(DTR)
X makedtr $(DTR) >$@
X
Xclean:
X rm -rf out.going bin
X rm -f *.o test.* togo togo.* batchparms batcher batchsm
X rm -f batchlog batchlog.* c7encode bencode
!
echo 'batch/viauux':
sed 's/^X//' >'batch/viauux' <<'!'
X#! /bin/sh
X# Transmit batch to $1. -r for no immediate call, -gd to specify grade
X# of traffic as fairly low. You may want to add -z to suppress return of
X# exit-status-0 report, depending on what uucp you run.
X#
X# The 'exec' cuts down the number of processes active for this simple case.
X
Xexec uux - -r -gd $1!rnews
!
echo 'batch/comp':
sed 's/^X//' >'batch/comp' <<'!'
X#! /bin/sh
X# Invoke compress, without silly 2.11-compatible header.
X# 12-bit compression is the lowest common denominator among news sites,
X# and is often almost as good as the much-more-costly 16-bit compression.
X
Xcompress -b 12
Xstatus=$?
Xcase "$status"
Xin
X 2)
X status=0 # compress stupidity
X ;;
Xesac
Xexit $status
!
echo 'batch/batchsplit':
sed 's/^X//' >'batch/batchsplit' <<'!'
X#! /bin/sh
X# Prepare some batches of size $1 in files named togo.[0-9] . We prepare a
X# total of 7 to stay within awk's limits on file descriptors (we need a
X# couple of other descriptors). Source is file togo.next if it exists, or
X# togo if not. If there is no togo.next and there is more in togo than will
X# fit in the numbered batches, put the next few lots in togo.next. This
X# avoids the need to paw through the whole togo file every time when a large
X# backlog has built up.
X#
X# Buglet: does not count the "#! rnews nnnnn" headers in sizes.
X#
X# If the togo file does not contain file sizes, we make an arbitrary guess
X# at an average size.
X
Xcase $#
Xin
X 0)
X echo 'Usage: batchsplit size' >&2
X exit 2
X ;;
Xesac
X
X# =()<. ${NEWSCONFIG-@<NEWSCONFIG>@}>()=
X. ${NEWSCONFIG-/usr/lib/news/bin/config}
X
XPATH=$NEWSCTL/bin:$NEWSBIN/batch:$NEWSBIN:$NEWSPATH ; export PATH
Xumask $NEWSUMASK
X
X# Locking.
Xlock="$NEWSCTL/LOCK"
Xltemp="$NEWSCTL/L.$$"
Xecho $$ >$ltemp
Xtrap "rm -f $ltemp ; exit 0" 0 1 2 15
Xwhile true
Xdo
X if newslock $ltemp $lock
X then
X trap "rm -f $ltemp $lock ; exit 0" 0 1 2 15
X break
X fi
X sleep 30
Xdone
X
Xrm -f togo.overflow togo.count
Xif test -s togo.next
Xthen
X input=togo.next
Xelse
X input=togo
Xfi
X
Xawk 'BEGIN { total = 0 ; ninbatch = 0 ; bno = 1 ; limit = '$1'
X batch = "togo." bno ; nbatches = 7 }
X {
X if (NF == 1)
X size = 3000 # Arbitrary guess.
X else
X size = $2
X if (total + size > limit && ninbatch > 0) {
X # Go to next batch.
X bno++
X if (bno <= nbatches) {
X batch = "togo." bno
X ninbatch = 0
X } else if (bno == nbatches+1 && FILENAME == "togo") {
X batch = "togo.next"
X limit = 4 * nbatches * limit
X } else {
X print NR - 1 >"togo.count"
X exit
X }
X total = 0
X }
X ninbatch++
X total += size
X print >batch
X }' $input
X
Xif test -s togo.count
Xthen
X sed "1,`cat togo.count`d" $input >togo.overflow
X rm togo.count
Xfi
X
Xif test -r togo.overflow
Xthen
X mv togo.overflow $input
Xelse
X >$input
Xfi
!
echo 'batch/bencode.1':
sed 's/^X//' >'batch/bencode.1' <<'!'
X.TH BENCODE 1 "9 January 1988"
X.UC 4
X.SH NAME
Xbencode, bdecode \- encode a binary file into printable ASCII; decode it back
Xinto binary.
X.SH SYNOPSIS
X.B bencode [source]
X.br
X.B bdecode [source]
X.SH DESCRIPTION
X.I Bencode
Xis used to a encode a binary file into printable ASCII, and
X.I bdecode
Xis used to decode an encoded file back into binary.
XThese functions are typically used to package a binary file for mailing.
X.PP
X.I Bencode
Xtakes the named source file (the default is standard input) and
Xproduces an encoded version on the standard output.
XThe encoding uses only the ASCII characters ``A'' \- ``Z'', ``a'' \- ``z'', ``0''
X\- ``9'', ``+'', and ``-''.
XThe ASCII characters blank, newline, and ``/'' also appear in the
Xencoded file, but do not represent encoded bits.
XThe encoded file is terminated with
Xa byte count and cyclic redundancy check for detecting corrupted files.
X.PP
X.I Bdecode
Xreads a file encoded by bencode (the default is standard input), strips off
Xany leading and trailing lines added by mailers,
Xand writes the decoded version to standard output.
X.PP
XThese functions are similar to
X.I uuencode
Xand
X.I uudecode,
Xbut are more robust because of the CRC check and because
Xthey don't send characters like ``^'' and ``\\'', which are likely to get
Xmangled if the file should happen to pass through a
Xnon-ASCII machine.
X.PP
X.SH SEE\ ALSO
Xuuencode(1)
X.SH AUTHORS
XKen Lalonde and Reg Quinton
X.SH BUGS
XThe file is expanded by 35% (3 bytes become 4 plus control information)
Xcausing it to take longer to transmit.
!
echo 'batch/compc7':
sed 's/^X//' >'batch/compc7' <<'!'
X#! /bin/sh
X# like comp except use the bizarre c7unbatch format
X
X# =()<. ${NEWSCONFIG-@<NEWSCONFIG>@}>()=
X. ${NEWSCONFIG-/usr/lib/news/bin/config}
X
XPATH=$NEWSCTL/bin:$NEWSBIN/batch:$NEWSBIN:$NEWSPATH ; export PATH
Xumask $NEWSUMASK
X
Xecho '#! c7unbatch'
Xcompress -b 12 | c7encode
Xexit 0 # compress exit status sometimes stupid
!
echo done
--
Please send comp.sources.unix-related mail to rsalz@uunet.uu.net.
Use a domain-based address or give alternate paths, or you may lose out.