sources-request@mirror.UUCP (07/14/86)
Submitted by: Rich $alz for Chris Lewis
Mod.sources: Volume 6, Issue 53
Archive-name: newbatchB
[ This is the second of two batched news utility programs (BNUP, to use
an acronym). It was written on a BSD4.3 machine, but it looks like
it would be fairly easy to convert to USG. Chris Lewis had posted
this to the net shortly before he left it, and I cleaned up the
documentation and am publishing it here. Both this program and
the previous one have some nice features; I look forward to the
next one that combines the best of both methods, portably. --r$ ]
#!/bin/sh
# This is a shell archive. Remove anything before this line,
# then unpack it by saving it in a file and typing "sh file".
# Contents: README batcher.1 batcher.ctrl batcher.c Makefile
echo x - README
sed 's/^XX//' > "README" <<'@//E*O*F README//'
XXThis program is a ``posthumous'' publication of softare Chris Lewis
XXposted to the net. I felt it was worthwhile, and decided to clean up
XXthe documentation a bit and publish it in mod.sources. This README,
XXthe manpage, and the sample control file come from Chris's original
XXposting, combined with additional information I gleaned from the source.
XXRead carefully -- there might be errors in the documentation.
XX /Rich $alz,
XX Moderator of mod.sources
XXNews admins! Sick of having millions of cron entries, differing
XX[cs7]sendbatch scripts, and blown out spool areas with your downstream
XXfeeds? Well, have I got something for you!
XXThe "batcher" program which works with 2.10.2 and 2.10.3 news, solves
XXall of these problems (if your're lucky!). You install it in
XX/usr/lib/news along with a batch control file, "batcher.ctrl". The
XXcontrol file contains one line per down-stream site which describes how
XXto send news to that site.
XXTo install, read these next few paragraphs and edit the source as
XXappropriate.
XX +Batching instructions for all downstream sites is contained in one
XX file, instead of having to make customized *sendbatches (e.g.,
XX differing bits in compress, different grades, different batch
XX sizes).
XX +You can speicfy the batch size for each site, tuning for various
XX link qualities.
XX +If you have 4.3BSD UUCP, you can completely eliminate the chance
XX of filling up your spool area. The control file lets you specify
XX the number of bytes allowable in the UUCP queue for each system;
XX the real batching is not invoked for a site until they make contact
XX and bring the queue size back down below the limit.
XX This is done by 'popen("uuq -l -s<sitename>", "r")' to find out the
XX size of the queue for the site. Non-4.3 sites can probably write
XX their own routine to scan UUCP "C." files to figure out how big the
XX queue is for a specific system. If you don't have this uuq command,
XX #undef UUQ in the source.
XX +If you are running on a 4.xBSD system, batcher will also not invoke
XX the real batching software if there is less than 5Mb free on the
XX UUCP spool file system. This is done by doing "df /usr/spool/uucp"
XX which I believe works only on 4.[23]BSD systems. If you can't do
XX this, #undef DFABLE in the source. It should be fairly easy to
XX kludge something up for other sites, however.
XXNext, modify your news sys file so that each system you want to control
XXthis way has batching turned on:
XX mnetor:....:F:/usr/spool/batch/mnetor
XXAdd entries for those sites in /usr/lib/news/batcher.ctrl. Information
XXon the control file is given in the manpage, and in the sample file
XXdistributed.
XXFinally, via cron, run batcher setuid news with either the name of a
XXsite to feed, or a "class" (which'll run all sites with that class).
XXThis software has made an incredible difference on this site. We give
XXfull feeds to 7 sites, plus smaller ones to about another ten. Until
XXthis was installed, we had to disable batching to one site or another
XXevery second day. And we blew spool areas on weekends. We haven't had
XXto touch *anything* since this was installed. Even when sites have
XXbeen unreachable for weeks on end.
XXChris Lewis
XX formerly at mnetor!clewis,
XX might now be reachable at utzoo!spectrix!clewis
@//E*O*F README//
chmod u=rw,g=rw,o=rw README
echo x - batcher.1
sed 's/^XX//' > "batcher.1" <<'@//E*O*F batcher.1//'
XX.TH BATCHER 1 LOCAL
XX.SH NAME
XXbatcher \- a new batching system for news
XX.SH SYNOPSIS
XX.B batcher
XX[
XX.IB \-c X
XX]
XX.br
XX.B batcher
XX[
XX.I site
XX]
XX.IR "(typically run out of " cron ".)"
XX.SH DESCRIPTION
XX.I Batcher
XXis designed to feed batched USENET neighbors in a controlled, orderly fashion.
XXIn particular, it is designed to replace multiple [cs7]sendbatch scripts,
XXand put an upper limit on the amount of data that may be queued for a site.
XXAt some sites, it is also possible to perform checks that will prevent
XXoverflowing the filesystem that contains the UUCP spool directory.
XX.PP
XX.I Batcher
XXreads the file
XX.RI NEWSLIB/batch.ctrl
XXto determine how to send data to each site.
XXThis control file contains one line per down\-stream site that describes how
XXto send news to that site.
XX.I Batcher
XXitself is run out of
XX.I cron
XXas necessary, setuid news.
XXIt should be given a single argument, either the name of a site to batch the
XXnews for, or a class
XX.RI ( batcher
XXwill then do work on all sites in that class).
XX.PP
XXIn the control file, lines beginning with a pound sign (``#'') are ignored.
XXData lines, one per site being fed, are comprised of seven colon\-separated
XXfields:
XX.RS
XXsite:class:compress:uux_flags:b_size:q_size:uux_command
XX.RE
XX.TP site
XXThe name of the site being fed. This is the same as the name in the news
XX.I sys
XXfile.
XXBy using the ``F'' protocol in news, the file
XX.IR /usr/spool/batch/ site
XXshould contain the full\-pathname list of articles intended for that site.
XX.TP class
XXA single letter; the default is ``z.'' Executing ``batcher \-cX'' works on all
XXsites in class ``X.''
XX.TP compress
XXThis is the command line to use to call
XX.IR compress ;
XXthe default is no compression.
XXIf something is specified here, it is used as an intermediate pipe, and the
XX``uux'' field defaults to ``uux\ ...\ !cunbatch.''
XX.TP uux_flags
XXThese are the flags to insert on the uux command line. As distributed, he
XXdefault is ``\-r\ \-z\ \gd'' (just queue work, only acknowledge failures,
XXgrade d transmission).
XX.TP b_size
XXThe size of batches, default is 100K.
XXThis is passed on to the standard
XX.I batcher
XXprogram.
XX.TP q_size
XXThe maximum UUCP queue size, default is 500K. As distributed, the program
XXparses the output of the 4.3BSD command ``uuq\ \-l\ \-sSITE.''
XXIf the queue is currently greater than ``q_size,'' no work is done for
XXthe site until it drops below that number.
XX.TP uux_command
XXThis can be a complete replacement for the uux command. The default is
XX``uux\ \-\ <uuxflags>\ site!<unbatch|cunbatch>.''
XX.PP
XXTo use
XX.IR batcher ,
XXmodify your news
XX.I sys
XXfile so that all (most?) (some??) sites use the batching protocol, e.g.:
XX.RS
XXmnetor:net.flame,net.rumor,net.bizarre:F:/usr/spool/batch/mnetor
XX.RE
XXNote that the batch filename must be the same name as the site being fed.
XXNext edit the control file, placing sites in classes and with queue sizes
XXthat seem appropriate.
XXThen, edit
XX.I /usr/lib/crontab
XXto replace calls to the old batching programs with calls to
XX.IR nbatcher .
XX(It should be running setuid news.)
XXYou will probably want to use both ``job class'' calls and explicitly\-named
XXsites.
XX.SH BUGS
XXQueue and spool\-fillage checking is done via
XX.I uuq
XXand
XX.IR df ,
XXrespectively. This is somewhat slow, but it is swamped by the execution
XXof the batching itself.
@//E*O*F batcher.1//
chmod u=rw,g=rw,o=rw batcher.1
echo x - batcher.ctrl
sed 's/^XX//' > "batcher.ctrl" <<'@//E*O*F batcher.ctrl//'
XX# SAMPLE BATCHER.CTRL FILE
XX#
XX# Lines beginning with pound signs are ignored. Data lines look like;
XX# site:class:compress:uux_flags:b_size:q_size:uux_command
XX# Where
XX# Site = Name of downstream neighbor
XX# Class = System class; "batcher -cX" does all sites in class X.
XX# Default class is z.
XX# Compress = Anything here is intermediate pipe to the uux, also
XX# changes default uux_command (see below).
XX# Uux_flags = Put in uux command if not given in field seven.
XX# Default is "-r -z -gd" (see below).
XX# B_size = Size of batches; default is 100K.
XX# Q_size = Size of UUCP queue; no work is done while # bytes
XX# in queue is greater than this.
XX# Uux_command = Complete replacement for uux command. Default is
XX# "uux - <uuxflags> site!<unbatch|cunbatch>" (see
XX# above).
XX# If a name does not exist, batcher won't send a batch.
XX#
XX# SAMPLE ENTRIES
XX#
XX# NORMAL NEWS FEEDS
XXgenat::compress -C::::
XXutcs::compress -C::::
XXradha::compress::::
XXtoram::compress:-n -r -gd:::
XXyetti:z:compress -C::::
XXlsuc::compress -C -b 13::::
XXmicomvax::::::
XX#
XX# NEW NEWS FEED COMING UP
XXsyntron::compress -C::::
XX#
XX# MOT NEWS FEEDS
XXmotsj1:A::-n -r -gd:::
XXmot:A::-n -r -gd:::
XX#
XX# "-cX" NEWS FEEDS
XXcxhq:A::-n -gB:::
XXcxsea:A::-n -gB:::
XXcxmso:A:compress -C:-n -gB:::
XXcxsch:A:::20000::mail cxsch!ewa
XXcxphx:A:::20000::mail cxphx!nms
XXcxmd:A:::20000::mail cxmd!joe
XXtest:A:::::mail clewis
@//E*O*F batcher.ctrl//
chmod u=rw,g=rw,o=rw batcher.ctrl
echo x - batcher.c
sed 's/^XX//' > "batcher.c" <<'@//E*O*F batcher.c//'
XX/* Chris Lewis, June 1986 */
XX#include <stdio.h>
XX#if defined(BSD4_2) || defined(BSD4_1C)
XX#include <strings.h>
XX#else
XX#include <string.h>
XX#endif
XXextern char *malloc();
XXextern void free();
XX#define NUMFLDS 7
XX#define NAME 0
XX#define CLASS 1
XX#define COMPRESS 2
XX#define UUXFLAGS 3
XX#define SIZE 4
XX#define QSIZE 5
XX#define UUX 6
XX#define UUQ /* define this if you have BSD 4.3 uuq utility */
XX#define DFABLE /* define this if "df /usr/spool/uucp" works on your system */
XX/* Take a look at these, too. */
XX#ifdef DEBUG
XX#define BATCHCTRL "batcher.ctrl"
XX#else
XX#define BATCHCTRL "/usr/lib/news/batcher.ctrl"
XX#endif
XX#define BATCH "/usr/spool/batch"
XX#define SBATCH "/usr/lib/news/batch"
XXint fldlen[NUMFLDS] = {10, 2, 20, 10, 7, 10, 30};
XXstruct desc {
XX char *flds[NUMFLDS];
XX struct desc *next;
XX} *head = (struct desc *) NULL,
XX *dptr = (struct desc *) NULL;
XX#if defined(BSD4_2) || defined(BSD4_1C)
XX#define strchr index
XX#endif
XXstruct desc *getflds();
XXint verbose = 0;
XXint class;
XXint spoolok = 1;
XXlong spoollim = 5000000;
XX/*
XX * main:
XX * - process arguments
XX * - read control file
XX * - for each system selected, see if there is any work,
XX * if so, go try to do it.
XX */
XXmain(argc, argv)
XXint argc;
XXchar **argv; {
XX register char *p;
XX register struct desc *curptr;
XX argc--; argv++;
XX for (;argc > 0 && **argv == '-'; argv++) {
XX for (p = (*argv)+1; *p; p++)
XX switch(*p) {
XX case 'v':
XX verbose = 1;
XX break;
XX case 'c':
XX class = *(p+1);
XX if (class == 0)
XX class = 'z';
XX else
XX p++;
XX break;
XX default:
XX fprintf(stderr, "batcher: Bad arg %s\n", *argv);
XX exit(1);
XX }
XX }
XX readctrl();
XX if (!checkspool()) {
XX exit(0);
XX }
XX if (verbose)
XX dumpctrl();
XX if (class) {
XX for (curptr = head; curptr && spoolok; curptr = curptr->next) {
XX if (*(curptr->flds[CLASS]) == class &&
XX (chkbatch(curptr->flds[NAME], "") ||
XX chkbatch(curptr->flds[NAME], ".work")) &&
XX (spoolok = checkspool()))
XX doit(curptr->flds[NAME]);
XX }
XX }
XX while(*argv && spoolok) {
XX if (chkbatch(*argv, "") || chkbatch(*argv, ".work")) {
XX doit(*argv);
XX }
XX argv++;
XX }
XX exit(0);
XX}
XX/*
XX * readctrl:
XX *
XX * Each line in the batch.ctrl file contains NUMFLDS colon-separated
XX * parameters. This function reads each line, and calls getflds
XX * to separate out the parameters. If getflds returns a system descriptor
XX * it is linked into the list of system descriptors.
XX */
XXreadctrl() {
XX char buf[BUFSIZ];
XX register char *p;
XX register struct desc *ptr;
XX struct desc *getflds();
XX register FILE *ctrl = fopen(BATCHCTRL, "r");
XX if (!ctrl) {
XX fprintf(stderr, "batcher: could not open %s file\n",
XX BATCHCTRL);
XX exit(1);
XX }
XX while (fgets(buf, sizeof(buf), ctrl)) {
XX if (buf[0] == '#')
XX continue;
XX p = buf + strlen(buf) - 1;
XX if (*p == '\n')
XX *p = '\0';
XX if ((ptr = getflds(buf)) && processctrl(ptr)) {
XX if (!head)
XX head = dptr = ptr;
XX else {
XX dptr->next = ptr;
XX dptr = ptr;
XX }
XX ptr->next = (struct desc *) NULL;
XX }
XX }
XX fclose(ctrl);
XX}
XX/*
XX * dumpctrl:
XX *
XX * If verbose is on, dump the tables
XX */
XXdumpctrl() {
XX register struct desc *p;
XX register int i;
XX for (p = head; p; p = p->next) {
XX for (i = 0; i < NUMFLDS; i++)
XX printf("%-*s", fldlen[i], p->flds[i]);
XX printf("\n");
XX }
XX}
XXchar *strsave();
XX/*
XX * getflds:
XX *
XX * This routine parses a single line from the batch.ctrl file,
XX * and if successfully parsed and checked out, returns a system
XX * descriptor pointer
XX */
XXstruct desc *
XXgetflds(buf)
XXchar *buf; {
XX register int cnt;
XX char b2[100];
XX char *curp, *p;
XX int len;
XX struct desc *dptr;
XX dptr = (struct desc *) malloc(sizeof (struct desc));
XX if (!dptr) {
XX fprintf(stderr, "batcher: Cannot malloc\n");
XX exit(1);
XX }
XX curp = buf;
XX for (cnt = 0; cnt < NUMFLDS; cnt++) {
XX if (cnt == (NUMFLDS - 1)) {
XX if (strchr(curp, ':')) {
XX fprintf(stderr, "batcher: too many colons: %s\n",
XX buf);
XX free(dptr);
XX return(NULL);
XX }
XX p = curp + strlen(curp);
XX } else
XX p = strchr(curp, ':');
XX if (p == NULL) {
XX fprintf(stderr, "batcher: invalid format (%d): %s\n",
XX cnt, buf);
XX free(dptr);
XX return(NULL);
XX }
XX len = p - curp;
XX if (len >= fldlen[cnt]) {
XX fprintf(stderr, "batcher: field %d too long: %s\n",
XX cnt+1, buf);
XX free(dptr);
XX return(NULL);
XX }
XX if (!(dptr->flds[cnt] = malloc(len + 1))) {
XX fprintf(stderr, "batcher: cannot malloc\n");
XX exit(1);
XX }
XX strncpy(dptr->flds[cnt], curp, len);
XX dptr->flds[cnt][len] = '\0';
XX curp = p + 1;
XX }
XX return(dptr);
XX}
XX/*
XX * strsave:
XX * returns pointer to malloc'd copy of argument
XX */
XXchar *
XXstrsave(s)
XXchar *s; {
XX register char *p = malloc(strlen(s) + 1);
XX if (!p) {
XX fprintf(stderr, "batcher: cannot malloc\n");
XX exit(1);
XX }
XX strcpy(p, s);
XX return(p);
XX}
XX/*
XX * chkbatch:
XX *
XX * return 1 if a batcher work file <batchdir>/<name><type> exists.
XX */
XXchkbatch(name, type)
XXchar *name;
XXchar *type; {
XX char batch[BUFSIZ];
XX sprintf(batch, "%s/%s%s", BATCH, name, type);
XX if (access(batch, 04) == 0)
XX return(1);
XX else
XX return(0);
XX}
XX/*
XX * doit:
XX *
XX * This routine is called with the name of the system that has
XX * been determined to have work for it. The system is searched
XX * for in the system descriptors. If found, a "system" line
XX * is contructed from the tables, and executed if system has not
XX * exceeded it's UUCP queue limit.
XX */
XXdoit(name)
XXchar *name; {
XX char cmdbuf[BUFSIZ];
XX int rc;
XX long queuesize;
XX long checkqueue(), checkspool();
XX long queuemax;
XX register struct desc *ptr;
XX if (verbose)
XX printf("batcher: doing %s\n", name);
XX for (ptr = head; ptr; ptr = ptr->next)
XX if (!strcmp(ptr->flds[NAME], name)) {
XX /* form the command line for batching */
XX sprintf(cmdbuf, "%s %s/%s %s",
XX SBATCH, BATCH, name, ptr->flds[SIZE]);
XX if (*(ptr->flds[COMPRESS]))
XX sprintf(cmdbuf + strlen(cmdbuf), "|%s",
XX ptr->flds[COMPRESS]);
XX /* Find the queue limit */
XX sprintf(cmdbuf + strlen(cmdbuf), "|%s", ptr->flds[UUX]);
XX if (1 != sscanf(ptr->flds[QSIZE], "%ld", &queuemax)) {
XX fprintf(stderr, "batcher: bad qmax field: %s\n",
XX ptr->flds[QSIZE]);
XX return;
XX }
XX queuesize = checkqueue(ptr->flds[NAME]);
XX rc = 0;
XX /* While we haven't exceeded the queue limit &
XX there's work to do, issue the command */
XX while (queuesize < queuemax && !rc &&
XX (chkbatch(ptr->flds[NAME], "") ||
XX chkbatch(ptr->flds[NAME], ".work")) &&
XX (spoolok = checkspool())) {
XX#ifdef DEBUG
XX printf("batcher: cmd: %s\n", cmdbuf);
XX rc = 1;
XX#else
XX rc = system(cmdbuf);
XX queuesize = checkqueue(ptr->flds[NAME]);
XX#endif
XX }
XX return;
XX }
XX fprintf(stderr, "batcher: no control line for %s\n", name);
XX}
XX/*
XX * processctrl:
XX *
XX * Check validity of batch.ctrl entries and supply defaults.
XX */
XXprocessctrl(ptr)
XXstruct desc *ptr; {
XX char buf[100];
XX register char *p, *uuxflags;
XX if (!ptr) return;
XX if (strlen(ptr->flds[NAME]) == 0) {
XX fprintf(stderr, "batcher: null system name\n");
XX free(ptr);
XX return(0);
XX }
XX if (strlen(ptr->flds[QSIZE]) == 0) {
XX free(ptr->flds[QSIZE]);
XX ptr->flds[QSIZE] = strsave("500000");
XX }
XX if (strlen(ptr->flds[CLASS]) == 0) {
XX free(ptr->flds[CLASS]);
XX ptr->flds[CLASS] = strsave("z");
XX }
XX if (strlen(ptr->flds[SIZE]) == 0) {
XX free(ptr->flds[SIZE]);
XX ptr->flds[SIZE] = strsave("100000");
XX }
XX if (strlen(ptr->flds[UUXFLAGS]) == 0) {
XX free(ptr->flds[UUXFLAGS]);
XX ptr->flds[UUXFLAGS] = strsave("-r -z -gd");
XX }
XX if (strlen(ptr->flds[UUX]) == 0) {
XX sprintf(buf, "uux - %s %s!%s", ptr->flds[UUXFLAGS],
XX ptr->flds[NAME],
XX *ptr->flds[COMPRESS] ? "cunbatch" : "rnews");
XX ptr->flds[UUX] = strsave(buf);
XX }
XX return(1);
XX}
XX/*
XX * checkqueue:
XX *
XX * Logically, all this code does is return the size of the UUCP queue
XX * for system "name".
XX * I've taken the easy way out and popen'd "uuq" (4.3 BSD UUCP utility)
XX * to parse the first line, which looks something like this:
XX *
XX * <systemname>: <n> jobs, <nnnn> bytes, ....
XX *
XX * I merely look for the first comma, and sscanf the number following.
XX * A proper solution would be to dive in and parse the UUCP queues
XX * themselves, but: it's moderately difficult, and it changes from
XX * system to system.
XX */
XXlong
XXcheckqueue(name)
XXchar *name; {
XX#ifdef UUQ
XX char buf[BUFSIZ];
XX long retval;
XX register char *p2;
XX FILE *p, *popen();
XX /* Gross, but the easiest way */
XX sprintf(buf, "uuq -l -s%s", name);
XX p = popen(buf, "r");
XX if (!fgets(buf, sizeof(buf), p)) {
XX return(0);
XX }
XX pclose(p);
XX p2 = strchr(buf, ',');
XX if (p2 && 1 == sscanf(p2+1, "%ld", &retval)) {
XX return(retval);
XX }
XX fprintf(stderr, "batcher: could not interpret %s\n", buf);
XX return(10000000);
XX#else
XX return(10000000);
XX#endif
XX}
XX/* This function returns the amount of free space on the spool
XX device, this may not work on your system - it reads the
XX second line from a "df /usr/spool/uucp" */
XX#define SPOOL "/usr/spool/uucp"
XXcheckspool() {
XX#ifdef DFABLE
XX char buf[100];
XX FILE *p, *popen();
XX long val;
XX sprintf(buf, "df %s", SPOOL);
XX if (!(p = popen(buf, "r"))) {
XX fprintf(stderr, "batcher: couldn't popen %s\n", buf);
XX return(0);
XX }
XX if (!fgets(buf, sizeof(buf), p)) {
XX fprintf(stderr, "batcher: no first line in df\n");
XX return(0);
XX }
XX if (!fgets(buf, sizeof(buf), p)) {
XX fprintf(stderr, "batcher: no second line in df\n");
XX return(0);
XX }
XX if (1 != sscanf(buf, "%*s %*ld %*ld %ld", &val)) {
XX fprintf(stderr, "batcher: couldn't get size from %s\n", buf);
XX return(0);
XX }
XX if (pclose(p)) {
XX fprintf(stderr, "batcher: DF failed\n");
XX return(0);
XX }
XX val *= 1024;
XX if (val < spoollim) {
XX printf("batcher: spool limit exceeded, %ld bytes left\n", val);
XX return(0);
XX } else
XX return(1);
XX#else
XX return(1);
XX#endif
XX}
@//E*O*F batcher.c//
chmod u=rw,g=rw,o=rw batcher.c
echo x - Makefile
sed 's/^XX//' > "Makefile" <<'@//E*O*F Makefile//'
XXDEFS=-DBSD4_2
XXCFLAGS = -O $(DEFS)
XXbatcher: batcher.c
XX cc -o batcher $(CFLAGS) batcher.c
@//E*O*F Makefile//
chmod u=rw,g=rw,o=rw Makefile
echo Inspecting for damage in transit...
temp=/tmp/sharin$$; dtemp=/tmp/sharout$$
trap "rm -f $temp $dtemp; exit" 0 1 2 3 15
cat > $temp <<\!!!
69 533 3297 README
102 557 3260 batcher.1
46 209 1374 batcher.ctrl
441 1376 9503 batcher.c
4 12 88 Makefile
662 2687 17522 total
!!!
wc README batcher.1 batcher.ctrl batcher.c Makefile | sed 's=[^ ]*/==' | diff -b $temp - >$dtemp
if test -s $dtemp
then echo "Ouch [diff of wc output]:" ; cat $dtemp
else echo "No problems found."
fi
exit 0