farren@gethen.UUCP (Michael J. Farren) (02/10/88)
Recently, I have seen a singularly unpleasant phenomenon - after
several days of no new news at all, ten to fifteen megabytes of news
will arrive in one big lump. What I think is happening is that an
upstream site has a nearly full spool directory, and when it starts
to send (or receive) news, cannot accept the transactions. When
expire is run, there is space, so all of the pent-up news is sent
downstream at once, which, unfortunately, may well cause the downstream
site's spool directory to become full, reproducing the problem. Bad
enough in itself, this problem can also cause articles, mail, and other
uucp transactions to be lost, as they fall out of the bottom of the
full spool directories.
If I'm way off base in my suppositions or my solutions, please let
me know.
--------
This is a set of context diffs which, when applied to the standard
'sendbatch' script, will produce a script which will attempt to limit
the total amount of news passed to a downstream site on any given
night. Properly applied, it should not cause anything to be lost,
just spread over time, so that variations in news flow will tend to
be evened out, avoiding overpopulation problems in spool directories.
There has been an additional flag added to the command line syntax of
sendbatch. This flag, -l, will invoke the limiting function. A typical
entry in crontab for the new sendbatch would look like:
sendbatch -c -l systemname
The limits are set with the DAILY parameter. Edit the script to fit
your own needs; the value is currently set at 5000000, which is somewhat
larger than the average daily news volume, but small enough to help.
Note that this doesn't represent the actual volume of news - it's
simply a counter, used in conjunction with the value batch uses to
limit the size of its batches. For example, the standard limitation
number used by batch is 50000. With this value, sendbatch invoked with
the limit flag will limit itself to 10 batches per night. On average,
this should come pretty close to the DAILY figure.
-------- Diffs follow - cut here --------
*** sendbatch.old Thu Sep 10 03:50:47 1987
--- sendbatch Tue Feb 9 14:25:00 1988
***************
*** 1,4
: '@(#)sendbatch.sh 1.10 9/23/86'
cflags=
LIM=50000
--- 1,5 -----
: '@(#)sendbatch.sh 1.10 9/23/86'
+ : 'Modified for batch limiting 2/09/88'
cflags=
LIM=50000
***************
*** 2,7
cflags=
LIM=50000
CMD='/usr/lib/news/batch /usr/spool/batch/$rmt $BLIM'
ECHO=
COMP=
--- 3,10 -----
cflags=
LIM=50000
+ DAILY=5000000
+ TOTAL=
CMD='/usr/lib/news/batch /usr/spool/batch/$rmt $BLIM'
ECHO=
COMP=
***************
*** 31,36
DOIHAVE=`uuname -l`
fi
continue;;
esac
if test -n "$COMP"
--- 34,41 -----
DOIHAVE=`uuname -l`
fi
continue;;
+ -l) TOTAL=0
+ continue;;
esac
if test -n "$COMP"
***************
*** 52,57
else
(eval $ECHO; eval $CMD $COMP $C7) |
if test -s /usr/spool/batch/$rmt.cmd
then
/usr/spool/batch/$rmt.cmd
--- 57,63 -----
else
(eval $ECHO; eval $CMD $COMP $C7) |
+
if test -s /usr/spool/batch/$rmt.cmd
then
/usr/spool/batch/$rmt.cmd
***************
*** 57,62
/usr/spool/batch/$rmt.cmd
else
uux - -r $rmt!$RNEWS
fi
fi
done
--- 63,81 -----
/usr/spool/batch/$rmt.cmd
else
uux - -r $rmt!$RNEWS
+ fi
+
+ if test -n $TOTAL
+ then
+ TOTAL=`expr $TOTAL + $LIM`
+ if test $TOTAL -ge $DAILY
+ then
+ if test -f /usr/spool/batch/$rmt.work
+ then
+ mv /usr/spool/batch/$rmt.work /usr/spool/batch/$rmt
+ fi
+ break
+ fi
fi
fi
done
--
Michael J. Farren | "INVESTIGATE your point of view, don't just
{ucbvax, uunet, hoptoad}! | dogmatize it! Reflect on it and re-evaluate
unisoft!gethen!farren | it. You may want to change your mind someday."
gethen!farren@lll-winken.llnl.gov ----- Tom Reingold, from alt.flame dmk@dmk3b1.UUCP (David Keaton) (02/11/88)
In article <669@gethen.UUCP> farren@gethen.UUCP (Michael J. Farren) writes: >There has been an additional flag added to the command line syntax of >sendbatch. This flag, -l, will invoke the limiting function. This is a neat hack. I'd like to add a word of caution, though. Don't use sendbatch -l if you're close to running out of spool space YOURSELF. If that's the case you want to flush everything out of your system as fast as possible. Also, don't use sendbatch -l when transmitting to a site that has as much disk space as you do or more. Otherwise you'll cause propagation delays that aren't necessary. -- David Keaton dmk%dmk3b1@uunet.uu.net uunet!dmk3b1!dmk
farren@gethen.UUCP (Michael J. Farren) (02/12/88)
In article <404@dmk3b1.UUCP> dmk@dmk3b1.UUCP (David Keaton) writes: >In article <669@gethen.UUCP> farren@gethen.UUCP (Michael J. Farren) writes: >>There has been an additional flag added to the command line syntax of >>sendbatch. This flag, -l, will invoke the limiting function. > >Don't use sendbatch -l if you're close to running out of spool space >YOURSELF. If that's the case you want to flush everything out of your >system as fast as possible. Perhaps not. If your downstream sites are being seriously flummoxed because you're overloading THEIR directories, then you might not want to flush yourself onto them so quickly. >Also, don't use sendbatch -l when transmitting to a site that has as >much disk space as you do or more. Otherwise you'll cause propagation >delays that aren't necessary. This is a drawback to the scheme, but I believe the problems caused by big fluctuations in the volume of news are greater than the propagation delay problems. Articles should still go out in the order received, just not all of them at once. With normal flow, the limiting function should never be invoked, and if it is invoked, then the delays should be fairly minimal. I would like to ask for feedback from people who are using sendbatch -l, once it has been in place long enough to see if it has some of the desired effect. It's always nice to get data to support your theories :-) -- Michael J. Farren | "INVESTIGATE your point of view, don't just {ucbvax, uunet, hoptoad}! | dogmatize it! Reflect on it and re-evaluate unisoft!gethen!farren | it. You may want to change your mind someday." gethen!farren@lll-winken.llnl.gov ----- Tom Reingold, from alt.flame
zeeff@b-tech.UUCP (Jon Zeeff) (02/16/88)
>In article <669@gethen.UUCP> farren@gethen.UUCP (Michael J. Farren) writes: >There has been an additional flag added to the command line syntax of >sendbatch. This flag, -l, will invoke the limiting function. > I did something like this a long time ago - it should definitely be a part of the standard distribution. It doesn't work so well if you run sendbatch hourly though. I finally ended up with a demon sendbatch script that spools news on the fly (as soon at it notices that someone picked up some of their news, it spools more). It runs from late evening to early morning and does allow a per day limit. The only other thing I use around here to keep life sane is a uukill script that cuts off any incoming uucp connection if disk space is about to run out. I wouldn't need it if all sites put limits on sendbatch though. -- Jon Zeeff Branch Technology, uunet!umix!b-tech!zeeff zeeff%b-tech.uucp@umix.cc.umich.edu