steve@nuchat.UUCP (Steve Nuchia) (01/23/88)
[news.software.b added to followup to news.admin article] In article <12312@oliveb.olivetti.com>, jerry@oliveb.olivetti.com (Jerry Aguirre) writes: > > With the variations in volume of news I was also having problems coming > up with with an expire time that made full use of the disk without > running over. The problem with such a "hand" solution is that it uses > my time and if I am unavailable then things break down. I went through the same series of observations.. > I finally came up with a script that maintains things without attention > from me. Using df and awk the script checks the available blocks and > inodes on /usr/spool/news. (Somewhat like what patch 14 adds to > sendbatch.) When this drops below a specified level (10K blocks and 3K > inodes) news is expired at 28 days. If there is still not enough space > news is expired again at 21 days and mail is sent to me about it. This but came up with a slightly different solution, which addresses another problem as well. My poor little 286 machine, with a half-dozen+ news neighbors, often couldn't unbatch news fast enough to prevent uuxqt from blowing away the lock and starting another unbatch, which slowed things down further... Anyway, it wasn' pretty. So, I replaced /usr/bin/rnews with: umask 0222 cat - >> /usr/spool/news/incoming/in.$$ ls -l /usr/spool/news/incoming/in.$$ >> /usr/lib/news/log and wrote a daemon to do the real rnews, serially and at low priority. It also serializes expire with respect to unbatching and the original also managed sendbatch. That version was, I believe, posted a while back. Since then I've done some work that makes it more stable and moved sendbatch back to cron's arena. I'll include my "sendsome" script below too - you may need to twiddle it to parse the df output properly on your system. /* * newsd.c - daemon for unbundling news and doing other * essential stuff, like running expire. Obviates certain * cron processing and allows the slow components of news * to run niced without playing uucp games. * * Steve Nuchia * 27 Sept 1987 */ #include <sys/types.h> #include <sys/stat.h> #include <ustat.h> #include <sys/dir.h> #include <stdio.h> #define INCOMING "/usr/spool/news/incoming" #define RNEWS "/usr/lbin/rnews" #define IHAVE "/files/news/.ihave" #define OUTGOING "/usr/spool/batch" #define NEWSDIR "/files/news" #define SPOOLDIR "/usr/spool" main() { int incoming; struct direct art; char aname[DIRSIZ+1], fullname[80], best[80], command[160]; long btime, xtime; int i; struct stat sbuf; struct ustat usbuf; int scount = 25, days = 15; time(&xtime); nice ( 20 ); if ( (incoming = open ( INCOMING, 0 )) < 0 ) perror ( INCOMING ); while ( 1 ) { sleep ( 60 ); /* see how the space situation looks */ stat ( NEWSDIR, &sbuf ); ustat ( sbuf.st_dev, &usbuf ); if ( usbuf.f_tfree > 1000 && usbuf.f_tinode > 500 ) { scount = 0; /* look around in INCOMING */ lseek ( incoming, 2L * sizeof(struct direct), 0 ); best[0] = 0; while ( read ( incoming, &art, sizeof(struct direct) ) > 0 ) { if ( ! art.d_ino ) continue; for ( i = 0; i < DIRSIZ; i++ ) aname[i] = art.d_name[i]; aname[i] = 0; sprintf ( fullname, "%s/%s", INCOMING, aname ); stat ( fullname, &sbuf ); if ( ! best[0] || btime > sbuf.st_mtime ) { btime = sbuf.st_mtime; strcpy ( best, fullname ); } } /* if there is anything, take care of oldest */ if ( best[0] ) { sprintf ( command, "%s < %s", RNEWS, best ); if ( ! system ( command ) ) unlink ( best ); continue; } } else { printf ( "space problem in NEWSDIR %d\n", ++scount ); fflush ( stdout ); sleep ( 120 ); } /* otherwise we are free to do housekeeping */ stat ( SPOOLDIR, &sbuf ); ustat ( sbuf.st_dev, &usbuf ); if ( usbuf.f_tfree > 5000 && usbuf.f_tinode > 500 ) { if ( scount > 30 ) /* 30 times around with no space */ { time(&btime); scount = 20; days = days - 1 + (btime - xtime) / (23 * 3600L); xtime = btime; sprintf ( command, "expire -e%d -v -a >> expire.log 2>&1", days ); printf ( "%s\n", command ); fflush ( stdout ); system ( command ); } } else { if ( scount > 25 ) scount = 25; printf ( "space problem in SPOOLDIR\n" ); fflush ( stdout ); sleep ( 180 ); } } } and here's sendsome, with some of the clients removed - I made two mods to sendbatch to make this work - one to make it exit after one pass through its loop and another to allow command-line specification of the compress program. The latter was added because the memory consumption for a full 16-bit compress is _much_ greater than that for a compress compiled for 13-bit, and it just isn't worth it for local links. : eval `df | grep /usr | awk '{ print "BLKS=" $3 "; INODS=" $5 }'` if test -z "$BLKS" -o -z "$INODS" -o "$BLKS" -lt 7000 -o "$INODS" -lt 100 then exit fi eval `uustat -q | tail +1 | awk '{print $1 "=" $3}'` if test -z "$sugar" -o "$sugar" -lt 4 then nice -20 /usr/lib/news/sendbatch -cp/usr/lib/news/comp12 sugar fi if test -z "$uunet" -o "$uunet" -lt 4 then nice -20 /usr/lib/news/sendbatch -c uunet fi if test -z "$uhnix1" -o "$uhnix1" -lt 6 then nice -20 /usr/lib/news/sendbatch -cp/usr/lib/news/comp12 uhnix1 fi Oh, and one more piece of technology - I've got the following being run as part of my rc startup script - it ensures that the spool files, with PID-based names, don't get overwritten. You'll notice in the daemon source that it bases its selection of which batch to process on the mtime so all is well. cd /usr/spool/news/incoming for x in * do for l in a b c d e f g h i j k l m n o p do if [ ! -f $l$x ] then mv $x $l$x break fi done done cd /usr/lib/news mv newsd.log onewsd.log PATH=/usr/lib/news:/usr/lbin:/usr/bin:/bin:. ; export path ../newsd > newsd.log 2>&1 & -- Steve Nuchia | [...] but the machine would probably be allowed no mercy. uunet!nuchat!steve | In other words then, if a machine is expected to be (713) 334 6720 | infallible, it cannot be intelligent. - Alan Turing, 1947