steve@nuchat.UUCP (Steve Nuchia) (03/15/88)
Here is a shar of the local code I've put together to make
news run faster on my poor little 286. It will take some
customization to get it going on your system but if you
are having trouble with disk space fluctuation and/or
running out of cycles before you run out of news it will
be worth it.
Oh, something I forgot to mention in the README:
You need to build a 13-bit version of compress. This will
run a whole lot faster and use a lot less memory than the
16-bit version. The penalty on uucp is only about 15% and
you save a whole lot of cpu and Kcore minutes. Use the
13 bit version for your local feeds (and convince them
to reciprocate) and keep the 16 bit compress around for
long distance feeds and long-term storage compression.
------
#! /bin/sh
# This is a shell archive. Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file". To overwrite existing
# files, type "sh file -c". You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g.. If this archive is complete, you
# will see the following message at the end:
# "End of shell archive."
# Contents: README crontab ifuncs.patch news.rc newsd.c recv.c
# rnews.sh sendsome
# Wrapped by usenet@nuchat on Mon Mar 14 21:42:16 1988
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f README -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"README\"
else
echo shar: Extracting \"README\" \(2555 characters\)
sed "s/^X//" >README <<'END_OF_README'
XThese changes are motivated by the observation that
Xthe news receiving process is painfully slow on non-dbm
Xsites. Given that dbm was introduced specifically to speed
Xsuch things up that should be no surprise. Nevertheless
XI wanted to do what I could to improve the non-dbm case.
X
XHere's the code.
X
XFirst, there is a minor hack in ifuncs.c that inproves the
Xhistory scan speed noticably. The old loop was reading in
Xa line, mapping the whole thing to lower case, then calling
Xthe STRCMP macro on it. Since the first character is '<'
Xthe optimization on the macro, comparing the first character
Xbefore investing in a function call, was wasted. And since
Xthe second character differs in an overwhelming proportion
Xof the cases the lower case mapping scan was a waste.
X
XThen there's the revised unbatch pipeline. The newsd
Xdaemon is started from news.rc at boot time and is
Xresponsible for serializing unbatching and expiring
Xactivities and for monitoring the space on the disks
Xinvolved in news handling. The rules embodied in it
Xare the result of extensive tuning in my environment.
XYou will have to tune it for your to get good stability.
X
XSubordinate to newsd is a new batch cracker, recv. This
Xone serializes the decompress with respect to the rnews
Xruns, which is a win on systems with limited memory. The
Xbig win in recv is that it sorts the batch by history
Xhash value so that the ten history database files are
Xread at most once each. Without this optimization the
Xblock cache is completely ineffective, and in my system
Xadding the optimization cut the disk accesses per news
Xarticle received in half.
X
XThe rnews.sh script is what I put on uuxqt's path, ie in
X/usr/bin. Real rnews, the link to inews, is in /usr/lbin
Xon my system. This has the side benefit of eliminating
Xthe uuxqt lock breakage problem many small systems have
Xbeen experiencing.
X
XThe other thing newsd does is dynamically adjust the
Xexpire horizon. This is a big win, as it allows you
Xto run arbitrarily close to max all the time safely.
X
XAlso included is my batch master script, which I run from
Xcron every ten minutes. What the heck, I'll throw in
Xmy crontab for completeness. Anyway, my batch sender
Xhelps keep me out of trouble by not spooling when there
Xisn't enough room or there is already too much spooled
Xfor a given neighbor. These features are in the latest
Xsendbatch.sh if you can figure them out, but I like
Xmine better.
X
XAll this code is original and is hereby placed in
Xthe public domain.
X
X Steve Nuchia
X 14 March 1988
X (713) 334 6720 steve@nuchat
END_OF_README
if test 2555 -ne `wc -c <README`; then
echo shar: \"README\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f crontab -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"crontab\"
else
echo shar: Extracting \"crontab\" \(158 characters\)
sed "s/^X//" >crontab <<'END_OF_crontab'
X8,18,28,38,48,58 * * * * /usr/lib/news/sendsome 2>&1 | (head; cat > /dev/null)
X5 4 15 * * TZ=CST6CDT; /usr/lib/news/arbitron
X35 1 * * * /usr/lib/news/trimlib
END_OF_crontab
if test 158 -ne `wc -c <crontab`; then
echo shar: \"crontab\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f ifuncs.patch -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"ifuncs.patch\"
else
echo shar: Extracting \"ifuncs.patch\" \(865 characters\)
sed "s/^X//" >ifuncs.patch <<'END_OF_ifuncs.patch'
X# Sorry, I don't have a context diff program yet.
X#
X# The snippet below should make it obvious what to
X# do, only the SWN lines and the else case are
X# new, and there are no other changes.
X#
X# steve
X#
X hfp = xfopen(histfile(lcident), "r");
X while (fgets(bfr, BUFLEN, hfp) != NULL) {
X#ifdef SWN_SLOW
X p = index(bfr, '\t');
X if (p == NULL)
X p = index(bfr, '\n');
X if (p != NULL) /* can happen if nulls in file */
X *p = 0;
X lcase(bfr);
X
X if (STRCMP(bfr, lcident) == 0)
X#else SWN_SLOW
X for ( p = bfr, q = lcident; *q; q++, p++ )
X {
X if ( *p == '\t' || *p == '\n' ) break;
X if ( isupper(*p) ) *p = tolower(*p);
X if ( *p != *q ) break;
X }
X if ( ! *q && ( ! *p || *p == '\t' || *p == '\n' ) )
X#endif SWN_SLOW
X {
X (void) fclose(hfp);
X idunlock();
X#ifdef DEBUG
X fprintf(stderr,"history returns true\n");
X#endif /* DEBUG */
X return TRUE;
END_OF_ifuncs.patch
if test 865 -ne `wc -c <ifuncs.patch`; then
echo shar: \"ifuncs.patch\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f news.rc -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"news.rc\"
else
echo shar: Extracting \"news.rc\" \(465 characters\)
sed "s/^X//" >news.rc <<'END_OF_news.rc'
X:
Xcd /usr/spool/news/incoming
X(echo news\\c > /dev/console)
Xfor x in `ls | grep '^in'`
Xdo
X for l in a b c d e f g h i j k l m n o p
X do
X if [ ! -f $l$x ]
X then
X mv $x $l$x
X break
X fi
X done
Xdone
X
Xcd /usr/lib/news
Xrm -f send.lock
Xmv newsd.log onewsd.log
XPATH=/usr/lib/news:/usr/lbin:/usr/bin:/bin:. ; export path
X(echo " daemon\\c" > /dev/console)
Xsu - usenet -c "cd /usr/lib/news; exec ./newsd &" < /dev/null > newsd.log 2>&1
X(echo " started" > /dev/console)
END_OF_news.rc
if test 465 -ne `wc -c <news.rc`; then
echo shar: \"news.rc\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f newsd.c -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"newsd.c\"
else
echo shar: Extracting \"newsd.c\" \(2870 characters\)
sed "s/^X//" >newsd.c <<'END_OF_newsd.c'
X/*
X * newsd.c - daemon for unbundling news and doing other
X * essential stuff, like running expire. Obviates certain
X * cron processing and allows the slow components of news
X * to run niced without playing uucp games.
X *
X * Steve Nuchia
X * 27 Sept 1987
X */
X
X#include <sys/types.h>
X#include <sys/stat.h>
X#include <ustat.h>
X#include <sys/dir.h>
X#include <stdio.h>
X
X#define INCOMING "/usr/spool/news/incoming"
X#define OOPS "/usr/spool/news/oops"
X#define UNPACK "/usr/lib/news/recv"
X#define IHAVE "/files/news/.ihave"
X#define OUTGOING "/usr/spool/batch"
X#define NEWSDIR "/files/news"
X#define SPOOLDIR "/usr/spool"
X
Xmain ( argc, argv )
X int argc;
X char *argv[];
X{
X int incoming;
X struct direct art;
X char aname[DIRSIZ+1], abest[DIRSIZ+1], fullname[80], best[80], command[160];
X long btime, xtime;
X int i;
X struct stat sbuf;
X struct ustat usbuf;
X int scount = 25, days = 15;
X
X if ( argc > 1 )
X days = atoi ( argv[1] );
X time(&xtime);
X nice ( 20 );
X if ( (incoming = open ( INCOMING, 0 )) < 0 ) perror ( INCOMING );
X
X while ( 1 )
X {
X sleep ( 60 );
X /* see how the space situation looks */
X stat ( NEWSDIR, &sbuf );
X ustat ( sbuf.st_dev, &usbuf );
X if ( usbuf.f_tfree > 1000 && usbuf.f_tinode > 500 )
X {
X scount = 0;
X /* look around in INCOMING */
X lseek ( incoming, 2L * sizeof(struct direct), 0 );
X best[0] = 0;
X
X while ( read ( incoming, &art, sizeof(struct direct) ) > 0 )
X {
X if ( ! art.d_ino ) continue;
X for ( i = 0; i < DIRSIZ; i++ ) aname[i] = art.d_name[i];
X aname[i] = 0;
X sprintf ( fullname, "%s/%s", INCOMING, aname );
X stat ( fullname, &sbuf );
X if ( ! best[0] || btime > sbuf.st_mtime )
X {
X btime = sbuf.st_mtime;
X strcpy ( best, fullname );
X strcpy ( abest, aname );
X }
X }
X /* if there is anything, take care of oldest */
X if ( best[0] )
X {
X sprintf ( command, "%s %s", UNPACK, best );
X if ( system ( command ) )
X {
X sprintf ( command, "%s/%s", OOPS, abest );
X link ( best, command );
X }
X unlink ( best );
X continue;
X }
X }
X else
X {
X printf ( "space problem in NEWSDIR %d\n", ++scount );
X fflush ( stdout );
X sleep ( 120 );
X }
X /* otherwise we are free to do housekeeping */
X stat ( SPOOLDIR, &sbuf );
X ustat ( sbuf.st_dev, &usbuf );
X if ( usbuf.f_tfree > 5000 && usbuf.f_tinode > 500 )
X {
X if ( scount > 30 ) /* 30 times around with no space */
X {
X time(&btime);
X scount = 20;
X days = days - 1 + (btime - xtime) / (23 * 3600L);
X xtime = btime;
X sprintf ( command, "echo %d > expire.days", days );
X system ( command );
X sprintf ( command, "expire -e%d -v -a >> expire.log 2>&1",
X days );
X printf ( "%s\n", command );
X fflush ( stdout );
X system ( command );
X }
X }
X else
X {
X if ( scount > 25 ) scount = 25;
X printf ( "space problem in SPOOLDIR\n" );
X fflush ( stdout );
X sleep ( 180 );
X }
X }
X}
END_OF_newsd.c
if test 2870 -ne `wc -c <newsd.c`; then
echo shar: \"newsd.c\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f recv.c -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"recv.c\"
else
echo shar: Extracting \"recv.c\" \(3392 characters\)
sed "s/^X//" >recv.c <<'END_OF_recv.c'
X#include <sys/types.h>
X#include <sys/stat.h>
X
Xchar buf[256];
X
Xmain ( argc, argv )
X int argc;
X char *argv[];
X{
X int fd, pfd[2], size;
X
X close ( 0 );
X fd = open ( argv[1], 0 );
X if ( fd != 0 )
X {
X perror ( argv[1] );
X exit(1);
X }
X read ( fd, buf, sizeof(buf) );
X if ( ! strncmp ( buf, "#! cunbatch\n", 12 )
X && buf[12] == 0037 && buf[13] == (char) 0235 )
X {
X size = buf[14] & 0x1f;
X lseek ( fd, 12L, 0 );
X pipe ( pfd );
X if ( ! fork() )
X {
X close ( 1 );
X dup ( pfd[1] );
X close ( pfd[0] );
X close ( pfd[1] );
X if ( size <= 13 )
X execl ( "/usr/lbin/comp12", "comp12", "-d", (char *) 0 );
X else
X execl ( "/usr/lbin/comp16", "compress", "-d", (char *) 0 );
X perror ( "compress" );
X exit(1);
X }
X close ( fd );
X dup ( pfd[0] );
X close ( pfd[0] );
X close ( pfd[1] );
X exit ( unbatch() );
X }
X /* never mind... */
X lseek ( fd, 0L, 0 );
X exit(unbatch());
X}
X
Xchar bbuf[1024];
X
Xextern long atol();
X
Xunbatch()
X{
X int new = 1;
X int len = 0, st, i;
X int ch;
X long alen;
X char *p, *q;
X int ofd[10];
X char fname[128];
X int rnews, xstat, rv = 0;
X int pfd[2];
X
X for ( ch = 0; ch < 10; ch++ )
X {
X sprintf ( fname, "/usr/spool/news/decomp/recv.%d", ch );
X ofd[ch] = creat ( fname, 0600 );
X }
X while ( len += bbfill ( bbuf + len, 1024 - len ) )
X {
X if ( strncmp ( bbuf, "#! rnews ", 9 ) )
X {
X if ( new )
X punt ( bbuf, len );
X printf ( "garbled news\n" );
X exit ( 1 );
X }
X new = 0;
X alen = atol ( bbuf + 9 );
X for ( st = 0; st < len && bbuf[st] != '\n'; st++ );
X st++;
X ch = 0;
X for ( p = bbuf + st; !ch && p < bbuf + len; p++ )
X {
X if ( strncmp ( p, "Message-ID: ", 12 ) )
X while ( p < bbuf + len && *p != '\n' ) p++;
X else
X while ( p < bbuf + len && *p != '\n' && *p != '\@' ) ch = *p++;
X }
X if ( ch < '1' || ch > '9' ) ch = '0';
X ch -= '0';
X while ( alen && len )
X {
X i = len - st;
X if ( i > alen ) i = alen;
X write ( ofd[ch], bbuf, st + i );
X alen -= i;
X bcopy ( bbuf + st + i, bbuf, len = len - st - i );
X st = 0;
X len += bbfill ( bbuf + len, 1024 - len );
X }
X }
X for ( ch = 0; ch < 10; ch++ ) close ( ofd[ch] );
X for ( ch = 0; ch < 10; ch++ )
X {
X sprintf ( fname, "/usr/spool/news/decomp/recv.%d", ch );
X if ( ! (rnews = fork()) )
X {
X struct stat sbuf;
X
X stat ( fname, &sbuf );
X if ( !sbuf.st_size ) exit ( 0 );
X close ( 0 );
X open ( fname, 0 );
X execl ( "/usr/lbin/rnews", "rnews", (char *) 0 );
X perror ( "rnews" );
X exit ( 1 );
X }
X while ( wait ( &xstat ) != rnews );
X unlink ( fname );
X rv |= xstat | (xstat >> 8);
X }
X return ( rv );
X}
X
Xbcopy ( f, t, l )
X char *f, *t;
X{
X while ( l-- ) *t++ = *f++;
X}
X
Xbbfill ( p, s )
X char *p;
X int s;
X{
X int len = 0, rv;
X
X while ( len < s )
X {
X if ( (rv = read ( 0, p + len, s - len )) > 0 )
X len += rv;
X else
X break;
X }
X return ( len );
X}
X
Xpunt ( buf, len )
X char buf[];
X int len;
X{
X int pfd[2], rnews, xstat;
X
X pipe ( pfd );
X if ( ! (rnews = fork()) )
X {
X close ( 0 );
X dup ( pfd[0] );
X close ( pfd[0] );
X close ( pfd[1] );
X execl ( "/usr/lbin/rnews", "rnews", (char *) 0 );
X perror ( "rnews" );
X exit ( 3 );
X }
X close ( pfd[0] );
X write ( pfd[1], buf, len );
X while ( (len = read ( 0, buf, 1024 )) > 0 )
X write ( pfd[1], buf, len );
X close ( pfd[1] );
X while ( wait ( &xstat ) != rnews );
X exit ( xstat | (xstat >> 8) );
X}
END_OF_recv.c
if test 3392 -ne `wc -c <recv.c`; then
echo shar: \"recv.c\" unpacked with wrong size!
fi
# end of overwriting check
fi
if test -f rnews.sh -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"rnews.sh\"
else
echo shar: Extracting \"rnews.sh\" \(109 characters\)
sed "s/^X//" >rnews.sh <<'END_OF_rnews.sh'
Xumask 0222
Xcat - >> /usr/spool/news/incoming/in.$$
Xls -l /usr/spool/news/incoming/in.$$ >> /usr/lib/news/log
END_OF_rnews.sh
if test 109 -ne `wc -c <rnews.sh`; then
echo shar: \"rnews.sh\" unpacked with wrong size!
fi
chmod +x rnews.sh
# end of overwriting check
fi
if test -f sendsome -a "${1}" != "-c" ; then
echo shar: Will not over-write existing file \"sendsome\"
else
echo shar: Extracting \"sendsome\" \(1589 characters\)
sed "s/^X//" >sendsome <<'END_OF_sendsome'
X:
X
Xeval `df | grep /usr | awk '{ print "BLKS=" $3 "; INODS=" $5 }'`
X
Xif test -z "$BLKS" -o -z "$INODS" -o "$BLKS" -lt 7000 -o "$INODS" -lt 100
Xthen
X exit
Xfi
X
Xif [ -f /usr/lib/news/send.lock ]
Xthen
X if kill -0 `cat /usr/lib/news/send.lock` 2> /dev/null
X then
X exit
X else
X rm /usr/lib/news/send.lock
X fi
Xfi
X
Xecho $$ > /usr/lib/news/send.lock
X
Xeval `uustat -q | tail +1 | awk '{print $1 "=" $3}'`
X
Xif test -z "$tness1" -o "$tness1" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 tness1
Xfi
X
Xif test -z "$sugar" -o "$sugar" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 sugar
Xfi
X
Xif test -z "$uunet" -o "$uunet" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -c uunet
Xfi
X
Xif test -z "$meow" -o "$meow" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 -b12 meow
Xfi
X
Xif test -z "$uhnix1" -o "$uhnix1" -lt 6
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 uhnix1
Xfi
X
Xif test -z "$splut" -o "$splut" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 splut
Xfi
X
Xif test -z "$flatli" -o "$flatli" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 flatline
Xfi
X
Xif test -z "$amdahl" -o "$amdahl" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -c amdahl
Xfi
X
Xif test -z "$siswat" -o "$siswat" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 siswat
Xfi
X
Xif test -z "$killer" -o "$killer" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 killer
Xfi
X
Xif test -z "$shell" -o "$shell" -lt 4
Xthen
X nice -20 /usr/lib/news/sendbatch -cp/usr/lbin/comp12 shell
Xfi
X
Xrm /usr/lib/news/send.lock
END_OF_sendsome
if test 1589 -ne `wc -c <sendsome`; then
echo shar: \"sendsome\" unpacked with wrong size!
fi
chmod +x sendsome
# end of overwriting check
fi
echo shar: End of shell archive.
exit 0
--
Steve Nuchia | [...] but the machine would probably be allowed no mercy.
uunet!nuchat!steve | In other words then, if a machine is expected to be
(713) 334 6720 | infallible, it cannot be intelligent. - Alan Turing, 1947