wcf@psuhcx (Bill Fenner) (11/22/88)
Does anyone have a good way to expire news automatically when the news partition gets full? We only have a 25 meg partition for news, and it often manages to fill up on weekends, and it's a big pain to come in to find a console log 5 inches thick with logs of /usr/spool/news: write failed, filesystem full. We're using nntp to receive articles from psuvax1 (main feed), and uucp to send back to psuvax1 as well as to send to and from hogbbs, a FidoNet BBS, which doesn't create many incoming messages... Bill
dg@lakart.UUCP (David Goodenough) (11/25/88)
From article <1066@psuhcx.psu.edu>, by wcf@psuhcx (Bill Fenner): > Does anyone have a good way to expire news automatically when the news > partition gets full? We only have a 25 meg partition for news, and it > often manages to fill up on weekends, and it's a big pain to come in > to find a console log 5 inches thick with logs of /usr/spool/news: write > failed, filesystem full. Lets see. I use the following shell script line to expire news on a daily basis: find /usr/spool/news -mtime +7 -type f -exec rm \{\} \; 2>&1 >/dev/null and the following on a weekly basis to prevent the history file from oozing all over /usr: expire -e 7 -i -r Now write an awk script (or a C program) that takes a look at the output from df, to see how crammed your news partition is, and adjust the '-mtime +7' bitty in the find command as appropriate (i.e. +5 if the partition is getting full, and +7 or more if it's empty) The find / expire combination is recommended to any sysadmins who are really getting fed up of running an hour long expire a night. If you set the expire to happen on Sunday morning (say at 9:00), you can let it take forever, and it won't dent the system quite as bad. Of course you may need to pick your time carefully if you're something like ncoast / killer / portal, cause those folks may be on line and active all the time. Still it's a suggestion ..... -- dg@lakart.UUCP - David Goodenough +---+ | +-+-+ ....... !harvard!xait!lakart!dg +-+-+ | AKA: dg%lakart.uucp@harvard.harvard.edu +---+
wcs@skep2.ATT.COM (Bill.Stewart.[ho95c]) (11/30/88)
In article <1066@psuhcx.psu.edu> wcf@psuhcx (Bill Fenner) writes: > Does anyone have a good way to expire news automatically when the news > partition gets full? We only have a 25 meg partition for news, and it Here's my "trashnews" script. I run it hourly from cron, which seems to be often enough. It uses "df" to find how many blocks are free, and if there aren't enough, it grinds through the history file looking for articles to trash (starts at the top, works down - it doesn't care when the article *should* have expired.) Caveats: - You need to be running a version of news with one history file. - If your "df" output format is different than System V you will have to modify the sed / awk script to pick out the right field. - it doesn't clean up the history file - expire will have to do this for you. I run expire -r weekly. ======================== cut here ============================================ ####### Zap netnews until disk space is adequate. TARGET="/usr/spool" remove="rm -f" ## remove="echo" for debugging debug=":" ## debug="echo" SPOOLDIR=/usr/spool/news ## Where the articles live LIBDIR=/usr/lib/news ## Where the data files live trashgroups="comp/mail/maps comp/binaries/atari talk/politics/misc comp/sys/atari" ## attack these brutally cd $SPOOLDIR ## Where the articles live echo "===================== `date`" limit=5000 export limit LIBDIR remove debug if [ "$1" = "-x" ] ; then set -x ; remove="echo remove"; debug=echo ; shift ; fi case "$1" in [0-9]*) limit=$1 ; shift ;; esac ######## Make sure $TARGET has inodes (evil System V bug!) if df $TARGET | sed 's/(/ (/' | awk ' { { print "df inodes ", $0 ; if ( $5 < 1000 ) exit 0 ; else exit 1 ; } } ' #>/dev/null then echo trashing inodes ; find comp/mail/maps -type f -print | xargs $remove ; find control talk rec/humor -type f -mtime +3 -print | xargs $remove else echo inodes ok fi ######## Make sure $LIBDIR has space if df /usr | sed 's/(/ (/' | awk ' { { print "df,limit ", $3, '$limit' ; if ( $3 < '$limit' ) exit 0 ; else exit 1 ; } } ' #>/dev/null then echo remove /usr/lib/news/ohis* /usr/lib/news/olog* ; $remove /usr/lib/news/ohis* /usr/lib/news/olog* else echo /usr ok fi ############### ( ## generate list of files to trash #echo $LIBDIR/olog* $LIBDIR/ohistory find $trashgroups -type f -mtime +2 -print 2>/dev/null sed -e 's/.* //' -e '/^$/d' -e '/cancel/d' -e 's/\./\//g' $LIBDIR/history ) | while read victim victims ; do if [ -f "$victim" ] ; then if df $TARGET | sed 's/(/ (/' | awk ' { { print "df,limit ", $3, '$limit' ; if ( $3 < '$limit' ) exit 0 ; else exit 1 ; } } ' #>/dev/null then echo remove $victim $victims ; $remove $victim $victims else echo enough ; break fi else $debug $victim already gone fi done #################################### cut here ################ exit 0 -- # Thanks; # Bill Stewart, AT&T Bell Labs 2G218 Holmdel NJ 201-949-0705 ho95c.att.com!wcs # # One Bell System - it works!
dtynan@sultra.UUCP (Der Tynan) (12/01/88)
From article <1066@psuhcx.psu.edu>, by wcf@psuhcx (Bill Fenner): > > Does anyone have a good way to expire news automatically when the news > partition gets full? We only have a 25 meg partition for news, and it > often manages to fill up on weekends, and it's a big pain to come in > to find a console log 5 inches thick with logs of /usr/spool/news: write > failed, filesystem full. I have a similar problem. Having given it some thought, I have come up with a clean solution that (someday) I will implement in 2.11 (or whatever). On the other hand, if any of the *new-and-improved* news software people are reading this, perhaps they'd care to comment? Anyway, the idea is this. In the NEWS/active file, a new field is introduced in the tradition of the 'm' field for 'moderated'. It is a boolean ('y'/'n'?), which indicates that the given newsgroup is not read at this site. In this way, a nightly (or weekly) cron program would zip through all the .newsrc files, to see what groups aren't subscribed to, and update the 'active' file. On the other hand, if someone subscribes to a currently unavailable group, the daemon would reactivate it. And vnews/readnews/whatever would inform the reader that the group isn't currently carried, but will appear in a few days. Of course, certain groups (such as comp.mail.maps) would have a special mark saying that they must ALWAYS be subscribed to ('a' perhaps?). Then, rnews as part of its processing, would look at this flag, and if necessary, dump the article. Currently, the two ways of doing this, are to remove the group from the active file, in which case the 'junk' group fills up like nobodys business. Or, conversely, to have the sysadmin at the remote feed modify the 'sys' file, so that certain groups weren't sent. This is awkward, because changes may occur very frequently. Both schemes also mean that the 'checkgroups' messages will bomb fairly severely. In this age of Trailblazers, I don't think anyone is worried about line bandwidth, but just disk space (20Mb/week), so this scheme would allow them to carry only those groups that people actually read. Comments? - Der -- dtynan@zorba.Tynan.COM (Dermot Tynan @ Tynan Computers) {apple,mips,pyramid,uunet}!Tynan.COM!dtynan --- If the Law is for the People, then why do we need Lawyers? ---
stu@jpusa1.UUCP (Stu Heiss) (12/03/88)
In article <2694@sultra.UUCP> dtynan@sultra.UUCP (Der Tynan) writes:
-From article <1066@psuhcx.psu.edu>, by wcf@psuhcx (Bill Fenner):
->
-> Does anyone have a good way to expire news automatically when the news
-> partition gets full?
-I have a similar problem. Having given it some thought, I have come up with
-a clean solution that (someday) I will implement in 2.11 (or whatever).
-Anyway, the idea is this. In the NEWS/active file, a new field is introduced
-in the tradition of the 'm' field for 'moderated'. It is a boolean ('y'/'n'?),
-which indicates that the given newsgroup is not read at this site. In this
-way, a nightly (or weekly) cron program would zip through all the .newsrc
-files, to see what groups aren't subscribed to, and update the 'active' file.
-On the other hand, if someone subscribes to a currently unavailable group,
-the daemon would reactivate it. And vnews/readnews/whatever would inform
-the reader that the group isn't currently carried, but will appear in a few
-days. Of course, certain groups (such as comp.mail.maps) would have a special
-mark saying that they must ALWAYS be subscribed to ('a' perhaps?).
We do something similar with a couple of shell scripts and no mods to the news
software - works quite nicely. I use the previously posted script (inactng.sh)
to get a list of inactive (nobody reads them) newsgroups and rm the articles in
the associated directories. In addition, we always junk 'junk' and never junk
'comp.mail.maps' and 'news.announce.important'. See article in news.software.b
for the source to the two scripts.
--
Stu Heiss {spl1,uchicago.edu!gargoyle,ddsw1}!jpusa1!stu