tneff@bfmny0.UUCP (Tom Neff) (05/09/89)
I used to put a lot of individual expires into the root crontab,
which meant every time I wanted to adjust things I had to mess with
cron. I wasn't satisfied with the logging of this either. So I
wrote a general purpose cleanup shell script where I can specify
all the individual expires, trims, logging etc for news. Perhaps
others may find it useful. I call it "cleannews":
---cut here------cut here------cut here------cut here------cut here---
#! /bin/sh
# '@(#)cleannews 1.2 05/03/89'
# Perform news cleanup. This script should be run nightly from the
# root crontab. The distributed version performs a general news expire
# and trims the log files. You can add newsgroup-specific expires and
# any other news related stuff you want to run nightly. Keeping
# everything together in one shell script lets you manage netnews
# flexibly without changing 'crontab' all the time. Cleannews also
# writes a logfile (including error codes). This logfile is not
# itself cleaned automatically but it should only grow by a few
# lines per night. To turn it off, set Logfile=/dev/null.
#
# This version was tested with the news 2.11 software but should
# work most anywhere.
# set -vx # uncomment for debugging
# Customize these values (if necessary) for your site.
Date=/bin/date
Expire=/usr/lib/news/expire
Trimlib=/usr/lib/news/trimlib
Logfile=/usr/lib/news/cleannews.log
# # # # # # # # # # # # # # # # # # # # # # # # # # # # #
#
# Add copies of the following block for any specific
# expires you want to run besides the general expire.
#
# Customize the following values for each expire:
KEEPNEWS=14
KEEPHIST=30
VERBOSE=0
OUTPUT="> /dev/null"
GROUPS=all
if $Expire -e $KEEPNEWS -E $KEEPHIST -v $VERBOSE -n $GROUPS $OUTPUT
then RC=;
else RC="?? exit code $? ??" ;
fi
$Date "+%D %T expire $GROUPS $RC" >> $Logfile
# End of expire block
#
# # # # # # # # # # # # # # # # # # # # # # # # # # # # # #
# This is the news logfile trimmer from the news 2.11 'misc' directory
if $Trimlib
then RC=;
else RC="?? exit code $? ??" ;
fi
$Date "+%D %T trimlib $RC" >> $Logfile
---cut here------cut here------cut here------cut here------cut here---
--
Tom Neff UUCP: ...!uunet!bfmny0!tneff
"Truisms aren't everything." Internet: tneff@bfmny0.UU.NETusenet@statware.UUCP (Usenet News) (05/10/89)
The following script is what I use to put all my expires in one place.
It is not as simple as "cleanup", but is more flexible for those sites
which may need it. I wanted to expire newsgroups at different rates, and
this permits me to do that. I run it every night, and have the advantage
of making all expire changes in one file. Note that I rebuild the
history file (-r) once a month, by testing if the date is the 15th.
You will have to adjust the list of newsgroups and other settable
parameters before you start using this script.
#!/bin/sh
# @(#) daily.SH Usenet daily script Version 1.1 September 1988
#
# Do the report using report.awk, run expire, and run sendbatch.
# That this script starts several processes running in the background.
#
# By Mathieu Federspiel, {hp-pcd,tektronix}!orstcs!statware!mcf
# Version 1.0: July 1988
# Customize Parameters
# Some systems do not use directory structure for spool directory,
# and need to comment out "find ... -type d ...".
# The options used on the TMAIL command may need to be changed.
# File "news/misc/report.awk" is used, and the mail header stuff needs to
# be commented out. Move and edit this file, set REPORT to working version.
NEWSUSR=usenet
NEWSGRP=news
DESTDIR=/usr/news
LIBDIR=/lib
SPOOLDIR=/spool
TMAIL=/usr/bin/mailx
REPORT=/usr/news/lib/report.awk
# see expire(8) when setting these
EXPIREOPTS="-e 13 -i -v2"
EXPIREGRPS="-n all,!control,!cor,!cs,!junk,!or,!orst,!pnw,!rec,!soc,!talk,!texhax,!comp.sources.unix,!comp.sources.wanted,!comp.doc,!news.admin,!news.groups"
# you may want to expire some groups after *short* time
EXPIREOPTS2="-e 6 -i -v2"
EXPIREGRPS2="-n control,cor,cs,junk,or,orst,pnw,rec,soc,comp.sources.wanted,news.admin,news.groups"
# you may want to expire some groups after *long* time
EXPIREOPTS3="-e 350 -i -v2"
EXPIREGRPS3="-n comp.doc,comp.sources.unix,texhax"
# see sendbatch(8) when setting these
SENDBATCHOPTS="-c -b16"
TOSYSTEM="orstcs"
# Go to directory with log files
cd ${DESTDIR}${LIBDIR}
# The list in the for controls how many old files are retained
for i in 5 4 3 2 1 0
do
j=`expr $i + 1`
mv log.$i log.$j
mv errlog.$i errlog.$j
done
mv log log.0
mv errlog errlog.0
# this may be "cp /dev/null log"
> log
> errlog
# If run by NEWSUSER, chown and chgrp not required
#chown ${NEWSUSR} log errlog ; chgrp ${NEWSGRP} log errlog
# Start reports
(awk -f $REPORT log.0 | ${TMAIL} -n -s "Usenet Report from log.0" ${NEWSUSR})&
#(cat errlog.0 | ${TMAIL} -n -s "Usenet Errfile from errlog.0" ${NEWSUSR})&
# Remove expired articles. Once a month, rebuild history file and
# do the expiration. Refer to expire(8).
if [ 15 -eq `date +%d` ]
then
# -p, expire old articles by date of posting, gets rid of
# articles with long expiration dates. use if -i not used.
# -r, rebuild history file.
# find, get rid of bad files, eg from disk problems
( {
date ; df /usr/news
echo "Standard Expire:\n"
${DESTDIR}${LIBDIR}/expire $EXPIREOPTS $EXPIREGRPS
echo "\n\nShort Expire:\n\n"
${DESTDIR}${LIBDIR}/expire $EXPIREOPTS2 $EXPIREGRPS2
echo "\n\nLong Expire:\n\n"
${DESTDIR}${LIBDIR}/expire $EXPIREOPTS3 $EXPIREGRPS3
echo "\n\nRemove empty files:\n"
# Find works to remove empty regular files.
# Rmdir with find/sed removes empty directories.
# Find does ls on directories under SPOOLDIR, using "SPOOLDIR/*" so
# SPOOLDIR/.rnews is not removed.
# Ls must report as follows for editing by sed:
# ls spool/comp
# total NN ...
# where NN is the size of the contents; anything may follow.
# Sed passes only directories with "total 0" to rmdir.
# Rmdir is used as it will remove only empty directories.
find ${DESTDIR}${SPOOLDIR} -size 0 -type f -exec rm {} \;
rmdir `find ${DESTDIR}${SPOOLDIR}/* -type d -depth -print -exec ls -s {} \; |\
sed -e "
1{
h
N
}
/total 0/{
x
n
}
h
d
"`
echo "\n\nRebuild history file:\n"
${DESTDIR}${LIBDIR}/expire -r -v2
date ; df /usr/news
} | ${TMAIL} -n -s "Usenet Expire, and history rebuild" ${NEWSUSR})&
else
( {
date ; df /usr/news
echo "Standard Expire:\n"
${DESTDIR}${LIBDIR}/expire $EXPIREOPTS $EXPIREGRPS
echo "\n\nShort Expire:\n"
${DESTDIR}${LIBDIR}/expire $EXPIREOPTS2 $EXPIREGRPS2
date ; df /usr/news
} | ${TMAIL} -n -s "Usenet Expire" ${NEWSUSR})&
fi
# Process articles being sent from here
(${DESTDIR}${LIBDIR}/sendbatch ${SENDBATCHOPTS} ${TOSYSTEM})&
# That's all, Folks!
exit 0
--
Mathieu Federspiel mcf%statware.uucp@cs.orst.edu
Statware {hp-pcd,tektronix}!orstcs!statware!mcf
260 SW Madison Avenue, Suite 109 503-753-5382
Corvallis OR 97333 USA