vern@zebra.UUCP (Vernon C. Hoxie) (01/06/90)
This collection of script and 'c' programs permit the archiving of news groups to 3b1 tape drives and permit retrieval by subject. #! /bin/sh # This is a shell archive. Remove anything before this line, then unpack # it by saving it into a file and typing "sh file". To overwrite existing # files, type "sh file -c". You can also feed this as standard input via # unshar, or by typing "sh <file", e.g.. If this archive is complete, you # will see the following message at the end: # "End of archive 1 (of 2)." # Contents: Digest Install MANIFEST Makefile README conv.prep.sh # conv.rd.sh conv.sq.sh convert.sh copy.sh group.c header.sh prep.sh # purge.sh rd.tape.sh read.vtoc.c save.list squirrel.sh strt.hdr.sh # Wrapped by root@quagga on Fri Jan 5 21:24:46 1990 PATH=/bin:/usr/bin:/usr/ucb ; export PATH if test -f Digest -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"Digest\" else echo shar: Extracting \"Digest\" \(4946 characters\) sed "s/^X//" >Digest <<'END_OF_Digest' X This is a collection of shell scripts and 'c' programs designed Xto aid in archiving news articles. The format permits these articles to be Xrecovered individually according to the 'Subject:' line of each article. X X The collection consists of the following programs: X X'squirrel.sh': This links the articles in the news directories selected Xby the user edited 'save.list' file. It is to be run by 'cron' and Xsends reports to the news administrator as to the status of the Xcollection. A support file containing the 'Message-Id:' of each article Xis maintained to preclude duplication of articles due to cross postings. X X'prep.sh': This performs the preliminary processing prior to writing to Xtape. A 'contents' file is generated from the 'Subject:' line of each Xarticle and then 'contents' is sorted and regrouped according to subject Xso the follow-up titles appear together in 'contents'. The articles are Xthen compressed and moved from the 'arcdir' to 'savdir'. X XBASIC SCRIPTS: X'wr.tape.sh': This performs the actual write operation after Xdetermining that there is room on the tape for the collection of Xarticles. If the collection is larger than the available space, the Xcollection is truncated. Articles successfully transferred to tape are Xpurged from the 'arcdir'. X X'rd.tape.sh': This reads the volume table of contents from the tape and Xpermits the user to select one chapter from the tape for review. The X'contents' file of the selected chapter is then presented to the user so Xthat individual articles can be selected. The desired articles are then Xcopied out into a subdirectory of the '/news/.tape.ops' directory for Xretrieval by the user. X X'copy.sh': This is included as an aid in recovering from a verification Xerror during a write operation. It copies existing chapters on a bad Xtape to another tape so that additions chapters can be added to the Xcollection. X X'header.sh': The definitions for the various file names and directories Xare listed here for use by all the shell scripts so that these names are Xconsistent for the complete set. X XSUPPORT 'C' PROGRAMS: X'read.vtoc.c': This reads the '/tmp/vtoc' file generated by 'Tgetname -b' Xcall. It is used by the 'rd.tape' and 'wr.tape' scripts for information Xneeded for accessing the tape. See the comments in 'read.vtoc.c' for Xmore information. X X'group.c': This performs the grouping of similar subject titles in Xthe 'contents' file before writing to tape. X X'pick.files.c': This is the display and selection program for retrieving Xthe desired articles. X XREFERENCE FILES: X'save.list': This is the user defined list of news directories which Xare recursively searched for articles to be "squirreled" for later Xtransfer to tape. X X'params': This is a generated file which is maintained by the various Xprograms to transfer data from one operation to the next. X X'link.list': Another generated file which contains the 'Message-Id:' Xlines from the articles which have 'squirreled'. These Id's are kept on Xfile for two months at which time the list is purged by 'squirrel.sh'. X XCONVERT SCRIPTS: X An addition set of scripts are included with this distribution Xto convert existing tapes of the '/news/...' articles to the format used Xby the 'rd.tape.sh'. These are run from '/news/.start' to prevent Xinterference with normal data collection processes. X X'strt.hdr.sh': The definitions for the various file names and directories Xare listed here for use by all the conversion scripts so that these file Xnames are used consistently for all the operations. X X'convert.sh': The state machine which controls the separate operations Xto convert regular tapes to this format. X X'conv.rd': This consecutively reads the chapters from the AT&T type Xtapes into '/news/.start/recover/news/*..' so that they can be processed Xby 'conv.sq' until sufficient articles are available to fill a chapter Xin the compressed format. X X'conv.prep': A copy of 'prep.sh' which accesses the special directories Xused by the conversion process. X X'conv.wr': A copy of 'wr.tape' for use by the conversion process. X X'rd.params': A generated file which controls the reading of the tapes. X X'wr.params': A generated file which controls the writing of the Xcompressed files to tape. It is identical to the 'params' file used by Xthe regular process but is maintained separately in order to preclude Xinterference between the processes. X XADDITIONAL SCRIPT: X'purge.sh': This is not necessary for the operation of any of the tape Xoperations. It searches the entire '/news/..' file systems for empty Xdirectories and deletes these derelicts generated by cross postings. XI run it once a week. X XINSTALLATION AIDS: X'install.sh': This shell script makes the necessary directories and Xtransfers the programs to the places they will be found by 'header.sh' Xand 'strt.hdr.sh'. It also calls the 'Makefile' to produce the 'c' code Xbinaries. The configuration produced is that which is to be used by Xnormal operations. X END_OF_Digest if test 4946 -ne `wc -c <Digest`; then echo shar: \"Digest\" unpacked with wrong size! fi # end of overwriting check fi if test -f Install -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"Install\" else echo shar: Extracting \"Install\" \(2194 characters\) sed "s/^X//" >Install <<'END_OF_Install' XBASE=/news/.tape.ops XGRP=news XUSER=netnews X Xif [ ! -d $BASE ] Xthen X mkdir $BASE X chgrp $GRP $BASE X chown $USER $BASE Xfi X Xfor i in header.sh squirrel.sh prep.sh wr.tape.sh rd.tape.sh Xdo X echo "Moving '$i' to '$BASE'" X cp "$i" $BASE Xdone Xchmod 755 $BASE/* Xecho "Moving 'save.list' to '$BASE'" Xcp save.list $BASE Xchgrp $GRP $BASE/* Xchown $USER $BASE/* X Xif [ ! -d $BASE/docs ] Xthen X mkdir $BASE/docs X chgrp $GRP $BASE/docs X chown $USER $BASE/docs Xfi X Xfor i in README Digest Theory Xdo X echo "Moving '$i' to '$BASE/docs'" X cp "$i" $BASE/docs Xdone Xchgrp $GRP $BASE/docs/* Xchown $USER $BASE/docs/* X Xif [ ! -d $BASE/src ] Xthen X mkdir $BASE/src X chgrp $GRP $BASE/src X chown $USER $BASE/src Xfi X Xfor i in read.vtoc.c pick.files.c group.c Makefile Xdo X echo "Moving '$i' to '$BASE/src'" X cp "$i" $BASE/src Xdone X Xhere=$PWD Xcd $BASE/src Xmake Xchgrp $GRP $BASE/src/* Xchown $USER $BASE/src/* Xcd $here X Xline="20 02 * * * /bin/su netnews % ksh -c /usr/lib/news/tape.ops/squirrel" Xecho "Do you want me to add:\n$line\nto '/usr/lib/crontab'? <y/n> \c" Xread ch Xif [ "$ch" = "y" ] Xthen X cp /usr/lib/crontab /tmp/cron X echo "$line" >> /tmp/cron X if [ $? = 0 ] X then X mv /tmp/cron /usr/lib/crontab X fi Xfi X Xecho "Do you want to run 'purge'? <y/n> \c" Xread ch Xif [ "$ch" = "y" ] Xthen X cp purge.sh /usr/local/bin X chmod 0755 /usr/local/bin/purge.sh X chgrp bin /usr/local/bin/purge.sh X chown bin /usr/local/bin/purge.sh X line="\ X10 03 * * 3 /bin/su netnews % ksh -c /usr/local/bin/purge > /dev/null" X echo "Do you want me to add:\n$line\nto '/usr/lib/crontab'? <y/n> \c" X read ch X if [ "$ch" = "y" ] X then X cp /usr/lib/crontab /tmp/cron X echo "$line" >> /tmp/cron X if [ $? = 0 ] X then X mv /tmp/cron /usr/lib/crontab X fi X fi Xfi X XSTART=/news/.start Xecho "Do you install the conversion scripts in '$START'? <y/n> \c" Xread ch Xif [ "$ch" = "y" ] Xthen X mkdir $START X chgrp $GRP $START X chown $USER $START X for i in strt.hdr.sh convert.sh conv.rd.sh conv.sq.sh \ Xconv.prep.sh wr.tape.sh X do X echo "Moving '$i' to '$START'" X cp $i $START X done X mv $START/wr.tape.sh $START/conv.wr.sh X chmod 0755 $START/* X echo "Moving 'save.list' to '$START'" X chgrp $GRP $START/* X chown $USER $START/* Xfi X Xecho "Installation complete." END_OF_Install if test 2194 -ne `wc -c <Install`; then echo shar: \"Install\" unpacked with wrong size! fi chmod +x Install # end of overwriting check fi if test -f MANIFEST -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"MANIFEST\" else echo shar: Extracting \"MANIFEST\" \(774 characters\) sed "s/^X//" >MANIFEST <<'END_OF_MANIFEST' X File Name Archive # Description X----------------------------------------------------------- X Digest 1 X Install 1 X MANIFEST 1 This shipping list X Makefile 1 X README 1 X Theory 2 X conv.prep.sh 1 X conv.rd.sh 1 X conv.sq.sh 1 X convert.sh 1 X copy.sh 1 X group.c 1 X header.sh 1 X pick.files.c 2 X prep.sh 1 X purge.sh 1 X rd.tape.sh 1 X read.vtoc.c 1 X save.list 1 X squirrel.sh 1 X strt.hdr.sh 1 X wr.tape.sh 2 END_OF_MANIFEST if test 774 -ne `wc -c <MANIFEST`; then echo shar: \"MANIFEST\" unpacked with wrong size! fi # end of overwriting check fi if test -f Makefile -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"Makefile\" else echo shar: Extracting \"Makefile\" \(254 characters\) sed "s/^X//" >Makefile <<'END_OF_Makefile' XCFLAGS=-O -v XLDFLAGS=-s X XSHAREDLIB=/lib/crt0s.o /lib/shlib.ifile X XTARGETS=group pick.files read.vtoc X XOBJS=$(TARGETS).o X Xall: $(TARGETS) X X$(TARGETS): $$@.o X $(LD) $(LDFLAGS) $(SHAREDLIB) $? $(LIBS) -o $@ X chgrp news $@ X chown netnews $@ X Xclean: X rm *.o; END_OF_Makefile if test 254 -ne `wc -c <Makefile`; then echo shar: \"Makefile\" unpacked with wrong size! fi # end of overwriting check fi if test -f README -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"README\" else echo shar: Extracting \"README\" \(2687 characters\) sed "s/^X//" >README <<'END_OF_README' X This collection of programs was designed to permit the archiving Xof news article directly to UNIX PC type tapes. Articles are protected Xfrom 'expire' by linking them to a separate directory. Because some Xusers have the news written to separate partition of the disk or to a Xsecond disk, all operations and the linking process is accomplished in a Xsub directory of /news. A dot file is used to avoid possible conflicts Xwith the regular news programs. X X News groups to be archived are selected by entering their names Xin full path name format in '/news/.tape.ops/save.list'. Note that the Xsearch algorithm is recursive and any directories encountered are Xsearched in turn. This means that an entry of '/news/unix-pc' Xin 'save.list' will search all the groups included under 'unix-pc' such Xas '/news/unix-pc/general', '/news/unix-pc/sources, etc. In some cases Xthis may result in archiving of groups not really wanted. X X The 'install.sh' script makes all the needed directories and Xtransfers the scripts to the locations the other scripts expects to Xfind them. It also runs 'make' to produce the binaries for the operations Xwhich were not readily implemented in shell. It should be noted that Xthese scripts must run by '/bin/ksh' as the bourne shell does not Xdo 'functions' or '(( .. ))' type calculations. X X 'Squirrel.sh' should be run by cron at least as often as the Xretention duration of 'expire' so that articles are not deleted before Xthey are 'squirreled'. There is a line in 'install.sh' which can be Xuncommented which will write an appropriate entry into your 'crontab'. X X Another set of scripts are included to convert tapes which were Xwritten by the regular processes. They are designed to be run from X'/news/.start' to keep them from interfering with the regular 'run' Xoperations. After all existing tapes have been converted, the entire X'/news/.start' directory may be deleted. These are set up as an option Xin 'install.sh'. X X Although not really part of the archiving procedures, 'purge' is Xincluded to delete empty directories from the /news heading. These Xdirectories are remnants of cross postings long since forgotten. This Xtoo is an option in 'install.sh'. X X Hopefully, the documentation included in "Theory" and "Digest" Xwill provide a guide into using and modifying these scripts. Comments Xin the scripts and in the '.c' files should also be referenced for more Xinformation. X X If you run into trouble, want additional information or have Xsuggestions, please e-mail at the following address. X XVernon C. Hoxie {ncar,nbires,boulder,isis}!scicom!zebra!vern X3975 W. 29th Ave. voice: 303-477-1780 XDenver, Colo., 80212 TB+ uucp: 303-455-2670 END_OF_README if test 2687 -ne `wc -c <README`; then echo shar: \"README\" unpacked with wrong size! fi # end of overwriting check fi if test -f conv.prep.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"conv.prep.sh\" else echo shar: Extracting \"conv.prep.sh\" \(2555 characters\) sed "s/^X//" >conv.prep.sh <<'END_OF_conv.prep.sh' Xif [ ! "$WRKDIR" ] Xthen X . /news/.start/strt.hdr.sh # Use this for start up Xfi X Xecho `date +%T`" Starting prep routines for transfer to tape." X X# Clean out any previous effort to generate a 'savdir' etc. Then X# create new ones. X Xif [ -d $SAVDIR ] Xthen X rm -r $SAVDIR Xfi Xmkdir $SAVDIR X Xif [ -f $TMP1 ] Xthen X rm $TMP1 Xfi Xif [ -f $TMP2 ] Xthen X rm $TMP2 Xfi X Xif [ -s $PARAMS ] Xthen X fnum=`awk - '{ if ( $0 ~ /Last fnum:/ ) print $3 }' $PARAMS` Xelse X echo "We are in trouble!" X echo "There is no $PARAMS file so the 'squirrel'ed data is" X echo "defective. Clean it up before proceeding." X read X exit 1 Xfi X X> $CONFILE Xls -1 $ARCDIR > $PNTRS X Xecho `date +%T`" Reading 'Subject:' lines from 'arcdir'. " X X# Go through all the files in 'arcdir' and read the 'Subject: ' line. X# Add a tab and the file number to the line and put it in 'contents'. Xwhile X read pntr Xdo X subj=`sed -n '/^Subject:/ { X s/Subject: //p X q X }' $ARCDIR/$pntr` X if [ -n "$subj" ] X then X echo "$subj""\t"$pntr >> $CONFILE X fi Xdone <$PNTRS X Xecho `date +%T`" Changing arrangement of the 'contents' file. " X# Sort the 'contents' then run 'group' to place followup articles X# following the original. 'Group' also produces 'c.pntrs' which is X# a listing of the files in the same order that they appear X# in the 'contents'. X Xsort -fd -o $CONFILE $CONFILE X$SRCDIR/group X Xecho "\tYou should have a destination tape installed in the drive at X\tthis point. It will then have completed the retension phase X\tbefore you need it." X Xecho `date +%T`" Moving and compressing files to be taped. " X# Get the 'arcdir' file names from 'contents' and transfer to 'savdir' X# while compressing. Be careful not to use up the last 5 percent of X# disk. Strip the "Path:" line from the heading in the process. X Xwhile X read pntr Xdo X sed '/^Path:/d' $ARCDIR/$pntr | compress > $SAVDIR/$pntr.Z X chgrp $GRP $SAVDIR/$pntr.Z X chown $OWN $SAVDIR/$pntr.Z X set `df $NEWSDEV` X if [ $3 -lt $PERCENT ] X then X total=`cat $PNTRS | wc -l` X lnum=`fgrep -nx $pntr $PNTRS | cut -f1 -d":"` X let needed="$total-$lnum" Xecho "You are running low on disk space. There are only $3 blocks Xremaining. So far, $lnum of $total files have been compressed. X A. Single step through the rest of the list. X B. Open another window and transfer data to floppy disks. X C. Go ahead and put what we have on tape, save the rest. X D. Quit for now. XEnter A, B, C, or D. <cr>" X read ans X if [ "$ans" = "d" -o "$ans" = "D" ] X then X cd $currdir X exit 1 X elif [ "$ans" = "c" -o "$ans" = "C" ] X then X break X fi X fi Xdone < $CPNTRS X Xexit 0 END_OF_conv.prep.sh if test 2555 -ne `wc -c <conv.prep.sh`; then echo shar: \"conv.prep.sh\" unpacked with wrong size! fi chmod +x conv.prep.sh # end of overwriting check fi if test -f conv.rd.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"conv.rd.sh\" else echo shar: Extracting \"conv.rd.sh\" \(3258 characters\) sed "s/^X//" >conv.rd.sh <<'END_OF_conv.rd.sh' X# This is the sequence of operations to read articles from a tape X# prepared using 'prep' and 'wr.tape'. X Xif [ ! "$WRKDIR" ] Xthen X . /news/.start/strt.hdr.sh Xfi X X X# This should be the start of the program X Xecho `date +%T`" Starting read operations. " Xtoday=`date '+%y-%m-%d'` Xif [ ! -f $RDPARMS ] Xthen X > $RDPARMS X chgrp $GRP $RDPARMS X chown $OWN $RDPARMS Xfi X X# Check that the 'params' file has the proper entries. Xawk -F" " - '{ if ( $0 ~ /Name of work tape:/ ) ok1 = 1 X if ( $0 ~ /Date last taped:/ ) ok2 = 1 X if ( $0 ~ /Next chapter number:/ ) ok5 = 1 X } X END{ X if ( ok1 != 1 ) print "Name of work tape: " X if ( ok2 != 1 ) print "Date last taped: 0" X if ( ok5 != 1 ) print "Next chapter number: 0" X }' $RDPARMS >> $RDPARMS X Xret=1 Xch=x Xwhile [ $ret != 0 ] Xdo X Tgetname -t X ret=$? X if [ $ret != 0 ] X then Xecho "Please insert a source tape. <cr> to continue, Q)uit. \c" X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X exit 1 X fi X ret=1 X else X# From here on, the tape drive is locked, any 'exit' must perform X# 'Tgetname -l' to unlock the drive befor leaving. X Tgetname +l X Tgetname -v > /dev/null X set `Tgetname -r | tr -d "'"` X name=$1 X $SRCDIR/read.vtoc -w9 > this.tape X if [ -f old.tape ] X then X diff this.tape old.tape > /dev/null X ret=$? X if [ $ret != 0 ] X then Xecho "This doesn't seen to be the tape that I have been reading. XWaiting for another tape to be inserted. X<cr> to continue. Q)uit \c" X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X Tgetname -l X exit 1 X fi X fi X else Xecho "There isn't any info about any previous tape. If this is the beginning Xof a tape to be converted, <cr> to continue. Q)uit \c" X Tgetname -l X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X exit 1 X fi Xawk -F" " - '{ if ( $0 ~ /Name of work tape:/ ) $2 = "'"$name"'" X if ( $0 ~ /Date last taped:/ ) $2 = "'"$today"'" X if ( $0 ~ /Next chapter number:/ ) $2 = "1" X print $1" "$2 >> "'$TMP1'" X }' $RDPARMS X mv $TMP1 $RDPARMS X ret=0 X fi X fi Xdone X X# Get params Xtdate=`sed -n 's/Date last taped:\ //p' $RDPARMS` Xchap=`sed -n 's/Next chapter number:\ //p' $RDPARMS` X Xset `$SRCDIR/read.vtoc -r$chap` Xoffset=$2 Xblocks=$3 Xcat this.tape Xbase=$PWD Xif [ -d recover ] Xthen X echo `date +%T`" Purging old 'recover' directory. " X rm -r recover Xfi Xmkdir recover Xcd recover X Xecho `date +%T`" Reading chapter $chap from tape '$name' starting \ Xat $offset offset." Xdbuf -iT248O$offset /dev/rft3 2>/dev/nul | \ X execStrip tapecpio -icdR"T248" >/dev/null 2>$WRKDIR/errs X Xret=$? Xif [ $ret != 0 ] Xthen X echo "ret = "$ret X echo "Whoops. There was an error during the reading of '$name'. XBailing out! Please rerun this program." X Tgetname -l X exit 1 Xfi Xecho `date +%T`" Rewinding tape." Xnohup Tgetname -v >/dev/null 2>/dev/null & # to rewind the tape X Xcd $base X X(( chap=chap+1 )) X$SRCDIR/read.vtoc -r$chap > /dev/null Xret=$? Xif [ $chap -ge $ret ] Xthen X rm old.tape X chap=1 X echo "This was the last chapter on this tape. Next time insert \ Xa different tape." X Tgetname -l Xelse X mv this.tape old.tape Xfi Xawk -F" " - '{ if ( $0 ~ /Name of work tape:/ ) $2 = "'"$name"'" X if ( $0 ~ /Next chapter number:/ ) $2 = "'"$chap"'" X print $1" "$2 >> "'$TMP1'" X }' $RDPARMS X mv $TMP1 $RDPARMS Xrm -f /tmp/tape.vtoc Xexit 0 END_OF_conv.rd.sh if test 3258 -ne `wc -c <conv.rd.sh`; then echo shar: \"conv.rd.sh\" unpacked with wrong size! fi chmod +x conv.rd.sh # end of overwriting check fi if test -f conv.sq.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"conv.sq.sh\" else echo shar: Extracting \"conv.sq.sh\" \(3938 characters\) sed "s/^X//" >conv.sq.sh <<'END_OF_conv.sq.sh' X# The purpose of this script is to 'squirrel' away files in the news X# files for transfer to tape. By linking these files to entries in X# the 'arcdir', they are protected from 'expire' yet do not use up an X# great amount of disk space. X Xif [ ! "$WRKDIR" ] Xthen X . /news/.start/strt.hdr.sh Xfi X Xecho `date +%T`" Running 'squirrel' to link files into 'arcdir'. " X X# To commence, let's find out the status of our files and generate X# the directories and file entries which are missing. X Xif [ ! -d $ARCDIR ] Xthen X mkdir $ARCDIR X chmod 755 $ARCDIR Xfi X Xif [ ! -s $PARAMS ] Xthen X > $PARAMS X chgrp $GRP $PARAMS X chown $OWN $PARAMS X fnum = 1 Xelse X fnum=`sed -n 's/Next fnum to link:\ //p' $PARAMS` Xfi X X# Check that the 'params' file has the proper entries. Xawk -F" " - '{ X if ( $0 ~ /Next fnum to link:/ ) ok1 = 1 X if ( $0 ~ /Number of bytes held:/ ) ok2 = 1 X if ( $0 ~ /Date last squirreled:/ ) ok3 = 1 X if ( $0 ~ /Name of work tape:/ ) ok4 = 1 X if ( $0 ~ /Blocks of offset:/ ) ok5 = 1 X if ( $0 ~ /Date last taped:/ ) ok6 = 1 X if ( $0 ~ /Blocks available:/ ) ok7 = 1 X if ( $0 ~ /Next chapter number:/ ) ok8 = 1 X if ( $0 ~ /Compression factor:/ ) ok9 = 1 X } X END{ X if ( ok1 != 1 ) print "Next fnum to link: 0" X if ( ok2 != 1 ) print "Number of bytes held: 0" X if ( ok3 != 1 ) print "Date last squirreled: 0" X if ( ok4 != 1 ) print "Name of work tape: News-000" X if ( ok5 != 1 ) print "Blocks of offset: 0" X if ( ok6 != 1 ) print "Date last taped: 0" X if ( ok7 != 1 ) print "Blocks available: 45566" X if ( ok8 != 1 ) print "Next chapter number: 0" X if ( ok9 != 1 ) print "Compression factor: 1000" X }' $PARAMS >> $PARAMS X Xtoday=`date '+%y-%m-%d'` X X# Check for a link list and either create a new one or copy X# the current one to wrk.links for upadating. X Xif [ -s $LINKS ] Xthen X cp $LINKS $KLINKS Xelse X > $KLINKS Xfi X Xx=fnum Xwhile [ -f $ARCDIR/$x ] Xdo X rm -f $ARCDIR/$x X (( x = x + 1 )) Xdone X Xrm -f $WRKDIR/contents Xrm -f $GOOFS X Xfunction recur X{ X if [ -d $1 -a `du -s $1 | cut -f1` = 1 ] X then X return -1 X fi X typeset this next X this=$1 X set `ls -1 $1` X while [ $1 ] X do X next=$this/$1 X if [ -d $next ] X then X recur $next X fi X if [ -f $next ] X then X id=`sed -n '/^Message-ID:/ { X s/Message-ID: //p X q X }' $next | tr -d "/*?^$\[\]|"` X if [ -n "$id" ] X then X if X [ -z "`sed -n '/'"$id"'/p' $KLINKS`" ] X then X if ln $next $ARCDIR/$fnum X then X let fnum="fnum+1" X echo "$id""\t"$today \ X >> $KLINKS X else X echo $next", "$ARCDIR/$fnum >> $GOOFS X fi X fi X fi X fi X shift X done X} X Xwhile X read fname Xdo X dname=`echo $fname | cut -f1 -d"#"` X if [ -n "$dname" ] X then X echo `date +%T`" Squirreling "$dname. X recur $WRKDIR/recover$dname X fi Xdone <$LIST X X# Fudge the sum by 80 bytes for each entry to account fot the 'contents' X# file and the 'ss.filelist' which will added to the tape during write X# operations. Xnbytes=`ls -l $ARCDIR | awk - '{ sum = sum + $5 + 80 } X END{ print sum }' - ` X Xsort -d -o $LINKS $KLINKS X X> $TMP1 Xawk -F" " - '{ if ( $0 ~ /Next fnum to link:/ ) $2 = "'"$fnum"'" X if ( $0 ~ /Number of bytes held:/ ) $2 = "'"$nbytes"'" X if ( $0 ~ /Date last squirreled:/ ) $2 = "'"$today"'" X print $1" "$2 >> "'$TMP1'" X }' $PARAMS Xmv $TMP1 $PARAMS Xchgrp $GRP $PARAMS Xchown $OWN $PARAMS Xchgrp $GRP $LINKS Xchown $OWN $LINKS Xrm -f $KLINKS X Xoffset=`sed -n 's/Blocks of offset:\ //p' $PARAMS` Xchap=`sed -n 's/Next chapter number:\ //p' $PARAMS` Xfactor=`sed -n 's/Compression factor:\ //p' $PARAMS` X X(( min_size = ( MAXOFFSET - offset ) / (10 - chap) )) X(( this = nbytes/$factor )) # blocks for this collection X Xecho `date +%T`" Squirrling completed. X Recommended size of next tape entry: $min_size X Approximate size of this collection: $this" X Xif [ -d recover ] Xthen X X echo `date +%T`" Purging 'recover' directory." X rm -r recover Xfi X Xif [ "$chap" -lt 9 ] Xthen X (( this = this + 100 )) Xfi X Xif [ "$this" -gt "$min_size" ] Xthen X exit 2 Xelse X exit 0 Xfi END_OF_conv.sq.sh if test 3938 -ne `wc -c <conv.sq.sh`; then echo shar: \"conv.sq.sh\" unpacked with wrong size! fi chmod +x conv.sq.sh # end of overwriting check fi if test -f convert.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"convert.sh\" else echo shar: Extracting \"convert.sh\" \(1505 characters\) sed "s/^X//" >convert.sh <<'END_OF_convert.sh' X# This is the sequence of operations to read articles from a tape X# prepared using 'prep' and 'wr.tape'. X Xif [ ! "$WRKDIR" ] Xthen X . /news/.start/strt.hdr.sh Xfi X X# This is the start of the program X Xawk -F" " - '{ if ( $0 ~ /State of operations:/ ) ok1 = 1 X } X END{ X if ( ok1 != 1 ) print "State of operations: 1" X }' $RDPARMS >> $RDPARMS X Xif [ ! -s $PARAMS ] Xthen X echo "Next fnum to link: 1" > $PARAMS X echo "Number of bytes held: 0" >> $PARAMS X echo "Date last squirreled: 0" >> $PARAMS X echo "Next chapter number: 1" >> $PARAMS X echo "Blocks of offset: 0" >> $PARAMS X chgrp $GRP $PARAMS X chown $OWN $PARAMS Xfi X Xstate=1 X Xwhile [ 1 ] Xdo X state=`sed -n 's/State of operations:\ //p' $RDPARMS` X case $state in X 1) conv.rd.sh X ret=$? X if [ "$ret" = "0" ] X then X state=2 X else X Tgetname -l X exit X fi;; X 2) conv.sq.sh X ret=$? X case $ret in X 0) state=1;; X 1) tgetname -l X exit;; X 2) state=3;; X esac;; X 3) Tgetname -l X conv.prep.sh X ret=$? X if [ "$ret" = 0 ] X then X state=4 X else X exit X fi;; X 4) conv.wr.sh X ret=$? X if [ "$ret" = 0 ] X then X state=1 Xecho `date +%T`" Finished writing to new tape. Reinsert the source tape. X<cr> to continue. Q)uit." X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X exit X fi X else X exit X fi;; X esac Xawk -F" " - '{ if ( $0 ~ /State of operations:/ ) $2 = "'"$state"'" X print $1" "$2 >> "'$TMP1'" X }' $RDPARMS X mv $TMP1 $RDPARMS X chgrp $GRP $RDPARMS X chown $OWN $RDPARMS X Xdone Xexit END_OF_convert.sh if test 1505 -ne `wc -c <convert.sh`; then echo shar: \"convert.sh\" unpacked with wrong size! fi chmod +x convert.sh # end of overwriting check fi if test -f copy.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"copy.sh\" else echo shar: Extracting \"copy.sh\" \(5516 characters\) sed "s/^X//" >copy.sh <<'END_OF_copy.sh' X# This script is presented as a recovery tool to be used if a verify X# failure occurs when writing to tape using these scripts. It copies X# data from one tape to another chapter by chapter. X X# It requires that the source tape be written by the 'wr.tape.sh' script X# which is included in this collection of scripts. Changes might be made X# to these scripts so that other data can be copied, but I haven't had X# need to generate such revisions. That is left as an exercise for you. X X# A test is first run to determine that there is sufficient disk space X# available to copy each chapter. If insufficient space is available, X# the amount of additional space required is reported. X X# Space can be attained by copying some files to floppy disk. Also, if X# you have the time to rerun the 'prep.sh' script, you can free disk space X# by issuing the 'rm -r savdir' command.. X X# These routines will overwrite existing tapes if you desire. This option X# is made at the begining of the 'wr.tape.sh' operation. If you want to X# add on to an existing tape, you must edit this script in the initialization X# of the $PARAMS routine. You can also edit 'cp.params' from another window X# while the read operation is running. X Xexport NEWSDEV=/dev/fp002 Xexport WRKDIR=/news/.tape.ops Xexport CACHE=$WRKDIR/cache Xexport ARCDIR=$WRKDIR/cache Xexport PARAMS=$WRKDIR/cp.params Xexport SAVDIR=$WRKDIR/cache Xexport CONFILE=$WRKDIR/c.contents Xexport SRCDIR=/news/.tape.ops/src Xexport PNTRS=$WRKDIR/cp.pntrs Xexport CPNTRS=$WRKDIR/cp.pntrs Xexport TMP1=$WRKDIR/tmpp1 Xexport TMP2=$WRKDIR/tmpp2 Xexport COPY=1 Xexport GRP=news Xexport OWN=netnews Xexport PERCENT=6200 # This is the last five percent of disk space Xexport MAXOFFSET=45566 # or B1FE 512 byte blocks = 23329792 bytes X# If we add 2 blocks for header, B200 512 byte blocks X# or 1DAA blocks per stream??? X Xecho `date +%T`" Checking available memory and setting up files. " X XTgetname -t Xret=$? Xpau=0 Xwhile [ $pau -eq 0 ] Xdo X if [ $ret -ne 0 ] X then X echo \ X"Please insert a source tape. Q)uit or <cr> to continue. \c" X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X exit 1 X fi X Tgetname -t X ret=$? X else X Tgetname -v > /dev/null X biggest=`$SRCDIR/read.vtoc -g` X set `df $NEWSDEV` X avail=$3 X (( dif = biggest + $PERCENT - avail )) X if [ $dif -gt 0 ] X then X echo "The copy operation requires $dif more blocks of disk space." X exit X fi X pau=1 X fi Xdone X Xif [ ! -s $PARAMS ] Xthen X > $PARAMS Xfi X X# Check that the 'cp.params' file has the proper entries. Xawk -F" " - '{ if ( $0 ~ /Number of bytes held:/ ) ok1 = 1 X if ( $0 ~ /Next fnum to link:/ ) ok2 = 1 X if ( $0 ~ /Name of work tape:/ ) ok3 = 1 X if ( $0 ~ /Blocks of offset:/ ) ok4 = 1 X if ( $0 ~ /Date last taped:/ ) ok5 = 1 X if ( $0 ~ /Blocks available:/ ) ok6 = 1 X if ( $0 ~ /Next chapter number:/ ) ok7 = 1 X if ( $0 ~ /Compression factor:/ ) ok8 = 1 X if ( $0 ~ /Next source chapter:/ ) ok9 = 1 X } X END{ if ( ok1 != 1 ) print "Number of bytes held: 0" X if ( ok2 != 1 ) print "Next fnum to link: 0" X if ( ok3 != 1 ) print "Name of work tape: blank" X if ( ok4 != 1 ) print "Blocks of offset: 0" X if ( ok5 != 1 ) print "Date last taped: 1-1-1" X if ( ok6 != 1 ) print "Blocks available: 45566" X if ( ok7 != 1 ) print "Next chapter number: 1" X if ( ok8 != 1 ) print "Compression factor: 1024" X if ( ok9 != 1 ) print "Next source chapter: 1" X }' $PARAMS >> $PARAMS X Xfunction clean X{ X rm -f $CONFILE X rm -f $CPNTRS X rm -r $CACHE X rm -f $PARAMS X rm -f $TMP1 X rm -f $TMP2 X rm -f $WRKDIR/errs X} X Xpau=0 Xwhile [ "$pau" -eq 0 ] Xdo X ret=1 X while [ $ret != 0 ] X do X echo `date +%T`" Starting read operations. " X today=`date '+%y-%m-%d'` X X Tgetname -t X ret=$? X if [ $ret != 0 ] X then Xecho "Please insert the source tape. <cr> to continue, Q)uit. \c" X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X exit 1 X fi X ret=1 X else X# From here on, the tape drive is locked, any 'exit' must perform X# 'Tgetname -l' to unlock the drive befor leaving. X Tgetname +l X Tgetname -v > /dev/null X set `Tgetname -r | tr -d "'"` X name=$1 X fi X done X X# Get params X chap=`sed -n 's/Next source chapter:\ //p' $PARAMS` X X set `$SRCDIR/read.vtoc -r$chap` X offset=$2 X blocks=$3 X base=$PWD X if [ -d $CACHE ] X then X echo `date +%T`" Purging old 'cache' directory. " X rm -r $CACHE X fi X mkdir $CACHE X cd $CACHE X $SRCDIR/read.vtoc -w9 X echo `date +%T`" Reading chapter $chap from tape '$name' starting \ Xat $offset offset." X dbuf -iT248O$offset /dev/rft3 2>/dev/nul | \ X execStrip tapecpio -icdR"T248" >/dev/null 2>$WRKDIR/errs X X ret=$? X if [ $ret != 0 ] X then X echo "ret = "$ret X echo "\tThere was an error during the reading of the \ Xsource tape. X\tBailing out! Please rerun this program." X Tgetname -l X exit 1 X fi X echo `date +%T`" Rewinding tape." X Tgetname -v >/dev/null 2>/dev/null X Tgetname -l X cd $base X X (( chap = chap + 1 )) X $SRCDIR/read.vtoc -r$chap > /dev/null X ret=$? X if [ $chap -ge $ret ] X then X pau=1 X echo "This was the last chapter on this tape." X fi X X > $TMP1 X awk -F" " - '{ if ( $0 ~ /Next source chapter:/ ) $2 = "'"$chap"'" X print $1" "$2 >> "'$TMP1'" X }' $PARAMS X mv $TMP1 $PARAMS X X cp $ARCDIR/contents $CONFILE X sed 's/\-//; s/^d//; /Z$/ !d; s/.Z//' < $ARCDIR/ss.filelist > $CPNTRS X echo "Insert the target tape now. <cr> to continue, Q)uit" X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X clean X exit X fi X $WRKDIR/wr.tape.sh X echo "Insert the source tape now. <cr> to continue, Q)uit" X read ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X clean X exit X fi Xdone Xclean Xexit 0 END_OF_copy.sh if test 5516 -ne `wc -c <copy.sh`; then echo shar: \"copy.sh\" unpacked with wrong size! fi chmod +x copy.sh # end of overwriting check fi if test -f group.c -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"group.c\" else echo shar: Extracting \"group.c\" \(3720 characters\) sed "s/^X//" >group.c <<'END_OF_group.c' X/* This is a 'c' version of a 'sed/awk' script I first used to group */ X/* similar titles together in the contents. It isn't as fast as some */ X/* other algorithm might be but it beats the devil out of the script. */ X/* That script took 20 minutes to process 800 titles!!! */ X X/* The 'contents' is first read into the memory space allocated to */ X/* 'r1'. 'r1' is then scanned to count the number of lines. Next */ X/* 'r2' is allocated of sufficient size to hold 'lncnt' entries of */ X/* 'item' and 'r3' is allocated to hold 'lncnt' pointers to 'item'. */ X/* The 'items' in 'r2' are filled in by a second scan of 'r1'. */ X/* 'line' contains a pointer into 'r1' to the start of each entry in */ X/* the 'contents' listing and 'leng' is the number of characters in */ X/* the title. */ X/* The top 'item' is then used as pattern for scanning the remainder */ X/* of 'r1' for entries which contain the pattern or which are contained */ X/* in the pattern. When a match is found, the '*line' is written to */ X/* 'r4' and the 'leng' value in 'r2' is set to zero to indicate to */ X/* subsequent searchs that that entry has been "found". After each */ X/* pass, the 'r4' array is sorted to length and the **char's are */ X/* added to 'r3'. */ X/* After the scan sequence is completed, the data is read out from 'r1' */ X/* to the reopened 'contents' in the newly ordered sequence. */ X X#include <fcntl.h> X#include <malloc.h> X#include <stdio.h> X X#define CFILE "contents" X#define PFILE "c.pntrs" X Xtypedef struct { char *line; X int leng; X } item; X XFILE *fds1, *fds2; Xmain() X X{ X int i, k, fd, lcnt, done; X long size; X char *c1, *c2, *c3, *c4; X item *i1, *i2, *i3; X char **l1; X char *r1; X item *r2; X char **r3; X item *r4[100]; X if ( access( CFILE, 04 )) X { X perror( "No \"contents\" file." ); X exit( 1); X } X fd = open("contents", O_RDONLY ); X size = lseek( fd, 0, 2 ); X lseek( fd, 0, 0 ); X r1 = malloc( size ); X read( fd, r1, size ); X close( fd ); X for( c1 = r1, lcnt = 0; c1 <= r1 + size ; c1++ ) X if ( *c1 == '\n' ) lcnt++; X lcnt += 2; X r2 = (item *)calloc( lcnt, sizeof( item )); X r3 = (char **)calloc( lcnt, sizeof( char * )); X for( c1 = r1, i1 = r2; c1 < r1 + size, *c1; i1++ ) X { X i1->line = c1; X c2 = c1; X while ( *++c1 ) X { X if ( *c1 == '\t' ) i1->leng = c1 - c2; X else if ( *c1 == '\n' ) X { X *c1++ = '\0'; X break; X } X } X } X i1->line = '\0'; X i1->leng = 0; X l1 = r3; X for ( i1 = r2; *i1->line; i1++ ) X { X if ( ! i1->leng ) continue; X i = 0; X for ( i2 = i1; i2->line; i2++ ) X { X if ( ! i2->leng ) continue; X if ( i1->leng <= i2->leng ) X { X k = i2->leng - i1->leng; X c1 = i1->line; X c2 = i2->line; X } X else X { X k = i1->leng - i2->leng; X c1 = i2->line; X c2 = i1->line; X } X for ( k++; k; c2++, k-- ) X { X c3 = c1; X c4 = c2; X while ( *c3++ == *c4++ ) X if ( *c4 == '\t' ) X { X r4[i++] = i2; X break; X } X } X } X done = 0; X /* Now reorder the strings to put the shortest first. */ X while ( ! done ) X { X done = 1; X for ( k = 0; k < i - 1; k++ ) X { X if ( r4[k]->leng > r4[k+1]->leng ) X { X i3 = r4[k]; X r4[k] = r4[k+1]; X r4[k+1] = i3; X done = 0; X } X } X } X /* Now move the pointers to the output array */ X for ( k = 0; k < i; k++ ) X { X *l1++ = r4[k]->line; X r4[k]->leng = 0; X /* leng = 0 says this item already queued for output */ X } X } X if (( fds1 = fopen( CFILE, "w" )) == 0 ) X { X perror( "Opening contents file" ); X exit( -1 ); X } X X if (( fds2 = fopen( PFILE, "w" )) == 0 ) X { X perror( "Opening pointers file" ); X exit( -1 ); X } X for ( l1 = r3; *l1; l1++ ) X { X fprintf( fds1, "%s\n", *l1 ); X c1 = *l1; X while ( *c1++ != '\t' ); X fprintf( fds2, "%s\n", c1 ); X } X} END_OF_group.c if test 3720 -ne `wc -c <group.c`; then echo shar: \"group.c\" unpacked with wrong size! fi # end of overwriting check fi if test -f header.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"header.sh\" else echo shar: Extracting \"header.sh\" \(1202 characters\) sed "s/^X//" >header.sh <<'END_OF_header.sh' X X# The purpose of this script is to prepare the 'squirrel'ed files X# for transfer to tape. This consists of building a 'contents' X# file from the "Subject" line in the posted articles. The 'contents' X# is sorted and similar titles are moved together for easier reference. X# The articles are then compressed and moved to the 'savdir' where X# they can copied to tape. X Xexport NEWSDEV=/dev/fp002 Xexport CMDDIR=/usr/lib/news/tape.ops Xexport WRKDIR=/news/.tape.ops Xexport SRCDIR=$WRKDIR/src Xexport ARCDIR=$WRKDIR/arcdir Xexport PARAMS=$CMDDIR/params Xexport LIST=$CMDDIR/save.list # Directory list to be saved Xexport LINKS=$CMDDIR/link.list Xexport KLINKS=$CMDDIR/old.links Xexport GOOFS=$CMDDIR/goofs Xexport SAVDIR=$WRKDIR/savdir Xexport CONDIR=$WRKDIR/condir Xexport CONFILE=$WRKDIR/contents Xexport PNTRS=$WRKDIR/pntrs Xexport CPNTRS=$WRKDIR/c.pntrs Xexport RPNTRS=$WRKDIR/r.pntrs Xexport GRP=news Xexport OWN=netnews Xexport TMP=$WRKDIR/tmps Xexport TMP1=$WRKDIR/tmpp1 Xexport TMP2=$WRKDIR/tmpp2 Xexport PERCENT=6200 # This is the last five percent of disk space Xexport MAXOFFSET=45566 # or B1FE 512 byte blocks = 23329792 bytes X# If we add 2 blocks for header, B200 512 byte blocks X# or 1DAA blocks per stream??? END_OF_header.sh if test 1202 -ne `wc -c <header.sh`; then echo shar: \"header.sh\" unpacked with wrong size! fi # end of overwriting check fi if test -f prep.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"prep.sh\" else echo shar: Extracting \"prep.sh\" \(2540 characters\) sed "s/^X//" >prep.sh <<'END_OF_prep.sh' Xif [ ! "$WRKDIR" ] Xthen X . /news/.tape.ops/header.sh Xfi X Xecho `date +%T`" Starting prep routines for transfer to tape." X Xcurrdir=$PWD Xcd $WRKDIR X X# Clean out any previous effort to generate a 'savdir' etc. Then X# create new ones. X Xif [ -d $SAVDIR ] Xthen X rm -r $SAVDIR Xfi Xmkdir $SAVDIR X Xif [ -f $TMP1 ] Xthen X rm $TMP1 Xfi Xif [ -f $TMP2 ] Xthen X rm $TMP2 Xfi X Xif [ -s $PARAMS ] Xthen X fnum=`awk - '{ if ( $0 ~ /Last fnum:/ ) print $3 }' $PARAMS` Xelse X echo "\t\tWe are in trouble!" X echo "\tThere is no $PARAMS file so the 'squirrel'ed data is" X echo "\tdefective. Clean it up before proceeding." X read X exit 1 Xfi X X> $CONFILE Xls -1 $ARCDIR > $PNTRS X Xecho `date +%T`" Reading 'Subject:' lines from 'arcdir'. " X X# Go through all the files in 'arcdir' and read the 'Subject: ' line. X# Add a tab and the file number to the line and put it in 'contents'. Xwhile X read pntr Xdo X subj=`sed -n '/^Subject:/ { X s/Subject: //p X q X }' $ARCDIR/$pntr` X if [ -n "$subj" ] X then X echo "$subj""\t"$pntr >> $CONFILE X fi Xdone <$PNTRS X Xecho `date +%T`" Changing arrangement of the 'contents' file. " X X# Sort the 'contents' then run 'group' to place followup articles X# following the original. 'Group' also produces 'c.pntrs' which is X# a listing of the files in the same order that they appear X# in the 'contents'. X Xsort -fd -o $CONFILE $CONFILE X$SRCDIR/group X Xecho "\nYou should have a destination tape installed in the drive at Xthis point. It will then have completed the retension phase Xbefore you need it.\n" X Xecho `date +%T`" Moving and compressing files to be taped. " X# Get the 'arcdir' file names from 'contents' and transfer to 'savdir' X# while compressing. Be careful not to use up the last 5 percent of X# disk. Xwhile X read pntr Xdo X sed '/^Path:/d' $ARCDIR/$pntr | compress > $SAVDIR/$pntr.Z X chgrp $GRP $SAVDIR/$pntr.Z X chown $OWN $SAVDIR/$pntr.Z X set `df $NEWSDEV` X if [ $3 -lt $PERCENT ] X then X total=`cat $PNTRS | wc -l` X lnum=`fgrep -nx $pntr $PNTRS | cut -f1 -d":"` X let needed="$total-$lnum" Xecho "You are running low on disk space. There are only $3 blocks Xremaining. So far, $lnum of $total files have been compressed. X A. Single step through the rest of the list. X B. Open another window and transfer data to floppy disks. X C. Go ahead and put what we have on tape, save the rest. X D. Quit for now. XEnter A, B, C, or D. <cr>" X read ans X if [ "$ans" = "d" -o "$ans" = "D" ] X then X cd $currdir X exit 1 X elif [ "$ans" = "c" -o "$ans" = "C" ] X then X break X fi X fi Xdone < $CPNTRS X X# Now do the tape operations. Xwr.tape.sh END_OF_prep.sh if test 2540 -ne `wc -c <prep.sh`; then echo shar: \"prep.sh\" unpacked with wrong size! fi chmod +x prep.sh # end of overwriting check fi if test -f purge.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"purge.sh\" else echo shar: Extracting \"purge.sh\" \(824 characters\) sed "s/^X//" >purge.sh <<'END_OF_purge.sh' X# The purpose of this script is 'purge' the /news files of unused X# directories. X XTMP=/tmp/purge X Xfunction recur X{ X if [ -d $1 -a `du -s $1 | cut -f1` = 1 ] X then X return -1 X fi X typeset this next X this=$1 X set `ls -1 $1` X while [ $1 ] X do X next=$this/$1 X if [ -d $next ] X then X recur $next X if [ $? -lt 0 ] X then X echo $next >> $TMP X rmdir $next X fi X fi X shift X done X} X X# This is the real start of the script, the functions had to come first. X X(echo "To: System Administrator" Xecho "Subject: '/news' directories\n" Xecho "\tThe following '/news' directories have been purged.") > $TMP X Xrecur /news X Xcat $TMP | mail root Xrm $TMP X X# Maybe we shouldn't have zapped this directory, put it back. Xif [ ! -d "/news/.rnews" ] Xthen X mkdir /news/.rnews X chgrp news /news/.rnews X chown netnews /news/.rnews Xfi Xexit END_OF_purge.sh if test 824 -ne `wc -c <purge.sh`; then echo shar: \"purge.sh\" unpacked with wrong size! fi chmod +x purge.sh # end of overwriting check fi if test -f rd.tape.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"rd.tape.sh\" else echo shar: Extracting \"rd.tape.sh\" \(2202 characters\) sed "s/^X//" >rd.tape.sh <<'END_OF_rd.tape.sh' X# This is the sequence of operations to read articles from a tape X# prepared using 'prep' and 'wr.tape'. X Xif [ ! "$WRKDIR" ] Xthen X . /news/.tape.ops/header.sh Xfi X X X# This should be the start of the program X Xecho `date +%T`" Starting read operations. " X Xwhile [ "$ret" != 0 ] Xdo X Tgetname -t 2/dev/null X ret=$? X if [ "$ret" != 0 ] X then X echo "Please insert a tape. Q)uit or C)ontinue. \c" X read -r ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X exit X fi X ret=1 X else X# From here on, the tape drive is locked, any 'exit' must perform X# 'Tgetname -l' to unlock the drive before leaving. X Tgetname +l X Tgetname -b > /dev/null 2>/dev/null X ret=$? X set `Tgetname -r | tr -d "'"` X name=$1 X offset=$2 X if [ $offset = 0 ] X then X echo "\nYou may have inserted the wrong tape. XThis tape has no data written to it. Enter \"g\" when you have Xanother tape inserted. Enter \"q\" to quit." X read -r ch X if [ "$ch" = "q" -o "$ch" = "Q" ] X then X Tgetname -l X exit X else Xecho "\nWaiting for another tape to be inserted. XPress any key to continue. Q)uit \c" X read -r ch X if [ "ch" = "q" -o "$ch" = "Q" ] X then X Tgetname -l X exit X fi X fi X ret=1 X fi X fi Xdone X X# 'read.vtoc -w9' will write all used chapters to standard out in column X# format. X Xecho `date +%T`" The contents of tape \""$name"\" are:" X $SRCDIR/read.vtoc -w9 Xecho "Which chapter do you wish to access? \c" Xread chap X Xset `$SRCDIR/read.vtoc -r$chap` Xoffset=$2 Xblocks=$3 X Xbase=$PWD Xif [ ! -d $name ] Xthen X mkdir $name Xfi Xcd $name Xif [ ! -d $chap ] Xthen X mkdir $chap Xfi Xcd $chap X Xtapecpio -icuxRT32O$offset contents < /dev/rft3 2>/dev/null Xret=$? Xif [ $ret != 0 ] Xthen X echo "ret = "$ret X echo "\tThere was an error during the reading of the source tape. X\tBailing out! Please rerun this program." X Tgetname -l X exit 1 Xfi X$SRCDIR/pick.files Xecho `date +%T`" Searching for files." Xlist=`cat r.pntrs` Xdbuf -iT124O$offset /dev/rft3 | \ X execStrip tapecpio -icvR"T124" $list 1> /dev/null 2>$WRKDIR/errs X Xecho `date +%T`" Rewinding tape." XTgetname -v # to rewind the tape XTgetname -l Xecho `date +%T`" Your recovered files are in:\n"$PWD X Xrm -f /tmp/tape.vtoc Xrm -f r.pntrs Xrm -f ss.filelist X Xcd $base Xexit END_OF_rd.tape.sh if test 2202 -ne `wc -c <rd.tape.sh`; then echo shar: \"rd.tape.sh\" unpacked with wrong size! fi chmod +x rd.tape.sh # end of overwriting check fi if test -f read.vtoc.c -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"read.vtoc.c\" else echo shar: Extracting \"read.vtoc.c\" \(3698 characters\) sed "s/^X//" >read.vtoc.c <<'END_OF_read.vtoc.c' X/* This is one way of getting around that damned 'TdspSSss' program */ X/* makes a window that you can't kill. */ X X/* Flags: */ X/* c print [num] chapter number and offset, for writing */ X/* d debug, prints data about all 9 chapters */ X/* g return the size of the larget chapter */ X/* w write, print data about all used chapters */ X/* r read, print chapter offset and size, only one */ X/* entry if a number is included. */ X/* - ignored */ X/* [0-9] select specific chapter data */ X/* if a chapter number is present with either 'c' or 'w' */ X/* data relevant to that entry number is written to */ X/* '/tmp/ss.osets'. */ X/* The return value is the chapter number of the next blank */ X/* chapter unless [num] is used. Then it is [num]. */ X X#include <stdio.h> X#include <fcntl.h> X#include <string.h> X X#define VTOC "/tmp/tape.vtoc" Xstruct names { X char text[46]; X short files; X int start; X int bytes; X}; Xint fd, cflag, dflag, gflag, wflag, rflag; Xstruct names entry; X Xmain(argc, argv) Xint argc; Xchar *argv[]; X{ X int i, num; X char *p, ch; X i = 1; X cflag = 0; X dflag = 0; X gflag = 0; X rflag = 0; X wflag = 0; X num = -1; X while ( i < argc ) X { X p = argv[i]; X while ( ch = *p++ ) switch( ch ) X { X case '0': X case '1': X case '2': X case '3': X case '4': X case '5': X case '6': X case '7': X case '8': X case '9':num = ch & 0x0f; break; X case 'c':cflag = 1; break; X case 'd':dflag = 1; break; X case 'g':gflag = 1; break; X case 'r':rflag = 1; break; X case 'w':wflag = 1; break; X case '-': X default: break; X } X i++; X } X if (( fd = open( VTOC, O_RDONLY)) == -1 ) X { X perror("Can't open /tmp/tape.vtoc"); X exit( -1 ); X } X lseek( fd, 520, 0 ); X if ( wflag | cflag ) i = w_info( num ); X else if ( rflag ) i = r_info( num ); X else if ( gflag ) i = greatest(); X close( fd ); X exit( i ); X} X Xint w_info( num ) Xint num; X{ X int fd2, i, j, k, m, n; X char *p; X char buf1[20]; X char name[10]; X char id[16]; X char comment[26]; X if ( num != '\0' ) j = num; X else j = -1; X k = 0; X if ( wflag ) printf( X"Chap Name Comment Files Offset Size\n"); X for (i=1; i <= 9; i++) X { X read( fd, &entry, sizeof(entry) ); X if ( entry.text[0] < '\0' ) X { X if ( dflag ){ X entry.text[0] = '\0'; X entry.text[8] = '\0'; X entry.text[21] = '\0'; X } X else X { X if ( cflag ) printf( "%d %d", i, k ); X close( fd ); X return( i ); X } X } X if ( wflag ) X { X m = 0; X for ( n=0; n < 8; n++ ) name[n]=entry.text[m++]; X name[n]='\0'; X for ( n=0; n < 13; n++ ) id[n]=entry.text[m++]; X id[n]='\0'; X for ( n=0; n < 24; n++ ) comment[n]=entry.text[m++]; X comment[n]='\0'; X printf( X "%3d %-8s %-24s %4d %6d %6d\n", i, X name, comment, entry.files, entry.start, X entry.bytes ); X } X if ( i == j ) X { X X sprintf( buf1,"%d:%d", X entry.start, entry.bytes ); X for ( m = 0, p = buf1; *p; m++, p++ ); X *p++=m; X *p = '\0'; X fd2 = open( "/tmp/ss.osets", X O_WRONLY | O_CREAT, 0666 ); X write( fd2, buf1, strlen( buf1 ) ); X close( fd2 ); X if ( cflag ) X { X printf( "%d %d", i, k ); X close( fd ); X return( i++ ); X } X return( i ); X } X k = entry.start + entry.bytes; X } X return( i ); X} X Xint r_info( num ) Xint num; X X{ X int i, k; X k = num; X for ( i = 1; i <= 9 ; i++ ) X { X read( fd, &entry, sizeof(entry) ); X if ( entry.text[0] < '\0' ) break; X if ( num == -1 ) k = i; X if ( i == k ) printf( "%d %d %d", i, X entry.start, entry.bytes ); X } X return( i ); X} X Xint greatest() X X{ X int i, j, k; X k = 0; X for ( i = 1; i <= 9 ; i++ ) X { X read( fd, &entry, sizeof(entry) ); X if ( entry.text[0] < '\0' ) break; X j = atoi( entry.bytes ); X if ( j > k ) k = j; X } X return( k ); X} END_OF_read.vtoc.c if test 3698 -ne `wc -c <read.vtoc.c`; then echo shar: \"read.vtoc.c\" unpacked with wrong size! fi # end of overwriting check fi if test -f save.list -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"save.list\" else echo shar: Extracting \"save.list\" \(365 characters\) sed "s/^X//" >save.list <<'END_OF_save.list' X# This is the list of directories or files to be scanned X# for 'squirrel'ing to 'arcdir'. Later, 'arcdir' is to be X# processed by 'prep.sh' for transfer to tape. X X/news/to.zebra # This should be changed to your local name. X/news/unix-pc X/news/comp/sys/att X/news/comp/sources X X# The following entries may be needed for converting old tapes. X#/news/.tape.ops X#/tmp END_OF_save.list if test 365 -ne `wc -c <save.list`; then echo shar: \"save.list\" unpacked with wrong size! fi # end of overwriting check fi if test -f squirrel.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"squirrel.sh\" else echo shar: Extracting \"squirrel.sh\" \(4537 characters\) sed "s/^X//" >squirrel.sh <<'END_OF_squirrel.sh' X# The purpose of this script is to 'squirrel' away files in the news X# files for transfer to tape. By linking these files to entries in X# the 'arcdir', they are protected from 'expire' yet do not use up an X# great amount of disk space. X Xif [ ! "$WRKDIR" ] Xthen X . /news/.tape.ops/header.sh Xfi X Xcd $WRKDIR X X# To commence, let's find out the status of our files and generate X# the directories and file entries which are missing. X Xif [ ! -d $ARCDIR ] Xthen X mkdir $ARCDIR X chmod 755 $ARCDIR Xfi X Xif [ ! -s $PARAMS ] Xthen X > $PARAMS X chgrp $GRP $PARAMS X chown $OWN $PARAMS X fnum = 1 Xelse X fnum=`sed -n 's/Next fnum to link:\ //p' $PARAMS` Xfi X X# Check that the 'params' file has the proper entries. Xawk -F" " - '{ X if ( $0 ~ /Next fnum to link:/ ) ok1 = 1 X if ( $0 ~ /Number of bytes held:/ ) ok2 = 1 X if ( $0 ~ /Date last squirreled:/ ) ok3 = 1 X if ( $0 ~ /Name of work tape:/ ) ok4 = 1 X if ( $0 ~ /Blocks of offset:/ ) ok5 = 1 X if ( $0 ~ /Date last taped:/ ) ok6 = 1 X if ( $0 ~ /Blocks available:/ ) ok7 = 1 X if ( $0 ~ /Next chapter number:/ ) ok8 = 1 X if ( $0 ~ /Compression factor:/ ) ok9 = 1 X } X END{ X if ( ok1 != 1 ) print "Next fnum to link: 0" X if ( ok2 != 1 ) print "Number of bytes held: 0" X if ( ok3 != 1 ) print "Date last squirreled: 0" X if ( ok4 != 1 ) print "Name of work tape: News-000" X if ( ok5 != 1 ) print "Blocks of offset: 0" X if ( ok6 != 1 ) print "Date last taped: 0" X if ( ok7 != 1 ) print "Blocks available: 45566" X if ( ok8 != 1 ) print "Next chapter number: 0" X if ( ok9 != 1 ) print "Compression factor: 1000" X }' $PARAMS >> $PARAMS X Xtoday=`date '+%y-%m-%d'` X X# If we have a 'link.list' file, remove entries over two months old Xif [ -s $LINKS ] Xthen X cp $LINKS $KLINKS X > $TMP X awk - 'BEGIN{ now = "'"$today"'" X split( now, old, "-" ) X old[2] -= 2 X if ( old[2] < 1 ){ X old[1] -= 1 X old[2] += 12 X } X if ( old[1] < 70 ) old[1] += 100 X daysofmonth = "31 28 31 30 31 30 31 31 30 31 30 31" X split( daysofmonth, days ) X if ( old[3] > days[ old[2] ] ) old[3] = days[ old[2] ] X FS = " " X } X { split( $2, was, "-" ) X if ( was[1] < 70 ) was[1] += 100 X if ( old[1] > was[1] ){ next } X if ( old[1] == was[1] && old[2] > was[2] ){ next } X if ( old[1] == was[1] && old[2] == was[2] && \ X old[3] > was[3] ){ next } X print $0 >> "'$TMP'" X }' $KLINKS X mv $TMP $KLINKS Xelse X > $KLINKS X > $LINKS Xfi X Xx=fnum Xwhile [ -f $ARCDIR/$x ] Xdo X rm -f $ARCDIR/$x X (( x = x + 1 )) Xdone X Xrm -f $WRKDIR/contents Xrm -f $GOOFS X Xfunction recur X{ X if [ -d $1 -a `du -s $1 | cut -f1` = 1 ] X then X return -1 X fi X typeset this next X this=$1 X set `ls -1 $1` X while [ $1 ] X do X next=$this/$1 X if [ -d $next ] X then X recur $next X fi X if [ -f $next ] X then X id=`sed -n '/^Message-ID:/ { X s/Message-ID: //p X q X }' $next | tr -d "/*?^$\[\]|"` X if [ -n "$id" ] X then X if X [ -z "`sed -n '/'"$id"'/p' $KLINKS`" ] X then X if ln $next $ARCDIR/$fnum X then X let fnum="fnum+1" X echo "$id""\t"$today >> $KLINKS X else X echo $next", "$ARCDIR/$fnum >> $GOOFS X fi X fi X fi X fi X shift X done X} X Xwhile X read fname Xdo X dname=`echo $fname | cut -f1 -d"#"` X if [ -n "$dname" ] X then X recur $dname X fi Xdone <$LIST X Xfunction commas X{ echo `echo $1 | awk - '{ n = length( $0 ) X s = n % 3 X if ( s == 0 ) s = 3 X m = substr( $0, 1, s ) X s++ X while ( n > s ){ X o = substr( $0, s, 3 ) X m = m","o X s += 3 X } X print m }' -` X} X Xnbytes=`ls -l $ARCDIR | awk - '{ sum = sum + $5 } X END{ print sum }' - ` Xcbytes=`commas $nbytes` X Xsort -d -o $LINKS $KLINKS X X> $TMP Xawk -F" " - '{ if ( $0 ~ /Next fnum to link:/ ) $2 = "'"$fnum"'" X if ( $0 ~ /Number of bytes held:/ ) $2 = "'"$cbytes"'" X if ( $0 ~ /Date last squirreled:/ ) $2 = "'"$today"'" X print $1" "$2 >> "'$TMP'" X }' $PARAMS Xmv $TMP $PARAMS Xchgrp $GRP $PARAMS Xchown $OWN $PARAMS Xchgrp $GRP $LINKS Xchown $OWN $LINKS Xrm -f $KLINKS X Xoffset=`sed -n 's/Blocks of offset:\ //p' $PARAMS` Xchap=`sed -n 's/Next chapter number:\ //p' $PARAMS` Xfactor=`sed -n 's/Compression factor:\ //p' $PARAMS` X X(( min_size = ( MAXOFFSET - offset ) / (10 - chap) )) X(( this = nbytes/factor )) # blocks for this collection X Xecho "To: netnews XSubject: File Squirreling.\n XYour 'savdir' currently has $cbytes bytes of news articles saved. X Recommended size of next tape entry: $min_size blocks X Approximate size of this collection: $this blocks" > $TMP Xif [ "$this" -gt "$min_size" ] Xthen X echo "\t\tYou better transfer these to tape soon.!" >> $TMP Xfi Xecho "\nThis is from your friendly file saver." >> $TMP Xcat $TMP | /bin/rmail netnews Xrm $TMP END_OF_squirrel.sh if test 4537 -ne `wc -c <squirrel.sh`; then echo shar: \"squirrel.sh\" unpacked with wrong size! fi chmod +x squirrel.sh # end of overwriting check fi if test -f strt.hdr.sh -a "${1}" != "-c" ; then echo shar: Will not over-write existing file \"strt.hdr.sh\" else echo shar: Extracting \"strt.hdr.sh\" \(1164 characters\) sed "s/^X//" >strt.hdr.sh <<'END_OF_strt.hdr.sh' X X# The purpose of this script is to prepare the 'squirrel'ed files X# for transfer to tape. This consists of building a 'contents' X# file from the "Subject" line in the posted articles. The 'contents' X# is sorted and similar titles are moved together for easier reference. X# The articles are then compressed and moved to the 'savdir' where X# they can copied to tape. X Xexport NEWSDEV=/dev/fp002 Xexport WRKDIR=/news/.start Xexport CACHE=$WRKDIR/cache Xexport ARCDIR=$WRKDIR/arcdir Xexport RDPARMS=$WRKDIR/rd.params Xexport PARAMS=$WRKDIR/wr.params Xexport SAVDIR=$WRKDIR/savdir Xexport CONFILE=$WRKDIR/contents Xexport SRCDIR=/news/.tape.ops/src Xexport PNTRS=$WRKDIR/pntrs Xexport CPNTRS=$WRKDIR/c.pntrs Xexport LIST=$WRKDIR/save.list # Directory list to be saved Xexport LINKS=$WRKDIR/link.list Xexport KLINKS=$WRKDIR/wrk.links Xexport GOOFS=$WRKDIR/goofs Xexport TMP1=$WRKDIR/tmpp1 Xexport TMP2=$WRKDIR/tmpp2 Xexport COPY=0 Xexport GRP=news Xexport OWN=netnews Xexport PERCENT=6200 # This is the last five percent of disk space Xexport MAXOFFSET=45566 # or B1FE 512 byte blocks = 23329792 bytes X# If we add 2 blocks for header, B200 512 byte blocks X# or 1DAA blocks per stream??? END_OF_strt.hdr.sh if test 1164 -ne `wc -c <strt.hdr.sh`; then echo shar: \"strt.hdr.sh\" unpacked with wrong size! fi # end of overwriting check fi echo shar: End of archive 1 \(of 2\). cp /dev/null ark1isdone MISSING="" for I in 1 2 ; do if test ! -f ark${I}isdone ; then MISSING="${MISSING} ${I}" fi done if test "${MISSING}" = "" ; then echo You have unpacked both archives. rm -f ark[1-9]isdone else echo You still need to unpack the following archives: echo " " ${MISSING} fi ## End of shell archive. exit 0 -- Vernon C. Hoxie {ncar,nbires,boulder,isis}!scicom!zebra!vern 3975 W. 29th Ave. voice: 303-477-1780 Denver, Colo., 80212 TB+ uucp: 303-455-2670
vern@zebra.UUCP (Vernon C. Hoxie) (01/09/90)
To those of you have tape machines and who pulled the scripts I posted the other day, the following correction should be made: In '/news/.tape.ops/header.sh, the line: From- export CMDDIR=/usr/bin/news/tape.ops To- export CMDDIR=/news/.tape.ops Note the dots. This error is a remnant of an earlier configuration I have since discarded. You can reinstate it if you wish, but other changes will also be required. vern -- Vernon C. Hoxie {ncar,nbires,boulder,isis}!scicom!zebra!vern 3975 W. 29th Ave. voice: 303-477-1780 Denver, Colo., 80212 TB+ uucp: 303-455-2670