[news.sysadmin] My script for mapfiles/pathalias updating

erict@flatline.UUCP (j eric townsend) (06/25/88)

Several people have asked about automating their mapfiles/pathalias
stuff.  I finally got around to commenting my script for uploading.

I run it with a cron entry of:

30 2 * * * /bin/su root -c "/u/bin/demap.sh | mail erict"

It has a whole *TWO* debugging status statements.  The first states
that the script has run (and gives the date/time), the second
only gets sent if the pathalias database was updated.  I just
mail 'em to myself for fun and profit. :-)

Here it is:
(Shar?  Who needs a shar?)


# History:
# demap.sh 7Jun88 JET -- wrote initial script
# 20Jun88 JET -- fixed the stupid find statement for pathalias database.
# 25Jun88 JET -- commented the script so others would be able to follow.

# erict@flatline.UUCP
# A really kludgey version that probably takes forever to run...
# but it was quick n dirty to write... 

echo "demap.sh run at" `date`
cd /usr/lib/mail/maps

# This pulls the maps from their spool/news directory, cleans them
# up, and puts them into the map directory...
# LastCheck is an empty file that I use as a date-keeper for the
# last time the spool directory was checked for new maps.

for i in `find /usr/spool/news/comp/mail/maps -type f \
 -newer /usr/lib/mail/maps/LastCheck -print`
do
sed -e "/^From: /D" -e "/^Path: /D" -e "/^Newsgroups: /D" -e "/^From /D" -e "/^Subject: /D" -e "/^Message-ID: /D" -e "/^Date: /D" -e "/^Expires: /D" -e "/^Sender: /D" -e "/^Lines: /D" -e "/^Approved: /D" -e "/^Supersedes: /D" $i \
| sh
done

touch /usr/lib/mail/maps/LastCheck


# do pathalias stuff
# thanks to apt@uhnix2.uh.edu (Alan Pfeiffer-Traum) for the find trick.
# this will rebuild the pathalias/smail database *if*, and only *if*
# new map files were processed.

latest_files=`find /usr/lib/mail/maps/[d,u]* -newer /usr/lib/mail/paths -print`

if [ -n "$latest_files" ]
then 
echo "Updating pathalias database."
cd /usr/lib/mail/maps
cat dir.form d.* u.* | pathalias -l flatline > /tmp/pathalias
cd /usr/lib/mail
mv paths paths.old
sort /tmp/pathalias > paths
chmod a+r paths
rm /tmp/pathalias
fi



Anyone who thinks they can do this better, please post your script.

Again, this is most definately a mondo-jock kludge.  But, it works. :-)
-- 
                                        Skate UNIX or go home, boogie boy...
"But why should I type "rm -r $HOME" if I want to play trek???"
J. Eric Townsend ->uunet!nuchat!flatline!erict smail:511Parker#2,Hstn,Tx,77007
             ..!bellcore!tness1!/

wcf@psuhcx.psu.edu (Bill Fenner) (06/27/88)

We use something slightly different here.  Instead of going through all
the /usr/spool/news/comp/mail/maps files, we have in our sys file

UUMAP:world,comp.mail.maps::/usr/lib/news/uumap

where /usr/lib/news/uumap is

#!/bin/sh
#
# uumap - unpack the usenet maps
#
umask 002
cd /usr/spool/uumap
/bin/sed -f /usr/lib/news/sedfile | /usr/local/bin/unshar
/usr/local/bin/compress -f *[0-9]

/usr/lib/news/sedfile is:

1,/^$/d
/^$/d
/^#/d

The first line strips the header, the second strips blank lines, and the
third strips comments (we don't have much disk space), and then it compresses
it.  We also use unshar instead of sh just in case someone posts something
like rm -r / to comp.mail.maps (not too horribly hard.)  I wonder how many
sites would go kaplooey after that happened, if it did?

Then every once in a while we run this.

#!/bin/sh
#
#  mkpaths  -- make a new pathalias "paths" file for smail
#
USAGE="Usage: mkpaths [-e errlogfile ]"

PATH=:/bin:/usr/bin:/usr/local/bin; export PATH

# where are the map files?
mapdir="/usr/spool/uumap"
# list of map files to use
mapfiles="d.* u.usa.* u.can.* Paths.*"
# where to place paths file (if -c given)
pathfile="/usr/lib/uucp/paths"
# default errlog file (if no -e)
deflog="/usr/lib/uucp/mkpaths.log"
# file that contains any dead links
deadfile="DEAD"

cd ${mapdir}

# parse command line
set -- `getopt cp:e: $*`
if [ $? != 0 ]; then
	echo ${USAGE}
	exit 2
fi
COPYTO=""
PATHSFILE=${pathfile}
ERRLOG=${deflog}
for i in $*
do
	case ${i} in
	   -e)  ERRLOG=${2}; shift; shift
		;;
	   --)  shift; break
		;;
	esac
done

uncompress * 2>${ERRLOG}

# get any dead links
list=""
if [ -r ${deadfile} -a -s ${deadfile} ]; then
	dead=`cat ${deadfile}`
	for sys in ${dead}
	do
		list="${list} -d ${sys}"
	done
fi

# run the sucker
(pathalias -v ${list} ${mapfiles} | sort > ${PATHSFILE}) 2>>${ERRLOG}

compress -f * 2>>${ERRORLOG}

exit 0

I got these scripts off the net somewhen a while back, and added some things
to compensate for our lack of disk space.  Anybody want to comment on this
one?

-- 
    Bitnet: wcf@psuhcx.bitnet     Bill Fenner     | "How can we dance
   Internet: wcf@hcx.psu.edu                      |  When the beds are burning"
  UUCP: {gatech,cmcl2,rutgers}!psuvax1!psuhcx!wcf |
 Fido: Sysop at 263/42                            | Now wait a second . . .

jlo@elan.UUCP (Jeff Lo) (06/27/88)

In article <957@flatline.UUCP> erict@flatline.UUCP (j eric townsend) writes:
># This pulls the maps from their spool/news directory, cleans them
># up, and puts them into the map directory...
># LastCheck is an empty file that I use as a date-keeper for the
># last time the spool directory was checked for new maps.
>
>touch /usr/lib/mail/maps/LastCheck
>
>Anyone who thinks they can do this better, please post your script.

Your script is very similar to the one I use here, the major difference
being that I keep two time stamp files.  I touch one (the new time stamp)
*before* I start extracting the maps, then use "-newer" on the other one (the
old time stamp).  When I am done, I move the new time stamp to the old one.
This way, if news happens to be coming in while I am extracting the maps
there is no race between find and rnews.  Better to get a map twice than
not at all.
-- 
Jeff Lo
..!{ames,hplabs}!elan!jlo
Elan Computer Group, Inc.
(415) 322-2450

vanpelt@unisv.UUCP (Mike Van Pelt) (06/28/88)

In article <957@flatline.UUCP> erict@flatline.UUCP (j eric townsend) writes:
[description of map-updating script]
>>Anyone who thinks they can do this better, please post your script.

I just put a line in my KILL file for comp.mail.maps, as follows:

/^/w | (cd /maps/newmaps ; sh)

I leave comp.mail.maps subscribed, so when I read the news, as soon as
I come to the map group, all the new maps are automatically copied to
the map directory.  All record-keeping is done by   rn  so there's
no need for timestamping and such.

I could also put a /^/j in there so it would automatically mark them
all as read after writing them to the map directory, but I preferr to
hit 'cy' manually after seeing them written out.
-- 
Mike Van Pelt                                 "Hey, hey, ho ho,
Unisys, Silicon Valley                        Western culture's got to go."
vanpelt@unisv.UUCP                            Stanford students and faculty.

vixie@palo-alto.DEC.COM (Paul Vixie) (06/29/88)

In article <320@elan.UUCP> jlo@elan.UUCP (Jeff Lo) writes:
# In article <957@flatline.UUCP> erict@flatline.UUCP (j eric townsend) writes:
# ># This pulls the maps from their spool/news directory, cleans them
# ># up, and puts them into the map directory...
# ># LastCheck is an empty file that I use as a date-keeper for the
# ># last time the spool directory was checked for new maps.
# >
# >touch /usr/lib/mail/maps/LastCheck
# >
# >Anyone who thinks they can do this better, please post your script.
# 
# Your script is very similar to the one I use here, the major difference
# being that I keep two time stamp files.  I touch one (the new time stamp)
# *before* I start extracting the maps, then use "-newer" on the other one (the
# old time stamp).  When I am done, I move the new time stamp to the old one.
# This way, if news happens to be coming in while I am extracting the maps
# there is no race between find and rnews.  Better to get a map twice than
# not at all.

I hacked up 'uuhosts -unbatch' to do a
	touch /usr/spool/uucpmap/comp.mail.maps/.time_stamp
every time it unpacked a map article; then I wrote a Makefile for /usr/lib/mail
that has paths.dir and paths.pag depending on paths which depends on the
.time_stamp file.  At 12:30am every night I run 'cd /usr/lib/mail;make',
which rebuilds the paths database if any map articles have been unpacked
since the last time the paths database was built.

It's actually a little more complicated, of course.  I depend on path.local
and glue.local, and rebuild the paths database if either of these changes
also.  I had thought to build a bunch of Makefile dependencies based on the
map files themselves, but I kept scrogging the Makefile :-(.  So I hacked
uuhosts as described above and use a static Makefile.

Note that I do not use uuhosts -index or uuhosts <host> for host lookups.
The "findhost" package is muchly better and faster for this.  Uuhosts is
great for unpacking the maps, though.  Unshar by itself will still exec a
/bin/sh once it has the headers out of the way; uuhosts execs a program
called "mapsh" which does a chroot(2) and then execs the real /bin/sh.  So
if someone puts "rm -rf /" in a comp.mail.maps article, the most I can lose
is the contents of /usr/spool/uucpmap.

I keep meaning to package all this stuff back up with my various hacks and
some documentation... someday... sigh...
-- 
Paul Vixie
Digital Equipment Corporation	Work:  vixie@dec.com	Play:  paul@vixie.UUCP
Western Research Laboratory	 uunet!decwrl!vixie	   uunet!vixie!paul
Palo Alto, California, USA	  +1 415 853 6600	   +1 415 864 7013

jfh@rpp386.UUCP (John F. Haugh II) (06/30/88)

In article <471@unisv.UUCP> vanpelt@unisv.UUCP (Mike Van Pelt) writes:
>In article <957@flatline.UUCP> erict@flatline.UUCP (j eric townsend) writes:
>[description of map-updating script]
>>>Anyone who thinks they can do this better, please post your script.

in the shell script which i use to run expire, i copy the maps out of
comp.mail.maps, unshar them, remove the article and THEN i run an
expire on comp.mail.maps.  it seems to work very well.

- john.
-- 
John F. Haugh II                 +--------- Cute Chocolate Quote ---------
HASA, "S" Division               | "USENET should not be confused with
UUCP:   killer!rpp386!jfh        |  something that matters, like CHOCOLATE"
DOMAIN: jfh@rpp386.uucp          |             -- with my apologizes

clewis@spectrix.UUCP (Chris "where are you this week?" Lewis) (06/30/88)

In article <3238@palo-alto.DEC.COM> vixie@palo-alto.DEC.COM (Paul Vixie) writes:

[ Regarding pulling the maps and using pathalias ]

>I hacked up 'uuhosts -unbatch' to do a
>	touch /usr/spool/uucpmap/comp.mail.maps/.time_stamp
>every time it unpacked a map article; then I wrote a Makefile for /usr/lib/mail
>that has paths.dir and paths.pag depending on paths which depends on the
>.time_stamp file.  At 12:30am every night I run 'cd /usr/lib/mail;make',
>which rebuilds the paths database if any map articles have been unpacked
>since the last time the paths database was built.

There's a simpler way (if you're running uuhosts):

This is a fragment of our uuhosts shell script.  The standard "-unbatch"
has been extended by the addition of the "$BIN/uuhosts -pathalias", and
the "-pathalias" has been added as a recursive call to do the path munging.  
Since unbatch doesn't do anything if there are no maps to process, pathalias 
will be run automatically only when new maps come in.  Since modifications 
to our local file are rare, I manually run "uuhosts -pathalias" when 
I change it - or simply wait for the next maps to come in.

	-unbatch)
		# Unbatch map articles batched by inews.  Called from cron.
		cd $MAPS/$UUCPMAP
		if [ -f Batch ]; then
			:
		else
			exit 1
		fi
		mv Batch Batch.working
		for f in `cat Batch.working`
		do
			$BIN/uuhosts -x < $f
			sleep 5
		done
		rm Batch.working

		$BIN/uuhosts -i

		$BIN/uuhosts -pathalias

		if [ -f $LIB/uuwhere ]; then
			$LIB/uuwhere
		fi
		exit 1
	;;

	-pathalias)
		cd $MAPS/$UUCPMAP
		$zcat d.* u.* path.* | \
			pathalias -f -dutstat -dcxhq -dihnp4 -dcbosgd -dhao -lspectrix \
			> /tmp/paths.1 2> $LIB/pathalias.err
		if [ ! -s /tmp/paths.1 ]; then
			echo "PATHALIAS FAILED!!!!" | $MAILER postmaster
			exit 1
		fi
		pathproc < /tmp/paths.1 > /tmp/paths.2
		mv /tmp/paths.2 $PATHS
		rm -f /tmp/paths.1
		;;

(BTW: we *are* running compressed maps.  pathproc comes from the smail
distribution and is just a small sed script wrapped around sort.)

I've been sort of wondering what all of the fuss about AT&T's dropping of
external forwarding of mail thru ihnp4 and cbosgd.  We've had them marked
dead for 6 months without any trouble.  We figgered that the load on those
machines was so high, and the delays are frequently so long that it wasn't
worth relying on them.  Especially given when many times other sites
exaggerated their links to ihnp4 so much that neighboring local-call site's 
mail got routed thru ihnp4.  And that Henry Spencer seemed to be telling 
us several times a month that his link to ihnp4 was down (pre-uunet<->mnetor).

And, AT&T's done yeoman's service for the net over the years - give 'em
a break!  Though, ihnp4 should never have let so many people connect to it -
maintaining a map that size would take several people full time just ensuring
that the other sites still exist from week to week!

We typically mark dead sites who:

	- Autoroute *all* mail and have out of date maps (rutgers 
	  occasionally rejects almost everything we send it - so I 
	  periodically mark rutgers dead.  Most of the time rutgers 
	  does a reasonable job - almost marked uunet dead once too...).
	- Insist on telling everybody about links outsider's are not 
	  permitted to use and bounce stuff (GRRRR!!!!).
	- Very overloaded or are having communication problems.
	  (often mentioned in news.config)
	- Mention that they're going out of service temporarily or 
	  permanently.  (often mentioned in news.config)
	- Sometimes minor sites get ridiculous about links to other 
	  machines (eg: I were to copy uunet's map and mark every cost
	  as zero ;-)  So, when a unknown machine pops into major paths,
	  I check to ensure that it's a moderately stable one and that
	  they intend to support the traffic.  If not, it gets marked dead.
-- 
Chris Lewis, 
These addresses will last only til June 30: (clewis@lsuc afterwards)
UUCP: {uunet!mnetor, utcsri!utzoo, lsuc, yunexus}!spectrix!clewis
Moderator of the Ferret Mailing List (ferret-list,ferret-request@spectrix)

sl@van-bc.UUCP (pri=-10 Stuart Lynne) (07/02/88)

In article <3386@rpp386.UUCP> jfh@rpp386.UUCP (The Beach Bum) writes:
>In article <471@unisv.UUCP> vanpelt@unisv.UUCP (Mike Van Pelt) writes:
>>In article <957@flatline.UUCP> erict@flatline.UUCP (j eric townsend) writes:
>>[description of map-updating script]
>>>>Anyone who thinks they can do this better, please post your script.
>

In my sys file:
	
	MAPS:world,comp.mail.maps:F

From a nightly cron script - getmap:


#!/bin/sh
#	@(#)getmaps.sh

#	Reads the netnews for the UUCP maps, extract them and
#	moves them to the storage area

DEST=/usr/spool/paths
ADM=mail
LOG=/local/lib/paths/maplog
SRC=/usr/spool/batch/MAPS
PATH=$PATH:/local/bin

cd $DEST
date >> $LOG

if [ -r $SRC ] 
then
	cat $SRC >> $SRC.work
	rm $SRC
fi

cat $SRC.work | while read i
do
	unshar $i
done


-- 
Stuart.Lynne@wimsey.bc.ca {ubc-cs,uunet}!van-bc!sl     Vancouver,BC,604-937-7532

daved@physiol.su.oz (Dave Davey) (07/04/88)

In article <3386@rpp386.UUCP>, jfh@rpp386.UUCP (John F. Haugh II) writes:
> In article <471@unisv.UUCP> vanpelt@unisv.UUCP (Mike Van Pelt) writes:
> >In article <957@flatline.UUCP> erict@flatline.UUCP (j eric townsend) writes:
> in the shell script which i use to run expire, i copy the maps out of
> comp.mail.maps, unshar them
We run pubindex(1) on the u.* files to support examining them with lookbib(1).
Searching is then very fast, and these utilities are better suited to the
multiline records than is grep(1).