clewis@ecicrl.UUCP (Chris Lewis) (01/09/91)
Archive-name: unpackmaps3.1/part1
#! /bin/sh
# This is a shell archive. Remove anything before this line, then feed it
# into a shell via "sh file" or similar. To overwrite existing files,
# type "sh file -c".
# The tool that generated this appeared in the comp.sources.unix newsgroup;
# send mail to comp-sources-unix@uunet.uu.net if you want that tool.
# Contents: README unpackmaps uuwhere
# Wrapped by clewis@ecicrl on Mon Dec 17 17:26:23 1990
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
echo If this archive is complete, you will see the following message:
echo ' "shar: End of archive 1 (of 1)."'
if test -f 'README' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'README'\"
else
echo shar: Extracting \"'README'\" \(7659 characters\)
sed "s/^X//" >'README' <<'END_OF_FILE'
X UNPACKMAPS V3.1
X Dec 17 1990
X Chris Lewis
X
X This README, unpackmaps and uuwhere are Copyright 1990 Chris Lewis,
X All Rights Reserved.
X
X You can do anything you want with it, provided that this copyright
X notice remains intact, you don't claim you wrote it yourself, and you
X don't make money selling it directly. You may redistribute this as
X you wish, but if you modify it, you should clearly indicate in this
X README file that it has been modified, by whom, and describe what
X has been changed. Reasonable copying charges are okay.
X
X Though I have taken pains to ensure that this program is reasonably
X reliable and secure, I cannot make any warrantee as to the reliability
X or security of this software when used on any computer. It is up to
X the user of this software to determine its suitability for their
X purposes and take responsibility for its operation.
X
XThis is source for a simple, *secure*, map unpacking facility.
X
XIt is much simpler than uuhosts, has roughly the same functionality,
Xis easier to install, probably faster and more efficient (at least
Xthe versions I've seen w.r.t. compression), and is considerably less
Xvulnerable to trojan maps than most map unpackers that other people use.
X(It has fewer vulnerabilities than uuhosts)
X
XThe intent is that any UNIX/XENIX/BSD system that can run news will
Xbe able to run this too, so I'm attempting to keep to greatest-common-
Xdenominator except for pathalias and things that I implement myself.
X
XThis release includes:
X
X - map unpacking
X - pathalias operation plus hooks for path customization.
X - a mechanism for viewing map entries for arbitrary sites.
X ("uuwhere")
X - automatic map article deletion if desired.
X - rebuild of where database.
X
XThis had been posted to comp.sources.misc about a year ago. The
Xprime differences here are that compress is run as a pipeline as stdin to
Xpathalias, so you don't need to modify pathalias anymore, and the
Xnew "uuwhere" tool. There is still no makefile because it wouldn't
Xdo anything useful.
X
XRegarding security: as many may remember, there's been a fair bit of
Xdiscussion on security of map unpacking on the net. Rather than play
Xaround with trying to make a secure *true* unshar, which probably noone would
Xtrust because it would be so big, I simply made a few simplifying assumptions
Xabout the map format and use an awk script to unpack a map article into
Xa map file. It checks for and refuses to unpack articles which have
Xslashes in their names. I sent off some mail to Mel asking whether the
Xassumptions I've made about map format are true, but never got any
Xresponse. I believe that this is *pretty* secure, in that it doesn't
Xhave to be run as root, doesn't use the Bourne shell for unpacking, and
Xis careful about the file names it creates. Please let me know if there
Xare any holes I didn't think of.
X
XGeneral operation:
X - your news is modified to batch incoming map article file names
X to a specific batch file (analogous to normal news batching).
X C-news users take note: you may have to utter magic incantations
X (hint "classes" in C-news Alpha) to get sendbatches to avoid
X trying to uux these... Another way is to use an explicit batch
X file name in the sys file that isn't under the usual "out.going"
X directories and point unpackmaps at it.
X - unpackmaps wakes up, usually once per day, and extracts the
X maps specified (if any) in the batch file into the map directory.
X Maps are extracted using a secure awk script without resort
X to setuid root or other wierdnesses. If you want the uuwhere
X facility, unpackmaps will extract site->map file correspondences.
X - If any maps were extracted, pathalias is fired off, and the
X resultant file put in the place specified.
X - If you've specified uuwhere, the where database will be regenerated.
X - if anything was done, unpackmaps sends you mail telling you
X what happened.
X
XInstallation:
X - If you want to compress the stored map files, set the COMPRESS
X variable to point your compress program. You should be using
X compress versions 3 or 4. A copy will have come with your
X news software.
X
X - I STRONGLY recommend that you build a special version of compress
X with 12 bit compression instead of 16 for this. When this
X is done, compress is considerably smaller (eg: bss of 32K instead
X of 400K+). The reason for this is obvious - pathalias is enormous
X when it's running, and so is a 16 bit compress. God help you if
X it starts to swap.
X
X Advantages:
X - the whole thing runs considerably faster
X - much less swap/paging
X - on our machine, 16 bit compress practically hangs
X everyone else when run at the same time as pathalias.
X
X Disadvantages:
X - the map directory is 10% (really! only 10%!) bigger.
X
X What I did was the following:
X - go to the source directory for compress
X - remove the binary if it is there.
X - say:
X make compress CFLAGS=-DUSERMEM=0
X - rename this to something like /usr/bin/compress12
X - make sure that uuwhere and unpackmaps shell scripts
X have the same name. Eg: /usr/bin/compress12
X
X - edit unpackmaps to set the variables at the beginning of
X the shell script. Note especially the batch file name
X (see below)
X - make the directory for the map files, owned by news.
X - put unpackmaps in a suitable place. Eg: /usr/lib/news
X - put uuwhere in a suitable place. Eg: /usr/local/bin.
X - unpackmaps should be run from the userid that owns and runs news.
X - su to the news userid, and run:
X unpackmaps -i
X This will build the initial path file.
X - insert into your crontab something like:
X 30 3 * * * /bin/su news -c "<path to unpackmaps>/unpackmaps > /dev/null"
X - insert into your news sys file something like:
X
X (C-news)
X
X maps:comp.mail.maps/all:f:/usr/spool/news/out.special/maps
X
X (B-news)
X
X maps:world,comp.mail.maps:F:/usr/spool/batch/maps
X
X - copy uuwhere to an accessible bin directory after modifying
X the configuration section at the beginning.
X
Xuuwhere "sitename" will give you the map name and line numbers where
Xthe site is defined (with #N comments). Adding a -v option will
Xinvoke compress (if necessary) and show you the map entry itself.
XThe uuwhere mechanism *only* works using the "#N" entries, and will
Xnot search for macro reassignments and other namings. One other drawback
Xis that uuwhere won't display whole map files directly, but that was
Xnever particularly useful anyways.
X
Xunpackmaps -i: will extract all map articles into the map spool area - useful
Xfor the first time you use it.
X
Xunpackmaps -p: runs pathalias even if no map articles were extracted.
X
Xunpackmaps -w: rebuilds uuwhere database even if no map articles were
Xextracted.
X
XIn order to push the paths file into /usr/lib/uucp, I created a file called
Xpaths in /usr/lib/uucp, with 644 permissions, owned by the userid that runs
Xunpackmaps.
X
XThe package will send mail to who you specify indicating which maps were
Xunpacked, and any error returns from pathalias.
X
XThe uuwhere database is built from the maps as they are unpacked. After
Xinstallation, it may take a while for the where database to be complete.
X(same as for path files themselves too). I have implemented a binary
Xsearch routine to make uuwhere go faster, but frankly, the sed's fast
Xenough on all but the slowest machine. The "-w" option permits you
Xto rebuild the where database immediately.
X
XLet me know of any changes you needed to make to get this to work.
XI'm also open to suggestions for new features....
X----------
XChris Lewis, Markham, Ontario, Canada
X{uunet!attcan,utgpu,yunexus,utzoo}!lsuc!{ecicrl|eci386}!clewis
END_OF_FILE
if test 7659 -ne `wc -c <'README'`; then
echo shar: \"'README'\" unpacked with wrong size!
fi
# end of 'README'
fi
if test -f 'unpackmaps' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'unpackmaps'\"
else
echo shar: Extracting \"'unpackmaps'\" \(8171 characters\)
sed "s/^X//" >'unpackmaps' <<'END_OF_FILE'
X:
X# Unpackmaps Copyright 1990, Chris Lewis, All Rights Reserved
Xtrap "rm -f /tmp/unp?$$; exit" 0 1 2 3 15
XIFS="
X"
Xexport IFS
XPATH=/bin:/usr/bin
Xexport PATH
X
X# The name of the file that you've caused your news system to
X# batch the file names of the map articles.
X# Eg: C-news
X#BATCH=/usr/lib/news/batch/b.maps/togo
X#Modern C-news (directory other than /usr/spool/news/out.going)
X#BATCH=/usr/spool/news/out.special/maps/togo
X# Eg: B-news
XBATCH=/usr/spool/batch/maps
X# News spool directory
XNEWSSPOOL=/usr/spool/news
X# Where you want the maps to go.
X# I like using /usr/spool/maps, but on our system, /usr/spool/news
X# is a separate file system, and /usr runs close to the limit...
XMAPDIR=/usr/spool/maps
X# Person to send results and error messages to
XNOTIFY=clewis
X# pathalias binary
XPATHALIAS=/usr/bin/pathalias
X# where you want the path files to go:
X# A convenient place is /usr/lib/uucp/paths which is the smail
X# default. If you're going to put this in /usr/lib/uucp, I suggest
X# (rather than make /usr/lib/uucp writeable by everybody), doing
X# the following:
X# su root
X# cd /usr/lib/uucp
X# touch paths
X# chown news paths (or usenet)
X# chmod 644 paths
XPATHFILE=/usr/lib/uucp/paths
X# Auxiliary options to pathalias. Tune to local tastes....
XPATHOPTS="-dwatmath -drobohack -decijmm -dcognos -dyunexus"
X# If you have a version[s] of your machine's map entry that is different
X# from what's published, change this variable to point at it/them.
X# (Eg: I publish the first entry here, and the second one is local tuning
X# and hidden connections)
XPATHLOCAL=/u/clewis/maps/path.local
X# If this variable is set to the compress binary, maps will be
X# compressed.
XCOMPRESS=/usr/bin/compress
X# 1 to strip comments from maps - don't do this if you want to use
X# uuwhere. However, this is a great space saver...
XNOCOMMENTS=0
X# Define to the name of a file where you want the where database
X# to be kept. Undef if you don't want uuwhere at all.
XWHEREFILE=$MAPDIR/where.db
X# Uncomment this if you want the map unpacker to remove the
X# News articles after the maps have been extracted from them.
X# DO NOT DO THIS IF YOU FORWARD MAP ARTICLES TO OTHER SITES!
X# This also relies on your awk returning "exit" codes properly.
X# Yours may not...
XUNLINK=1
X# PS: there is *one* possible edit that you might want to make
X# below - the maps used to generate wierd domains, but most of that
X# appears to be gone now (don't ask me, I never particularly understood
X# it, but since Peter Honeyman recommended it...). If you object
X# to these wierd domains, uncomment the egrep.
X
X# Edit no more....
X
Xumask 022
X
Xif test ! -d $MAPDIR -o ! -w $MAPDIR
Xthen
X echo "$MAPDIR missing, unwritable or not a directory" >&2
X exit 1
Xfi
X
Xfor i
Xdo
X case $i in
X -p)
X forcepath=true
X ;;
X -P)
X forcepath=false
X ;;
X -i)
X cd /
X rm -f $BATCH.work
X # using find/sort instead of ls just in case there's lots of
X # articles....
X find $NEWSSPOOL/comp/mail/maps -type f -print | sort > $BATCH
X ;;
X -w)
X forcewhere=true
X ;;
X *)
X echo "usage: unpackmaps [-i] [-p]" >&2
X exit 1
X ;;
X esac
Xdone
X
Xcd $MAPDIR
XWHERETMP=/tmp/WHERE$$
Xrm -f $WHERETMP
X
Xwhile test -f $BATCH -o -f $BATCH.work
Xdo
X # There is no window of vulnerability here as long as noone else is
X # creating $BATCH.work.
X if test ! -f $BATCH.work
X then
X mv $BATCH $BATCH.work
X fi
X
X while read i stuff
X do
X # Using stuff to capture remaining junk on line.
X # Eg: C-news article sizes.
X
X if test -z "$i"
X then
X break
X fi
X
X if test ! -r $i
X then
X echo "$i apparently superseded or expired"
X continue
X fi
X
X # This awk script depends on the following map article format:
X # <don't cares>
X # cat << 'something' > filename
X # map body
X # something
X # <don't cares>
X # "something" doesn't have to be enclosed in quotes in the cat line.
X # This isn't particularly fast - could be dramatically speeded up
X # if written in C, but I was trying to ensure that this is as simple
X # and self-evident as possible.
X
X awk '
X BEGIN {
X where = "'"$WHEREFILE"'"
X }
X $1 == "cat" && collecting == 0 {
X recno = 1
X endtoken=$3;
X if (substr(endtoken, 1, 1) == "'"'"'")
X endtoken=substr(endtoken, 2, length(endtoken)-2);
X collecting = 1;
X foundone = 1;
X name = $5;
X if (index(name, "/") != 0) {
X printf("Security violation attempt in %s!\n", "'$i'");
X exit 1;
X } else
X printf("extracting %s from %s\n", name, "'$i'");
X next;
X }
X
X {
X if (!collecting)
X next;
X if ($1 == endtoken) {
X line = "rm -f " name ".Z"
X print "" | line
X collecting = 0;
X next
X }
X if ($1 ~ /^#N/ && where) {
X for (i = 2; i <= NF; i++) {
X sname = $i
X if (p = index(sname, ","))
X sname = substr(sname, 1, p-1)
X printf "@%s %s %d\n", sname, name, recno >> \
X "'$WHERETMP'";
X }
X }
X if ("'$NOCOMMENTS'" == 1 && $0 ~ /#/)
X print substr($0, 1, index($0, "#")) > name
X else {
X print $0 > name
X }
X recno++
X }
X
X END {
X if (collecting) {
X printf("Non-terminated map in %s\n", "'$i'");
X exit 1;
X }
X if (!foundone) {
X printf("%s does not contain a properly formed map\n", "'$i'");
X exit 1;
X }
X }' $i
X
X if test $? = 0 -a -n "$UNLINK"
X then
X rm -f $i
X fi
X
X done < $BATCH.work
X rm $BATCH.work
Xdone > /tmp/unpA$$ 2>&1
X
Xif test -n "$COMPRESS"
Xthen
X files=`ls ?.* | sed -e '/\.Z$/d'`
X if test -n "$files"
X then
X $COMPRESS -f $files
X fi
Xfi
X
Xif test -f "$PATHALIAS" -a "$forcepath" != false
Xthen
X if test -s /tmp/unpA$$ -o "$forcepath" = true
X then
X
X (
X if test -n "$COMPRESS"
X then
X $COMPRESS -dc [ud].*.Z | cat - $PATHLOCAL
X else
X cat [ud].* $PATHLOCAL
X fi |
X
X $PATHALIAS -f $PATHOPTS |
X
X # format of the pathalias -f output is
X # cost host route
X #
X # format of a 'paths' file for smail is
X # host route first_hop_cost
X #
X # move cost field to end of line:
X
X sed 's/\(.*\) \(.*\) \(.*\)/\2 \3 \1/' |
X
X # convert target domain/host to lower case:
X
X #lcasep |
X
X # remove some additional wierdnesses (per Peter Honeyman):
X # You can leave it in or not.
X
X # egrep -v '(\.(com|edu|mil|gov|net|org|arpa|[a-z][a-z]) .*!.*!)|(.\.(com|edu|mil|gov|net|org|arpa|[a-z][a-z]) )' |
X
X # sort the stream:
X
X sort > /tmp/paths ) > /tmp/unpB$$ 2>&1
X
X if test ! -s /tmp/paths
X then
X echo "Pathalias failed no map file created" >> /tmp/unpB$$
X else
X cat /tmp/paths > $PATHFILE 2>> /tmp/unpB$$
X if test $? != 0
X then
X echo "Copy to $PATHFILE failed" >> /tmp/unpB$$
X else
X rm /tmp/paths
X fi
X echo "Map remade" >> /tmp/unpB$$
X ls -l $PATHFILE >> /tmp/unpB$$
X fi
X
X if test -s /tmp/unpB$$
X then
X echo "Pathalias output:" >> /tmp/unpA$$
X cat /tmp/unpB$$ >> /tmp/unpA$$
X fi
X fi
Xfi
X
Xif test -n "$forcewhere"
Xthen
X for i in ?.*
X do
X fname=`basename $i .Z`
X case $i in
X *.Z)
X $COMPRESS -dc $i
X ;;
X *)
X cat $i
X ;;
X esac |
X awk '
X BEGIN {
X name = "'"$fname"'"
X }
X {
X if ($1 ~ /^#N/) {
X for (i = 2; i <= NF; i++) {
X sname = $i
X if (p = index(sname, ","))
X sname = substr(sname, 1, p-1)
X printf "@%s %s %d\n", sname, name, NR >> \
X "'$WHERETMP'";
X }
X }
X
X }'
X done
Xfi
X
Xif test -n "$WHEREFILE" -a -s $WHERETMP
Xthen
X if test ! -f $WHEREFILE
X then
X touch $WHEREFILE
X fi
X
X # First awk: throws away WHERE references in $WHEREFILE that
X # are now in $WHERETMP
X # Sort: sort by site name
X # Second awk: coalesce references to same site/file to one line.
X awk '
X BEGIN {
X mapseen[""] = 1
X }
X $1 ~ /^@/ {
X printf("%s %s %s\n", substr($1, 2), $2, $3);
X mapseen[$2] = 1
X next;
X }
X {
X if (mapseen[$2])
X next
X printf("%s %s %s\n", $1, $2, $3);
X }' $WHERETMP $WHEREFILE |
X sort |
X awk '
X {
X if (site != $1 || map != $2) {
X if (site)
X printf("\n");
X site = $1
X map = $2
X printf("%s %s %s", $1, $2, $3);
X } else
X printf(",%s", $3);
X }
X END {
X printf("\n");
X }' > /tmp/TMP2
X
X if test -s /tmp/TMP2
X then
X cat /tmp/TMP2 > $WHEREFILE
X fi
X echo "Where database ($WHEREFILE) rebuilt" >> /tmp/unpA$$
Xfi
Xrm -f /tmp/TMP2 $WHERETMP
X
Xif test -s /tmp/unpA$$
Xthen
X mail $NOTIFY < /tmp/unpA$$
Xfi
END_OF_FILE
if test 8171 -ne `wc -c <'unpackmaps'`; then
echo shar: \"'unpackmaps'\" unpacked with wrong size!
fi
chmod +x 'unpackmaps'
# end of 'unpackmaps'
fi
if test -f 'uuwhere' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'uuwhere'\"
else
echo shar: Extracting \"'uuwhere'\" \(863 characters\)
sed "s/^X//" >'uuwhere' <<'END_OF_FILE'
X:
X# Copyright 1990, Chris Lewis, All Rights Reserved
XMAPDIR=/usr/spool/maps
XWHEREFILE=$MAPDIR/where.db
XCOMPRESS=/usr/bin/compress
X
Xif [ $1 = "-v" ]
Xthen
X verbose=1
X shift
Xfi
Xfor i
Xdo
X echo Searching for $i
X if [ -n "$verbose" ]
X then
X list=`sed -n -e "/^$i[ ]/s/[ ][ ]*/:/gp" \
X -e "/^$i[ ]/q" $WHEREFILE`
X for j in $list
X do
X eval `echo $j | sed -e 's/\([^:]*\):\([^:]*\):\(.*\)/s=\1 f=\2 l=\3/p'`
X echo $s $f $l
X numlist=`echo $l | sed -e 's/[, ][, ]*/ /g'`
X for l in $numlist
X do
X echo "Found $s in $f at line $l"
X if [ -n "$f" -a -n "$l" ]
X then
X if [ -r $MAPDIR/$f ]
X then
X cat $MAPDIR/$f
X elif [ -r $MAPDIR/$f.Z ]
X then
X $COMPRESS -dc $MAPDIR/$f.Z
X else
X echo "No such file $f" >&2
X fi | sed -n -e "$l,/^#N/p"
X fi
X done
X done
X else
X grep "^$i" $WHEREFILE
X fi
Xdone
END_OF_FILE
if test 863 -ne `wc -c <'uuwhere'`; then
echo shar: \"'uuwhere'\" unpacked with wrong size!
fi
chmod +x 'uuwhere'
# end of 'uuwhere'
fi
echo shar: End of archive 1 \(of 1\).
cp /dev/null ark1isdone
MISSING=""
for I in 1 ; do
if test ! -f ark${I}isdone ; then
MISSING="${MISSING} ${I}"
fi
done
if test "${MISSING}" = "" ; then
echo You have the archive.
rm -f ark[1-9]isdone
else
echo You still must unpack the following archives:
echo " " ${MISSING}
fi
exit 0
--
Chris Lewis, Phone: (613) 832-0541
UUCP: uunet!utai!lsuc!ecicrl!clewis
Moderator of the Ferret Mailing List (ferret-request@eci386)
Psroff mailing list (psroff-request@eci386)