allbery@uunet.UU.NET (Brandon S. Allbery - comp.sources.misc) (09/24/89)
Posting-number: Volume 8, Issue 47
Submitted-by: clewis@eci386 (Chris Lewis)
Archive-name: unpkmaps2
Periodically, people ask for a map unpacker. I've posted this
before in news.admin, but I guess it's time to make it semi-official.
The one big advantage to this one is that it's intended to be secure,
and is in fact somewhat better than uuhosts (not to mention a little
less of a CPU and disk hog). It's been in use for about 8 months
on a couple of machines.
I call it "unpackmaps V2.0"
#! /bin/sh
# This is a shell archive. Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file". To overwrite existing
# files, type "sh file -c". You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g.. If this archive is complete, you
# will see the following message at the end:
# "End of archive 1 (of 1)."
# Contents: MANIFEST README path.patch unpackmaps
# Wrapped by clewis@eci386 on Tue Sep 19 15:27:00 1989
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'MANIFEST' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'MANIFEST'\"
else
echo shar: Extracting \"'MANIFEST'\" \(275 characters\)
sed "s/^X//" >'MANIFEST' <<'END_OF_FILE'
X File Name Archive # Description
X-----------------------------------------------------------
X MANIFEST 1 This shipping list
X README 1 Readme
X path.patch 1 patch for pathalias
X unpackmaps 1 map unpacker
END_OF_FILE
if test 275 -ne `wc -c <'MANIFEST'`; then
echo shar: \"'MANIFEST'\" unpacked with wrong size!
fi
# end of 'MANIFEST'
fi
if test -f 'README' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'README'\"
else
echo shar: Extracting \"'README'\" \(6341 characters\)
sed "s/^X//" >'README' <<'END_OF_FILE'
X UNPACKMAPS V2.0
X
XThis is source for a simple, *secure*, map unpacking facility.
X
XIt is much simpler than uuhosts, though it will grow somewhat.
XThe intent is that any UNIX/XENIX/BSD system that can run news will
Xbe able to run this too, so I'm attempting to keep to greatest-common-
Xdenominator except for pathalias and things that I implement
Xmyself.
X
XThis release includes a patch to pathalias 9 to allow pathalias to
Xhandle compressed files in its argument list. You don't need to
Xpatch pathalias, but if you don't, you can't keep the maps in compressed
Xformat (yet).
X
XThe next release (if there ever is one) will contain such things as
Xmore efficient path file modification, a method for viewing the map
Xfiles analogously to uuhosts (from which this software derives a
Xconsiderable amount of inspiration), and more automated installation.
X
XA slightly older version of this was mailed to Neil Gorsuch (of Zardoz
Xsecurity mailing list fame) who said that he would be working on it,
Xpossibly merging it with other map unpackers and officially posting it.
XI'm posting it because I occasionally see requests for it, and Neil
Xhasn't posted his yet.
X
XRegarding security: as many may remember, there's been a fair bit of
Xdiscussion on security of map unpacking on the net. Rather than play
Xaround with trying to make a secure *true* unshar, which probably noone would
Xtrust because it would be so big, I simply made a few simplifying assumptions
Xabout the map format and use an awk script to unpack a map article into
Xa map file. It checks for and refuses to unpack articles which have
Xslashes in their names. I sent off some mail to Mel asking whether the
Xassumptions I've made about map format are true, but never got any
Xresponse. I believe that this is *pretty* secure, in that it doesn't
Xhave to be run as root, doesn't use the Bourne shell for unpacking, and
Xis careful about the file names it creates. Please let me know if there
Xare any holes I didn't think of.
X
XGeneral operation:
X - your news is modified to batch incoming map article file names
X to a specific batch file (analogous to normal news batching).
X C-news users take note: you may have to utter magic incantations
X (hint "classes" in C-news Alpha) to get sendbatches to avoid
X trying to uux these...
X - unpackmaps wakes up, usually once per day, and extracts the
X maps specified (if any) in the batch file into the map directory.
X Maps are extracted using a secure awk script without resort
X to setuid root or other wierdnesses.
X - If any maps were extracted, pathalias is fired off, and the
X resultant file put in the place specified.
X - if anything was done, unpackmaps sends you mail telling you
X what happened.
X
XInstallation:
X - If you want to compress the stored map files, patch your
X version 9 pathalias source using the path.patch file. DO NOT
X define COMPRESS in unpackmaps unless you've done so.
X All this patch does is allow pathalias to handle files arguments
X that may be compressed - it does it by "system"'ing off COMPRESS when
X files with ".Z" are found. (I wanted to "popen" COMPRESS, but
X the changes looked a little extreme, besides, having both
X compress and pathalias running on a pipeline seemed to me to
X be a little extreme w.r.t. memory.) You should, after patching
X pathalias, modify the COMPRESS define in pathalias's config.h
X to contain the same pathname as the COMPRESS in unpackmaps,
X except that the COMPRESS define in config.h should also include
X "-d" (it's being used as if it was zcat)
X
X uuhosts does compression by completely uncompressing the map
X database, and running pathalias, and then recompressing the whole
X thing again. Which means you need an enormous amount of disk
X (approx 3 times the compressed map directory), plus be willing
X to put up with a compression run on every map file too. (not
X to mention intermediate path files)
X
X My approach only needs enough space for one additional uncompressed
X map entry plus a copy or two of the final map database.
X
X - I STRONGLY recommend that you build a special version of compress
X with 12 bit compression instead of 16 for this. When this
X is done, compress is considerably smaller (eg: bss of 32K instead
X of 400K+). The reason for this is obvious - pathalias is enormous
X when it's running, and it ain't nice to have to fork/exec
X a 500K+ process at the same time for *every* map file. Ugh.
X
X Advantages:
X - the whole thing runs considerably faster
X - much less swap/paging
X - on our machine, 16 bit compress practically hangs
X everyone else when run at the same time as pathalias.
X Disadvantages:
X - the map directory is 10% (really! only 10%!) bigger.
X
X What I did was the following:
X - go to the source directory for compress
X - remove the binary if it is there.
X - say:
X make compress CFLAGS=-DUSERMEM=0
X - rename this to something like /usr/bin/compress12
X - make sure that the config.h for pathalias and unpackmaps
X shell script has the same name. Eg: /usr/bin/compress12
X
X - edit unpackmaps to set the variables at the beginning of
X the shell script.
X - make the directory for the map files, owned by news.
X - put unpackmaps in a suitable place. Eg: /usr/lib/news
X - unpackmaps should be run from the userid that owns and runs news.
X - su to the news userid, and run:
X unpackmaps -i
X This will build the initial path file.
X - insert into your crontab something like:
X 30 3 * * * /bin/su news -c "<path to unpackmaps>/unpackmaps > /dev/null"
X
Xunpackmaps -i: will extract all map articles into the map spool area - useful
Xfor the first time you use it.
X
Xunpackmaps -p: runs pathalias even if no map articles were extracted.
X
XIn order to push the paths file into /usr/lib/uucp, I created a file called
Xpaths in /usr/lib/uucp, with 644 permissions, owned by the userid that runs
Xunpackmaps.
X
XThe package will send mail to who you specify indicating which maps were
Xunpacked, and any error returns from pathalias.
X
XLet me know of any changes you needed to make to get this to work.
XI'm also open to suggestions for new features....
X----------
XChris Lewis, Markham, Ontario, Canada
X{uunet!attcan,utgpu,yunexus,utzoo}!lsuc!{ecicrl|eci386}!clewis
XFerret Mailing list: ...!lsuc!gate!eci386!ferret-request
X(or lsuc!gate!eci386!clewis or lsuc!clewis)
END_OF_FILE
if test 6341 -ne `wc -c <'README'`; then
echo shar: \"'README'\" unpacked with wrong size!
fi
# end of 'README'
fi
if test -f 'path.patch' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'path.patch'\"
else
echo shar: Extracting \"'path.patch'\" \(2225 characters\)
sed "s/^X//" >'path.patch' <<'END_OF_FILE'
X*** config.old.h
X--- config.h
X**************
X*** 11,16
X /* default place for dbm output of makedb (or use -o at run-time) */
X #define ALIASDB "/usr/local/lib/palias"
X
X
X
X /**************************************************************************
X--- 11,19 -----
X /* default place for dbm output of makedb (or use -o at run-time) */
X #define ALIASDB "/usr/local/lib/palias"
X
X+ /* place for zcat for uncompressing maps inline */
X+ #define COMPRESS "/usr2/clewis/maps/compress12 -d"
X+
X
X
X /**************************************************************************
X*** parse.old.y
X--- parse.y
X**************
X*** 402,407
X fixprivate(); /* munge private host definitions */
X Lineno = 1;
X while (optind < Argc) {
X if (freopen((Cfile = Argv[optind++]), "r", stdin) != 0)
X return 0;
X sprintf(errbuf, "%s: %s", Argv[0], Cfile);
X--- 402,439 -----
X fixprivate(); /* munge private host definitions */
X Lineno = 1;
X while (optind < Argc) {
X+ #ifdef COMPRESS
X+ static char Fname[512];
X+ char compressed[512];
X+ char cmdbuf[512];
X+ int dotidx;
X+
X+ strcpy(Fname, Cfile = Argv[optind++]);
X+ strcpy(compressed, Cfile);
X+
X+ dotidx = strlen(Cfile) - 2;
X+ if (dotidx >= 0 && strcmp(&Fname[dotidx], ".Z") == 0)
X+ Fname[dotidx] = '\0';
X+ else
X+ strcat(compressed, ".Z");
X+
X+ if (access(Fname, 04) != 0) {
X+ if (access(compressed, 04) == 0) {
X+ unlink("/tmp/pathfile");
X+ sprintf(cmdbuf, "%s < %s > /tmp/pathfile", COMPRESS,
X+ compressed);
X+ system(cmdbuf);
X+ if (freopen("/tmp/pathfile", "r", stdin) != 0) {
X+ unlink("/tmp/pathfile");
X+ return 0;
X+ }
X+ unlink("/tmp/pathfile");
X+ }
X+ } else {
X+ if (freopen(Fname, "r", stdin) != 0)
X+ return 0;
X+ }
X+ #else
X if (freopen((Cfile = Argv[optind++]), "r", stdin) != 0)
X return 0;
X #endif
X**************
X*** 404,409
X while (optind < Argc) {
X if (freopen((Cfile = Argv[optind++]), "r", stdin) != 0)
X return 0;
X sprintf(errbuf, "%s: %s", Argv[0], Cfile);
X perror(errbuf);
X }
X--- 436,442 -----
X #else
X if (freopen((Cfile = Argv[optind++]), "r", stdin) != 0)
X return 0;
X+ #endif
X sprintf(errbuf, "%s: %s", Argv[0], Cfile);
X perror(errbuf);
X }
END_OF_FILE
if test 2225 -ne `wc -c <'path.patch'`; then
echo shar: \"'path.patch'\" unpacked with wrong size!
fi
# end of 'path.patch'
fi
if test -f 'unpackmaps' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'unpackmaps'\"
else
echo shar: Extracting \"'unpackmaps'\" \(5874 characters\)
sed "s/^X//" >'unpackmaps' <<'END_OF_FILE'
X# Written by: Chris Lewis, clewis@eci386
X# unpackmaps release 2.0 (alpha)
Xtrap "rm -f /tmp/unp?$$; exit" 0 1 2 3 15
XIFS="
X"
Xexport IFS
XPATH=/bin:/usr/bin
Xexport PATH
X
X# The name of the file that you've caused your news system to
X# batch the file names of the map articles.
X# Eg: C-news
XBATCH=/usr/lib/news/batch/b.maps/togo
X# Eg: B-news
X#BATCH=/usr/spool/batch/maps
X# News spool directory
XNEWSSPOOL=/usr/spool/news
X# Where you want the maps to go.
X# I like using /usr/spool/maps, but on our system, /usr/spool/news
X# is a separate file system, and /usr runs close to the limit...
XMAPDIR=/usr/spool/news/maps
X# Person to send results and error messages to
XNOTIFY=clewis
X# pathalias binary
XPATHALIAS=/usr/lbin/pathalias
X# where you want the path files to go:
X# A convenient place is /usr/lib/uucp/paths which is the smail
X# default. If you're going to put this in /usr/lib/uucp, I suggest
X# (rather than make /usr/lib/uucp writeable by everybody), doing
X# the following:
X# su root
X# cd /usr/lib/uucp
X# touch paths
X# chown news paths (or usenet)
X# chmod 644 paths
XPATHFILE=/usr/lib/uucp/paths
X# Auxiliary options to pathalias. Tune to local tastes....
XPATHOPTS="-dwatmath"
X# If you have a version[s] of your machine's map entry that is different
X# from what's published, change this variable to point at it/them.
X# (Eg: I publish the first entry here, and the second one is local tuning
X# and hidden connections)
XPATHLOCAL="/usr2/clewis/maps/path.local /usr2/clewis/maps/path.nonpublic"
X# If this variable is set to the compress binary, maps will be
X# compressed. DO NOT define this unless you've applied the accompanying
X# patch to your version 9 pathalias source, and you've set the path
X# in config.h to point at the same place.
XCOMPRESS=/usr2/clewis/maps/compress12
X# 1 to strip comments from maps - don't do this if you ever try
X# to read the maps and figure out where the site is, their contacts
X# etc.
XNOCOMMENTS=0
X# PS: there is *one* possible edit that you might want to make
X# below - the maps used to generate wierd domains, but most of that
X# appears to be gone now (don't ask me, I never particularly understood
X# it, but since Peter Honeyman recommended it...). If you object
X# to these wierd domains, uncomment the egrep.
X
X# Edit no more....
X
Xumask 022
X
XPATH=/bin:/usr/bin
Xexport PATH
X
Xif test ! -d $MAPDIR -o ! -w $MAPDIR
Xthen
X echo "$MAPDIR missing, unwritable or not a directory" >&2
X exit
Xfi
X
Xif [ $# = 1 ]
Xthen
X case $1 in
X -p)
X forcepath=true
X ;;
X -i)
X cd /
X rm -f $BATCH.work
X # using find/sort instead of ls just in case there's lots of
X # articles....
X find $NEWSSPOOL/comp/mail/maps -type f -print | sort > $BATCH
X ;;
X *)
X echo "usage: unpackmaps [-i] [-p]" >&2
X exit 1
X ;;
X esac
Xfi
X
Xcd $MAPDIR
X
Xwhile [ -f $BATCH -o -f $BATCH.work ]
Xdo
X # There is no window of vulnerability here as long as noone else is
X # creating $BATCH.work.
X if [ ! -f $BATCH.work ]
X then
X mv $BATCH $BATCH.work
X fi
X
X while read i stuff
X do
X # Using stuff to capture remaining junk on line.
X # Eg: C-news article sizes.
X
X if [ -z "$i" ]
X then
X break
X fi
X
X if [ ! -r $i ]
X then
X echo "$i apparently superseded or expired"
X continue
X fi
X
X # This awk script depends on the following map article format:
X # <don't cares>
X # cat << 'something' > filename
X # map body
X # something
X # <don't cares>
X # "something" doesn't have to be enclosed in quotes in the cat line.
X # This isn't particularly fast - could be dramatically speeded up
X # if written in C, but I was trying to ensure that this is as simple
X # and self-evident as possible.
X
X awk '
X $1 == "cat" {
X endtoken=$3;
X if (substr(endtoken, 1, 1) == "'"'"'")
X endtoken=substr(endtoken, 2, length(endtoken)-2);
X collecting = 1;
X foundone = 1;
X name = $5;
X if (index(name, "/") != 0) {
X printf("Security violation attempt in %s!\n", "'$i'");
X exit;
X } else
X printf("extracting %s from %s\n", name, "'$i'");
X next;
X }
X
X {
X if (!collecting)
X next;
X if ($1 == endtoken) {
X line = "rm -f " name ".Z"
X print "" | line
X collecting = 0;
X next
X }
X if ("'$NOCOMMENTS'" == 1 && $0 ~ /#/)
X print substr($0, 1, index($0, "#")) > name
X else
X print $0 > name
X }
X
X END {
X if (collecting) {
X printf("Non-terminated map in %s\n", "'$i'");
X }
X if (!foundone) {
X printf("%s does not contain a properly formed map\n", "'$i'");
X }
X }' $i
X done < $BATCH.work
X rm $BATCH.work
Xdone > /tmp/unpA$$ 2>&1
X
Xif test -n "$PATHALIAS" -a -x "$PATHALIAS"
Xthen
X if test -s /tmp/unpA$$ -o -n "$forcepath"
X then
X cd $MAPDIR
X
X (
X $PATHALIAS -f $PATHOPTS ?.* $PATHLOCAL |
X
X # format of the pathalias -f output is
X # cost host route
X #
X # format of a 'paths' file for smail is
X # host route first_hop_cost
X #
X # move cost field to end of line:
X
X sed 's/\(.*\) \(.*\) \(.*\)/\2 \3 \1/' |
X
X # convert target domain/host to lower case:
X
X #lcasep |
X
X # remove some additional wierdnesses (per Peter Honeyman):
X # You can leave it in or not.
X
X # egrep -v '(\.(com|edu|mil|gov|net|org|arpa|[a-z][a-z]) .*!.*!)|(.\.(com|edu|mil|gov|net|org|arpa|[a-z][a-z]) )' |
X
X # sort the stream:
X
X sort > /tmp/paths ) > /tmp/unpB$$ 2>&1
X
X if test ! -s /tmp/paths
X then
X echo "Pathalias failed no map file created" >> /tmp/unpB$$
X else
X cat /tmp/paths > $PATHFILE 2>> /tmp/unpB$$
X if test $? != 0
X then
X echo "Copy to $PATHFILE failed" >> /tmp/unpB$$
X else
X rm /tmp/paths
X fi
X echo "Map remade" >> /tmp/unpB$$
X fi
X
X if test -s /tmp/unpB$$
X then
X echo "Pathalias output:" >> /tmp/unpA$$
X cat /tmp/unpB$$ >> /tmp/unpA$$
X fi
X fi
Xfi
X
Xif test -x "$COMPRESS"
Xthen
X files=`ls ?.* | sed -e '/\.Z$/d'`
X if [ -n "$files" ]
X then
X $COMPRESS -f $files
X fi
Xfi
X
X
Xif test -s /tmp/unpA$$
Xthen
X mail $NOTIFY < /tmp/unpA$$
Xfi
END_OF_FILE
if test 5874 -ne `wc -c <'unpackmaps'`; then
echo shar: \"'unpackmaps'\" unpacked with wrong size!
fi
chmod +x 'unpackmaps'
# end of 'unpackmaps'
fi
echo shar: End of archive 1 \(of 1\).
cp /dev/null ark1isdone
MISSING=""
for I in 1 ; do
if test ! -f ark${I}isdone ; then
MISSING="${MISSING} ${I}"
fi
done
if test "${MISSING}" = "" ; then
echo You have the archive.
echo "Read README to install unpackmaps"
rm -f ark[1-9]isdone
else
echo You still need to unpack the following archives:
echo " " ${MISSING}
fi
## End of shell archive.
exit 0
--
Chris Lewis, R.H. Lathwell & Associates: Elegant Communications Inc.
UUCP: {uunet!mnetor, utcsri!utzoo}!lsuc!eci386!clewis
Phone: (416)-595-5425