kent@ssbell.UUCP (Kent Landfield) (06/02/89)
USENET Sources Archiver
Tired of archiving sources by hand ?? Tired of missing postings to
comp.sources.all just 'cause you finally took a day off ? Can't
remember the filename that you saved those posted sources to ?
Not too sure how long you will be able to hold that 'old archiver'
together before the tires give out ?
Ever wonder how people archive their comp.sources.all newsgroups ?
Where are all these automated source archivers that you see mentioned
on the net ? Where are they available from ? What do they do ? How
can you *get* one ?
If you answered yes to any of the above, don't feel alone. There are
many people out there with the same problems. I know 'cause I was
one of them......
It had gotten to the point of frustration. I had to do something!
Shutting down the newsfeed was not a possibility, I was addicted...
Running
find /usr/spool/news/comp/sources -type f -print | xargs rm -f
from cron just so I didn't see them was the cowards way out.
I thought I had the answer, I'd ask the net!
"Hey buddy, got an archiver ?"
Week 1, three mail messages came in..
Week 2, four mail messages came in..
Week 3, one mail messages came in..
And then the messages stopped coming.......
Did the received messages hold the answer .... ?
"Uh, if you have any luck, could you please send
the info on to me ? InadvanceThanks. I sure could
use it too ya know.."
I was at my wits end! Users were asking for posted sources that had just
past through our machine and I DIDN'T HAVE IT.... Jumping was starting
to look real good...
Finally, on the verge of breakdown, I had an attack of logic.
"I'm looking for a program. I am a programmer. If I can not find the
program I want, I could write it!" For the next few days, I walked around the
office mumbling encouragements to myself. People started to look at me
funny... My secretary went to visit my boss after a week. Shortly
there after I heard, "Kent, get in here!" After settling down in the
rock hard folding chair in front of his desk, I could see he was disturbed.
"Kent, Donna tells me that you seem to have a problem and that you are
wandering around all day saying "I drink a can, I drink a can. What are
you, drunk ?" "Only with an obsession sir", I responded. "What is that,
some sort of imported beer ?" he asked. "No sir, what I have been saying
is 'I think I can'." "Well just what is it that you think you can do ? "
"Write a piece of code." I *knew* that I had made a mistake there...
"WHAT! you are a programmer right? You better be able to write a piece of
code. What the hell do I pay you for anyway?" I didn't answer that, my
foot was squarely in my mouth. I *really* didn't want it shoved down my
throat... "Now get out of here and go write that code!!" he said pointing
his fat finger at the door. "THANK YOU sir!" I said as I backed out of the
room. The rest, as you say is history....
So much for the **fiction** :-) :-) NOW for the FACTS :-)
Last night I sent the rkive USENET Sources Archiver to rich salz for
posting to comp.sources.unix. It should appear in a spool directory
near you *real* soon now. (Is there a catch-22 here, need an archiver to
catch the sources to an archiver so you can archive the archiver
sources :-) )
The rkive package was initially designed for archiving comp.sources.all
newsgroups. It does however, support archiving of non-moderated,
non-sources newsgroups. What follows is a *long* explanation of just
what it is and what it does. (This might be one to send to the printer
and read later.. :-)) You have been warned! :-)
------------------------------------
rkive - A USENET Sources Archiver
------------------------------------
rkive is used to archive the USENET sources groups to an alternate
location as specified in an rkive configuration file. Archives can be
maintained in one of three ways:
Archive-Name - The moderators of *most* sources groups assign
an official Archive-Name to each article that gets submitted
to the net. In this manner, each file has a "new-login" or
"elm/part06" type of format. For multi-part postings, a sub-
directory is created (as indicated in the elm example) to
hold the separate "parts". This format is used by many large
archive sites because it is easier for retrieval via mail
request software such as netlib and the filenames give hints
as to what the software is.
Volume-Issue - Software sent via *most* moderated groups have
an assigned Volume and Issue number. This allows the modera-
tors to track and reference the individual items that have
been posted to the group. Each individual article is given
an "Issue" number. The Issues are grouped together into a
"Volume". There are roughly 100 articles in each Volume but
this is an arbitrary split totally up to the moderator.
This format is extremely useful when the software archives
are cataloged. It makes searching of the files quicker and
verification of complete volumes easier. This archive format
is recommended for any site that will be doing *massive*
searches of the individual volumes since it keeps the qua-
dratic nature of directory searches from making your life
miserable.
Article Number - The news software stores the articles
locally by naming the news article by a number generated on
every site. The Article Number ordering is unique to each
site. If an Article Number archive is requested (or required
by the newsgroup), the news article file is copied to the
directory specified in the archive configuration file. The
name of the archived article will match the original name
generated by the news software.
By means of a configuration file, the archive administrator is able to
control how archiving is performed. The administrator can specify on a
per newsgroup basis:
o The type of the archiving, such as Volume-Issue
Archive-Name, or Article Number archiving,
o Where the newsgroup archive is to be stored on disk,
o The location of log file for the newsgroup,
o The format of the logfile records,
o The location of index file for the newsgroup,
o The format of the index file records,
o A list of users to be sent mail when an article is archived,
o The owner/group and modes of each archived member, and
o Whether the archived members should be compressed or not.
o How to deal with REPOSTs to archived members,
o How to deal with patches to posted sources (only in
newsroups that support the Patch-To: line),
It is intended that rkive be run by cron on a daily basis. In this manner,
the sources are archived and available for retrieval from the archives on the
day it reaches the machine instead of having to wait for expire -a to run.
It allows for the archives to be managed by the same or different people
(or accounts). It supports the building of indexes for later review or to
interface to the netlib type of mail retrieval software. It also supports
mailing notifications of the archiving to a specified list of users or
aliases. The indexes and log file record formats are specifiable by the
person configuring the rkive configuration file.
----------------
REPOST Handling:
----------------
ADD_REPOST_SUFFIX
This define allows the administrator to configure the software to
add "-repost" (or whatever is defined in REPOST_SUFFIX) to the
end of all files that are marked as REPOST by the newsgroup moderator.
The suffix is added prior to compression. This feature should only be
configured/exist on systems whose filename limits are greater than 14.
MV_ORIGINAL
This define allows the administrator to configure the software to
move the original article into a "originals" directory in the
problems directory. The inbound reposted article is placed into
the archive in the correct position.
If neither define is specified then the inbound article is placed into
the archive in the correct position only if the initial article is not
in the archive. Otherwise the reposted article is placed in the problems
directory as normal duplicate articles are now.
-----------------
PATCHES Handling:
-----------------
rkive supports the new Auxiliary header "Patch-To:" that is going to
be used in c.s.u. The Patch-To: line exists for articles that are patches
to previously posted software. The Patch-To: line only appears in articles
that are posted, "Official", patches. The initial postings do not contain
the Patch-To: auxiliary header line.
Auxiliary Headers For Patch Postings:
Submitted-by: Kent Landfield <kent@ssbell.UUCP>
Posting-number: Volume 23, Issue 14
-> Patch-To: Volume 22, Issue 122
Archive-name: rkive/patch1
There are two different types of handling with regards to patches.
Package - This type of archiving of patches places the patches
in the same directory that the initial source was
posted to. This type of archiving is only available
to newsgroup archives that are using Archive-Name
archiving as well.
Historical - This type of archiving patches is done by sites that
want to place the the patches in the volume/issue
directory as specified by the moderator when the patch
was initially posted.
Archive recognizes that the Patch-To: line indicates the article is
a patch. For Archive-Name archiving which has specified "Package"
patches archiving in the configuration file, rkive puts the article
into the directory that contained the initial posting (volume22/rkive).
For Archive-Name that has not specified Package archiving or for
Volume/Issue archiving, the article would still be labeled as
volume23/rkive/patch01 or volume23/v23i014 respectively.
rkive also writes a .patchlog file in the BASEDIR for the newsgroup
that is used to track patches to originally posted software. The
.patchlog is going to be used for the "random software downloader :-)"
so that complete software packages (sources and patches) can be requested
from sites that do not use combined Archive-Name and Package archiving.
The format of the .patchlog file is:
#
# Patchlog for comp.sources.whoknows
#
# Path To Initial Initial Current Current
# Patchfile Volume Issue Volume Issue
#
bb/patch01 22 105 23 77
or if volume issue format..
v47i022 22 105 23 77
-------------------------
Article Header Reduction:
-------------------------
Articles that are stored just as they arrived on your system are potentially
wasting disk space. Certain rfc822/rfc1036 header lines are of little use
after the article is archived. If you wish to have the headers "trimmed"
when the file is archived, assure that REDUCE_HEADERS is defined. Currently
all header lines that are *not* either;
From:, Newsgroups:, Subject:, Message-ID:, and Date:
will be removed. This can produce a savings of as much as 200 to 500
bytes per archived article.
---------
Security:
---------
rkive sets the ownership, group and modes on the archived members according
to the information specified in the configuration file. Currently though,
rkive uses the default umask for creating the log and index files.
rkive will not archive files outside of the BASEDIR specified in the
configuration file so a "prankster" can not do nasty things to your
system files by having an Archive-name line like:
Archive-name: ../../../../../../etc/passwd
It will also not overwrite duplicate files. They are stored underneath
the problems directory specified in the configuration file. The admin
is alerted to the fact and it then becomes a manual cleanup problem.
-------------------------------------------------
article - Format News Article Header Information
-------------------------------------------------
The accompanying article program allows you to view the article headers in
much the same manner that you use a printf statement. This was initially
done for debugging purposes but I quickly found that it was *extremely*
useful in dealing with news articles in general. It works great in shell
scripts to view articles that need to be read.... Also super for perusing
the archives directly and generating indexes to the archives in *many*
different ways...:-)
--------
CREDITS:
--------
I have to give credit where credit is due 'cause they earned it!
I used the code in header.c of the News 2.11 as the basis of ideas for
dealing with the article headers. The code I have written is not the same
but most of the concepts and some of the flow control resulted from reviewing
how it was "suppose to be done". (rfcs only go so far.. :-)) For that I
thank rick adams and the authors of news for the *excellent* code to study
from.. :-)
I would also like to thank my beta testers for the headaches of dealing
with me, with forcing different ideas on me at a time when I was "almost"
willing to listen :-) and for the many different "full redistribution of
sources" every time I had a new version. Specifically I want to thank
eric@amperif (Eric Johnson) and denny@mcmi (Denny Page) for putting up
with me.. :-) and for rick@ssbell (Rick Ohnemus) for his excellent debugging
techniques.. :-)
------------------------------------------------------------------------
This software set was developed under an archiving model similar to
that maintained currently on uunet. It was intended that the archiving
facilities were more of a "site" facility and not an individuals
facility. (That is unless the individual owned the site :-)). I have
not tried to use rkive for maintaining a private (many dups on a single machine)
archive. There does not seem to be any reason why it would not work. It
just hasn't been done. rkive will accept an rkive.cf file specified on the
command line so it would be possible for an individual to have their own
mini archive directory structure. This is *not* recommended if the site is
doing archiving since the software will store multiple copies thus wasting
more disk space than it is worth. Aside from that, if someone does try it,
let me know how it turns out. :-) :-)
VERY IMPORTANT! If you have a problem, there's someone else out there
who either has had or will have the same problem. Please send all
patches, ideas, etc to kent@ssbell (or uunet!ssbell!kent) so that I
can continue to improve the functionality and portability of this
package.
kent@ssbell.UUCP
uunet!ssbell!kent