brad@looking.ON.CA (Brad Templeton) (12/20/89)
Posting-number: Volume 9, Issue 81
Submitted-by: brad@looking.ON.CA (Brad Templeton)
Archive-name: newsclip/part12
#! /bin/sh
# This is a shell archive. Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file". To overwrite existing
# files, type "sh file -c". You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g.. If this archive is complete, you
# will see the following message at the end:
# "End of archive 12 (of 15)."
# Contents: doc/append.mm.2
# Wrapped by allbery@uunet on Tue Dec 19 20:10:04 1989
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'doc/append.mm.2' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'doc/append.mm.2'\"
else
echo shar: Extracting \"'doc/append.mm.2'\" \(46220 characters\)
sed "s/^X//" >'doc/append.mm.2' <<'END_OF_FILE'
XIf you wish to set these, do so in your initialization section or at the
Xvery start of article processing. It does no good to set them once
Xany reference to the body of an article has been made.
X.xd "int" "preserve_case"
X.P
XNormally all sections of the article body are mapped to lower case before
Xpattern matching, so that patterns will match upper, lower and mixed
Xcase instances of the desired strings. If you do not desire this, set this
Xvariable to TRUE.
X.P
XSetting this flag true also stops the mapping of header sections to lower
Xcase. Most header sections, such as \fBsubject\fP and \fBfrom\fP are usually
Xmapped to lower case.
X.xd "int" "paragraph_scan"
X.P
XYour program has the option of scanning the body as a series of paragraphs
Xrather than a series of lines. If you set this variable to be true,
Xthe body will be read in and lines will be grouped into paragraphs before
Xthey are scanned. Paragraph breaks are marked at blank lines, changes in
Xindentation or a switch from new text to included text.
X.P
XParagraph scanning has one big advantage. If you're searching for a
Xpattern like ``ronald reagan,'' then you won't find it if the text
Xlooks like:
X.in 0.5in
X
X.nf
X.P
XAnd furthermore, I blame it all on Ronald
XReagan, who was President in 1987.
X.fi
X.in -0.5inYou will match such a pattern if the paragraph_scan flag is on.
X.P
XParagraph scanning does consume extra time, of course, and should only
Xbe used if you have multi-word patterns to search for.
X.xd "int" "white_compress"
X.P
XYour program has the option of scanning the body in such a way that
Xall spans of ``white space,'' namely spaces, tabs, formfeeds, and (in
Xthe case of paragraph_scan mode (see above)) line breaks are compressed
Xinto a single space. Set this variable to TRUE and this will be done
Xbefore any pattern matches on the article body.
X.P
XThis does not apply to matches on the \fBsubject\fP or other normal
Xstrings.
X.H 2 "Global Control"
X.P
XYou can set your program to do an automatic accept or reject on all articles
Xin a given newsgroups. Set one of these variables to TRUE (in, for example,
Xyour \fBSTARTGROUP\fP routine, and all further articles will be accepted
Xor rejected (according to the variable) until you reset the variable or
Xprocessing of the current newsgroup finishes.
X.P
XThese variables should never be used in \fIfilter\fP mode. These variables
Xprovide a way to ``unsubscribe'' to a newsgroup entirely within the context
Xof the NewsClip language.
X.xd "int" "accept_all"
X.P
XIf true, accept all articles until this variable is cleared or processing
Xof the current newsgroup terminates.
X.xd "int" "reject_all"
X.P
XIf true, reject all articles until this variable is cleared or processing
Xof the current newsgroup terminates.
X.xd "int" "named_group( newsgroup )"
X.P
XThis returns true if the newsgroup is one of those named as a newsgroup
Xconstant (\fB#comp.foo\fP) or inside an \fBis\fP operator within your
Xprogram. It is false otherwise.
XThis is a quick way to tell if the group is one of the
Xones you are interested in, ie. named somewhere in your program.
X.xd "int" "newsrc_group( newsgroup )"
X.P
XReturns true if the group is one of those named in the \fB.newsrc\fP file.
XThis will only be true in a \fB.newsrc\fP processing mode, like \fInewsrc\fP
Xor \fIbatch\fP mode. Unsubscribed groups are not included, unless the
X\fI+unsubscribed\fP option has been used. This can be useful in eliminating
Xcrossposting.
X.H 2 "Outside World"
X
X.xd "string array" "options"
X.P
XThis array contains all the command line options provided to the news
Xfiltering program through use of the \fIo=\fP command line option. It will
Xbe empty if no options were provided. The string ``o='' does not form
Xpart of any of the option strings.
X.xd "string" "my_domain"
X.P
XA string that contains the domain name of the host machine as used to
Xgenerate mail reply addresses in news articles. This will only be valid
Xif the person who installed the NewsClip system on your machine defined
Xit, as there is no portable way of finding this out.
X.P
XIf you're not sure, and you want to refer to the domain of your own
Xmachine, just type it in yourself. This variable exists only for those wishing
Xto write general newsclip programs that can move from machine to machine.
X.xd "string" "my_mail_address"
X.P
XThis gives the mail address of the user running the news filtering program.
XIt depends on \fBmy_domain\fP, so note the warnings given above.
X.xd "datetime" "time_now"
X.P
XThis gives the date/time that the newsclip program was invoked. Note that
Xif the program somehow runs for hours, this variable will still always have
Xthe same value -- the date/time at startup.
X.H 2 "Handy Constants"
X
X.xd "int" "true"
X.P
XThe integer 1. (\fBPre-declared\fP)
X.xd "int" "false"
X.P
XThe integer 0. (\fBPre-declared\fP)
X.xd "datetime" "day"
X.P
XThe number of seconds in a day. (\fBPre-declared\fP)
X.xd "datetime" "week"
X.P
XThe number of seconds in a week. (\fBPre-declared\fP)
X.xd "datetime" "month"
X.P
XThe number of seconds in an ``average'' month. (\fBPre-declared\fP)
X.xd "string" "nilstring"
X.P
XThe value of string header variables that have no string assigned to them. (\fBPre-declared\fP)
X.xd "userid" "niluserid"
X.P
XThe value of userid header variables that have no userid assigned to them. (\fBPre-declared\fP)
X.xd "database" "nildatabase"
X.P
XA nil database can be searched in (you will find nothing in it) but you can't
Xadd items to it. (\fBPre-declared\fP)
X.xd "newsgroup" "nilnewsgroup"
X.P
XA newsgroup number that corresponds to no actual newsgroup. (\fBPre-declared\fP)
X.xd "array" "nilarray"
X.P
XThe value of an array header variable that was not present. Do not attempt
Xto index into such an array, as a \fBnilarray\fP is not the same as
Xan empty array. (\fBPre-declared\fP)
X.H 2 "Special Predeclared Functions"
X
X.xd "int" "count( array )"
X.P
XThe number of elements in an array. Do not use on a nil array. (\fBPre-declared\fP)
X.xd "string" "realname( userid )"
X.P
XWhen applied to a variable of type userid, this function returns the
X``full name,'' if any, that was associated with the userid. For example,
Xif an article contains the line: \fBFrom: user@foo.bar (Joe User)\fP then
X\fBfrom\fP will look like ``\fBuser@foo.bar\fP'' and \fBrealname(from)\fP will
Xbe the string ``\fBjoe user\fP.'' (\fBPre-declared\fP)
X.H 2 "Database Routines"
X.P
XThe concept of the database is described elsewhere in the manual. Databases
Xcan be accessed through array indexing, as well as the \fBin\fP and
X\fBhas\fP operators. These routines are used to initialize, read and write
Xdatabases.
X.P
XDatabase read and write routines take strings for filenames. These strings
Xcan contain special escape codes, all starting with the character tilde.
XThese codes can be used to expand important information into the filename,
Xso that database filenames can be based on the current newsgroup, the user's
Xhome directory, and other things.
X.P
XThe escape codes are:
X.P
X \fB~\fP (followed by slash) User's home directory
X \fB~.\fP User's ``dot'' (hidden file) directory (location of .newsrc)
X \fB~n\fP current newsgroup name, with dots changed to slashes
X \fB~N\fP current newsgroup name, mapped to a short single directory form.
X \fB~d\fP newsgroup in \fBdir_newsgroup\fP with dots changed to slashes
X \fB~D\fP newsgroup in \fBdir_newsgroup\fP mapped to short single directory form.
X \fB~s\fP News spool directory (/usr/spool/news, usually)
X \fB~l\fP News lib directory (/usr/lib/news, usually)
X \fB~~\fP A real tilde in the filename
X \fB~u\fP The userid of the person running the program
X \fB~u\fP The decimal process id of the news clipping process.
X.P
XYou can also build filenames on your own with the \fBconcat\fP function.
X.P
XThe \fB~N\fP and \fB~D\fP escapes
Xcan be used when you want a file name that does not take
Xup multiple directory levels. You will get a name that does its best
Xto be unique in the length of a single filename -- 14 characters
Xon most Unix systems. There may also be a check character present to
Xhelp, but the form may vary from system to system.
X.P
XThis will normally be unique, but there is no guarantee of it. As new
Xnewsgroups are created, two might collide. Use this form at your own
Xrisk.
X
X.xd "database" "read_database( string filename )"
X.P
XReads in a database from the specified file. The database can be one
Xwritten out with the \fBwrite_database\fP procedure, or it can be a
Xhand created one, or a combination of the two. If the database file is
Xmissing or empty, an empty (but not nil) database will be returned.
X.P
XDatabase records are lines in the database file. Such lines normally
Xconsist of a greater than sign (``>''), an integer field value, a comma,
Xa long integer ``last access'' time, another comma, and then the
Xdatabase key string. Lines that don't match this format are considered
Xto be just the database key string. A default integer field of 1 and
Xlast access time of \fBtime_now\fP will be provided.
X.P
XDatabase files may also start with a special record that indicates the
Xnumber of records in the database. (This will only be created by
Xthe NewsClip database writing procedure.)
X.P
XThis function reads the entire database into memory, and returns a
Xfresh database descriptor.
X(\fBPre-declared\fP)
X
X.xd "procedure" "write_database( database d, string filename, datetime expire )"
X.P
XThis procedure writes out an existing database to the specified file.
XAll records are written out in the proper format described above, and
Xa database size record is written first.
X.P
XIf the database is nil or empty, and the specified file does not exist,
Xthen the file will not be created. This allows you to avoid creating large
Xnumbers of empty files.
X.P
XAny records that have not been accessed since the specified expire date
Xwill not be written out to the database file. For example, to write out
Xall records that have been accessed or created in the last month, use
Xa time of \fBtime_now - month\fP. To write out all records, use a time
Xof zero.
X.P
XThe expiry feature lets you gradually eliminate all old, unused records
Xfrom database as time passes.
X(\fBPre-declared\fP)
X.xd "procedure" "db\_delete( database d, string index )"
X.P
XThis procedure deletes an index from the database. If the index was
Xnot present, nothing happens. (\fBPre-declared\fP)
X.xd "procedure" "free_database( database d )"
X.P
XThis frees up the memory associated with your database. Once you have
Xwritten out a database, you should free it if you will no longer need it.
XBe sure to nil out the database variables that refer to that database as
Xwell.
X.P
XThere is no real need to free databases if your program is just about
Xto terminate.
X(\fBPre-declared\fP)
X.xd "database" "fresh_database( int predicted_size )"
X.P
XThis creates and returns a descriptor for a new empty database. You should
Xprovide an estimate of the eventual number of records that the database
Xmight contain. The estimate need not be right. If it is too small, searches
Xwill become slow. If it is too large, memory will be wasted.
X(\fBPre-declared\fP)
X.xd "int" "exists( string filename )"
X.P
XThis tests whether a file exists or not. You may want to perform some
Xdatabase operations only if certain files exist.
X.H 2 "Useful Functions"
X
X.xd "string" "getenv( string variable )"
X.P
XThis is the standard C library routine to fetch the value of an environment
Xvariable. See the C library man page for more details.
X.xd "int" "strlen( string s )"
X.P
XThis is the standard C library routine to find out the number of characters
Xin a string. Do not use it on \fBnilstring\fP.
X.xd "int" "chindex( string s, int index )"
X.P
XThis allows you to index a specific character in a string. The character
Xis returned as an integer. If you index one past the last character, you
Xwill get a zero. Beyond that, the result is undefined.
X.P
XThe first character is index 0, the next is index 1, etc. If there are
X10 characters in the string, then index 10 will return the integer 0.
X.P
XYou may only look at the characters, you may not change them.
X.xd "string" "left( string s, int parts )"
X.P
XThis parses a string delimited by dots, and returns
Xthe leftmost \fIparts\fP parts of the string, not including the dot at the
Xend. For example, in the string s = ``comp.sys.ibm.pc,''
X\fBleft(s,2)\fP returns ``comp.sys'' and \fBleft(s,1)\fP returns ``comp''
Xwith no dot on the end.
X.xd "string" "right( string s, int parts )"
X.P
XThis parses a string delimited by dots, and returns
Xthe rightmost \fIparts\fP parts of the string, not including the dot at the
Xfront. For example, in the string s = ``iuvax.cs.indiana.edu,''
X\fBright(s,2)\fP returns ``indiana.edu'' and {right left(s,1)} returns ``edu''
X.P
XIf you ask for more parts than are there, you get the whole string.
X.xd "string" "clipfront( string s, int num )"
X.P
XReturns a string with the frontmost \fInum\fP characters removed.
X.xd "string" "leftmost( string s, int num )"
X.P
XReturns a string with only the frontmost (left) \fInum\fP characters. The
Xresultant string thus has a maximum length of \fInum\fP.
X.xd "string" "domain( string mailaddress )"
X.P
XThis returns everything in a string after the first @-sign. For example,
Xgiven ``fbaggins@bagend.shire.midearth,'' this returns just
X``bagend.shire.midearth.''
X.xd "string" "concat( string s1, string s2 )"
X.P
XThis function returns a new string that is the concatenation of the two
Xargument strings.
X.xd "string" "drop_re( string subject )"
X.P
XThis function returns the argument string with any spaces and ``Re:''
Xprefixes removed from the front. It is meant to be applied to subject
Xlines to remove those ever present prefixes. No matter how many there
Xare, it removes them all.
X.xd "string" "literal\_pattern( string pat )"
X.P
XThis function returns the argument string with any regular expression
X``metacharacters'' escaped so that they will match only the literal
Xcharacter they represent. This allows the use of strings you import
Xfrom the outside to be used as patterns without worrying about
Xregular expression characters.
X.xd "string" "permstring( string s )"
X.P
XNormally all strings and arrays created in the process of dealing with
Xan article are temporary. They become undefined at the start of each
Xnew article. You can create a permanent string with the use of this
Xfunction.
X.P
XBeware that creating too many permanent strings can quickly waste memory.
X.xd "string" "lower( string s )"
X.P
XReturns a string with
Xall upper case characters in the source string mapped to lower case.
X.xd "procedure" "dprintf( string format, arg, ... )"
X.P
XThis very special procedure can help you debug your NewsClip programs.
XIt performs a C style ``printf'' to the standard error. It is only
Xfor use by C programmers.
X.P
XBe warned that the automatic conversion of newsgroup and userid values to
Xstrings does not apply here. Newsgroup values will show up as integers,
Xand userid values will not work at all. If you wish these as strings,
Xassign them to string variables before printing them.
X(\fBPre-declared\fP)
X.xd "procedure" "subscribe( string groupname )"
X.P
XThis special procedure adds the argument string to the \fB.newsrc\fP
Xfile as a subscribed group. Naturally, this only works in the various
X\fInewsrc\fP processing modes, it particular \fIbatch\fP mode. The group
Xwill be entered with no articles read.
X.P
XIf the group was already a subscribed group, nothing is done. If the
Xgroup was an unsubscribed group already present in the \fB.newsrc\fP,
Xand the \fI+unusubscribed\fP option has been used, the group will be
Xresubscribed to, with the list of seen articles intact. If that option
Xis not used, the group name will be added to the end of the \fB.newsrc\fP
Xeven though it already exists, making two entires, one subscribed and
Xone not subscribed. This is not recommended.
X.P
XThis procedure is primarily intended for programs that feed other sites.
XIn such cases, you will wish to scan the special \fBcontrol\fP newsgroup
Xfor \fBnewgroup\fP control messages, subscribing the site to all new
Xgroups that match the patterns desired. You can also do this when
Xfiltering in the stand-alone \fInewsrc\fP mode.
X.P
XSee the sample feed program that comes with the system for an example of
Xuse of this procedure.
X
X
X
X
X.H 1 "Installation"
X.P
XIf a site licence for the NewsClip system has been arranged, installation
Xshould be performed by a system administrator, as it is a good idea to
Xinstall a couple of NewsClip's files into system directories.
X.P
XIf you have a binary single-user licence, you need only create a
Xdistribution list (and that only if you want to check article distribution
Xlevels). You may then place the \fBncc\fP compiler anywhere in your
Xcommand \fB$PATH\fP and compile, so long as the \fBnewsclip.a\fP and
X\fBucode.h\fP files are in your current directory when you compile.
XSee below for more details.
X
X.H 2 "Distribution"
X.P
XThe program is usually distributed via uucp dialup. A compressed
X``tar'' file is provided. Manuals are sent via normal means, although
Xan \fBn/troff\fP source for the manual is available to site licence
Xcustomers.
X.P
XXenix customers can receive the distribution on floppy disk, in which case
Xthe disk(s) will either be 360K regular floppies or 1.2 Megabyte high
Xdensity floppies. They will be in ``tar'' format.
X.P
XIf no other means is available, the distribution may also be had on
XMS-DOS floppies, as many users have a means of copying up from an MS-DOS
Xmachine to their Unix system. Other forms of distribution can be
Xarranged upon customer request.
X
X.H 2 "Source Code Installation"
X.P
XFirst create a directory for NewsClip source and unpack the source
Xdistribution in it. Usually, this comes as a ``tar'' file, so you issue
Xthe command:
X.Bb
Xtar xf newsclip.tar
X.Be
Xin that directory. If the package is compressed (the file is
X\fBnewsclip.tar.Z\fP) you need to issue the command:
X.Bb
Xuncompress newsclip.tar
X.Be
Xfirst, or if you want to do it all at once:
X.Bb
Xcompress -d <newsclip.tar.Z | tar xf -
X.Be
X(The source code to a decompressing program is available upon request.)
X.P
XYou will now have a directory full of NewsClip source, special files and
Xa subdirectory containing the NewsClip compiler.
X.P
XTo begin, you must pick a definitions file for your variant of the Unix
Xoperating system. These files are of the form \fBsysdefs.\fP\fIOS\fP.
XCopy the appropriate file to the file \fBsysdefs.h\fP. The files you
Xmay choose from are:
X.sp 0.5V
X.in 0.4in
X
X.sp 0.7V
X.ti -0.4in
X\fBsysdefs.sysv\fP
X.P
XFor System III or System V AT&T Unix systems, including Xenix/386
X.sp 0.7V
X.ti -0.4in
X\fBsysdefs.bsd\fP
X.P
XFor most recent BSD, Mach and SunOS systems. This assumes that you
Xhave the \fBgethostname()\fP system call, which some early BSD systems
Xdo not have. It also assumes you don't have the System V string library.
X.sp 0.7V
X.ti -0.4in
X\fBsysdefs.x286\fP
X.P
XFor Xenix/286, this arranges for large model operation. Small model is
Xalso possible, but can easily run out of memory. Use \fBsysdefs.sysv\fP
Xif this is what you wish. You will also have to alter the
X\fBMakefile\fP to use the Xenix/286 \fBCFLAGS\fP and \fBLFLAGS\fP.
X.sp 0.7V
X.ti -0.4in
X\fBsysdefs.v7\fP
X.P
XFor older systems without the \fBgethostname\fP or \fBuname\fP system
Xcalls. If this is you, you should either create a file called
X\fB/etc/systemid\fP that contains your UUCP site name, or
Xedit \fBsysdefs.h\fP to include the file \fBwhoami.h\fP so that
Xthe \fBsysname\fP symbol is defined to give that site name.
X.in -0.4in0
X.P
XAfter you have created \fBsysdefs.h\fP you should go through it with an
Xeditor to make sure that everything is defined properly. The comments
Xin that file should lead you through any changes. Unless you are in the
X``UUCP'' mail domain, you will definitely have to go in to define and
Xedit the \fBMAILDOMAIN\fP symbol. If you keep your news spools or
Xnews library in some place other than the standards (\fB/usr/spool/news\fP
Xand \fB/usr/lib/news\fP) you will have to edit defined symbols
X\fBNEWSLIB\fP and \fBNEWSSPOOL\fP).
X.P
XNow you are almost ready to ``make.'' Edit the \fBMakefile\fP and make sure
Xeverything is Ok. In particular, if you are compiling for Xenix/286
Xwith large model, edit the \fBCFLAGS\fP definition in the \fBMakefile\fP
Xappropriately -- comments will tell you what to do. Also edit the
Xdefinition for \fBLIBDIR\fP if your news library is not \fB/usr/lib/news\fP.
X.P
XEdit the \fBCFLAGS\fP variable in \fBcomp/Makefile\fP as well.
X.P
XEdit the variable \fBCOMDIR\fP to name the local commands directory where
Xthe \fBncc\fP compiler executable should go, and edit the variable
X\fBMANDIR\fP to indicate where the \fBnroff\fP source for the man
Xpages should be placed.
X.P
XNow type:
X.Bb
Xmake all
X.Be
Xand assuming all is well, the newsclip library \fBcliplib.a\fP will be
Xmade. Then the compiler \fBncc\fP will be made, and the \fB.newsrc\fP
Xgenerating program \fBmknewsrc\fP.
X.P
XYou can test the compiler and almost all features of the system now by
Xperforming compiles in the current directory. Before installing, there
Xis one more task to perform.
X.H 2 "Distribution List"
X.P
XThe distribution level feature of NewsClip requires that you create
Xa file listing the sizes of the standard NewsClip distributions and
Xall USENET distributions that are visible on your machine. A sample
Xfile called \fBdistlist\fP exists, which you should use as the basis for
Xyour file. Copy the file to your news library directory, ie. to
X\fB/usr/lib/news/distlist\fP.
X.P
XNow edit it. The file consists of lines with a USENET distribution,
Xsome whitespace, and an integer that estimates the number of sites in
Xthat distribution. The estimates don't have to be correct, but it is
Ximportant that they be ordered, so that larger distributions have larger
Xnumbers.
X.P
XIt is also important that the NewsClip special distributions have numbers
Xthat exactly match their corresponding distribution at your site. For
Xexample, \fBcity\fP should match the number you give for your local
Xcity-wide distribution.
X.P
XYou can leave the numbers for the global distributions as they are, or
Xupdate them. Don't ever use a number larger than an integer can hold.
XYou should delete the sample local distributions (\fBkw\fP, \fBuw\fP,
X\fBont\fP, \fBcan\fP, \fBna\fP) unless they apply to you.
X.P
XIf you can't install this file in the news library directory, install
Xit in the same directory as your \fB.newsrc\fP file. In fact,
XNewsClip programs look there first, so users can provide their own
Xdistribution lists as desired.
X
X.H 2 "Full Install"
X.P
XOnce you are satisfied everything is working, become ``root'' and:
X.Bb
Xmake install
X.Be
XWhen you are done, you can:
X.Bb
Xmake clean
X.Be
Xto remove generated object files and executables.
X
X.H 2 "Modifying RN & Other Readers"
X.P
XWith the distribution you will find a file called \fBrnpatches\fP which
Xlists the small set of changes you need to apply in the RN source
Xdirectory. There are only half a dozen or so.
X.P
XYou also have to copy the files \fBfiltpipe.c\fP and \fBpipemode.h\fP to the
XRN source directory, so that they may be linked in with RN when you
Xcompile it.
X.P
XBe sure to add the \fBNEWSFILTER\fP definition to ``config.h'' or
Xsome other appropriate header file. If you want ``real time''
X(as you read) filtering, define \fBRTFILTER\fP there as well. If
Xyou don't, filtering will all be done at the start of each newsgroup,
Xmuch like the kill file.
X.P
XCompile and test your new RN, and when you are satisifed, install it
Xas the official version.
X.P
XEven single user licencees of NewsClip can modify their RN program,
Xeven if the system administrator is not interested. The RN program
Xis a ``freeware'' program -- anybody can get and compile the source
Xcode. Just ensure that you don't have 30 people on the same machine
Xwho all have their own privately modified RN programs -- that would be
Xa large waste of your system's resources.
X
X.H 2 "Single User"
X.P
XIf you are not a system administrator or you don't have source code, you
Xwill have to place NewsClip files within your own personal directory
Xstructure.
X.P
XThe easiest thing to do is just to keep all your files -- the compiler,
Xthe library and other associated files -- in one directory, and do all
Xyour compiling there.
X.P
XThe file \fBcliplib.a\fP should either be in the current directory when
Xyou compile, or its name can be specified as a command line argument to
Xthe \fBncc\fP compiler.
X.P
XThe file \fBucode.h\fP should also be in the current directory when
Xyou compile, or the directory in which it is found should be specified
Xwith the \fIInclude=dir\fP option to the \fBncc\fP compiler.
X.P
XWhen all else fails, you can always compile with the \fI-link\fP option
Xand perform the C compile on your own, for example in a \fBmake\fP file.
X.P
XThe \fBdistlist\fP file should reside in the same directory as your
X\fB.newsrc\fP file when you run your NewsClip programs. This is
Xeither your home directory, or the directory specified with the
X\fBDOTDIR\fP environment variable or other means. It can also reside
Xin the current directory when you run your news clipping program.
X.P
XYou may keep your \fBncc\fP and other executable files in any suitable
Xplace, such as a private directory listed in your \fBPATH\fP environment
Xvariable.
X
X
X
X
X.H 1 "RN"
X.P
XThe most popular news reading program is probably the \fBRN\fP newsreader
Xby Larry Wall. This is a highly interactive, screen oriented newsreader.
X.P
XWe have provided an extra source file and a small set of source code
Xpatches so that the RN newsreader can be adapted to talk to NewsClip
Xfilter programs in the \fIpipe\fP mode. This is the best way to
Xuse NewsClip.
X.P
XSome machines don't have USENET news files on them, but they have access
Xto fast (usually ethernet) network connections to other machines that
Xdo, usually using the ``TCP/IP'' network protocol. If the machine with
Xthe news files runs a special program called an ``NNTP server,'' then
Xusers on the news-less machine can run a special variant of RN called
X``RRN.'' The RRN program obtains all its news articles by asking the
XNNTP server to provide them over a network connection. It is possible
Xto use NewsClip with such systems in a more limited way.
X.P
XIf you're a system administrator with access to the RN source, check
Xour special on-disk file for information on how to make a new version
Xof RN.
X.P
XThere are other newsreaders that can use the special \fIpipe\fP mode
Xto talk to NewsClip programs. One that is forthcoming is the
X``TMN Netnews'' or ``News 3.0'' system by Eric Raymond. Documentation on
Xhow to talk to a filter program like NewsClip is provided in the manual
Xfor TMN.
X
X.H 2 "Basic Use"
X.P
XWrite a NewsClip program that filters your news. Compile it to an
Xexecutable called \fBnclip\fP, which is the default.
X.Bb
Xncc myprog.nc
X.Be
X.P
XNow place the \fBnclip\fP program in the same directory as your
X\fB.newsrc\fP file -- this is usually your home directory. If you
Xdon't want it there, you can also place it in any of the command search
Xdirectories specified in your \fBPATH\fP environment variable. Finally,
Xyou can make an environment variable called \fBNCLIP\fP which contains
Xthe pathname of your news filtering program.
X.P
XNow run RN or RRN. When you read in normal sequence, rejected
Xarticles will be quickly skipped over. The message,
X``Filtering \fIarticle-num\fP'' will appear for each rejected article.
X.P
XThis only works when you read in normal sequence. If you read out
Xof sequence, by searching or deliberately typing in article numbers
Xor moving back and forth with RN commands like ``\fB-\fP,'' ``P'' etc.,
Xyou will still be able to read articles that would otherwise be
Xrejected.
X
X.H 3 "Two Adaptations"
X.P
XWe have arranged NewsClip to filter as you read. That means articles
Xare not rejected until you actually try to read them. If you ask RN
Xfor a subject list, (the ``='' command) you will see articles in the
Xlist that might be rejected. You won't see them when it comes
Xtime to read them later.
X.P
XOur adaptations to RN can optionally be made to work like RN's
X``kill'' files. In this case, all articles in a group are filtered
Xwhen you enter it. This takes time, but requires no processing as
Xyou read. It also means rejected articles won't show up in subject
Xlists. In general, this fits in better with RN's way of doing things,
Xbut it also involves an annoying delay every time you enter a group.
XIt is up to your system administrator to pick the adaptation desired.
X
X.H 2 "NewsFilter Bugs"
X.P
XIf your filtering program should crash for some reason -- perhaps you
Xindexed into an array beyond its bounds -- then RN will give up
Xtalking to it and accept all articles. You will get a brief message
Xindicating that the filter has shut down, but this can easily be lost
Xin RN's rapid screen updates.
X.P
XYou should check your filter from time to time in \fIlist\fP mode to
Xensure that it is working as you expect.
X
X.H 2 "Sending Commands"
X.P
XThere is a special entry point procedure called \fBcommand\fP which
Xyou can define in your newsclip programs. This gets a string
Xargument, which is intended to be a command from the newsreader.
X.P
XWe have added an extra command to RN that works at the article
Xlevel. It is ``\fBT\fP,'' because it ``talks'' to the news filter.
X(Ok, so that's not a great mnemonic. All the nice ones were taken.)
X.P
XIf you type a command to RN that starts with a \fBT\fP, the entire string
Xof that command will be passed to your NewsClip \fBcommand\fP
Xprocedure. Try something simple like this:
X.Bb
Xprocedure
Xcommand( string com )
X{
X dprintf( "The command was %s\\n", com );
X accept; /* means this command was valid */
X}
X.Be
XYou will find that when you issue a \fBT\fP command, your news filtering
Xprogram prints it out.
X.P
XThe trick is to interpret the string into commands for your newsfilter.
XIn general, these will be ``kill'' commands, requesting that your
Xfilter update databases to accept or reject certain kinds of articles.
XFor example, \fBTKF <1234.123@foo.bar>\fP might be a request to kill
Xall followups to the article with message-id \fB<1234.123@foo.bar>\fP.
X\fBTKU bad@worse.com (Evil Guy)\fP might be a request to kill all articles
Xby the specified user.
X.P
XIt is entirely up to you to interpret such commands, and entirely up
Xto the RN user (usually that's you as well) to send them.
X
X.H 2 "Interpreting Commands"
X.P
XHere's a sample procedure to interpret some commands like the ones
Xabove. We'll show you how to send them later.
X.Bb
Xprocedure
Xcommand( string com )
X{
X userid fromwho;
X string arg; /* argument to kill command */
X extern int chindex( string, int );
X extern string lower( string );
X extern string clipfront( string, int );
X string array artrefs; /* passed references array */
X if( chindex( com,1 ) != 'K' )
X reject;
X arg = clipfront(com,4);
X switch( chindex( com,2 ) ) {
X case 'F': /* followups */
X badmes[arg] = true;
X accept;
X case 'T': /* all followups of top parent */
X parse artrefs = arg, " ";
X if( count(artrefs) > 0 )
X badmes[artrefs[0]] = true;
X else
X reject;
X accept;
X
X case 'U': /* a user */
X parse fromwho = lower(arg);
X baduser[fromwho] = true;
X accept;
X case 'S': /* a subject with "re:" stripped off */
X badsubject[lower(arg)] = true;
X accept;
X case 's': /* subject pattern */
X killpatterns[lower(arg)] = true;
X accept;
X }
X reject; /* otherwise was an invalid command */
X}
X.Be
X.P
XFirst of all, we insist that the second letter of the command be
Xa ``K,'' as these are kill commands. Then we switch on the 3rd letter
Xto decide what command it really is. The command argument starts on the 5th
Xcharacter (index 4), as the 4th will be a space.
X.P
XThese commands update databases. One, \fBbadmes\fP is a database of
Xthe message-ids of articles we don't like. Articles with references to
Xthese IDs will be rejected in the \fBarticle\fP procedure.
X.P
XWe also have a database called \fBbaduser\fP which lists users we don't
Xlike the postings of. All postings from those users will be rejected.
XIn addition, we put the message-ids of those rejected postings into
Xthe \fBbadmes\fP database, so that we don't see the followups, either.
X.H 3 "Kill Topic"
X.P
XThe ``T'' command (kill a topic) is a variant of the ``F'' command, which
Xkills all followups. This expects to get passed the entire
X\fBReferences:\fP list for the article. It will pick the first reference,
Xwhich is the oldest parent article (in theory the original) and put it
Xin the database.
X.P
XThis causes not just the followups to our current article to be rejected,
Xbut all the followups to the original article that started the discussion
Xwhich it is part of. In theory, that eliminates an entire topic from
Xyour reading. Unfortunately, not all followups really contain a true
X\fBReferences:\fP chain, so life isn't perfect.
X.P
XYou could easily add your own commands to this list. You might want
Xto mark good messages as well as bad ones, and good users. You could
Xmark subject lines, manually entered killing patterns or any other
Xinformation you can store. Usually you store these things in a
Xdatabase which you will write out at the end of your session, and read
Xin at the beginning of the next one.
X.P
XYou can also store information that only lasts for the duration
Xof the current session, of course.
X
X.H 3 "Using the Databases"
X.P
XOf course, to use the databases defined above, you need to read and
Xwrite them:
X.Bb
X/* Header symbols */
Xextern string subject;
Xextern string message_id;
Xextern string array references;
Xextern userid from;
X/* Databases */
Xdatabase badmes; /* bad message-ids */
Xdatabase baduser; /* users I don't like */
Xdatabase gooduser; /* users I do like */
Xdatabase badsubject; /* subject line database */
Xdatabase killpatterns; /* kill patterns for subject lines */
Xprocedure init()
X{
X badmes = read_database( "~/News/badmes" );
X baduser = read_database( "~/News/baduser" );
X gooduser = read_database( "~/News/gooduser" );
X badsubject = read_database( "~/News/badsubject" );
X killpatterns = read_database( "~/News/killpatterns" );
X}
Xprocedure terminate()
X{
X extern datetime time_now;
X /* save out my updated databases */
X write_database( badmes, "~/News/badmes", time_now-2*week );
X write_database( baduser, "~/News/baduser", time_now-month );
X write_database( badsubject, "~/News/badsubject", time_now-2*week );
X write_database( killpatterns, "~/News/killpatterns", time_now-2*week );
X /* Gooduser gets updated by hand */
X}
X.Be
X.P
Xand you also need an article routine to deal with them:
X.Bb
Xprocedure
Xarticle()
X{
X extern string drop_re( string );
X /* accept articles from desired users immediately */
X accept if from in gooduser;
X /* is this a followup to a killed chain? */
X reject if references != nilarray && references in badmes;
X /* If from a hated user, reject and kill the chain too */
X if( from in baduser ) {
X /* reject all followups, too */
X badmes[message_id] = true;
X reject;
X }
X /* if the subject is explicitly a bad one, or a keyword from our
X pattern list is found in it, reject */
X if( drop_re(subject) in badsubject || subject has killpatterns )
X reject;
X}
X.Be
X
X.H 2 "Sending the Commands"
X.P
XYou could send these commands yourself, by typing them in starting with
Xa \fBT\fP. RN, however, has a powerful macro facility which you should
Xuse to send your kill commands. In RN, you place macros in a file
Xcalled \fB.rnmac\fP, in the same directory as the \fB.newsrc\fP file.
X(Usually your home directory)
X.P
XFor the full story on RN macros, see the RN manual. In general, macro
Xdefinitions consist of a string that lists the key sequence required
Xto activate the macro, followed by white space and the key sequence
Xthe macro expands to. Type \fB&&\fP to RN to see your macros.
X.P
XLet's say you want the \fBK\fP key to issue the Kill Topic command that
Xwe have defined. (Normally this causes RN to store the subject
Xof the current message in its ``kill'' file, but you probably won't be
Xusing kill files much at this point. Your macro might look like:
X.Bb
XK TKT %R\\n
X.Be
X.P
XThis means that when you press ``K,'' RN acts like you typed the string
Xon the right. In RN macros, a ``%'' indicates a special code is to
Xbe expanded. In this case, ``%R'' expands to a proper \fBReferences\fP
Xline for a followup to the current message. Our ``TKT'' command takes
Xthe first ID from this list, and puts it in the bad message database.
X.P
XTry this out. Put a \fBdprintf\fP statement in your command processor
Xto make sure everything's happening as you like it. A full set
Xof macros might look like this -- this time with 2 letter macros.
X.Bb
X# Kill all followups to current article
XKF TKF %i\\nn
X# Kill all messages from poster of this article
XKU TKU %f\\n!mail '%f' <%p/killmessage\\nn
X# Kill all followups to immediate parent article (cousins)
XKC TKF %r\\nn
X# Kill all followups to the original parent of this discussion. (topic)
XKT TKT %R\\nn
X# Kill all articles with exactly the same subject, and all followups
XKS TKS %s\\nTKF %i\\nn
X# kill all articles with the user-provided pattern in the subject
XKs TKs %"Kill Pattern? "\\n
X.Be
X.P
XYou might adapt these macros to add a regular ``k'' on the end, so that
XRN performs its own local kill on all articles with the same subject
Xas your article when you issue the command.
X
X.H 4 "Eliminating a User"
X.P
XThe above sample ``KU'' command sends mail to the user whose postings you have
Xdecided to no longer read. We suggest you send a polite message of
Xthe following form should you do this. Put text like this in the
Xfile \fBNews/killmessage\fP: (\fBNews\fP is your private News directory.)
X.in 0.5in
X.P
XThis is an automatic message:
X.P
XI have decided, for various reasons, to filter out all USENET news
Xarticles posted by you from now on. This will be done by automatic
Xsoftware.
X.P
XThere are many reasons why I decide to no longer read the writings of
Xcertain people from time to time. The most common one is simply a lack
Xof time. It may be as 'personal' as the decison to no longer pick up
Xa magazine one once read regularly.
X.P
XThis message is being sent to you because you may or may not realize that
Xcertain aspects of your writings are driving people away from them.
XIf this is the case, you may wish to reconsider your actions on USENET.
X.P
XOnce again, this is an automatic message, and you need not reply unless
Xyou are unsure what USENET activities you have done which would cause
Xme to choose to no longer read your writings.
X.in -0.5inYou might also decide to prompt for a further reason and add that to
Xthe text of the message.
X.H 3 "Local vs. Global"
X.P
XThis sample control program updates global databases, but it could
Xalso be set up to update databases on a per-newsgroup basis. You could
Xhave different commands for local killing and global killing, as desired.
X
X.H 2 "Other Ideas"
X.P
XYou can do a great deal with this feature. While normally you would use
Xit to reject articles, other things are possible.
XInstead, you might decide, when you
Xsee an article you like, to have a command that ensures you see all
Xthe followups to that article. Later, when you get bored, you could
Xhave another message to cancel this request.
X.P
XYou could set up your NewsClip program to read something like:
X.Bb
X/* accept original articles */
Xaccept if references == nilarray;
X/* accept followups to good articles */
Xaccept if references in goodlist;
Xreject;
X.Be
XThis would only show you original, non-followup articles, and articles
Xthat are followups to originals that you liked. You would then see only
Xa very short set of articles in your news reading sessions. If you saw
Xone you were interested in, you would send a command down to your
Xnewsfilter to put the message-id or subject of that article in your
Xdatabase of good articles.
X.P
XYou would then see that chain of articles
Xuntil you sent down a message to remove the article from your database.
X.P
XBecause our normal configuration is to filter as you read, you would have
Xto make your decision the moment you saw the original article. Because
Xof the immediate communication, the followups to that article would get
Xaccepted right there in the same session, as you come to them.
XThis would not work if followups appear before their parent articles,
Xas sometimes happens due to funny news transmission problems.
X
X.H 2 "Using \fInewsrc\fP Mode"
X.P
XWhile real-time filtering in partnership with RN is great, you may
Xalso find use for automatic filtering in \fInewsrc\fP mode.
X.P
XFor example, if you have groups where you filter almost all the
Xarticles out, why wait to do this during your newsreading session?
X.P
XInstead, arrange to run your NewsClip program at night, perhaps every
Xnight, and have it filter for you. Just say:
X.Bb
Xnclip mode=newsrc
X.Be
Xand it will do the work. You can't run RN while this is taking place.
X.P
XYou may decide to only do night-time filtering of certain groups. In
Xthis case, create a special ``las'' file that lists those groups, and
Xuse the command:
X.Bb
Xnclip mode=newsrc las=lasfile +only
X.Be
X.P
XYou might even have a different program for night-time filtering than
Xyou use for newsreading -- it's up to you. You could use the same
Xprogram, and have it check command line options set with \fIoption=\fP.
XYou might want to use the special \fI+unsubscribed\fP option to scan
Xunsubscribed groups late at night. You might arrange to only scan
Xthe unsubscribed groups once a week, and scan the other groups more
Xfrequently.
X.P
XOf course, if new articles arrive during your newsreading session, they
Xwon't be scanned by your late night filter session, but rather will be
Xprocessed by your live filter. This should not normally be a problem.
X
X.H 3 "RRN"
X.P
XIf you use RRN, and don't have news spool files on your machine, you
Xcan't make use of \fInewsrc\fP mode. Live filtering is the only option
Xavailable to you.
X
X.H 2 "Kill Files"
X.P
XRN's ``Kill File'' facility has not been disabled by our adaptations.
XYou can use RN kill files in combination with NewsClip filtering.
XIn fact, you can filter some groups with kill files and other
Xgroups with NewsClip, as you desire.
X.P
XRN Kill files get to go first. NewsClip filtering comes after that.
XThis can be changed by the system administrator if you desire to
Xhave NewsClip filtering work as a pre-pass, like the kill file.
X
X
X
X.H 1 "Licence"
X.P
XThe NewsClip\*(Tm system is Copyright {copyright} 1989 by Looking Glass
XSoftware Limited. This manual and other associated documents, unless
Xspecified otherwise, are Copyright {copyright} 1989 by Looking Glass
XSoftware Limited. These programs and documents may be used only by licenced
Xcustomers of the NewsClip system, or those performing a limited
Xduration evaluation of the system.
X.P
XSince this is a programming language system, you might assume that you
Xare free to give out or sell compiled programs. This is not the case.
XThe compiler in this system is actually fairly simple, and just checks
Xand maps types and statements. It makes the language more readable and
Xreliable.
X.P
XThe real work is done by the library, which is quite complex. If you
Xlook at your executables, you will find that the library makes up the
Xbulk of them.
X.P
XYou are free to do whatever you wish with your newsclip source programs,
Xthe intermediate C programs the compiler outputs, and the object modules
Xyour C compiler makes from those. You can't, however, give out our
Xlibrary in any form without our permission. We will arrange permission
Xunder certain circumstances -- send us mail.
X
X.H 2 "Single User"
X.P
XA single user licence is just that -- a licence for a single person
Xto use NewsClip to filter news. You can't let other people use it, and
Xyou can't filter feeds with it. You can, however, run it on more
Xthan one machine, as long as it's just you using it on each machine.
X
X.H 2 "Site"
X.P
XA site licence lets everybody on the licenced computer use NewsClip
Xtools on that computer. Everybody can compile their own programs, or
Xone person can compile programs for other people on that computer.
XFeeds can be filtered, both to and from.
X.P
XSite licences come in different sizes -- workstation, minicomputer,
XBBS, network and mainframe. If you grow in a dramatic way, an upgrade
Xcan be easily arranged.
X
X.H 2 "Feeds"
X.P
XNaturally it's easy to filter your outgoing feeds with NewsClip. You
Xwill also want to filter the feed that feeds you. That means running
Xyour NewsClip feed program on the feeding machine.
X.P
XTo do this, you are permitted to move the library source to your feed
Xmachine, and compile it there. Please inform the administrators of that
Xmachine that they must pay a licence fee if they wish to use the software
Xother than for feeding your site.
X
X.H 2 "Evalu-Ware"
X.P
XSome versions of NewsClip are released as what we term ``Evalu-Ware'' --
Xour name for what some people have called ``Shareware.'' (We use the
Xdifferent name simply because the ``Share'' in \fIShareware\fP has led
Xmany people to believe such programs are not regular commercial software
Xproducts. In fact, Evalu-ware products are fully commercial software
Xsystems. They simply allow an unusual form of distribution that benefits
Xboth the customer and the publisher.
X.P
XIn particular, users can access the software far more easily (particularly
Xin the case of Unix software, for which there is no single medium) and they
Xcan perform a no-risk evaluation of the actual product.
X.P
XYou can copy the Evalu-Ware versions of the NewsClip system freely. You
Xcan leave the source code available for network access and distribute it to
Xyour friends and associates.
X.P
XWhen you get the source code, you may compile it and try out the system.
XUse it and experiment with it for a while, then make your own judgement
Xabout whether you wish to continue using it. If you do, call or write
Xus to arrange a licence. You can phone 519-884-7473, or 800-265-2782
Xin the USA. Or mail to \fBcliporder@clarinet.com\fP. Read the file
X\fBLicence\fP in the source directory for full details.
X.P
XIf you don't wish to continue using it, simply erase the binaries and
Xyou have lost nothing.
X
X.H 2 "Notes"
X.P
XNewsclip is a trademark of Looking Glass Software Limited.
X.P
XClariNet is a trademark of Clarinet Communications Corp.
X.P
XUnix is a trademark of AT&T Bell Labs.
END_OF_FILE
if test 46220 -ne `wc -c <'doc/append.mm.2'`; then
echo shar: \"'doc/append.mm.2'\" unpacked with wrong size!
fi
# end of 'doc/append.mm.2'
fi
echo shar: End of archive 12 \(of 15\).
cp /dev/null ark12isdone
MISSING=""
for I in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ; do
if test ! -f ark${I}isdone ; then
MISSING="${MISSING} ${I}"
fi
done
if test "${MISSING}" = "" ; then
echo You have unpacked all 15 archives.
rm -f ark[1-9]isdone ark[1-9][0-9]isdone
else
echo You still need to unpack the following archives:
echo " " ${MISSING}
fi
## End of shell archive.
exit 0