rsalz@uunet.uu.net (Rich Salz) (06/27/89)
Submitted-by: utzoo!henry Posting-number: Volume 19, Issue 78 Archive-name: cnews2/part01 [ This is a rewrite of the transport and expire portions of the Usenet software. It was designed to be fast and flexible. --r$ ] : ---CUT HERE--- mkdir doc mkdir conf mkdir notebook mkdir batch mkdir contrib mkdir contrib/nntpmail mkdir contrib/nntpmail/post_via_mail mkdir contrib/nntpmail/mailing_lists mkdir contrib/nntpmail/mailing_lists/appendfile mkdir contrib/nntpmail/nntp_support mkdir contrib/nntpmail/mail_to_group mkdir contrib/rn.mod mkdir expire mkdir h mkdir hfake mkdir input mkdir libbig mkdir libbsd42 mkdir libc mkdir libcnews mkdir libfake mkdir libsmall mkdir libstdio mkdir libusg mkdir libv7 mkdir libv8 mkdir man mkdir misc mkdir nntpdiffs mkdir nntpdiffs/diff mkdir nntpdiffs/src mkdir nntpdiffs/src.allnew mkdir relay mkdir relay/regress mkdir relay/regress/out mkdir relay/regress/out/test mkdir relay/regress/out/test/a mkdir relay/regress/out/test/b mkdir relay/regress/out/test/c mkdir relay/regress/out/control mkdir relay/regress/master mkdir relay/ads mkdir relay/anews mkdir relay/ctl mkdir relay/aux mkdir relay/altctl mkdir relay/sh mkdir rna mkdir rna/lib echo 'COPYRIGHT': sed 's/^X//' >'COPYRIGHT' <<'!' X/* X * Copyright (c) University of Toronto 1985, 1986, 1987, 1988, 1989. X * All rights reserved. X * Written mostly by Geoffrey Collyer and Henry Spencer. X * This software is not subject to any license of the American Telephone X * and Telegraph Company or of the Regents of the University of California. X * X * Permission is granted to anyone to use this software for any purpose on X * any computer system, and to alter it and redistribute it freely, subject X * to the following restrictions: X * X * 1. The authors are not responsible for the consequences of use of this X * software, no matter how awful, even if they arise from flaws in it. X * X * 2. The origin of this software must not be misrepresented, either by X * explicit claim or by omission. Since few users ever read sources, X * credits must appear in the documentation. X * X * 3. Altered versions must be plainly marked as such, and must not be X * misrepresented as being the original software. Since few users X * ever read sources, credits must appear in the documentation. X * X * 4. This notice may not be removed or altered. X */ ! echo 'PATCHDATES': sed 's/^X//' >'PATCHDATES' <<'!' X23-Jun-1989 ! echo 'README': sed 's/^X//' >'README' <<'!' XThis is C News, superseding assorted preliminary releases. 9 June 1989 X XC News is a reimplementation of the transport and storage subsystems of the Xnews software -- basically, everything except news readers. We supply a Xsimple news reader (written by Michael Rourke, included by permission, Xslightly modified [so bugs are probably our fault]) as a replacement for XB-News readnews suited to use by occasional users. For regular news users, Xthere are several more sophisticated readers widely available, and all should Xwork with C News. We use Larry Wall's "rn" ourselves; we have not included Xit because this distribution is already rather big. X XC News's major advantage over B News is that it is much faster. Timings Xquite a while ago gave C News a speed advantage of roughly a factor of 25 Xin processing incoming batches. This has probably improved a bit since. XC News is now, on good machines with good C libraries, mostly system-call Xbound. Use of system calls has been optimized with some care, so it's Xunlikely that further big speed improvements can be made at user level Xwithout sacrificing backward compatibility. See our paper in the Winter '87 XUsenix proceedings for some discussion of how the speed was achieved. X XC News also wins over B News on simplicity and robustness. We provide X(in our opinion) everything that's necessary, and avoid the frills that Xrun up the complexity and decrease reliability. We have not attempted to Xprovide every feature anyone can think of, and have no plans to do so Xin future either. (This is one reason why we've stayed out of the news- Xreader business, which generally has a bad case of feature-of-the-week Xsyndrome.) X XC News's files are fully compatible with those of B News for any program Xthat does not read log files and does not inspect the middle field of the Xhistory file closely. (The one major program that does is "nntp"; we include Xdiffs for NNTP 1.5 which handle our middle-field format, interface it to Xour input subsystem, and generally make it quite a bit faster than the Xoriginal.) C News complies fully with RFC 1036 (nee RFC 850), the official Xdefinition of news interchange format. X XC News is, by intent and *we think* in practice, compatible with B News Xat the level of most interfaces to the normal user, which basically means Xthe semantics and options of "inews". It is *not* compatible on the Xsystem-administration level, although we think most of the changes are Ximprovements or worthwhile simplifications. The "postnews" that we supply Xis not compatible with that of B News; it is purely interactive, as news Xthat is already formatted can simply be fed to "inews -h". X XFor those who now run one of our ancient pre-alpha versions, many things Xhave changed, and in particular the four-field history file format is gone. XC News has also changed quite a bit since the alpha release that went out Xon Usenet some time ago. X XFor our beta testers, build and the Makefiles have changed quite a bit but Xthe software itself needed only fairly trivial fixes. X XWe know of three things that could still use work in this release: X X 1. The documentation could use work, especially for naive customers. X As it stands, it pretty much assumes a general knowledge of news X software. X X 2. There are a great number of small improvements that could X be made to the installation process, especially to permit still more X customization via the "build" program. X X 3. The fgetmfs function (in the libc directory) assumes that fgets X does not alter the buffer beyond the end of the string. We are not X sure how portable this is, although it works on all our beta-test X systems, and may revise fgetmfs someday. X XThe active file format is the 4-field one that B news introduced midway Xthrough 2.10, with minor additions: an `x' in the 4th field means X``discard articles in this newsgroup'', and `=group' in the 4th field Xmeans ``file articles for this newsgroup under `group' instead''. X XThe history file format is like B with one exception: the second field, Xwhich few programs ever look at, now consists of two subfields separated Xby a tilde (~). The first is the arrival date as a decimal number, the Xsecond is the expiry date (if any) as a human-readable date (as emitted by Xrnews) or a decimal number (after expire has gotten its hands on it once). XExpire is tolerant of human-readable dates in both those places, but other Xthings may not be. The best way to get the history file into the new Xformat is to rebuild it completely (this is RELATIVELY quick). X XThe sys file format is like a late-model B news with two extensions. XFirst, the second field (groups and distributions) may optionally be Xsplit into two subfields (newsgroups and distributions, respectively) Xwith a slash. This permits solutions to various tricky problems that can Xarise in odd situations if it is impossible to tell what's a newsgroup Xname and what's a distribution. Second, there is a new flag in the Xthird field: f is like F except that its output has the size Xinformation that the C batcher wants for accurate limiting of batch Xsize. X XThe way the news articles themselves are stored is totally unchanged; we Xhave been unable to think of any changes that are worth the trouble. X XThere are some new control files in /usr/lib/news, to control the smarter Xexpire and batcher. (Old C News sites, note some format changes.) X XFile organization: the one change is that programs are now kept mostly Xin /usr/lib/newsbin, with /usr/lib/news reserved for control files etc. X XB News sites note: /bin/rnews is now a front end for the input spooler. XThe real news-filing program is called relaynews and is not in /bin. X XC News is meant to run adequately on a 16-bit machine, although this has Xnot been tested thoroughly since utzoo became a Sun. X XMost (by intent all) of the programs understand seven key environment Xvariables: NEWSARTS specifies location of articles (default X/usr/spool/news), NEWSCTL specifies location of control files (default X/usr/lib/news), NEWSBIN gives location of programs (default X/usr/lib/newsbin), NEWSUMASK gives the umask to be used in creating Xfiles (default 002), NEWSPATH gives the path used to find "normal" programs X(default /bin:/usr/bin), NEWSMASTER is the address to which problem Xreports should be mailed (default usenet), and NEWSCONFIG is the full Xpathname of a little file that shell programs can use to pick up local Xdefault settings for all these things (the equivalent for the C programs Xis in the C News library) (default /usr/lib/news/bin/config). The Xenvironment variables override the defaults for testing and for operation Xin funny situations. Note that one or two things (e.g. relaynews), as Xdistributed, will insist on renouncing setuid privileges if invoked with Xthese overrides. X XBe warned that the nntp stuff in nntpdiffs, and the simple news reader in Xrna, have not been gone over very well to make sure that they use the Xstandard configuration mechanisms. Hardwired pathnames may be present there, Xand in general the stuff there is not well fitted into our automatic-install Xmachinery. X XSee ROADMAP for a run-down on what's where in the distribution. See Xdoc/install for how to install it. Conf/build is the interactive setup Xprogram that does most of the work, or rather, sets up shell files which, Xwhen run, do most of the work. Even if your system is odd enough that Xyou don't want to run the shell files conf/build generates -- doit.bin and Xfriends -- as is, you are most strongly advised to run conf/build and use Xthe resulting shell files as guides, rather than trying to "wing it" Xyourself. Getting the library built correctly is *NOT* trivial, because Xwe try to cater to all 57000 different variants of Unix and there are a lot Xof decisions that have to be made. This is why we supply a 31KB shell Xprogram to generate the configure+compile+install instructions, rather than Xjust sending a complete pre-cooked recipe embodied in a Makefile: there are Xtoo many variables affecting how it should be done. X XC News has been tested pretty thoroughly. We're also thoroughly sick of Xit and make no promises that there will ever be another release. We may, Xrepeat *may*, provide updates via some appropriate newsgroup (currently Xthe best choice is "news.software.b", although there is some sentiment for Xfolding all the subgroups there into just "news.software"; we oppose Xcreation of "news.software.c" because we don't think there will be enough Xtraffic to justify a whole newsgroup). X XIf you've found a problem, we definitely do want to hear about it. But, Xwe *do not* want to see 2000 lines of diff listing! What we want to see Xis a concise human-readable description of what the problem is and how, Xif at all, you solved it. If we want the diff listing, we will ask. XSimilarly, we are interested in hearing about changes and improvements, Xbut want to see terse descriptions first. X XIf you want us to consider changes/fixes/etc, send them to us, don't just Xpost them to the net. We don't necessarily read all possibly-relevant Xgroups. Only postings from us are officially part of C News. X XTo send comments, complaints, problem reports, etc., do *not* mail to XGeoff or Henry personally, but to: X X c-news@utstat.toronto.edu Xaka c-news@utstat.utoronto.ca Xaka utzoo!utstat!c-news X XUtstat.toronto.edu is directly on the Internet but does not have a lot of XUUCP connections. However, it does talk to utzoo. Utzoo connects to half Xthe known universe (well, not quite, but try via allegra, att, attcan, Xattunix, decvax, dptcdc, floyd, hoptoad, kitty, linus, mnetor, pyramid, Xsuncan, utai, utgpu, watmath, or yunexus). X X Geoff Collyer X Henry Spencer ! echo 'ROADMAP': sed 's/^X//' >'ROADMAP' <<'!' Xbatch Output batcher. It should work with B News, but not as well -- X it really wants to be told sizes of articles as well as names. Xconf Configuration stuff, including the "build" shell program that X does most of the work of installing C News. Xcontrib Software from other contributors, possibly useful but not really X an official part of C News. Xdoc User documentation, including "install" which discusses how to X install C News. Xexpire Expire and friends, including history rebuilding and active-file X updating (neither of which is done by expire in C News). Ought X to work with B News. Fast, and permits control of expiry time X and such on a per-group basis. Xh Header files for include (see below). Xhfake Header files that your system ought to have but possibly doesn't, X for include (see below). Xinclude Actually not in distribution -- "build" constructs this from h and X hfake as required. Xinput Input spooler and processing. Includes the "rnews" that goes in X /bin or wherever. Xlibsmall Xlibbig Libraries for a couple of key functions that care whether they X have a large address space to work with. Xlibbsd42 Xlibusg Xlibv7 Xlibv8 Libraries for details that are Unix-variety-specific. Xlibc Stuff that ought to be in your C library but probably isn't. Xlibcnews General C News library functions. Xlibfake Stuff that might be in your C library, but might not be. Xlibstdio Fast re-implementations of some crucial stdio functions. X These are compatible with most Bell-derived stdios, and quite a X bit faster than most of the pre-System-V ones (which includes X those of many 4BSD variants). Xman Manual pages. Xmisc Miscellaneous internal utilities and maintenance programs. Xnntpdiffs Amendments to NNTP version 1.5, for Internet users, to work X with C News history-file format and to be a good deal faster. Xnotebook The C News Implementor's Notebook -- pieces of documentation X aimed more at gurus than novices. Xrelay Relaynews and friends. The heart of C News -- actual reception X and filing of articles. Xrna A simple version of readnews (by Michael Rourke at UNSW) for X naive news users. ! echo 'doc/install': sed 's/^X//' >'doc/install' <<'!' X.de Fn X\\&\\$1\\fB\\$2\\fP\\$3 X.. X.TL XInstalling ``C News'' Network News Software X.AU XGeoff Collyer X.AI XDepartment of Statistics XUniversity of Toronto X.AB XThis document describes the flow Xof network news within and between machines, Xwhat each component of C news does, Xand Xhow to install C news. X.AE X.SH XIntroduction X.PP XNetwork news X(or X.I netnews Xfor short) Xconsists of a collection of messages formatted similarly to XARPAnet mail X(see ARPA Internet RFC 1036 for details), Xwidely spread. XThe logical network, Ximposed on top of various real networks, Xformed by the set of all interconnected sites Xexchanging network news is called ``Usenet'' Xand was formed in 1979, Xradiating out from Duke University. XNetnews is propagated Xbetween cooperating machines Xby a flooding algorithm, Xwith some loop prevention heuristics: Xeach machine sends its neighbours news that the Xneighbours have (probably) not yet seen. X.PP XFlow of netnews between machines Xmay be achieved by use of any network Xwhich can transmit an arbitrary stream of X(at least 7-bit) XASCII code, unmodified. XIf a network cannot transmit ASCII intact X(e.g. BITNET), Xit is possible to encode netnews Xbefore transmission across the network Xand Xdecode it upon reception from the network. XSince one cannot be certain that Xone's network neighbours will be up and Xreachable at all times, Xoutgoing netnews must be queued, Xat least in the case of network trouble. XTo date, Xat least Xthese networks, Xprotocols and media Xhave been used to transmit netnews correctly: XUUCP, XRS232, XNNTP, XEthernet\(rg, Xthe ARPA Internet, XDatakit\(rg, XACSnet, Xmagnetic tape, XSMTP, Xand XBITNET, Xthough at least the last two require some form of encapsulation Xto avoid corruption of articles; XSMTP because some common implementations get the newline-CRLF Xmappings wrong, Xthus throwing off byte counts in batches, Xand XBITNET because of its Procrustean chopping, Xexpanding, Xmapping, Xbashing Xand Xsmashing Xof all data sent through it X(sending lines of 80 or fewer characters of Xletters of either case and digits and plus and minus Xappears to be safe). X.PP XNetnews arrives via some network, Xwhich causes a command to be executed upon arrival X(e.g. X.I rnews ). X.I rnews Xtypically spools the inbound netnews for later processing. XEventually X(often within the hour Xor at the end of the business day), Xthe input queue is run Xand the netnews is stored locally, Xtypically under X.I /usr/spool/news , Xand queued for transmission to netnews neighbours. XOnce stored on disk, Xnetnews may be read by any of a collection of unprivileged news readers, Xincluding X.I cat (1). X.I Expire Xis run typically each night Xto remove old netnews from disk. X.PP XC News was originally written to provide a Xmuch faster and smaller, Xmore robust, Xreliable Xand Xcorrect Ximplementation of netnews software than B 2.11 news. XWe believe that C News is also Xfaster, Xsmaller Xand Xmore robust Xthan B 3.0 news. XB 3.0 news has many more features; Xtake that as you will. X.SH XC News Components. X.PP X.I Rnews Xinvokes Xthe input subsystem, Xwhich Xspools incoming netnews in its original form, Xas received, Xtypically in compressed batches. XPeriodically, Xthe input queue should be run Xby invoking X.I newsrun , Xthus uncompressing any compressed input Xand feeding it to X.I relaynews . X.PP X.I Relaynews Xfiles incoming netnews as articles on disk Xand Xinitiates spooled transmission to other machines, Xoften by simply writing the names of the disk files Xcontaining the articles on the ends of `batch' files of file names. XQuite a bit of policy from RFC 1036 is embedded in X.I relaynews . X.I inews Xis a complex front-end for X.I relaynews Xwhich implements much of the per-site policy on news posting. X.I inews Xis a fairly-slow shell script which is adequately fast for most sites, Xbut it will need to be replaced by something more streamlined Xfor sites which gateway mailing lists into netnews, Xfor example. X.PP XThe output X.I batcher Xreads lists of file names Xand Xgenerates news batches X(see RFC 1036 or X.I news (5) Xfor the format) Xof prescribed sizes Xand queues the batches Xfor transmission to other sites. XThe batcher is run asynchronously with X.I relaynews , Xtypically once an hour Xoutside of business hours. X.PP X.I Expire Xis generally run once per night Xto remove from disk news articles older than a few days. X.I Expire Xcan use different expiry criterion for different newsgroups Xand can archive articles instead of removing them. X.I Expire Xalso runs asynchronously Xwith X.I relaynews . X.PP XThere are many news readers. XC News Xcomes with a limited news reader X(\c X.I readnews Xby Michael Rourke) Xsufficient to replace Xlong-winded X.I /etc/motd s Xbut you will want a heavy-duty news reader Xif you plan to read more than local news groups. XWe recommend X.I rn Xby Larry Wall, Xavailable from Xyour netnews neighbours Xor Xyour nearby X.B comp.sources.unix Xarchive site. XThere are others: X.I vn Xand X.I vnews Xare two. X.SH XPreparation for Installation X.PP XNetnews consumes a lot of disk space Xand Xoften a lot of transmission time. XHere are some relevant measurements regarding a full netnews feed Xas of the time of writing X(January 1989), Xtaken from X.I news.lists : Xa day's netnews is about 3Mb and 1,400 articles Xin 450 newsgroups. XYears ago, Xsites often kept 14 days of netnews on disk, Xbut now many sites keep news for 3 to 5 days, Xthus allowing for the occasional long weekend. XThus a full news feed expired after 4 days will consume Xabout 12Mb. XSome people feel that news volume is doubling roughly every 16 months. XIf this is true, Xwe can expect volume to increase by a factor of 10 Xin about 4 years Xto 30Mb per day or 115Mb for 4 days. XIt is thus wise planning to set aside a lot of disk space for netnews. XThere are two ways to cope with ever-increasing volumes of netnews: Xrefuse to accept more newsgroups, Xor Xexpire news after shorter intervals on disk. XA current full feed takes just over 7 hours to transmit at 1200 baud, Xso for dial-up connections Xone clearly wants the fastest modems one can afford. X.PP XClearly, Xtransmitting a full news feed is a non-trivial commitment of resources, Xso you may have some difficulty in finding a site willing to supply one. XSuch a site may in turn expect you to feed yet other sites. XYou will need to agree with your prospective netnews neighbour(s) Xupon transfer media, Xprotocols Xand Xnetworks. X.PP XBefore proceeding to install C News, you should read this document through Xto the end, probably read the companion document X\fIThe Interface Between C News And The Outside World\fR, Xand possibly read selected items in the X\fIC News Implementor's Notebook\fR and the manual pages. X.PP XYou will need to Xassign a user id and group id to netnews X(often new ids called ``news''); Xinitialise these directories: XNEWSCTL X(typically X.B /usr/lib/news ), XNEWSBIN X(\c X.B /usr/lib/newsbin ), Xand XNEWSARTS X(\c X.B /usr/spool/news ); Xand Xinstall each subsystem. XNEWSCTL and NEWSARTS Xare logically one subtree, Xdefining a news data base, Xbut are split for backward compatibility with Xolder news software, Xparticularly old news readers. XNEWSBIN Xcontains programs and shell scripts Xwhich might be common amongst machines sharing Xa common architecture X(e.g. Sun 3's); X.Fn NEWSCTL /bin Xmay override these. XThe goal is to install the subsystems, Xintegrate them into a working news system, Xand Xconfigure the news system to communicate with other news systems. X.PP XThere are a few key files that must exist before any serious Xattempt may be made to operate the news software. X.Fn NEWSCTL /active Xis the list of newsgroups that this site knows X(is willing to accept or individually reject), Xand must be owned by the XNEWS Xuserid X(the userid that owns X.Fn NEWSBIN /relay/relaynews , Xtypically X.I news ). XYou will probably want to get your initial X.Fn "" active Xfile from your upstream feed Xand edit it to suit the set of groups you wish to receive. XBe sure to make the second field more than five digits wide, Xby adding leading zeroes X(ten digits is a conservative width). X.Fn NEWSCTL /sys Xdefines the newsgroups that this site is willing to accept Xand describes how they are to be forwarded to its neighbours. X.Fn NEWSCTL /server Xcontains the hostname of your file server, Xif you have multiple machines sharing news Xvia a network file system. X.Fn NEWSCTL /whoami Xsimilarly contains the name by which a cluster of machines Xsharing news is to be known for purposes of news. X.Fn NEWSCTL /mailname Xis optional and contains the full X(possibly dotted) Xname by which your cluster is known for purposes Xof mail. X.Fn NEWSCTL /organisation X(or X.Fn NEWSCTL /organization Xif you insist) Xcontains the name of your organisation, Xwhich will be copied into the X.B Organization: Xheader of articles posted locally, Xby default. X.Fn NEWSCTL /mailpaths Xdefines mail routes to machines which contain aliases Xfor postings to moderated newsgroups. X.Fn NEWSCTL /log , X.Fn NEWSCTL /errlog , X.Fn NEWSCTL /history , X.Fn NEWSCTL /history.dir , Xand X.Fn NEWSCTL /history.pag Xmust exist and be owned by the XNEWS Xuserid. XTentative versions of all these files are built by the installation Xprocedures, but it is quite likely that you'll have to edit some of them. X.SH XC News Installation X.PP XProceed to the X.Fn \& conf Xdirectory of the distribution. XThere is a major shell file here, named X.Fn \& build , Xthat will interrogate Xyou at length and construct shell files to do the real work. XYou may need to do X``chmod\ +x\ build'' before running it, to make it executable. X.PP XYou will probably need your system's manuals on hand to answer X.Fn \& build 's Xquestions. XAnother terminal (or another window, on a bitmap display) would also be Xuseful. XYou'd best be prepared to take notes, also, as X.Fn \& build Xwill occasionally suggest that something be checked when you're done. X.PP X.Fn \& Build Xitself does not alter any files or perform any installation Xchores: all it does is create shell files in the X.Fn \& conf Xdirectory. XIf you already know something about news software, or are merely Xsuspicious that X.Fn \& build Xhasn't done everything right, you should Xprobably read the shell files before running them. XThere are four of them: X.Fn \& doit.root , X.Fn \& doit.bin , X.Fn \& doit.news , Xand X.Fn \& again.root . X.PP X.Fn \& Doit.root Xsets up the major directories for news, and sets their Xownerships correctly. XIt typically will have to be run as \fIroot\fR to have adequate permissions Xfor all of this. XIt is brief. X.PP X.Fn \& Doit.bin Xdoes most of the work of building and installing the programs. XIt should be run as the user who owns the distribution directories and will Xown the executable programs under NEWSBIN. X.PP X.Fn \& Doit.news Xdoes some other small chores and installs control files. XIf any of the control files already exist, it will complain and refuse Xto overwrite them, as a safety precaution. XIt should be run as the owner of the news files. XSince many of the files it is installing are built by X.Fn \& doit.bin , Xthat should be run first. X.PP XFinally, X.Fn \& again.root Xtends to ownership and permission changes on Xa few programs that need to run set-userid. XIt requires the ability to change ownerships and to set permissions on Xthe files afterwards, which usually means running it as \fIroot\fR. XIt too is brief. X.PP XThere are undoubtedly strange systems out there that X.Fn \& build Xand friends are not smart enough to cope with. XIn such cases it will be necessary to edit the shell files before running Xthem, or to use them as guides and do the work by hand. XIn particular, systems that require strange options in X.Fn \& Makefile s Xwill need to have those inserted by hand. X.PP XIf you want to test pieces of C News without installing them, some X(not all) of the subsystems have a ``make\ r'' feature that runs a Xregression test. XNote: almost all of these require that X.Fn NEWSCTL /bin/config , Xor its local equivalent, be in place already so that shell files Xcan find out what PATH (etc.) they should be using. X.PP XNote that it is easy to build a test news system which is completely Xindependent of other existing news systems on the same machine, Xor Xto build one which shares its X.Fn NEWSBIN "" Xwith another C news system, Xor shares input of articles X(e.g. running an old B news system Xand a new C news system off the same stream of input Xuntil you are confident that your new C news system is operating Xto your satisfaction). XSee X.I subst (1) Xfor the mechanism which permits quickly changing Xall the places that know the location of X.Fn NEWSCTL /bin/config. XAfter that, Xedit this news system's X.Fn NEWSCTL /bin/config Xand things should be set up for separate existence. X.SH XFirst News X.PP XWhen you arrange to get a news feed from your neighbor, you should also Xask him to send you the current set of articles in the newsgroup X\fBnews.announce.newusers\fR. XSeveral of these are very important reading for people who are new to the net. X.SH XUnusual Systems X.PP XWe believe that C News runs fine on 16-bit machines, but it hasn't been Xtested very thoroughly on them lately. XIt will not perform quite as well with the more limited space. X.PP XMachines with very old compilers can be a headache. XThere are some hooks in X.Fn \& h/news.h Xfor doing without ``void'' and ``unsigned long'', Xtwo particular problem areas, but they have to be arranged manually; X.Fn \& build Xdoes not know about them. X.PP XSome very old systems cannot do \fIsetuid(geteuid())\fR, which makes it Ximpossible for a daemon to make directories and get the ownership right. XWe provide a small program, X.Fn \& setnewsids , Xto run setuid-root. X.Fn \& Relaynews Xknows about it and invokes it if \fIsetuid(geteuid())\fR fails; Xit then sets permissions correctly and re-invokes X.Fn \& relaynews . XThe code is short enough to be read and understood in full, so that the Xsuspicious system administrator can be sure that this setuid-root program Xis not going to do something awful. ! echo 'doc/README': sed 's/^X//' >'doc/README' <<'!' XThis is user documentation. See ../notebook for implementor's notes. X XEverything here is set up to use the -ms macros. No complex formatting Xis attempted, and only the three standard fonts (R, I, B) are used. X Xinstall somewhat sketchy how-to-install document Xinstall.out nroffed version of install Xinterface technical summary of how C News works and its interface to X the rest of the system ! echo 'doc/interface': sed 's/^X//' >'doc/interface' <<'!' X.DA "9 June 1989" X.TL XThe Interface Between C News And The Outside World X.AU XHenry Spencer X.AI XDept. of Zoology XUniversity of Toronto X.SH XIntro and Generalities X.PP XC News relates to the ``outside world'', Xthe system it is installed on, in a number of ways. XThis document attempts to enumerate them and explain what goes on. X.PP XIn general, C News attempts to rely only on ``common basic Unix'', Xand in particular it is not particularly BSD-specific or System-V-specific. XSpecifically, Xit makes no use of ornate locking mechanisms, silly interprocess-communication Xschemes, peculiar networking primitives, Xextensions to C, or other annoyances. X.SH XDirectories And Userids X.PP XMost of the components of C News live in \fI/usr/lib/newsbin\fR, Xwith control files in \fI/usr/lib/news\fR and spooling areas X(for current news, inbound traffic, and outbound traffic) in X\fI/usr/spool/news\fR. XSee the document X\fIDirectory Layout and PATH in C News\fR Xfor elaboration on the structure, X\fIFiles in /usr/lib/news (aka NEWSCTL)\fR for a summary of control files, Xand X\fIConfiguration Mechanisms in C News\fR Xfor how to change the locations of the directories. X.PP XThere is an extensive subdirectory structure under \fI/usr/lib/newsbin\fR, Xwith only a few heavily-used utilities in the top-level directory. XIn the following, Xprograms not explicitly described as going elsewhere are all under there Xsomewhere. X.PP XC News's spool areas and major control files need to be owned by a Xspecific userid, normally \fInews\fR. X(We believe that nothing except some of the Makefiles knows this name.) XWe suggest that this userid should not own anything else on the system. XThe programs in \fI/usr/lib/newsbin\fR can be Xowned by \fIbin\fR (or anyone else) except for those that need to be setuid. X(On our systems the non-setuid ones are owned by \fIbin\fR.) X.SH XUnix Utilities X.PP XC News requires a fairly complete Unix or equivalent. X(We take no position on whether 4BSD, or System V, is Unix or not; Xour private opinion is that neither truly deserves the name any more, Xalthough we occasionally change our minds about which is less deserving.) X(Note also that ``Unix'' is used here as an abbreviation for X\fBThe UNIX Operating System\fR\(rg, Xa registered trademark of AT&T; Xwe acquired our bad habits and incorrect capitalization from XUnix [sic] documentation supplied by the Bell System in the mid-1970s.) X.PP XIn particular, C News relies heavily on shell files. X``Shell'' here means, of course, the standard shell, written by XSteve Bourne. XIf your \fI/bin/sh\fR is not a Bourne shell or \fIvery\fR good imitation, Xyou're in trouble. XNote, in particular, that some old versions of the Korn shell differ from Xthe Bourne shell in some important details of quoting behavior, and Xwe \fIknow\fR this causes problems with C News. XYou need a standard shell. X.PP XTo the best of our ability, all our shell files begin with ``#!\ /bin/sh''. XIf your shell misinterprets this as a request to run the C shell, you're Xin trouble. XIf your shell doesn't know about ``#'' comments at all, you're in trouble. XWe hope that no modern shell makes either of these mistakes. X.PP XWe believe that none of our stuff relies on relatively modern shell features Xlike shell functions or \fIgetopts\fR (as distinct from \fIgetopt\fR). XIf \fItest\fR and \fIecho\fR are not built-in commands in your shell, Xefficiency will suffer Xbut everything should still run. XIt's possible that a few obscure things will break if your shell does not Xsupport ``['' as a synonym for \fItest\fR, although we try to avoid this usage. X.PP XWithin the shell files, C News makes heavy use of a wide variety of Unix Xutilities, notably \fIsed\fR and \fIawk\fR. XAny \fIawk\fR should do; in particular, nothing needs the ``new \fIawk\fR'' X(we don't run it yet). XAlthough we have tried to avoid it, it is possible that some things depend Xon \fIawk\fR recognizing ``\et'' inside strings, which very old \fIawk\fRs Xdidn't. X.PP XIf your Unix does not put all standard Unix programs in Xthe standard directories\(em\fI/bin\fR and X\fI/usr/bin\fR\(emyou will need to modify C News's default PATH to include Xwhatever bizarre directories are involved. XSee the \fIConfiguration Mechanisms\fR document for details. XIn particular, for some insane reason, 4BSD relocated a few standard Xutilities like \fIwc\fR to \fI/usr/ucb\fR, which causes trouble X(and not just to C News!). XWe simply put some links in \fI/usr/bin\fR to fix this on our systems, Xand this is what we recommend you do, but if you can't, you'll have to Xchange the PATH. X.PP XSeveral parts of C News rely on being able to send mail to an administrative Xaccount (the name of which can be chosen; see \fIConfiguration Mechanisms\fR). XThe assumption is that a message, without any RFC822 headers, can be piped Xinto \fI/bin/mail\fR (or whatever \fImail\fR is first in the PATH) Xinvoked with a single argument which is the addressee, Xand be delivered to the addressee's mailbox intact, possibly embellished Xwith headers. XThat is, with news's standard PATH, if X.DS Xecho hi there Joe | mail joe X.DE Xdoes not result in the message ``hi there Joe'' arriving in the mailbox X``joe'', you're going to have to fake it. XSee \fIDirectory Layout\fR for insight on where to put a fake \fImail\fR Xso that C News components will use it rather than \fI/bin/mail\fR. X.PP XSimilarly, several parts of C News rely on being able to invoke \fIhostname\fR Xwithout arguments, and have it return a single word which is the name of the XCPU the code is running on. XAgain, you'll have to fake it if you don't have \fIhostname\fR Xor if it outputs something strange. X.PP XInput reception and output batching both want to use the \fIcompress\fR Xdata-compression program. X\fICompress\fR has been published on Usenet and is shipped with many Unix Xsystems these days, but if you don't have it, we recommend that you get Xit from your neighbors or the \fIcomp.sources.unix\fR archives. XIt is possible to configure C News so that it doesn't use \fIcompress\fR Xto compress news being transmitted, but you don't want to. X\fICompress\fR is good and it is fast. X.SH XLibraries X.PP XC News is mostly self-contained as far as libraries go. XIt does need some things that are not yet universal, like a full set Xof string functions (e.g. \fIstrtok\fR); Xwe provide reasonably portable versions of these for places that lack them. X.PP XOne thing that C News needs, because it is both useful Xand a user-visible part of B News, Xis the \fIdbm\fR library. XAT&T has stupidly omitted it from System V. XWe include an emulation or two that are claimed to work reasonably well. XIf your system doesn't have a real \fIdbm\fR, however, our recommendation Xis that you harass your supplier about it. X.SH XNetworking X.PP X(We're talking here about networking in the sense of NFS and all those fun Xthings, not \fIuucp\fR or TCP/IP.) XC News is designed to run on systems which consist of a conglomeration of Xmachines on a local network, with only one of them acting as news server. XBe warned, though, that if you're doing this, you need to have \fIrsh\fR Xor some reasonably equivalent way of executing a program on another CPU. XIf you've only got one machine, Xjust make sure you \fIdon't\fR have a \fI/usr/lib/news/server\fR file and Xforget the whole issue. X.SH XHighly System-Specific Things X.PP XThere are two small utilities within C News that are inevitably highly Xsystem-specific and have a high probability of needing changes to match Xyour system. XBoth normally live in \fI/usr/lib/newsbin\fR proper. X.PP X\fISpacefor\fR is invoked by various components of C News to find out Xhow much disk space is available. XThe space margins in it are ones we find reasonable, but you may need Xto adjust them. XOn an old System V system in particular (one where \fIdf\fR can't be applied Xto any directory name, but needs to be given a name in \fI/dev\fR), you Xwill also probably need to modify the locations to be \fIdf\fRed. XYou will also need to fix \fIspacefor\fR if your system's \fIdf\fR yields Xresults in some strange format or in some strange units. XWe believe that we get it right for stock 4BSD, Xmany (but probably not all) System Vs, modern Irix, and real Unix (Version 7). XBeware introducing major inefficiencies: X\fIspacefor\fR is called a lot. X(Our current one could stand to be faster, in fact.) XBeware, also, that \fIspacefor\fR is not intended to permit routine operation Xusing filesystems that are on the brink of being full; Xthe limits it imposes are only approximate, and no attempt has been made Xto tune its clients to exploit every last available block. X.PP X\fIQueuelen\fR is invoked by the batcher to determine how long the current X\fIuucp\fR queues are, so it can judge whether it should suspend batching Xof output to a given site. XThere is too much diversity in \fIuucp\fRs for us to try to do this for all Xpossible versions. XWe think we get it right for the two most common flavors X(HDB, aka BNU, and old subdirectory versions). XOur versions measure queue length in batches, not bytes, Xbut it would not be hard to change this: X\fIqueuelen\fR, the \fIbatchparms\fR control file, and the \fIsendbatches\fR Xhow-many-batches-should-I-try-to-add logic need to agree on the units of Xmeasurement, but (we think) nothing else cares. X.PP XIt is just possible that you might have to modify \fInewshostname\fR, Xwhich also lives in \fI/usr/lib/newsbin\fR itself. X\fINewshostname\fR delivers the name which should be applied to the Xwhole system (not just the particular CPU) for news purposes. XIt tries to be fairly clever about different ways of getting the name, Xand in particular will take it out of \fI/usr/lib/news/whoami\fR if Xthat file exists, but if you're doing something odd on a strange system, Xchanges may be needed. X.SH XInput X.PP XInput from the net via \fIuucp\fR (or equivalent) shows up as batches of Xnews to be fed to \fIrnews\fR or its obsolete synonym \fIcunbatch\fR. XThese two programs normally live in \fI/bin\fR, although nothing in the Xcode knows about this and they can go elsewhere (one of our systems does this). XBoth are simple front ends that invoke \fIinput/newsspool\fR to store the Xbatch, while taking precautions to report trouble and not to overflow disks. XNeither \fIrnews\fR nor \fIcunbatch\fR needs to be setuid. X.PP XInput via NNTP over the Internet (or equivalent) uses rather different Xmachinery but ends up creating a saved batch in much the same way as X\fIinput/newsspool\fR does. X.PP X\fIInput/newsspool\fR is a small C program that saves a batch, Xwriting into a file in \fI/usr/spool/news/in.coming\fR. XIt must be able to create files there, and \fIinput/newsrun\fR (see below) Xneeds to be able to read them and delete them. XThis gets a little tricky because \fInewsspool\fR will usually be Xrun by \fIuuxqt\fR as userid \fIuucp\fR (or something like that), Xnot as \fInews\fR, which \fInewsrun\fR needs to run as. XThe recommended solution is to have X\fInewsspool\fR owned by \fInews\fR and setuid. XAn alternative is to give the \fIin.coming\fR directory Xthe userid of \fInews\fR and the groupid of \fIuucp\fR, or vice versa, Xand set permissions so that either can access it. XOne of our systems ran that way for a while. X.PP XTo actually process incoming news, \fIinput/newsrun\fR gets invoked Xto decompress the spooled batches and Xfeed them to \fIrelay/relaynews\fR (see below). XThere is an option for \fInewsspool\fR to invoke X\fInewsrun\fR when a batch is spooled, Xbut a (usually) Xpreferable method is to have \fIcron\fR invoke \fInewsrun\fR once an hour. X\fINewsrun\fR does its own locking to prevent multiple occurrences running Xsimultaneously. XThere is a related program, \fIinput/newsrunning\fR, that can be used Xto set or clear a flag that stops \fInewsrun\fR; Xthis may be a useful tactic if \fInewsrun\fR should not run at Xcertain times. XBoth \fInewsrun\fR and \fInewsrunning\fR must be run as \fInews\fR. X.PP XWhen a user posts news, he (or his news reader) does it by feeding the Xarticle to \fI/bin/inews\fR. XIn C News, \fIinews\fR is a complex shell file that attends to preliminaries Xand then invokes \fIrelay/relaynews\fR. X\fIInews\fR does not need to be setuid (indeed, we make no use of setuid Xshell files at all, since they are grossly insecure). X\fIRelaynews\fR is the heart of C News, the program that actually pulls Xbatches apart and places articles into the database. XIf users are to be able to post news, \fIrelaynews\fR should be owned Xby \fInews\fR and setuid. X.SH XNews Readers X.PP XC News is fully compatible with B News to any news-reader program that Xdoes not inspect the middle field of \fI/usr/lib/news/history\fR too closely. XStandard B News news readers work fine. XWe supply a simple news reader X(written by, and included with permission of, Michael Rourke) Xas a naive-user replacement for the B News X\fIreadnews\fR. XMore complex programs are preferable for serious news enthusiasts. XWe recommend Larry Wall's \fIrn\fR X(which we use, unmodified), Xbut there are others. X.SH XOutput X.PP X\fIRelay/relaynews\fR normally queues up news for transmission to other Xsystems by appending article names and sizes to batch files Xin subdirectories under \fI/usr/spool/news/out.going\fR. XThese are then processed by \fIbatch/sendbatches\fR, which should be run Xregularly, as \fInews\fR, by \fIcron\fR. X\fISendbatches\fR can be configured to use a variety of transmission Xmechanisms, the usual being \fIuux\fR. X.SH XExpiry And Related X.PP XNews articles are removed, possibly with archiving to an archive area, Xby the expiry subsystem. X\fIExpire/doexpire\fR should be invoked now and then, Xas \fInews\fR, by \fIcron\fR. XWe suggest nightly. X\fIDoexpire\fR actually invokes \fIexpire/expire\fR to do the dirty work. X.PP XC News \fIexpire\fR does not have an option to rebuild the X\fI/usr/lib/news/history\fR file from scratch, Xsince that has nothing to do with expiry. XTo rebuild \fIhistory\fR, Xe.g. if it has been destroyed, Xuse \fIexpire/mkhistory\fR. X.PP XSome news readers need to have the third field of \fI/usr/lib/news/active\fR Xupdated occasionally to show the lowest article number still present in each Xnewsgroup. XFrankly, we think such news readers simply need to be fixed. XC News \fIexpire\fR does not do this updating. XFor those who use such news readers, however, \fIexpire/upact\fR will do Xsuch an update. XIt should be run as \fInews\fR. X.PP X\fIRelay/relaynews\fR Xdoes not implement the ``Supersedes:'' header, which we loathe Xas a crude solution to the wrong problem. XAlas, the network-map distribution in X\fIcomp.mail.maps\fR relies heavily on it. XOur preferred approach is to process map articles as they arrive and Xthen expire them normally (using \fIexpire\fR's expiry-date-override Xfeatures to insist that they expire promptly). XHowever, for those who don't do this, and don't want to have megabytes Xof obsolete map data piling up, X\fIexpire/superkludge\fR will remove superseded files in specified Xnewsgroups. XIt should be run as \fInews\fR by \fIcron\fR, perhaps nightly. X.SH XReboots and Administration X.PP XIf the system crashes, things like locks must be cleaned out if C News Xis to function properly after reboot. X\fI/etc/rc\fR, or equivalent, should run \fImaint/newsboot\fR during reboot, Xas \fInews\fR. X.PP XCertain log files can grow without bounds if not renamed/removed now and Xthen. XWe recommend running \fImaint/newsdaily\fR once a day. XIt tends to logs, keeping a generation or two for use in trouble tracking, Xand also sends mail to the news administrator in the event that something Xfunny has happened. X.PP XIn general, C News does not attempt to break locks, Xon the philosophy that a stale lock may mean something is badly wrong. X(See X\fILocking in C News\fR for details on locking methods.) XThe various programs will either give up, to return later, or wait Xpatiently for the lock to go away. XIf one doesn't keep an eye on things, a problem of some kind can hang up Xthe news system for quite a while. XRunning \fImaint/newswatch\fR once in a while\(emwe recommend a few times Xa day\(emwill alert administrators to signs of trouble. XExcept on grossly slow systems, C News locks should never hang around for Xany great length of time. ! echo 'doc/Makefile': sed 's/^X//' >'doc/Makefile' <<'!' Xinstall.out: install X nroff -Tdumb -ms install >$@ ! echo done -- Please send comp.sources.unix-related mail to rsalz@uunet.uu.net. Use a domain-based address or give alternate paths, or you may lose out.