bd@hpda.UUCP (Bob Desinger) (01/11/86)
I recently installed 2.10.3 news on hpda; Greg Fowler of hplabs showed
me how to set up a compressed and batched newsfeed to his system and
back. The black magic of that afternoon was impressive, and I wanted
to share the knowledge. So I wrote down these (hopefully easy to
follow) steps detailing how to transmit and receive compressed news
batches.
The people who tested these instructions were also using 2.10.3beta, so
this cookbook applies to that version. I believe it should also work
on 2.10.2, but in honesty I haven't brought up 2.10.2 news. (Perhaps
some 2.10.2 wizard will post mods to this document; hope springs
eternal.)
Not long after setting up news, I decided to expire some of the
newsgroups sooner than the default in order to save disk space. This
10-minute task took an afternoon because of the gotchas. So one of the
files below explains how to expire some groups sooner. You can tell
which groups are the biggest hogs with the commands:
cd /usr/spool/news # or wherever you keep your articles
du | sort -rn # show biggest directories first
Hopefully this will save netnews administrators some time and grief.
Rick Adams has this shar, so perhaps these documents (or forms of them)
will appear in the 2.10.3 documentation.
Bob Desinger ihnp4!ucbvax!hpda!bd hpda!bd@BERKELEY.EDU
# This is a shell archive. Remove anything before this line,
# then unpack it by saving it in a file and typing "sh file".
#
# Wrapped by hpda!bd on Fri Jan 10 16:44:20 PST 1986
# Contents: README batch.lpr batch.ms expire-sooner
echo x - README
sed 's/^@//' > "README" <<'@//E*O*F README//'
Three files are included below:
1. batch.lpr The post-nroff cookbook, suitable for feeding to lpr (or
lp, depending on your Unix version) directly.
2. batch.ms The pre-nroff cookbook; process via `nroff -ms batch.ms'.
3. expire-sooner A plain file describing how to expire some of the
less-useful newsgroups sooner than normal.
@//E*O*F README//
chmod u=rw,g=r,o=r README
echo x - batch.lpr
sed 's/^@//' > "batch.lpr" <<'@//E*O*F batch.lpr//'
_0._1. _C_o_o_k_b_o_o_k: _S_e_t_t_i_n_g _u_p _a _C_o_m_p_r_e_s_s_e_d, _B_a_t_c_h_e_d _N_e_w_s_f_e_e_d
First, BATCH must have been #define'd when you built
the news system. To check, look in the file defs.h in the
news source directory. BATCH should be defined there as a
program name (by default, "unbatch"). If it's undefined or
commented out, define it, re-make the news system, and
install the new software.
You'll also need a working _c_o_m_p_r_e_s_s program. Use the
one shipped with this news distribution, which is based on
version 3.0; or use compress 4.0, which was posted to
mod.sources a while back. Your news neighbors should be
running a compatible version of compress. Versions 3.0 and
4.0 are compatible with each other, but both are incompati-
ble with versions 2.0 and before.
Update your _s_y_s file. First, add the F flag to the
other news system's line. For instance, if your
compressed-and-batched news feed is named frobozz, and its
_s_y_s file entry looks like:
frobozz:net,mod,na,usa,ca,to.frobozz::
then add the F flag as the third (colon-separated) field:
frobozz:net,mod,na,usa,ca,to.frobozz:F:
Now the pathnames of articles to be sent will be stashed in
a file. This file is named in the fourth field of the _s_y_s
entry; add it now. Use an entry of the form
_B_A_T_C_H_D_I_R/_s_y_s_t_e_m, where _B_A_T_C_H_D_I_R is usually /usr/spool/batch
(the actual value is defined in the news Makefile), and _s_y_s_-
_t_e_m is the name of the remote system, in this example fro-
bozz. A name of that form is necessary: the _c_s_e_n_d_b_a_t_c_h
script, which compresses and sends the batched news, looks
for a file name of this form to decide if there's news for
the remote system.
Your completed _s_y_s file line should look something
like:
frobozz:net,mod,na,usa,ca,to.frobozz:F:/usr/spool/batch/frobozz
In /usr/lib/crontab, find or create at least two news
lines: one that runs nightly, and one that runs every hour
or so. The nightly-run script should run _e_x_p_i_r_e, trim log
files, and perhaps compile weekly statistics that you post
to a local-area newsgroup one day a week. The hourly-run
script should complete the transmitting task with a line
like:
csendbatch frobozz
January 3, 1986
- 2 -
Make sure the script knows how to get to the directory in
which _c_s_e_n_d_b_a_t_c_h lives. You can either mention the direc-
tory in the script's PATH-setting line, or replace "csend-
batch" with its full pathname. _C_s_e_n_d_b_a_t_c_h reads the files
mentioned in /usr/spool/batch/frobozz, batches them,
compresses them, sends them to the remote system, and
arranges for remote processing.
This remote processing is directed by another file in
_B_A_T_C_H_D_I_R. Make a file with a name of the form
_B_A_T_C_H_D_I_R/_s_y_s_t_e_m.cmd (for this example,
/usr/spool/batch/frobozz.cmd). Put a line in it specifying
the command that the remote system should execute to unpack
the news batches that your system will send. Sample con-
tents for frobozz.cmd are:
uux - -r -z -n -gd frobozz!cunbatch
Now your system will transmit compressed batches. The
receiving side of the business is handled largely by a pro-
gram called _c_u_n_b_a_t_c_h.
Make sure there is an executable file called _c_u_n_b_a_t_c_h
in the BINDIR directory (check the Makefile for its actual
location). It must be reachable by uucp or by whatever
transport you'll use to transfer the netnews. If you
defined BINDIR as /usr/bin, you should have no problems
because uuxqt can already get there. If you defined it as a
different directory, you may have to teach uuxqt to look in
that directory; accomplishing this varies from system to
system. On 4.2BSD, add the directory to the "PATH=" line of
your uucp L.cmds file. On System V, on the _c_u_n_b_a_t_c_h line of
your L.cmds file, add a comma followed by the remote
system's name on that line. If yours is in
/usr/bin/news/cunbatch, your L.cmds file will look like:
[For 4.2BSD]
PATH=/bin:/usr/bin:/usr/bin/news
cunbatch
[For System V]
/usr/bin/news/cunbatch,frobozz
Other systems have a similar file in the /usr/lib/uucp
directory by which you can specify added programs and paths
different from the defaults. HP-UX, for example, has a
/usr/lib/uucp/COMMANDS file which expands uuxqt's horizons.
In more restrictive cases, paths are compiled into uuxqt.
If you can't modify any uucp files, just put _c_u_n_b_a_t_c_h in
/usr/bin.
You must also have a _c_u_n_b_a_t_c_h in _L_I_B_D_I_R (wherever your
Makefile defines it), because _r_n_e_w_s will eventually try to
January 3, 1986
- 3 -
exec that copy.
Tell the person at the other end of your newsfeed to
use _c_s_e_n_d_b_a_t_c_h and a uux command of _c_u_n_b_a_t_c_h to send you
news. Once that's in place, watch your uucp LOGFILE and
your news _l_o_g and _e_r_r_l_o_g files to ensure that news is being
correctly received and unpacked on your system.
January 3, 1986
@//E*O*F batch.lpr//
chmod u=rw,g=r,o=r batch.lpr
echo x - batch.ms
sed 's/^@//' > "batch.ms" <<'@//E*O*F batch.ms//'
@.if t .po 0.5i
@.if t .ll 7.5i
@.NH 2
Cookbook: Setting up a Compressed, Batched Newsfeed
@.PP
First, BATCH must have been #define'd when you built the news system.
To check, look in the file
defs.h
in the news source directory.
BATCH should be defined there as a program name (by default,
\*Qunbatch\*U).
If it's undefined or commented out,
define it, re-make the news system, and install the new software.
@.PP
You'll also need a working
@.B compress
program.
Use the one shipped with this news distribution,
which is based on version 3.0;
or use compress 4.0, which was posted to mod.sources a while back.
Your news neighbors should be running a compatible version of compress.
Versions 3.0 and 4.0 are compatible with each other,
but both are incompatible with versions 2.0 and before.
@.PP
Update your
@.B sys
file.
First, add the F flag to the other news system's line.
For instance, if your compressed-and-batched news feed is named frobozz,
and its
@.B sys
file entry looks like:
@.DS
frobozz:net,mod,na,usa,ca,to.frobozz::
@.DE
then add the F flag as the third (colon-separated) field:
@.DS
frobozz:net,mod,na,usa,ca,to.frobozz:F:
@.DE
Now the pathnames of articles to be sent will be stashed in a file.
This file is named in the fourth field of the
@.B sys
entry; add it now.
Use an entry of the form
@.I BATCHDIR/system ,
where
@.I BATCHDIR
is usually
/usr/spool/batch
(the actual value is defined in the news Makefile), and
@.I system
is the name of the remote system,
in this example
frobozz.
A name of that form is necessary: the
@.B csendbatch
script,
which compresses and sends the batched news,
looks for a file name of this form
to decide if there's news for the remote system.
@.PP
Your completed
@.B sys
file line should look something like:
@.DS
frobozz:net,mod,na,usa,ca,to.frobozz:F:/usr/spool/batch/frobozz
@.DE
@.PP
In /usr/lib/crontab, find or create at least two news lines:
one that runs nightly, and one that runs every hour or so.
The nightly-run script should run
@.B expire ,
trim log files,
and perhaps compile weekly statistics
that you post to a local-area newsgroup one day a week.
The hourly-run script should complete the transmitting task
with a line like:
@.DS
csendbatch frobozz
@.DE
Make sure the script knows how to get to the directory in which
@.B csendbatch
lives.
You can either mention the directory in the script's
PATH-setting line, or replace
\*Qcsendbatch\*U
with its full pathname.
@.B Csendbatch
reads the files mentioned in /usr/spool/batch/frobozz,
batches them, compresses them, sends them to the remote system,
and arranges for remote processing.
@.PP
This remote processing is directed by another file in
@.I BATCHDIR .
Make a file with a name of the form
@.I BATCHDIR/system .cmd
(for this example,
/usr/spool/batch/frobozz.cmd).
Put a line in it specifying the command that the remote system
should execute to unpack the news batches that your system will send.
Sample contents for
frobozz.cmd
are:
@.DS
uux - -r -z -n -gd frobozz!cunbatch
@.DE
@.PP
Now your system will transmit compressed batches.
The receiving side of the business is handled largely by a program called
@.B cunbatch .
@.PP
Make sure there is an executable file called
@.B cunbatch
in the
BINDIR
directory (check the Makefile for its actual location).
It must be reachable by uucp
or by whatever transport you'll use to transfer the netnews.
If you defined
BINDIR
as
/usr/bin,
you should have no problems because uuxqt can already get there.
If you defined it as a different directory,
you may have to teach uuxqt to look in that directory;
accomplishing this varies from system to system.
On 4.2BSD, add the directory to the
\*QPATH=\*U
line of your uucp
L.cmds
file.
On System V, on the
@.B cunbatch
line of your
L.cmds
file, add a comma followed by the remote system's name on that line.
If yours is in
/usr/bin/news/cunbatch,
your
L.cmds
file will look like:
@.DS
[For 4.2BSD]
PATH=/bin:/usr/bin:/usr/bin/news
cunbatch
@.sp
[For System V]
/usr/bin/news/cunbatch,frobozz
@.DE
Other systems have a similar file in the
/usr/lib/uucp
directory by which you can specify added programs
and paths different from the defaults.
HP-UX, for example, has a
/usr/lib/uucp/COMMANDS
file which expands uuxqt's horizons.
In more restrictive cases, paths are compiled into uuxqt.
If you can't modify any uucp files,
just put
@.B cunbatch
in
/usr/bin.
@.PP
You must also have a
@.B cunbatch
in
@.I LIBDIR
(wherever your Makefile defines it),
because
@.B rnews
will eventually try to exec that copy.
@.PP
Tell the person at the other end of your newsfeed
to use
@.B csendbatch
and a uux command of
@.B cunbatch
to send you news.
Once that's in place, watch your uucp LOGFILE and your news
@.B log
and
@.B errlog
files to ensure that news is being correctly received and unpacked
on your system.
@//E*O*F batch.ms//
chmod u=rw,g=r,o=r batch.ms
echo x - expire-sooner
sed 's/^@//' > "expire-sooner" <<'@//E*O*F expire-sooner//'
Some administrators prefer to squeeze every ounce of disk space
possible from their disk drives. Because the netnews takes up so much
space, non-technical but high-volume groups are sometimes expired more
rapidly than technical newsgroups. You can do this by forcing `expire'
to delete articles in certain newsgroups after N days, where N is
typically equal to 7.
As an example, the news expiration lines in the nightly news-cleanup
job run by cron are:
expire -v -e7 -n net.flame,net.jokes,net.politics,net.rec,
expire -v
The first line expires articles in the specified newsgroups after 7
days. The second line expires articles in all newsgroups after the
default period, which is 14 days.
Important things to note about running `expire' on a selected list of
newsgroups are:
1. After the "-n" flag, commas (not spaces) separate the newsgroups in
the list.
2. A comma after the last newsgroup in the list is absolutely necessary.
3. Newsgroups on the `expire' argument line specify a name-pattern for
newsgroups, and are not necessarily actual newsgroup names
themselves. Using the example on the above line, the `net.jokes'
pattern causes articles in all groups beginning with the title
"net.jokes" to be expired faster. Thus articles in net.jokes,
net.jokes.d, and other newsgroups beginning with "net.jokes" will
expire sooner. Likewise with the `net.rec' specifier: net.rec,
net.rec.bird, net.rec.nude, and other "net.rec*" newsgroups will
expire sooner.
@//E*O*F expire-sooner//
chmod u=rw,g=r,o=r expire-sooner
echo Inspecting for damage in transit...
temp=/tmp/shar$$; dtemp=/tmp/.shar$$
trap "rm -f $temp $dtemp; exit" 0 1 2 3 15
cat > $temp <<\!!!
9 50 338 README
198 756 5714 batch.lpr
184 796 4762 batch.ms
34 223 1515 expire-sooner
425 1825 12329 total
!!!
wc README batch.lpr batch.ms expire-sooner | sed 's=[^ ]*/==' | diff -b $temp - >$dtemp
if [ -s $dtemp ]
then echo "Ouch [diff of wc output]:" ; cat $dtemp
else echo "No problems found."
fi
exit 0david@ukma.UUCP (David Herron, NPR Lover) (01/14/86)
Everything is basically correct, except you don't need to do uux of cunbatch. If you look at the source code for inews (Is checkbatch() the right routine? or is the check delayed until it reaches unbatch?) you'll see that it checks for the "magic number" that compress puts in and if it's compressed sends the article through cunbatch. What that boils down to is that you use csendbatch to send the stuff but you don't need to use that system.cmd file as you said to do in your help file. Oh. I'm talking here about version 2.10.3 alpha 4/3/85. But I doubt anything changed before your version reached you. -- David Herron, cbosgd!ukma!david, david@UKMA.BITNET, soon, david@uky.csnet. Experience is something you don't get until just after you need it.
grr@cbm.UUCP (George Robbins) (01/18/86)
Having just set up news 2.10.3 with batched feeds and downstream sites, I would
like to make the following corrections/comments about Bob Designer's notes:
Bob recommends using 'uux...frobozz!cunbatch' to send the batched files, then
gives some tortured explanation of how to make this work. If you simply use
the default 'uux...frobozz!rnews', rnews(inews) on the receiving end sniffs at
the file, sees that it has to run cunbatch, which tries to run out of LIBDIR.
There are also some concerns about compress 3.0 applicable to smaller system.
As compiled by news, it assumes that you have a lot of memory available. On
a 68000 system, I did a 'size compress' and got about 700K. If you don't have
this kind of memory, or trying to use that much puts your system out to lunch,
you should recompile compress. Defining the symbol 'pdp11' creates a much
smaller program at the cost of some compression efficiency.
If you do this, sites that send you batched news must specify the -b 12 switch
to csendbatch or you will not be able to decode their news. Also, if you are
sending news to a 2.10.2 site that has not since installed compress 3.0 or 4.0,
you must use the -C option to generate batches that they can process.
When trying to set up expire, it's a good idea to do a (tar/dump/cpio) dump of
your LIBDIR and SPOOLDIR directories, since there's a good chance expire will
start blowing away articles you wanted to keep. In addition to the features
that Bob mentions, expire likes to delete cross-posted articles from all the
newsgroups they are posted to, even though you wished to preserve a particular
group.
Note that when trying to install news, you should try to work with someone
that has already installed 2.10.3. The installation instructions leave you
hanging about 2/3's of the way through the process, apparently since the
document wasn't completely updated when most of the 2.10.2 auto-installation
stuff was chopped out.
--
George Robbins - now working with, uucp: {ihnp4|seismo|caip}!cbm!grr
but no way officially representing arpa: cbm!grr@seismo.css.GOV
Commodore, Engineering Department fone: 215-431-9255 (only by moonlite)