red@redpoll.uucp (Richard E. Depew) (07/05/90)
On the chance that this may be helpful to someone trying to set
up their first ihave-sendme link, I offer the following notes collected
to help our news neighbors do the same. Critiques and suggestions are
welcome.
Dick Depew
red@redpoll.uucp
::::::::::::::
ihave.notes
::::::::::::::
ihave.notes - Version 1.3
red@redpoll.uucp (Richard E. Depew)
Wed Jul 4 10:29:00 EDT 1990
Purpose: a how-to guide for setting up an ihave-sendme link in cnews,
distilling the experience from four different sites.
Contributers: jb@aablue.com (John B. Scalia), macy@fmsystm (Macy M.
Hallock, Jr.), sam@uhura.neoucom.edu (Scott A. Mason), and
wfd@uhura.neoucom.edu (William F. Dorsey).
Application: ihave-sendme is useful if you have more than one newsfeed.
With it, you only get articles that you don't already have, rather than
transporting and then discarding lots of duplicates. The price is a delay
in receiving the articles, and increased traffic in control messages
between sites. The ihave-sendme protocol is easier to set up than it
first seems.
Example: a two way ihave-sendme link with redpoll for newsgroup rec.birds.
These instructions have been tested with cnews at a latest-patchlevels of
12-Mar-1990, 16-Apr-1900 and 25-May-1990.
Step 1. Create the following directories with full news ownership and
permissions, i.e. something like: "drwxrwxr-x 2 cnews news".
/usr/spool/news/out.going/redpoll.ihave
/usr/spool/news/out.going/redpoll.sendme
/usr/spool/news/out.going/redpoll
Step 2. Create newsgroups "control" and "to.redpoll", if you don't already
have them.
( as cnews/news, "/usr/lib/newsbin/maint/addgroup control y", etc.)
Step 3. Add the following lines to your /usr/lib/news/sys file,
following the "louie" model that is already in there. If you
have other groups or distributions that you don't want to "leak"
out, be sure to exclude them in the first line. If you have
recently switched from bnews, note that cnews distinguishes
between newsgroups (that are in the news directory tree), and
distributions (generally geographic limits). "comp" is a set
of newsgroups, "ihave" and "local" (probably) are distributions.
The distributions are included or excluded following the slash
("/") in the ":newsgroups/distributions:" field of a sys line.
=-=-=-=-=
# Send ihave telling redpoll what we have -- batcher turns the batch into a
# giant control message and posts it to "to.redpoll". (#1)
redpoll:rec.birds,!to/all,!sendme,!ihave,!local:I:redpoll.ihave/togo
# Send sendme in response to ihave from redpoll -- again, turned by batcher
# into giant control message posted to "to.redpoll". (#3)
redpoll-send-ids:to.redpoll/ihave:I:redpoll.sendme/togo
# Transmit said giant control messages by normal batching. (#2,#4)
redpoll-ctl:to.redpoll/all,!sendme,!ihave:f:redpoll/togo
# Send articles in response to sendme messages from redpoll. (#5)
redpoll-real:to.redpoll/sendme:f:redpoll/togo
=-=-=-=-=
Step 4. Add the following lines to your /usr/lib/news/batchparms
file, in this order:
=-=-=-=-=
redpoll.ihave 1000000 20 batchih nocomp viainews
redpoll.sendme 1000000 20 batchsm nocomp viainews
redpoll 100000 20 batcher compcun viauux
=-=-=-=-=
(Note 1: the size of the "ihave" and "sendme" batches is
increased by a factor of 10 over the default because sendbatches
thinks each "article" is 3000 characters, while it is really
only the article ID of about 60 characters. The default leads
to a bunch of small "ihave" or "sendme" control messages that
contain a maximum of only 33 message IDs. If you have enough
disk space (redpoll doesn't) you could increase this to 5000000.
The hitch here is that "spacefor" thinks these are really
huge articles and may refuse to batch them unless you have a lot
of free space on your disk. You might think of this as a safety
feature... you can't offer or request news unless you have enough
space to batch or unbatch it. I expect this may be altered in a
future cnews patch... there has been some discussion of
ihave-sendme alterations in news.software.b. If this does get
revised, it will probably be necessary to reduce the "size" back
to the original default.)
(Note 2: The site order in batchparms makes a difference.
Batching will be in the order given here if not overridden in
the sendbatches command. This is important because batching the
"ihave" and "sendme" "fake" sites before the "real" site can
save a whole sendbatches cycle which is usually an hour.)
Step 5. Make sure that the crontab entry for sendbatches will run the
new "sites" redpoll.ihave and redpoll.sendme. This is not a
problem if you run sendbatches without explicit filenames,
but will be a problem if you are trying to limit batching for
some sites to particular times.
In the latter case you need to explicitly name the whole list
of "sites" that are in /usr/spool/news/to.go, and you will need
at least two "sendbatches" commands. Multiple "sendbatches" must
be run at different times, otherwise they will try to lock each
other out.
For example, to send out ihave batches every two hours at night
(if you have received new articles) on weekdays, and all day Sat
and Sun, without disrupting other feeds (such as foo and bar), you
might put something like this in crontab (the lines have been split
for ease of reading):
=-=-=-=-=
# batch ihaves for redpoll every two hours, 6 pm to 6 am on weekdays
10 0,2,4,6,18,20,22 * * 1-5 su cnews -c \
'/usr/lib/newsbin/batch/sendbatches redpoll.ihave'
# batch ihaves for redpoll every two hours, all day on weekends
10 0,2,4,6,8,10,12,14,16,18,20,22 * * 0,6 su cnews -c \
'/usr/lib/newsbin/batch/sendbatches redpoll.ihave'
# batch everything else every hour, every day
15 * * * 0-6 su cnews -c '/usr/lib/newsbin/batch/sendbatches \
redpoll.sendme redpoll foo.ihave foo.sendme foo bar'
=-=-=-=-=
(note: as in the batchparms file, order is important... make
sure that the "ihave" and "sendme" sites are batched before
the "real" site... otherwise the control messages will have to
wait for another sendbatches cycle before they are submitted.)
Step 6. Make sure that you expire the "control" newsgroup after a
short time... the control messages accumulate quickly.
Use something like the following in /usr/lib/news/explist:
=-=-=-=-=
# real noise gets thrown away fast
junk,control x 2 -
=-=-=-=-=
Step 7. Check whether the correct "inews" is found by "viainews".
It probably is, but one of our test sites had two different
versions of "inews" in two different places, and the "wrong"
one was found first. If this is a problem, change the search
path in /usr/lib/news/bin/config, or move "inews", or hard-code
its location into the /usr/lib/newsbin/batch/viainews script.
If you do the latter, be warned that your fix will probably be
overwritten the next time you patch cnews, so save a copy in a
safe place.
Optional Step 8. /usr/lib/newsbin/queuelen doesn't work for the "fake"
systems redpoll.ihave and redpoll.sendme, so here is a modified
queuelen posted by Mike Murphy that does work for System V HDB
(if you are running Xenix, you will need to remove the space from
between the -d flag and the "." delimiter). Once again, save a
copy in a safe place.
=-=-=-=-=-=-=-=
Newsgroups: news.software.b
Subject: ihave/sendme problems and fixes
Message-ID: <1990Jun28.222640.12674@sceard.Sceard.COM>
Date: 28 Jun 90 22:26:40 GMT
>From: mrm@sceard.Sceard.COM (M.R.Murphy)
Reply-To: mrm@Sceard.COM (M.R.Murphy)
Organization: Sceard Systems, Inc., San Marcos, CA 92069
Lines: 88
... (removed elegant source code fix for ihave-sendme size problem)...
Also, ${NEWSBIN}/queuelen doesn't do what I expected for ihave/sendme.
queuelen always returns zero for ihave/sendme , since neither the directory for
somesystem.ihave nor somesystem.sendme is in the /usr/spool/uucp tree. The
directory for somesystem would be. I modified queuelen to be the following
(System V(tm) HDB flavo(u)red, sorry, other environments left as an exercise
for the interested reader :-)
-----
#! /bin/sh
# Find size of current queue of news outbound to $1. HDB/BNU version.
# =()<. ${NEWSCONFIG-@<NEWSCONFIG>@}>()=
. ${NEWSCONFIG-/usr/lib/news/bin/config}
PATH=$NEWSCTL/bin:$NEWSBIN:$NEWSPATH ; export PATH
umask $NEWSUMASK
cd /usr/spool/uucp
sys=`echo $1|cut -d "." -f1`
if test -d $sys
then
cd $sys
grep "news" C.* 2>/dev/null|grep -v X|wc -l
else
echo 0
fi
-----
--
Mike Murphy Sceard Systems, Inc. 544 South Pacific St. San Marcos, CA 92069
mrm@Sceard.COM {hp-sdd,nosc,ucsd,uunet}!sceard!mrm +1 619 471 0655
--
=-=-=-=-=-=-=-=
References: The best description of how this protocol is supposed to work
is found in the cnews source directory in notebook/ihave. If you
can't print the flowchart that is there, ihave.pic, perhaps this
ascii version will be helpful:
::::::::::::::
ihave.diagram
::::::::::::::
ihave/sendme processing diagram
utzoo utstat
match #1
article->rnews
|
v
batcher on
msgid file
|
(deadlock | avoidance) +------------------+
| | ihave processing |
v +------------------+
ihave control match #3 I
match #2 F or -----+-----------> rnews to.utzoo/ihave
inews to.utstat / |
| / |
| / |
v / |
batcher on ---+ v
filenames (opt.) batcher on
msgid file
|
+-------------------+ (deadlock | avoidance)
| sendme processing | |
+-------------------+ v
match #5 F sendme control
<-----------+--------- match #4
rnews to.utstat/sendme \ inews to.utzoo
\ |
| \ |
| \ v
| \ batcher on
| +--- filenames (opt.)
v
batcher on articles
filenames -------------------------------->
=-=-=-=-=-=-=-=
Disclaimer: These notes reflect the steps needed to set up ihave-sendme
links under four different operating systems. However, it is
likely that there are other hitches that we've not run into or
possibly just not noticed yet. Any errors in these instructions
are mine. Use them at your own risk.
Please let me know of any corrections or improvements that you may
have for these notes, based on your experience. Experience IS the
best teacher.
--
Richard E. Depew, Village of Munroe Falls, OH. red@redpoll.uucp
uunet!aablue!redpoll!red redpoll!red@uhura.neoucom.edu