[news.sysadmin] Keeping multiple news machines in sync

mcooper@oberon.USC.EDU (Michael A. Cooper) (01/17/88)

	[ We are running B news 2.11.14.  NNTP 1.3c is used to
	  transfer almost all news. ]

	I'd like to know if anyone has any ideas on keeping several
news hosts "sync'ed" up so as to allow users to read news on any
machine?

	Specifically, we have 2 seperate news hosts here at USC; our
gateway machine Oberon, and a secondary machine named Nunki.  All
incoming news arrives on Oberon.  Nunki gets all it's news from
Oberon.  There are several hundred machines NFS mounting
/usr/spool/news (the news "lib" directory is also under
/usr/spool/news) from these machines.  Users have the same home
directory on machines which get /usr/spool/news from Oberon or Nunki.

	Currently, a user must either choose to read news on hosts
that get /usr/spool/news from Oberon *or* Nunki.  They cannot just
read news on any old host.  What I'm after is some way of syncronizing
articles between Oberon and Nunki such to the affect that it doesn't
matter what machine /usr/spool/news is mounted from.  To complicate
matters, we'ed like to add 2 or 3 more news machines to ease the load
on the 2 current machines.

	There are several problems that I have thought of.  First is
that there is no way to force an article from news host A to be named
(numbered) the same on news host B.  The best you can do is to
initially copy everything from host A to host B.  Then pray everytime
you transfer an article that it will be numbered the same on host B.

	Another problem is propagation.  Even if we could be assured
that the articles would be named (numbered) appropriately, then you
still have problems in propagating the articles quickly.  Suppose
someone reads news from Oberon's /usr/spool/news.  Since Oberon gets
news almost constantly (NNTP and the ARPANET make a great pair!),
Nunki is always going to be behind.  A few minutes after the user
finishes reading news, he/she moves to a host who has Nunki's
/usr/spool/news mounted.  They read news and find nothing new.  There
.newsrc has now been "fixed" up.  The next time the user reads news
from Nunki's /usr/spool/news, they have to re-read all the articles
that came in to Oberon and weren't on Nunki when they read news on
Nunki.

	We currently NNTP everything from Oberon to Nunki.  I could
crank up the number of "polls" to Nunki to every 15 minutes or so.
This still leaves a large enough window for significant problems.  A
possible solution would be to crank up a nntp xmit everytime an
article for Nunki arrived on Oberon.  But there is just too much
overhead involved (Oberon's only a VAX 750).

	If anybody has any suggestions or experience with what I have
outlined, I would really like to hear from you.


-- 
Michael A. Cooper, University Computing Services, U of Southern California
  UUCP: {sdcrdcf, cit-vax}!oberon!mcooper   BITNET: mcooper@kylara
  ARPA: mcooper@oberon.USC.EDU              PHONE: (213) 743-2957

jerry@oliveb.olivetti.com (Jerry Aguirre) (01/19/88)

In article <6263@oberon.USC.EDU> mcooper@oberon.usc.edu (Michael A. Cooper) writes:
>	I'd like to know if anyone has any ideas on keeping several
>news hosts "sync'ed" up so as to allow users to read news on any
>machine?

To summarize; the problem is, of course,  that rnews/inews may assign
different numbers to articles when they arrive or are posted.  Even if
machine B's only feed is A it is difficult to guarantee that the
articles will be transmitted in the same sequence to B as they arrive on
A.  And then local postings on B would throw that out of sync.  Like
most things once sync is lost it will tend to get worse.  The result is
that a readers .newsrc file will not correctly list what articles have
been read if a different server is used.

Given that there is only one externally connected "master" system I can
think of one way to avoid the problem.  First is to have ALL systems
except the master (including the secondary servers) post to the master
using the nntp inews.  This avoids the problem of local postings
throwing the article numbers out of sync.

The second part of the sollution is not to use the news mechanisms to
send the articles to the secondary servers.  An obvious method would be
to use rdist to distribute the news files (including lib/active) to the
secondary servers.  The overhead of running rdist on such a large file
structure several times a day might be prohibitive.

Lacking rdist you could be very careful with your news transmission and
hope that the articles stay in sync.  Using nntp to transfer between the
systems should keep them in sync.  An occasional brute force copy from
the master to the secondaries would force them back into sync.

Another possiblity is to "find" all articles newer than the last
time, create a "tar" of those articles, copy it to the secondary, and
extract it there.  (Tar would preserve the link structure of cross
posted articles.)  The difficulty here is in handling the case where a
seconday server is down when the copy and extraction is run.

If I had to choose a method I would use find/tar during the day with an
rdist at night to handle dropped articles.  This would minimize the
overhead while staying in sync.

Having the news readers use the article numbers to keep track of what
has been read is the heart of the problem.  Unfortunately, keeping track
of article IDs consumes a lot more overhead.  Rnews has essectially the
same problem of keeping track of what it has "read" (actually received
already).  The history files using article IDs consume several
megabytes.  If everyone used consistant IDs like <nnn@site> then it
would be easy but many IDs are arbitrary strings.  For each user to
consume several megabytes just to keep track of what they have read is
probably not acceptable.

					Jerry Aguirre @ Olivetti ATC

bytebug@felix.UUCP (Roger L. Long) (01/19/88)

In article <6263@oberon.USC.EDU> Michael A. Cooper (mcooper) writes asking
how to keep multiple news machines in sync by article number.

In article <13176@oliveb.olivetti.com> Jerry Aguirre (jerry) writes:
>Given that there is only one externally connected "master" system I can
>think of one way to avoid the problem.  First is to have ALL systems
>except the master (including the secondary servers) post to the master
>using the nntp inews.  This avoids the problem of local postings
>throwing the article numbers out of sync.

Correct.  This also has the effect of "hiding" the internal network of
machines within an organization, since it really isn't important where
a user's home machine is, as long as mail gets to the gateway machine.
Here at FileNet, "felix" is the gateway, while all of the users reside
on "fritz".

Jerry goes on to suggest a couple of mechanisms to then distribute news
to other machines on the network.  The first, using rdist to distribute
news and lib/active, the second using find and tar.

>If I had to choose a method I would use find/tar during the day with an
>rdist at night to handle dropped articles.  This would minimize the
>overhead while staying in sync.

Jerry apparantly failed to note that Michael stated that oberon was a 750.
Felix, too, is a 750.  And after being spoiled by the speed of 780s and
785s, doing anything significant on a 750 seems to take forever.  Doing
a find on a news filesystem on a 750 while at the same time the 750 is
receiving news via NNTP and probably doing one or two UUCP transfers is
bound to be S-L-O-W.

Thus, an alternate suggestion, which I've been mulling about from time
to time here at FileNet:
- modify NNTP's ihave/sendme to use the article pathname instead of 
  Message-ID.

- modify inews (or integrate this functionality into the NNTP server) to
  do little more than extract the Xref line from the news header and use
  that to cross-link the article, if required, and update the lib/active
  file.  You'd also have to update the machine name in the Xref header
  when storing on the local machine, so that rn would recognize it.

This leaves the problem of updating the user's .newsrc files located on
various machines.  Since article numbers will be the same on all of the
machines, I'd suggest using rdist for this.

Jerry goes on to state:
>Having the news readers use the article numbers to keep track of what
>has been read is the heart of the problem.  Unfortunately, keeping track
>of article IDs consumes a lot more overhead.  Rnews has essectially the
>same problem of keeping track of what it has "read" (actually received
>already).  The history files using article IDs consume several
>megabytes.  If everyone used consistant IDs like <nnn@site> then it
>would be easy but many IDs are arbitrary strings.  For each user to
>consume several megabytes just to keep track of what they have read is
>probably not acceptable.

Well, not quite.  Using

	awk '{print $1}' </usr/lib/news/history | wc

I got 31643 article-ids totaling 716225 bytes.  Given that perhaps half
of those are already expired which wouldn't need to be stored, we could
assume a number of around a third of a megabyte (also assuming that the
reader had time to actually read all of the incoming news).  Perhaps a
better figure would be from the list of articles I've read that are still
available on the system.  I don't read a lot of news; it would appear that
I read 486 unique articles for a total of 10429 bytes of article-ids in
the last two weeks.  Not unreasonable, and in fact somewhat impressive,
given that my .newsrc file is nearly that large.  So maybe not as
unacceptable as you might think!
--
	Roger L. Long
	FileNet Corp
	{hplabs,trwrb}!felix!bytebug

jerry@oliveb.olivetti.com (Jerry Aguirre) (01/20/88)

In article <19044@felix.UUCP> bytebug@felix.UUCP (Roger L. Long) writes:
>Jerry apparantly failed to note that Michael stated that oberon was a 750.
>Felix, too, is a 750.  And after being spoiled by the speed of 780s and
>785s, doing anything significant on a 750 seems to take forever.  Doing
>a find on a news filesystem on a 750 while at the same time the 750 is
>receiving news via NNTP and probably doing one or two UUCP transfers is
>bound to be S-L-O-W.
Roger failed to note that oliveb is a 750...  I am intimately familiar
with the blazing speed of a 750 :-)  I just did a:

    "nice find /usr/spool/news -type f -newer filename -print"

and it took about 35 minutes (412 cpu seconds) to find 422 files.
(Searching 28 days of news, your mileage may vary.)  Of course tar is a
pig but it only has to deal with the new files, not the whole news
directory.

>Thus, an alternate suggestion, which I've been mulling about from time
>to time here at FileNet:
>- modify NNTP's ihave/sendme to use the article pathname instead of 
>  Message-ID.
I played with something like this.  I had the inews ihave/sendme send
both the article ID and pathname.  The advantage was that the receiver
could "uux !rnews sender!pathname" to fetch the article without the
delay and overhead of using "sendme".

>- modify inews (or integrate this functionality into the NNTP server) to
>  do little more than extract the Xref line from the news header and use
>  that to cross-link the article, if required, and update the lib/active
>  file.  You'd also have to update the machine name in the Xref header
>  when storing on the local machine, so that rn would recognize it.
A neat idea except articles that are not cross posted don't get an xref.
I news wouldn't be able to figure out the article number for those.  The
combination of getting the filename and checking for an xref line should
handle that though.

Perhaps even simpler would be to have a new option ("X") in the sys
file that worked like "F" but included all the cross posted file names,
not just the first link.  This would avoid the "find" because you could
just "tar" those files to the secondary server.

>Well, not quite.  Using
>
>	awk '{print $1}' </usr/lib/news/history | wc
>
>I got 31643 article-ids totaling 716225 bytes.  Given that perhaps half
>of those are already expired which wouldn't need to be stored, we could
>assume a number of around a third of a megabyte (also assuming that the
>reader had time to actually read all of the incoming news).  Perhaps a
>better figure would be from the list of articles I've read that are still
>available on the system.  I don't read a lot of news; it would appear that
>I read 486 unique articles for a total of 10429 bytes of article-ids in
>the last two weeks.  Not unreasonable, and in fact somewhat impressive,
>given that my .newsrc file is nearly that large.  So maybe not as
>unacceptable as you might think!

You are right, because one only has to keep track of what the user has
read this reduces the volume considerably.  (I probably have a larger
history file here because I expire at 28 days.)

However just keeping only the articles IDs is not enough.  First of all
you need random access to them so that as each article is scanned you
can decide whether to skip it or not.  If they were stored by newsgroup
then you could read just the IDs for that group into memory and hash on
them.  Otherwise you need the overhead of an access method.

What really changes is how one sequences through the articles.  If the
file numbers are not used then there is no information to tell where to
start or in what sequence to read.  Perhaps rnews could maintain a file
for each newsgroup with a list of current article IDs and corresponding
filenames.  This would allow opening one file and quickly checking
article IDs to find unread articles.

This would represent some pretty radical changes to the way news kept
track of articles.  Anybody for "D" news?
				Jerry Aguirre