steve@nuchat.UUCP (Steve Nuchia) (02/07/88)
After several months of observing news transport resopurce
consumption I would like to propose a new method of exchanging
articles over bathc-mode links. I am not sufficiently familiar
with nntp to draw detail analogies, but my goals here are similar
to those of nntp, applied to a different transport technology.
There are several inefficiencies and other problems in the present
system, which I will discuss after I sketch the proposed protocol.
When an article is localized and inews decides it needs to be
sent to a neighbor, an expanded ihave message is generated,
to include the Newsgroups: header and the pathname of the
article file (as seen in batching) as well as the article ID.
On receipt of the expanded ihave message you spool the information
to a hold file. Additional information required per line includes
the name of the "ihave" site and a timestamp.
A sendme debatcher can then sort the spool file to allow a linear
scan (classic merge algorithm) against the history file. The
debatcher also implements an ihave aging policy which attempts
to get articles from the lowest cost source. The proposed standard
aging policy is that each link has associated with it a cost stated
in terms of latency - the higher the cost of the link the longer
we are willing to wait to see if a lower cost link can supply the
article instead.
Once the debatcher decides to order an article from a neighbor it
does so by article file name, eliminating the history lookup at
the neighbor site.
Now I turn to the problems this addresses:
Redundant transmitions: because this has a lower cost
than the current ihave/sendme protocol it would be
used more, eliminating some redundant transmitions.
Also, there is a race in the current ihave/sendme
protocol than can result in ordering articles from
several sources - this can be dealt with while we're
at it. (probably need a timeout so we _can_ reorder).
Locality of newsgroup mask: For links which use this
mechanism, the newsgroup mask can reside entirely
on the destination site, elimitaing a distributed
administration headache. (and cutting down on the
bulk of the sys file for high-fanout sites.)
Lowest-cost routing: Obviously the latency mechanism
will give site admins very fine control over the
cost/latency/reliability tradeoffs, potentially
on a newsgroup-by-newsgroup basis.
Micro-optimizations: the proposed scheme is more efficient
than ihave/sendme in several small ways, particularly
for non-dbm implementations.
Anyone interrested in collaborating on an implementation of this
scheme is invited to respond, and of course criticism and suggestions
are welcome. I'll post an update on the project after the mail
comes in.
I have been running a batched ihave for my uunet downlink for
a few months, using a latency figure of about 6 days, and it works
pretty well for a prototype.
--
Steve Nuchia | [...] but the machine would probably be allowed no mercy.
uunet!nuchat!steve | In other words then, if a machine is expected to be
(713) 334 6720 | infallible, it cannot be intelligent. - Alan Turing, 1947