kolk@shelby.Stanford.EDU (Dan Kolkowitz) (12/19/89)
The news configuration at Stanford has a master host feeding several other hosts, which don't have any other news feeds. For most of these hosts we send article batches with rcp, for speed and efficient use of the network. We'd like to switch to NNTP both for reasons of security and more importantly to avoid holding the batches on the master host, when one of the others is down. An interesting compromise of the two methods occurred to us: we can create batches but use NNTP to transmit an entire batch as if it is a single article. As long as the message-id is chosen so that there is no possibility that the batch is rejected, it will be fed to rnews, and then the batch will be unpacked. This saves the IHAVE/SENDME overhead of the protocol and likewise reduces the rnews overhead by the number of execs of rnews that would have occurred. The testing that I have done seems to indicate that on a local Ethernet for an average batch of articles (~200k) it saves about 30% in elapsed time over sending it through an unmodified nntpxmit. (Our nntpxmit is changed to use a given message-id instead of one taken from the first article.) Slower links should benefit even more from the reduction in protocol overhead. The key point is that none of the articles in the batch ever get rejected, since the server is the sole news source for the recipient machine. It would not pay if articles were rejected. I'm just modifying our shell scripts now to use this method. If there is interest I'll post the C code and shell script changes.
lamy@ai.utoronto.ca (Jean-Francois Lamy) (12/20/89)
You could acheive a similar effect thing by using C news batching on the nntpd at the remote end and telling it to look at an empty file instead of the real history file. The remote nntpd can easily be made to invoke rnews on the resulting batch. On a similar note, we decided here that we did not want our NNTP relay to do any batching. So we set up an NNTP feed to a machine with an NNTP daemon that will accept anything (we pointed its notion of the history file at an empty file). By using C news batching, batches get created in the in.coming directory. One could of course feed those to news, but since we were in a position where all batching feeds could take the same stuff, we simply feed them directly to uux directly. Of course this is only convenient if you can multicast the batches to the sites (i.e. they all get the same stuff). This has allowed us to avoid running batching and outgoing UUCP on our news relay, and we share the load another host handle UUCP feeds (said host only has to run nntpd, not the whole news set-up). Jean-Francois Lamy lamy@ai.utoronto.ca, uunet!ai.utoronto.ca!lamy AI Group, Department of Computer Science, University of Toronto, Canada M5S 1A4
weening@Gang-of-Four.Stanford.EDU (Joe Weening) (12/20/89)
In article <89Dec19.120249est.2645@neat.cs.toronto.edu> lamy@ai.utoronto.ca (Jean-Francois Lamy) writes: You could acheive a similar effect thing by using C news batching on the nntpd at the remote end and telling it to look at an empty file instead of the real history file. The remote nntpd can easily be made to invoke rnews on the resulting batch. The problem with this is that you still have to wait for the handshaking on each message: >>> IHAVE <7004@shlump.nac.dec.com> 335 Ok >>> . 235 Thanks. This is one of the things we are trying to avoid. -- Joe Weening Computer Science Dept. weening@Gang-of-Four.Stanford.EDU Stanford University
brian@ucsd.Edu (Brian Kantor) (12/22/89)
Please don't; we're adding batching to the standard and I'd hate to see incompatable implementations out there. - Brian
lamy@cs.utoronto.ca (Jean-Francois Lamy) (12/22/89)
brian@ucsd.Edu (Brian Kantor) writes: >Please don't; we're adding batching to the standard and I'd hate to see >incompatable implementations out there. > - Brian Is this related in any way to streaming (as advocated/implemented by Jerry Aguire (sp?))? If not, is streaming being considered as a separate addition? Jean-Francois Lamy lamy@cs.utoronto.ca, uunet!cs.utoronto.ca!lamy Department of Computer Science, University of Toronto, Canada M5S 1A4
brian@ucsd.Edu (Brian Kantor) (12/23/89)
Yes, it's a variant on streaming. - Brian