root@olivey.olivetti.com (The Superuser) (06/08/89)
The current NNTP transmits news in lock-step between server and client. While this results in several benefits its performance suffers on links where there is a long turn around time (rtt). The impact is greatest where the server already has most of the articles and the Ihave/Gotit exchange predominates. The current handshaking would seem to preclude any kind of buffering of multiple requests across the link. I have been thinking about a negotiated option to the protocol that would allow the client to send multiple Ihave requests to the server. The primary need on the server end is for the "Gotit", "Enter" and "Thanks" messages to include the relevant message ID. The client would require a new message to indicate that the contents particular message ID will follow immediately and the server would trigger on that as the beginning of the article and use the given ID in the ack. This would allow decoupling the transmissions of the client and server. While the server would still process a single request at a time the client could buffer up several message ID/file name combinations and send them to the server before looking for responses. Requested IDs could be sent and refused or acknowledged IDs could be removed from the buffer and replaced by the next entry from the batch file. The disadvantages that I see for this are a slightly greater overhead due to including the message ID in more commands. There is also a slight increase in the window during which duplicates can be sent. Sending multiple commands in the same packet could offset the increased amount of data. The advantage would be an increase in the transmission rate. In the extreem case of all articles being duplicates there should be a steady stream of Ihaves with a stream of Gotits flowing back. Does this "streaming" mode seem like a viable option for NNTP? The alternative would be to send blocks of message IDs and have the server request only those it wanted. While more efficient this would require additional storage in the server and a negotiation to decide how many it could accept. It would also be more complex with regard to error recovery.
amanda@intercon.UUCP (Amanda Walker) (06/08/89)
In article <42723@oliveb.olivetti.com>, root@olivey.olivetti.com (The Superuser) writes: > Does this "streaming" mode seem like a viable option for NNTP? Well, some of this could be accomplished within the NNTP protocol as it stands. For example, I don't see any problem with sending multiple "IHAVE"s (or whatever), and collecting the responses as they come back, sort of like a sliding-window communications protocol. This would help a lot over links with high RTT. It would require some mods on the client side, but the current servers should be able to handle it, since incoming data on the TCP connection will just be buffered until the server reads it. On a similar note, I've been debating experimenting with a variation on the "HEAD" command that takes a range of article numbers (or a list of message-ids) and sends back the article headers as one big response (with blank lines between headers). This wouldn't be much more of a load on the server than XHDR is now, and would be a lot nicer than doing bunches of HEAD requests. Comments? -- Amanda Walker <amanda@intercon.UUCP> -- "Some of the worst mistakes in history have resulted from trying to apply methods that work fine in one field to another where they don't." -James Hogan
david@ms.uky.edu (David Herron -- One of the vertebrae) (06/08/89)
I've recently had similar thoughts but hadn't thought them beyond:
extend IHAVE to include a whole list of MID's
sender telling receiver some of the messages sender has to send
extend SENDME to " " " " " "
receiver telling sender what messages of the previous list
it would like to receive.
Then once the servers have negotiated a list of MID's to send the
sender sends each one as "normal".
Here we've got bunches of NNTP links. Our sequent does a real fine
job of doing lots of things ... so we tend to be free and easy about
making NNTP links, and we also make connection attempts every 10 minutes
or so. But we get widely variable times that the connections last,
*and* often have lots of nntpxmit's in the background.
It's pretty useful that the nntpxmits usually are throttled back by the
network. The systems guy here makes rude noises on occasion when he
sees all those nntpxmits hanging around -- but then he just likes to
make rude noises.
If the nntpxmits were to only do quickies (see, they would get in and
out quickly ... ) there wouldn't be as many hanging around anymore.
But each one would put greater load on the system.
--
<- David Herron; an MMDF guy <david@ms.uky.edu>
<- ska: David le casse\*' {rutgers,uunet}!ukma!david, david@UKMA.BITNET
<-
<- New word for the day: Obnoxity -- an act of obnoxiousnesslear@NET.BIO.NET (Eliot Lear) (06/09/89)
There have been discussions on this, before. Among other things, apparently, the current protocol causes telebits to have nervous breakdowns because of the ``interactive'' nature of the protocol. The only problem with what you've suggested is a matter of article id locking - specifically, if an article id is locked, the server cannot simply GOTIT because the article may be garbled on the other transmission line (another NNTP). My solution to this problem is to simply have the NNTP waiting for the IHAVE response sleep for a few seconds. Another possible solution is to have a temporary error of MaybeIGotIt. Maybe we should have both. -- Eliot Lear [lear@net.bio.net]
amanda@intercon.UUCP (Amanda Walker) (06/09/89)
I just realized that my previous posting may have sounded somewhat silly :-). I had momentarily forgetten quite how IHAVE works (must have been a stray cosmic ray...). I was actually thinking about buffering up ARTICLE commands, and forgot that it wouldn't work in reverse... Anyway, this afternoon I tried playing with a version of the HEAD command that takes an article range in addition to a single article number/message-id, and although it is quite nice to use from a newsreader, it's not as much of a performance boost over successive conventional HEAD commands as I'd hoped. This may well be more a matter of using a gerbil-powered computer for my development NNTP server than anything else, though :-). The mods are very simple, so if anyone else wants to play with this, send me email and I'll send them off to you. -- Amanda Walker <amanda@intercon.UUCP> -- "Some of the worst mistakes in history have resulted from trying to apply methods that work fine in one field to another where they don't." -James Hogan
jerry@olivey.olivetti.com (Jerry Aguirre) (06/09/89)
In article <08-Jun-89.110238@192.41.214.2> amanda@intercon.UUCP (Amanda Walker) writes: >it stands. For example, I don't see any problem with sending multiple >"IHAVE"s (or whatever), and collecting the responses as they come back, But the current responses don't tell you which IHAVE they refer to. You could assume that they were in order but that seems risky to me. A single off by one problem would throw the rest of the sesion out of sync.
brian@ucsd.EDU (Brian Kantor) (06/10/89)
Right now the NNTP protocol enhancements we're considering involve either making the server run in NORMAL or STREAM mode, with the IHAVE command in STREAM mode NOT prompting for the article but awaiting a BATCH command to indicate that a whole batch of articles is coming, or else a new command IGOT or YOUWANT or something like that which would perform the same article lookup as IHAVE but expect that the actual batch article transmission would be delayed until enough of them had been gathered up to make it worthwhile. Note that you save very little NNTP overhead this way; it's mainly for the news posting software that's being fed by NNTP that this whole thing exists. Additional proposed extensions have to do with security and usage measurement options. More on this later. Please note that if you make some change to the servers and clients that affects the syntax of any of the commands or responses, you MUST use an Xxxx command so that you will not be issuing any commands that an unmodified server will barf on. That is why XHDR is an X command (but will probably be adopted into the mainstream in the next protocol revision). Brian Kantor UCSD Office of Academic Computing Academic Network Operations Group UCSD C-010, La Jolla, CA 92093 USA brian@ucsd.edu ucsd!brian BRIAN@UCSD
amanda@intercon.UUCP (Amanda Walker) (06/12/89)
In article <1759@ucsd.EDU>, brian@ucsd.EDU (Brian Kantor) writes: > Please note that if you make some change to the servers and clients that > affects the syntax of any of the commands or responses, you MUST use an > Xxxx command so that you will not be issuing any commands that an unmodified > server will barf on. That is why XHDR is an X command (but will probably be > adopted into the mainstream in the next protocol revision). Yup. Even my currently-being-played-with software won't break unless someone else has implemented an "XHEAD" command with a different syntax, and has stolen software off of my Mac II :-). I suppose I could even change it to "XHEAD-INTERCON", just to make sure :-)... I may end up throwing this out anyway, since it only really wins on a network with a very long RTT with respect to the speed of the server, and just sending out "HEAD" commands while the previous header is still coming across does just about as well. -- Amanda Walker <amanda@intercon.UUCP> -- "Some of the worst mistakes in history have resulted from trying to apply methods that work fine in one field to another where they don't." -James Hogan
jerry@olivey.olivetti.com (Jerry Aguirre) (06/24/89)
I have finished my changes to the server and xmit routines for the support of a "streaming" mode. A test of the new code used 40.8 seconds of real time vs. 123.4 seconds for the old. (A 67% reduction!) This test was on an internal feed with a RTT of about 700ms. All the articles were previously sent and so were refused. This is the case I hoped to improve and I am pleased with the results. In the case where articles are not duplicates and must be transmitted there will also be improvement though processing speed and transmission bandwidth will effect the results. Setting up an accurate test is difficult because different articles must be used otherwise the second test will have its articles refused. (A simple test showed a 50% reduction in real time.) The new mode is negotiated on connection. Old servers complain about an unknown command causing the xmit to fall back to the old protocol. This makes the new xmit completely compatible. The server operates as before unless the new commands are used so it is also backward compatible. The receiving daemon is still unbuffered and so only forks off one rnews at a time. What changes is the delay waiting for articles. Any volunteers to beta test? Jerry Aguirre