reggers@UWOVAX.UWO.CA (Reg Quinton) (02/03/90)
There's been some considerable discussion of late on the issue of how to
get your ANU/News NNTP client machine synch'ed with your Unix NNTP server
machine. I'd like to make some observations since I'm also fighting this
issue too.
First, I think there's a consensus that the daily SKIM isn't good enough
for two reasons
a) it's daily, not every 15 min (old news is history, not news).
b) it's tediously slow.
The problem is the strategy for getting the client and server in synch.
What's going on is the client is connecting to the server and then, for
every group, it says "what have you got?". A lot of work goes on at both
sides with this strategy.
The usual method for dispersing News on Usenet is a flooding strategy.
nntp support uses the 'ihave' protocol for flooding news articles. The
client side will respond to 'ihave x' by saying either 'sendme x' or
something to the effect of 'tell me more' (I don't know the command). In
any case, the point of all of this is the nntp implementations on the Unix
side synchronizes data bases by having the server call the client (unlike
ANU/NEWS) and flood with the 'Ihave/Sendme' protocol. This is the reality
you have to keep in mind when you're dealing with Unix servers -- they've
been optimized to do it this way.
I've been trying to use this kind of strategy to keep my ANU/NEWS client in
synch with my UNIX server. What I've done is marked each group as NNTP
served off my server and I broadcast all articles received to VMS side
using the noted ihave/sendme protocol (the Unix procedure to do this is
called nntpsend and is usually invoked out by cron every 15 minutes or so).
This results in articles being flooded quickly to the VMS side but has
a few problems:
1. the ANU/NEWS nntp server is taking the *ENTIRE* article (header
and body) and squirreling it away in a batch file for later
posting.
- it would be wiser to ask for just the header
(as in the skim operation).
- and we all want the nntp server to post right away
instead of squirreling things in a file.
2. I still need to run a daily skim but if I've got the
two machines expiring news at the same rate then there's
no need to do a skim/served.
I know that lots of people are working with Goeff's NNTP code; I hope
these scratchings are kept in mind.gih900@CSC1.ANU.OZ.AU (Geoff Huston) (02/10/90)
>There's been some considerable discussion of late on the issue of how to >get your ANU/News NNTP client machine synch'ed with your Unix NNTP server >machine. I'd like to make some observations since I'm also fighting this >issue too. > >First, I think there's a consensus that the daily SKIM isn't good enough >for two reasons > > a) it's daily, not every 15 min (old news is history, not news). > > b) it's tediously slow. > >The problem is the strategy for getting the client and server in synch. >What's going on is the client is connecting to the server and then, for >every group, it says "what have you got?". A lot of work goes on at both >sides with this strategy. There are two issues here that you are dealing with in relation to the use of NNTP: 1 - using NNTP as a mechanism to suppport USENET's flooding algorithm, where, as is stated in the posting, the system which has new articles calls the downstream site and used the IHAVE command to push the news to the other site. A variant of this is the pull strategy where the downstream site uses the NEWNEWS command to ask what identifiers are held in the other site (since a given date/time) and then uses the ARTICLE command to get copies of the articles which it doesn't have. (The issue here is what date/time value to use, cause you may be slipping over time zones) 2 - using NNTP as a mechanism for clients slaved of a server. The unix software approach uses the concept of a truly diskless client, and all information is supplied by the server. When I looked at this for VMS I found that there needs to be a large amount of traffic to pass accross the link in order to paint the screens correctly, so I opted to implement a hybrid approach, where a local database is stored which contains a description of all available items. The problem is that the NNTP protocol is not well suited to syncing the local and remote news systems. Essentially the local database needs to be informed of which items are no longer held at the server, and which items need to be added to the local database. I used a private command in the VMS implementation to make VMS - VMS updates relatively efficient. Here the client passes over a list of ids, and the server responds with the relevant extracts of any news items for addition to the local database and also with a list of items to be deleted from the client. I then provided an alternate command stream using the defined NNTP commands to achieve the same result (NEWNEWS, followed by XHDR or HEADER depending on whether the remote system supports the XHDR command. This alternate command stream takes longer to execute, and on many unix implementations the NEWNEWS command is not that quick. About the only way I can think of to speed this up is to sync the client to the server to such an extent that both systems use the same item number index. The NNTP commands are then faster, as the NEWNEWS check can be replaced with a faster GROUP command and a check on the number range. I'll look at this in a bit more detail and see where it leads Geoff Huston