tytso@athena.mit.edu (Theodore Y. Tso) (09/22/89)
I've got an idea for nntpd, and like to know what other people think about it. Is it dumb? unnecessary? implemented already? The basic problem is that if you are a well-connected site, two or three hosts might be able to simultaneously blast with you with more or less the same list of articles. Since articles that are already processed are quickly skipped over, what tends to happen is that the two or three (or more!) sending sites quickly get syncronized and begin offering the same articles almost simultaneously. What then happens is the nntpd's on your site get the offer and spawn off rnews processes, where all but one of the offered articles get punted. This syncronicity causes a lot of unnecessary CPU crunching. Now, this problem can be solved by sys file hacking, but that tends to decrease redundancy. So how about the following solution: after nntpd determins that an article isn't in the history file, it tries to creat() some lock file based on the message-id in /tmp. If it succeeds, it writes its pid to that file and runs rnews. When the rnews finishes, the lock file gets unlinked. If it fails, it checks the pid in the file to see if the process still exists. If it doesn't, it assumes something has gone wrong and runs rnews anyway. If the process specified in the lock file exists, then we assume that the article is being processed by another file, and it can be skipped (i.e., it sends 435 back to the sending host). What do people think? Would this be a good idea, or not? =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Theodore Ts'o bloom-beacon!mit-athena!tytso 3 Ames St., Cambridge, MA 02139 tytso@athena.mit.edu Everybody's playing the game, but nobody's rules are the same!
tytso@athena.mit.edu (Theodore Y. Ts'o) (09/23/89)
In article <FLEE.89Sep22130759@shire.cs.psu.edu> flee@shire.cs.psu.edu (Felix Lee) writes: >In article <14506@bloom-beacon.MIT.EDU>, > Theodore Y. Tso <tytso@athena.mit.edu> proposes >a scheme to create a lock file for each article while nntpd is >receiving it. > >The question is, what do you tell site B when the article it tries to >offer you is being sent by site A? "Gotit" is wrong, if the A xmit >fails. Blocking until the lock goes away is unnecessary delay. The >best thing is to tell site B to ask again later. Actually, something which I forgot to mention is that currently the news system does currently block until site A finishes processing the article. The difference is that the nntpd forks a rnews that checks for the existence of /tmp/L<msg-id> and blocks until that file goes away. It would be more efficient for this to be handled at the NNTP layer. However, I'm not sure there's a NNTP code to send back that will cause nntpxmit to defer an article until later without aborting the entire transfer session. Is there a way to do such a thing, or will it require modification of nntpxmit as well? I was hoping for something that didn't require modification of nntpxmit, so I could run it on my site without worrying about what my neighbors run. =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-= Theodore Ts'o bloom-beacon!mit-athena!tytso 3 Ames St., Cambridge, MA 02139 tytso@athena.mit.edu Everybody's playing the game, but nobody's rules are the same!
flee@shire.cs.psu.edu (Felix Lee) (09/23/89)
In article <14506@bloom-beacon.MIT.EDU>,
Theodore Y. Tso <tytso@athena.mit.edu> proposes
a scheme to create a lock file for each article while nntpd is
receiving it.
I was considering doing this, especially in connection with
C-News-style NNTP batching. Haven't gotten around to doing it yet.
(Locking within a multithreaded NNTP server would be cheaper.)
The question is, what do you tell site B when the article it tries to
offer you is being sent by site A? "Gotit" is wrong, if the A xmit
fails. Blocking until the lock goes away is unnecessary delay. The
best thing is to tell site B to ask again later.
--
Felix Lee flee@shire.cs.psu.edu *!psuvax1!fleeflee@shire.cs.psu.edu (Felix Lee) (09/23/89)
In article <14527@bloom-beacon.MIT.EDU>, Theodore Y. Ts'o <tytso@athena.mit.edu> writes: > the news system does currently block until site A finishes > processing the article. Yes, for B news. I was more interested in article locking in conjunction with C News and NNTP batching, which doesn't require blocking, and would in fact be painful to block for. -- Felix Lee flee@shire.cs.psu.edu *!psuvax1!flee