bin@primate.wisc.edu (Brain in Neutral) (07/03/89)
From article <1884@papaya.bbn.com>, by rsalz@bbn.com (Rich Salz): > The next topic was running NNTPXMIT. Most people run it every 30 to 60 > minutes, a few run it every one to 10 minutes, and Scott Bradner at > Harvard runs it continuously. **Care to post your script that does this, > Scott?** After pleading from Erik Fair, most people agreed to run it more > often. **Please run NNTPXMIT at least every 10 minutes.** What is the reason for this? To reduce average propagation time? To increase size of already-large log files? Paul DuBois dubois@primate.wisc.edu
wisner@mica.Berkeley.EDU (Bill Wisner) (07/04/89)
bin@primate.wisc.edu (Brain in Neutral) writes: >What is the reason for this? To reduce average propagation time? To >increase size of already-large log files? To reduce average propagation time. Unless you're running on floppy disks or never prune your log files, their size should not be an issue. When Ed Vielmetti was at the U of Michigan's computing center, he had mailrus and sharkey running nntpxmit from cron every minute. This may or may not still be the case; Ed's now with UM's math department. Bill Wisner wisner@mica.berkeley.edu ucbvax!mica!wisner I'm not the NRA either.
fair@Apple.COM (Erik E. Fair) (07/04/89)
Running nntpsend often is entirely an exercise in making news flow more quickly. If everyone is running with what we recommended in the original NNTP release (every 10 minutes), then it takes an hour to go six hops. I'd like to see it take six minutes (i.e. run nntpsend every minute). If you read Rick Adams' UUNET statistics posted in news.lists every two weeks, I'm looking to push the initial wave of traffic to the first hour, and spread it out more evenly. This isn't a strictly statistical goal either; netnews is fast becoming the computer conferecing system of the Internet, but it is still not as fast as Internet mail. Fortunately, with your help, this can be fixed. Currently, ucbvax and apple are both running 1 minute nntpsend (I just took that first field in the /usr/lib/crontab file and turned it into a star "*"). I know that other backbone sites are running faster than 10 minutes too. It hasn't put much of a load on the systems because we're moving just as many bytes as before - the load is just spread out more evenly, rather than spiking every 10 minutes... Erik E. Fair apple!fair fair@apple.com P.S. let's move the discussion out of comp.org.usenix to news.software.nntp.
ed@mailrus.cc.umich.edu (Ed Anselmo) (07/04/89)
>When Ed Vielmetti was at the U of Michigan's computing center, he had mailrus >and sharkey running nntpxmit from cron every minute. This may or may not >still be the case; Ed's now with UM's math department. This is no longer the case: Looks like mailrus is running nntpxmit every 30 min.; sharkey once an hour. -- Ed Anselmo (ed@cardiology.ummc.umich.edu, mailrus!clip!ed)
emv@math.lsa.umich.edu (Edward Vielmetti) (07/04/89)
In article <1128@mailrus.cc.umich.edu> ed@cardiology.ummc.umich.edu (Ed Anselmo) writes: >>When Ed Vielmetti was at the U of Michigan's computing center, he had mailrus >>and sharkey running nntpxmit from cron every minute. This may or may not >>still be the case; Ed's now with UM's math department. > >This is no longer the case: > >Looks like mailrus is running nntpxmit every 30 min.; sharkey once an hour. > > -- Ed Anselmo (ed@cardiology.ummc.umich.edu, mailrus!clip!ed) Running every minute out of cron, especially if you have a lot of incoming feeds, means that there's going to be a lot of activity if you go down for any amount of time and come back up again. Imagine going down for 4 hours, then getting 4 hour's feed from ten sites simultaneously in the first few minutes of going back up. Tends to push up the load average a bit, and if there's anything wrong with the workings of other important software on the system (i.e. mail) that can make it difficult to address those problems. I had thought to put something in rc.local to delay accepting news from most sites for say 30 minutes after boot; you'd have two nntp_access files, and boot with a real restrictive one which only allows a few favorite close-by sites to feed you, which you'd replace with the real one after things quieted down sufficiently. --Ed
brian@ucsd.EDU (Brian Kantor) (07/04/89)
I've shipped a series of patches off to Stan Barber that permit you to
limit the number of transfer-mode NNTPd processes running on a system
based on load average. In particular, nntpd checks two load average
figures and will disallow startup or discontinue transfers if the load
average climbs above the appropriate limits.
The patches also include locking to prevent incoming transfers whilst
'expire' is running; this is done because there can be no duplicate
article detection when incoming news is spooled into .rnews and with
multiple feeds that's a real waste of resources.
Stan tells me he'll get those changes out when he has time. Please be
patient.
Below I've included our nntpsend which runs continously; it has load
limiting stuff in it too. Adjust the load limits and sleeps to suit
your environment. (UCSD is a poor tired vax/750 and thus not much
faster than an IBM-PC; our primary task is mail so news gets shut off
when the load average impacts mail flow.) This is started hourly from
crontab in case it exits; shell locking prevents more than one from
running at a time.
Enjoy.
- Brian
---
#
# What we have here is a csh script for sending netnews to NNTP sites.
#
set stoplim=10
set slowlim=6
set batchdir=/usr/spool/batch libdir=/usr/lib/news
set path=( $libdir /usr/ucb /usr/bin /bin $path )
#
# Go to where the action is
#
cd $batchdir
umask 022
set pname=$0
set pname=$pname:t
#
# Make sure only one of us is running
#
if { shlock -p $$ -f NNTPSEND } then
echo ${pname}: "[$$]" started `date`
else
echo ${pname}: "[$$]" already running "[`cat NNTPSEND`]"
exit 0
endif
while ( 1 )
set loadavg=( `uptime` )
# check the 15-minute load average
if ( $loadavg[$#loadavg]:r > $stoplim ) then
echo "load avg too high; quitting"
echo $loadavg
rm NNTPSEND
exit 1
endif
set sentsome=0
echo ${pname}: "[$$]" begin cycle `date`
foreach host ( \
rutgers.edu \
ucbvax.berkeley.edu \
tut.cis.ohio-state.edu \
)
# check the 5-minute load average
set loadavg=( `uptime` )
@ n = $#loadavg - 2
while ( "$loadavg[$n]:r" > $slowlim )
sleep 600
# check the 5-minute load average
set loadavg=( `uptime` )
@ n = $#loadavg - 2
end
set lock=NNTP_LOCK.${host}
set tmp=${host}.tmp
set send=${host}.nntp
if { shlock -p $$ -f ${lock} } then
if ( -e ${tmp} ) then
cat ${tmp} >> ${send}
rm ${tmp}
endif
if ( -e ${host} ) then
if ( -e ${send} ) then
mv ${host} ${tmp}
else
mv ${host} ${send}
endif
endif
if ( -e ${send} ) then
( echo ${pname}: "[$$]" begin ${host}:${send};\
time nntpxmit ${host}:${send};\
rm -f ${lock};\
echo ${pname}: "[$$]" end ${host}) &
set sentsome=1
else
rm -f ${lock}
endif
else
echo ${pname}: "[$$]" ${host} lock by "[`cat ${lock}`]"
endif
# let the load level a bit
sleep 30
end
echo ${pname}: "[$$]" end cycle `date`
# if there wasn't any news to send this time round,
# hibernate for a while
if (${sentsome} == 0) then
sleep 300
endif
endbin@primate.wisc.edu (Brain in Neutral) (07/05/89)
From article <714@stag.math.lsa.umich.edu>, by emv@math.lsa.umich.edu (Edward Vielmetti): | Running every minute out of cron, especially if you have a lot of incoming | feeds, means that there's going to be a lot of activity if you go down for | any amount of time and come back up again. Imagine going down for 4 hours, | then getting 4 hour's feed from ten sites simultaneously in the first few | minutes of going back up. Tends to push up the load average a bit, and if | there's anything wrong with the workings of other important software on | the system (i.e. mail) that can make it difficult to address those problems. | | I had thought to put something in rc.local to delay accepting news from most | sites for say 30 minutes after boot; you'd have two nntp_access files, and | boot with a real restrictive one which only allows a few favorite close-by | sites to feed you, which you'd replace with the real one after things quieted | down sufficiently. Before booting: mv /etc/nntpd /etc/nntpd~ After you've been up for a bit: mv /etc/nntpd~ /etc/nntpd Simple, effective. It doesn't prevent the load average hike from happening, but does at least delay it until you want it. Paul DuBois dubois@primate.wisc.edu
leres@ace.ee.lbl.gov (Craig Leres) (07/07/89)
Erik E. Fair writes: > quickly. If everyone is running with what we recommended in the > original NNTP release (every 10 minutes), then it takes an hour to go > six hops. I'd like to see it take six minutes (i.e. run nntpsend every If I run nntpsend every 10 minutes, I would expect it to take 5 minutes (on average) for me to propagate a new article. I would also expect it to take 30 minutes (on average) to travel six hops. I suppose my posting has the potential for starting a minor probability and statistics war (like the one on engine counting in comp.risks) so perhaps I should state my assumptions in an attempt to cut out the simpletons. My first assumption is that not everyone clocks are synchronized. A small sampling of internet hosts at several sites shows a spread of about plus or minus 3 minutes. My other assumption is that not everyone who runs nntpsend with a 10 minute interval starts on the hour. Even if you (stupidly) assume that every single nntp peer's clock is synchronized, I don't see how you can assume that they all start their cycles on the hour (my own system starts at 3 minutes past the hour or 7 minutes before the hour, depending on how you look at it). Note that even if every host starts their cycle on the hour, the un-synchronized clock syndrome will make it take almost 10 minutes for an article to travel in one direction and almost 0 minutes for it to travel in the other direction. This averages out to 5 minutes. Craig