[news.software.b] C News: batchrunning

ronald@robobar.co.uk (Ronald S H Khoo) (11/28/90)

Anyone else feel the need for such a beast besides me ?
After all, if you're feeding a number of sites,
batchrunning can be just as loading as newsrunning, and sometimes
one does feel the need to stop it :-)

Or should I just add "echo $$ > $NEWSCTL/sendbatches.pid" to sendbatches
and kill `cat $NEWSCTL/sendbatches.pid` ?
-- 
ronald@robobar.co.uk +44 81 991 1142 (O) +44 71 229 7741 (H)

henry@zoo.toronto.edu (Henry Spencer) (11/28/90)

In article <1990Nov27.161854.24449@robobar.co.uk> ronald@robobar.co.uk (Ronald S H Khoo) writes:
>Anyone else feel the need for such a beast besides me ?
>After all, if you're feeding a number of sites,
>batchrunning can be just as loading as newsrunning, and sometimes
>one does feel the need to stop it :-)

This is in the to-be-done queue; it's been suggested to us (by mail) before.
-- 
"The average pointer, statistically,    |Henry Spencer at U of Toronto Zoology
points somewhere in X." -Hugh Redelmeier| henry@zoo.toronto.edu   utzoo!henry

urlichs@smurf.sub.org (Matthias Urlichs) (11/28/90)

In news.software.b, article <1990Nov27.161854.24449@robobar.co.uk>,
  ronald@robobar.co.uk (Ronald S H Khoo) writes:
< Anyone else feel the need for such a beast besides me ?
< After all, if you're feeding a number of sites,
< batchrunning can be just as loading as newsrunning, and sometimes
< one does feel the need to stop it :-)
< 
You might want to put checks for the "stop" file into your sendbatches script
at some strategic locations.

Cleanly shutting down a system where there are files named LOCK, and
LOCKinput, and LOCKbatch, and LOCKmthreads, seems to be a problem...
especially because th chance that you catch all of them not being present
approaches zero with increasing news volume.

< Or should I just add "echo $$ > $NEWSCTL/sendbatches.pid" to sendbatches
< and kill `cat $NEWSCTL/sendbatches.pid` ?

This could only result in one duplicate batch, and therefore seems to be
safe. Killing (or crashing a machine with) a running "relaynews" or mthreads
seems to be more critical.
Does the dbz database have a flag to indicate that the database has been
closed cleanly? If not, it should probably be added..?

-- 
Matthias Urlichs -- urlichs@smurf.sub.org -- urlichs@smurf.ira.uka.de     /(o\
Humboldtstrasse 7 - 7500 Karlsruhe 1 - FRG -- +49+721+621127(0700-2330)   \o)/

henry@zoo.toronto.edu (Henry Spencer) (11/29/90)

In article <q!6mg2.e=4@smurf.sub.org> urlichs@smurf.sub.org (Matthias Urlichs) writes:
>Does the dbz database have a flag to indicate that the database has been
>closed cleanly? If not, it should probably be added..?

Unfortunately, it's impossible to do right.  I thought about this a bit,
but the combination of stdio buffering, buffer caches, and the Nightmare
File System make it virtually impossible to make such a flag trustworthy.

However, there basically isn't much room for inconsistency.  Particularly
if you are not using the in-core facility, writes go out immediately anyway.
-- 
"The average pointer, statistically,    |Henry Spencer at U of Toronto Zoology
points somewhere in X." -Hugh Redelmeier| henry@zoo.toronto.edu   utzoo!henry

ronald@robobar.co.uk (Ronald S H Khoo) (11/29/90)

urlichs@smurf.sub.org (Matthias Urlichs) writes:
 
> Cleanly shutting down a system where there are files named LOCK, and
> LOCKinput, and LOCKbatch, and LOCKmthreads, seems to be a problem...
 
Hmm..  Does this mean that we really ought to have STOP, STOPinput,
STOPbatch, STOPmthreads, etc in $NEWSCTL instead rather than lots of
little stop files scattered about the filesystem ?

H'm  I'm not sure what I'd do with a STOPinput :-)
-- 
ronald@robobar.co.uk +44 81 991 1142 (O) +44 71 229 7741 (H)

urlichs@smurf.sub.org (Matthias Urlichs) (12/01/90)

In news.software.b, article <1990Nov29.144403.12729@robobar.co.uk>,
  ronald@robobar.co.uk (Ronald S H Khoo) writes:
< urlichs@smurf.sub.org (Matthias Urlichs) writes:
<  
< > Cleanly shutting down a system where there are files named LOCK, and
< > LOCKinput, and LOCKbatch, and LOCKmthreads, seems to be a problem...
<  
< Hmm..  Does this mean that we really ought to have STOP, STOPinput,
< STOPbatch, STOPmthreads, etc in $NEWSCTL instead rather than lots of
< little stop files scattered about the filesystem ?
< 
Actually, one STOP file should suffice because (at least for me, your mileage
may vary) the main reason to stop processing is to shut down the system,
and then it doesn't make much sense to stop only one of the above.

-- 
Matthias Urlichs -- urlichs@smurf.sub.org -- urlichs@smurf.ira.uka.de     /(o\
Humboldtstrasse 7 - 7500 Karlsruhe 1 - FRG -- +49+721+621127(0700-2330)   \o)/

urlichs@smurf.sub.org (Matthias Urlichs) (12/01/90)

In news.software.b, article <1990Nov29.041256.26775@zoo.toronto.edu>,
  henry@zoo.toronto.edu (Henry Spencer) writes:
< In article <q!6mg2.e=4@smurf.sub.org> urlichs@smurf.sub.org (Matthias Urlichs) writes:
< >Does the dbz database have a flag to indicate that the database has been
< >closed cleanly? If not, it should probably be added..?
< 
< Unfortunately, it's impossible to do right.  I thought about this a bit,
< but the combination of stdio buffering, buffer caches, and the Nightmare
< File System make it virtually impossible to make such a flag trustworthy.
< 
Well, I'm not using NFS, and an fflush() plus fsync() should take care of most
cases. You wouldn't entirely eliminate the risk of getting a database marked
clean in error, but it would be reduced. Given that crashing a system is also
a low-probability event (or at least it should be), that might be enough.

However,

< However, there basically isn't much room for inconsistency.  Particularly
< if you are not using the in-core facility, writes go out immediately anyway.

Good news. (On my site, INCORE is used with expire only, and with expire it
doesn't matter if there's a crash.)

-- 
Matthias Urlichs -- urlichs@smurf.sub.org -- urlichs@smurf.ira.uka.de     /(o\
Humboldtstrasse 7 - 7500 Karlsruhe 1 - FRG -- +49+721+621127(0700-2330)   \o)/