[news.software.b] Huge logfiles

blair@obdient.CHI.IL.US (Doug Blair) (09/09/89)

Here's an interesting condition. Site obdient runs B news 2.11.17 on
a 386 clone with 45 meg for news articles and a 30 meg /usr file system
(and a couple other drives not relevent to this one). Our upstream newsfeed
has been erratic in the past couple of weeks... nothing for days and then
15-18 meg all at once. As a result of this one night we fell below the
5000 free block minimum we needed to process news, and news correctly
put further incoming batches in the news/.rnews directory with the
"too low on space, batch spooled for later processing" message entered
into the log and errlog files.

This worked fine for the first 5000 blocks of stuff coming in, but eventually
we ran out of blocks (not inodes!). News could then still make directory
entries, but could not copy the actual batch to .rnews, so the .rnews
directory became filled with entries for files with zero lengths, and some
of the news batches were lost. (Well, actually they were archived on 
another file system as part of something I keep running in case the
Alzheimers out of inodes bug strikes, but that's another story).

Next event of interest is the nightly expire run, which ends with an
rnews -U to process any news that came in while expire was running.
As it turned out, expire didn't free up 5000 blocks that night, since
most of the news was quite recent, so the rnews -U (which received
all those file names with zero lengths) wrote the too low on space
message into the log and errlog files and made a new entry in the
.rnews directory. It then received this same new entry again and again!

In fact, the log file grew to 12 megabytes and the errlog grew to 6
megabytes and the /usr file system also ran out of blocks! The system
was VERY reluctant to let me in this morning and has been quite
constipated.

I bring this up only to call it to the attention of those designing
new C and B news (Hey Eric, where are you????). The race condition
could be avoided if rnews -U checks for file length or does NOT change
the file name of the incoming batch if there's no space....

Hope your day started better :-)


Doug
 ___  _           _  _             _    
|   || |_  ___  _| ||_| ___  __  _| |_  Doug Blair    Obedient Software Corp.
| | ||  .\/ ._\/.  || |/ ._\|  \|_   _| 1007 Naperville Rd, Wheaton IL  60187 
|___||___/\___/\___||_|\___/|_|_| |_|   obdient!blair blair@obdient.chi.il.us

henry@utzoo.uucp (Henry Spencer) (09/11/89)

In article <5868@obdient.CHI.IL.US> blair@obdient.CHI.IL.US (Doug Blair) writes:
>I bring this up only to call it to the attention of those designing
>new C and B news (Hey Eric, where are you????). The race condition
>could be avoided if rnews -U checks for file length or does NOT change
>the file name of the incoming batch if there's no space....

C News doesn't have this problem.  We don't process spooled incoming batches
at all when space is low.  When space is *really* low, we drop incoming
news rather than trying to store it -- a properly-configured C News should
never run a filesystem down to zero.
-- 
V7 /bin/mail source: 554 lines.|     Henry Spencer at U of Toronto Zoology
1989 X.400 specs: 2200+ pages. | uunet!attcan!utzoo!henry henry@zoo.toronto.edu