csu@alembic.acs.com (Dave Mack) (06/26/91)
Twice now, I've run into a situation where the C News batcher
manages to produce news batches containing no news.
This is caused by an interaction between batchparms, expire,
the togo lists and the frequency of news transfer. If a site
calls relatively infrequently to get its news (or is down for
several days), and the batching site expires news fairly quickly,
it is relatively easy to get into a situation where the articles
referenced in the togo lists have expired before they are batched.
This is particularly easy with the default batch size of 20.
The batcher, as a result, produces batches containing nothing
but a "#! cunbatch" line followed by four garbage characters.
When the receiving site finally calls for its news, relaynews
faithfully delivers these junk batches ("inbound news garbled"
on the receiving end.) sendbatches, the next time it's called,
assembles another 20 (+/-) batches of articles which have expired.
By the time the calling site has called back for these, even more
articles have expired, and the cycle continues - for two weeks in
the most recent case.
In order to clear this, I either up the batch size and force
a more frequent poll from the calling site, or to nuke the togo
lists and start over.
The problem goes unreported because the batcher doesn't produce any
error output if the files in the togo list aren't there, unless
debugging (batcher -x) is turned on, which it normally isn't.
After staring at the code in sendbatches and batcher for a while,
I suspect that a major rewrite of sendbatches would be necessary
to fix this.
Context: C News Patchdate Mar. 24, 1991, SunOS4.1
--
Dave Macktimk@wynnds.xenitec.on.ca (Tim Kuehn) (06/26/91)
In article <1991Jun26.035746.14008@alembic.acs.com> csu@alembic.acs.com (Dave Mack) writes: >Twice now, I've run into a situation where the C News batcher >manages to produce news batches containing no news. > [batcher error description deleted] >After staring at the code in sendbatches and batcher for a while, >I suspect that a major rewrite of sendbatches would be necessary >to fix this. > >Context: C News Patchdate Mar. 24, 1991, SunOS4.1 What is really needed is some way to set the queue limit in bytes instead of pending uux's to keep the spool from bursting at the seams, but get out a reasonable amount of news in such cases. This should only require adding another field in the batchparms file "max spool bytes pending" and a modification to the "queuelen" program to check that as well as the strict number of pending files going out to a site. ------------------------------------------------------------------------ Tim Kuehn TDK Consulting Services (519)-888-0766 timk@wynnds.xenitec.on.ca -or- !{watmath|lsuc}!xenitec!wynnds!timk Valpo EE turned loose on unsuspecting world! News at 11! "You take it seriously when someone from a ballistics research lab calls you." Heard at a Unix user's meeting discussing connectivity issues.
henry@zoo.toronto.edu (Henry Spencer) (06/26/91)
In article <1991Jun26.035746.14008@alembic.acs.com> csu@alembic.acs.com (Dave Mack) writes: >Twice now, I've run into a situation where the C News batcher >manages to produce news batches containing no news. Yup, known problem. Tricky to solve within the structure of the existing batcher, at least if you don't want to introduce substantial overhead into the normal (everything-flowing-fine) case. It *is* on the to-be-fixed list, however, and that may happen as part of a major overhaul to the batcher. -- "We're thinking about upgrading from | Henry Spencer @ U of Toronto Zoology SunOS 4.1.1 to SunOS 3.5." | henry@zoo.toronto.edu utzoo!henry
gary@sci34hub.sci.com (Gary Heston) (06/27/91)
In article <1991Jun26.035746.14008@alembic.acs.com> csu@alembic.acs.com (Dave Mack) writes:
=Twice now, I've run into a situation where the C News batcher
=manages to produce news batches containing no news.
= [ description deleted ]
=In order to clear this, I either up the batch size and force
=a more frequent poll from the calling site, or to nuke the togo
=lists and start over.
If you want the site to get the news, run sendbatches daily. The
batches will then be built, and sit in your uucp spool directory,
where expire will neither know or care about them. Whenever your
downstream site bothers to call, it'll be waiting.
If disc space is a problem, delete everything over 7 days old
from their uucp spool dir.
=The problem goes unreported because the batcher doesn't produce any
=error output if the files in the togo list aren't there, unless
=debugging (batcher -x) is turned on, which it normally isn't.
=After staring at the code in sendbatches and batcher for a while,
=I suspect that a major rewrite of sendbatches would be necessary
=to fix this.
Go to it. Post it when you're done.
--
Gary Heston System Mismanager and technoflunky uunet!sci34hub!gary or
My opinions, not theirs. SCI Systems, Inc. gary@sci34hub.sci.com
I support drug testing. I believe every public official should be given a
shot of sodium pentathol and ask "Which laws have you broken this week?".chip@tct.com (Chip Salzenberg) (06/28/91)
According to csu@alembic.acs.com (Dave Mack): >After staring at the code in sendbatches and batcher for a while, >I suspect that a major rewrite of sendbatches would be necessary >to fix this. Not at all. Just have sendbatches check for and throw away any zero-length batches before compression/transmission. Here are diffs: Index: sendbatches *************** *** 153,156 **** --- 153,164 ---- for f in $them do + # Verify that at least one file in the batch exists. + # If not, ignore the batch. + any= + for ff in `sed 's/ .*$//' $f` + do + test -f $ff && { any=yes; break; } + done + # Sigh... sh -x on this won't work, because the -x # output ends up in /tmp/nb$$, and there is no way *************** *** 159,164 **** # processes in a pipeline) or (b) turn off -x for # a moment in a portable way. ! ( ( cd $NEWSARTS ; $batcher $here/$f ) | $muncher | ! $sender $sys ) >/tmp/nb$$ 2>&1 if test $? -eq 0 -a ! -s /tmp/nb$$ then --- 167,175 ---- # processes in a pipeline) or (b) turn off -x for # a moment in a portable way. ! if test -n "$any" ! then ! ( ( cd $NEWSARTS ; $batcher $here/$f ) | ! $muncher | $sender $sys ) ! fi >/tmp/nb$$ 2>&1 if test $? -eq 0 -a ! -s /tmp/nb$$ then -- Chip Salzenberg at Teltronics/TCT <chip@tct.com>, <uunet!pdn!tct!chip> "I want to mention that my opinions whether real or not are MY opinions." -- the inevitable William "Billy" Steinmetz
csu@alembic.acs.com (Dave Mack) (06/29/91)
In article <1991Jun27.153255.18008@sci34hub.sci.com> gary@sci34hub.sci.com (Gary Heston) writes: >In article <1991Jun26.035746.14008@alembic.acs.com> csu@alembic.acs.com (Dave Mack) writes: >=Twice now, I've run into a situation where the C News batcher >=manages to produce news batches containing no news. >= [ description deleted ] >=In order to clear this, I either up the batch size and force >=a more frequent poll from the calling site, or to nuke the togo >=lists and start over. > >If you want the site to get the news, run sendbatches daily. The >batches will then be built, and sit in your uucp spool directory, >where expire will neither know or care about them. Whenever your >downstream site bothers to call, it'll be waiting. Nope, sendbatches calls queuelen to check the uucp queue. If you have more than the number of batches specified in batchparms awaiting transmission, no more batches are created. So you end up with 20+ batches of valid news sitting in your uucp queue, a continuously growing list of articles in out.going/sitename, and the articles referenced in that list being expired out from under it. The remote site calls in for news, gets its chunk of valid articles, then the batcher starts creating the empty batches. Someone sent me a patch to batchsplit that may solve this problem. I'll let you know after I try it out. -- Dave Mack