[news.admin] Ways to deal with the additional load...

gnu@hoptoad.uucp (John Gilmore) (11/23/87)

wcs@ho95e.ATT.COM (Bill.Stewart) wrote:
> How have people handled the additional load?  I've had to expire
> articles much faster than before, mainly because my history file is
> limited to 1 megabyte (System V ulimit braindamage), though total disk
> volume has also been a problem.  Similarly, I've had rnews workload
> problems during the day (ho95e is an elderly 1MIPS machine which has
> real users on it); I hope CNews will help when I get it running.

I thought the ulimit braindamage was easily solved by making /etc/init
a little program that sets the ulimit to something sane and then exec's
the real init...but I don't believe in Missed'em V, except as implemented
by Guy Harris atop SunOS, so I've never had to try it.

C news definitely helps; I've been running an ancient version of C
"rnews" for months (with a 2.11 "vnews" and a 2.10.3 "inews" and expire,
and Bog knows what version of "rn"), and it's a serious win.

I am also expiring quicker than before (from 14 to 9 days), and my batching
scripts quit batching when the spool partition goes below 3MB free.
(Henry's C news batcher can do much better, cutting off an unresponsive
site while still getting the news out to everybody else, but in typical
C news fashion it changes the interface "just a little bit" so you can't
conveniently drop it in as a B news batcher replacement.  Worse, the
reason for the interface change is so Henry could write most of it as
a shell script.  I would make it meet the interface spec first and go
for the small-is-bootiful ideology second.  And I will, before I install
it here...)

One thing I did that cuts out a bunch of the load is to stop receiving
and forwarding comp.binaries.  I smile every time I see how many K show
up in the "top 25 news posters" lists but never appear on hoptoad...

A final technique which is partly implemented here is to run news feeds
on Telebit modems.  My feed to unisoft is on a Telebit link, and while
it only gets hundreds of chars/sec instead of the nominal 1200 (because
unisoft is incredibly overloaded and can't RECEIVE 1200 chars/sec),
it's a win.  When ptsfa figures out how to make Datakit send breaks and
work at 19200 baud, my incoming news will be on a Telebit too.  Let's
see, 2.4MB a day, that's about 35 minutes on the modem...
-- 
{pyramid,ptsfa,amdahl,sun,ihnp4}!hoptoad!gnu			  gnu@toad.com
Love your country but never trust its government.
		      -- from a hand-painted road sign in central Pennsylvania

henry@utzoo.UUCP (Henry Spencer) (11/26/87)

> (Henry's C news batcher can do much better, cutting off an unresponsive
> site while still getting the news out to everybody else, but in typical
> C news fashion it changes the interface "just a little bit" so you can't
> conveniently drop it in as a B news batcher replacement...

Nonsense.  Utzoo ran for months using that batcher with an ancient B news.
The *only* adverse effect of this is to make batch-length limiting rather
imprecise.  It has absolutely no effect on the disk-space conservation code,
which is totally independent of it.

> ... I would make it meet the interface spec first and go for the small-is-
> bootiful ideology second.  And I will, before I install it here...)

If you install all of C news, it meets the interface spec just fine, since
C rnews is prepared to generate the extra information which the batcher
would like (note:  "would like", as opposed to "needs").  However, it works
just fine as is even with a B rnews feeding it.  Apart from "meeting the
interface spec" being slower, it tends to lead to an indigestible monolithic
mess that is relatively difficult to change and parameterize for new needs.
I know, I wrote a couple of them:  the C batcher as distributed was not
our first attempt.  "Small-is-bootiful" was not our only motive here.

Um, would it be too much to suggest that perhaps your time (which, if it's
anything like mine, is in short supply) is more usefully spent fixing things
that are really broken...?
-- 
Those who do not understand Unix are |  Henry Spencer @ U of Toronto Zoology
condemned to reinvent it, poorly.    | {allegra,ihnp4,decvax,utai}!utzoo!henry