biocca@bevsun.bev.lbl.gov (Alan Biocca) (01/18/91)
I received email requesting my patch limiting the rate at which trn's
database builder 'mthreads' can read headers. Prior to installing this
we experienced massive nntp overload when building the database from
scratch. During this process trn reads every header of every message
of every newsgroup that the thread selection is enabled for. Running
mthreads on a Sun Sparcstation 1+ produced a pretty substantial network
load and the nntp server (a Sun 3/50) couldn't keep up with normal news
unbatching and finally died of unbatched news overflow.
The addition of a one second delay per header slows the initial database
construction significantly, but this is preferable to crashing the
server. Once the initial build is over we see 2-3 minutes of realtime
per update run with around 100 newsgroups threaded.
The patch is to mt-process.c, and consists of adding the one line
indicated below. It worked on both the original and the patched version of
trn.
mt-process.c:
85 if( caught_interrupt ) {
86 return;
87 }
88 #ifdef SERVER
>>> 89 sleep(1); /* akb slowdown 12/14/90 */
90 sprintf( buff, "HEAD %ld", (long)i );
91 put_server( buff );
92 if( get_server( buff, sizeof buff ) < 0 || *buff == CHAR_FATAL ) {
93 last_article = i - 1;
94 extra_expire = FALSE;
95 break;
96 }
97 if( *buff != CHAR_OK ) {
98 added_count--;
99 continue;
100 }
Alan K Biocca
WB6ZQZ