stevo@elroy.jpl.nasa.gov (Steve Groom) (05/16/91)
Recently, there was a discussion, I think in this group, on how
excessive NNTP "group" commands generated by some news readers (NN?)
could cause heavy CPU utilization on the NNTP server. There
was some discussion of this and a few patches posted. Not having this
problem, I skimmed the material and moved on.
Well, this morning I discovered that as of very recently we DO have
this problem, and that the discussion has already expired here.
Could someone please forward any relevant information and patches?
Thanks.
-steve
--
Steve Groom, Jet Propulsion Laboratory, Pasadena, CA
stevo@elroy.jpl.nasa.gov {ames,usc}!elroy!stevo
"... and the babe, she needs a chaaaa--nging..." (apologies to Bob Dylan)kherron@ms.uky.edu (Kenneth Herron) (05/16/91)
stevo@elroy.jpl.nasa.gov (Steve Groom) writes: >Recently, there was a discussion, I think in this group, on how >excessive NNTP "group" commands generated by some news readers (NN?) >could cause heavy CPU utilization on the NNTP server. This discussion actually took place on the nntp-managers mailing list. I'm posting here since it is probably of wider interest. The nntp GROUP command causes the server to (a) validate the requested group against the site's group list and the client's list of restricted groups, (b) chdir() to the group's article directory, (c) scan the directory to determine what article numbers are present, and (d) send a message back to the client giving the information gleaned in c. This is a lot of work, obviously. I had two client sites, one a VMS machine running some as-yet-undetermined package, and the other a NeXT running something called NewsGrazer, which were issuing several GROUPs per second for the same group. Typically there were 100 or more GROUPs for the same group all in the space of 5-10 seconds. It appeared that the client was issuing one GROUP per (new?) article, just before getting the article's header, as a way to make sure it was still in the right group. The enclosed patch causes the nntp server to save the output of the last GROUP command, as well as the time it was requested. If a GROUP command comes in for the same group as the last GROUP, within two minutes, nntp simply reprints the data, skipping all disk activity. The patch was developed for nntp 1.5.11; I don't know how cleanly it will apply to earlier versions. Apply the patch to server/group.c, decide which of the two defines to keep, and remake nntpd. The patch was originally developed as a quick fix for an in-house problem; I never dreamed I'd be distributing it until I started receiving requests from other managers. It is not a steller example of programming, and I haven't cleaned it up for release because I haven't yet had time to (I'm particularly unhappy with the fixed-size buffers). The PC_SLOWDOWN option exists to prevent large bursts of network traffic from the now-optimized GROUP, and to subtly discourage poorly-written software. Take it out if you want. For those who want to see if it's working, optimized GROUPs will be logged with "groupx" instead of "group". Let me add that I have heard from the author of NewsGrazer. He is almost ready to release an updated version which supposedly optimizes GROUPs. -------------------------------- This is a patch to buffer the group command for poorly-written client packages -- KJH *** group.c.orig Wed May 8 12:49:02 1991 --- group.c Wed May 8 13:55:43 1991 *************** *** 14,19 * remains selected. */ group(argc, argv) int argc; char *argv[]; --- 14,40 ----- * remains selected. */ + /* Local change: If a client issues a group command more than once + * for the same group within a short time (2 minutes), just send the + * same info sent for the original group--don't do the chdir() or read + * the directory. + * Done to prevent some poorly-written clients from hogging resources + * -- Ken Herron, 8-May-1991 + */ + + /* Poor Client optimization -- KJH */ + #define PC_OPTIM + #define PC_TIMEOUT (60 * 2) + /* Teach 'em a lesson */ + #define PC_SLOWDOWN + + #ifdef PC_OPTIM + time_t time(); + time_t lg_time; /* time of last group command */ + char lg_name[80]; /* name of last group */ + char lg_data[100]; /* data returned for last group command */ + #endif + group(argc, argv) int argc; char *argv[]; *************** *** 42,47 return; } if (find_group(argv[1], num_groups, &low_msg, &high_msg) < 0) { printf("%d Invalid group name (not in active).\r\n", ERR_NOGROUP); --- 63,85 ----- return; } + #ifdef PC_OPTIM + if ((*argv[1] == *lg_name) && (!strncmp(argv[1], lg_name, 80)) + && ((time(NULL) - lg_time) < PC_TIMEOUT)) { + /* same group */ + #ifdef LOG + syslog(LOG_INFO, "%s groupx %s", hostname, argv[1]); + ++grps_acsd; + #endif + fputs(lg_data, stdout); + fflush(stdout); + #ifdef PC_SLOWDOWN + sleep(1); + #endif + return; + } + #endif + if (find_group(argv[1], num_groups, &low_msg, &high_msg) < 0) { printf("%d Invalid group name (not in active).\r\n", ERR_NOGROUP); *************** *** 100,105 while ((cp = index(argv[1], '/')) != (char *) NULL) *cp = '.'; printf("%d %d %d %d %s\r\n", OK_GROUP, num_arts, --- 138,154 ----- while ((cp = index(argv[1], '/')) != (char *) NULL) *cp = '.'; + #ifdef PC_OPTIM + lg_time = time(NULL); + (void)sprintf(lg_data, "%d %d %d %d %s\r\n", + OK_GROUP, + num_arts, + (num_arts > 0 ? art_array[0] : 0), + (num_arts > 0 ? art_array[num_arts-1] : 0), + argv[1]); + (void)fputs(lg_data, stdout); + (void)strncpy(lg_name, argv[1], 80); + #else printf("%d %d %d %d %s\r\n", OK_GROUP, num_arts, *************** *** 106,110 (num_arts > 0 ? art_array[0] : 0), (num_arts > 0 ? art_array[num_arts-1] : 0), argv[1]); (void) fflush(stdout); } --- 155,160 ----- (num_arts > 0 ? art_array[0] : 0), (num_arts > 0 ? art_array[num_arts-1] : 0), argv[1]); + #endif (void) fflush(stdout); } -- Kenneth Herron kherron@ms.uky.edu University of Kentucky +1 606 257 2975 Department of Mathematics "So this won't be a total loss, can you make it so guys get to throw their mothers-in-law in?" "Sure, why not?"