dsill@ark1.nswc.navy.mil (Dave Sill) (11/18/89)
Well, it's been a helluva week.
In the beginning, I was running B News 2.10 and NNTP 1.5, and for the
most part everything worked as expected. But occasionally we'd get an
article posted invalidly to a moderated group which would cause ark1
to try to respond to the poster by mail. For several days. Hundreds
of attempts. I eventually posted a query here and learned how to stop
this condition. So I said to myself, this must be a result of running
such an old version of News; it's time to switch to B News 2.11.17
(which was handy).
The upgrade to 2.11.17 was fairly uneventful. Too uneventful, in
fact, because the aforementioned problem remained. I also started
getting *lots* of duplicate articles. Again, I decided that
upgrading, to 2.11.19, was the solution to my problems. I fetched the
patches from tut, installed them, and built and installed the new
version. Yes, I ran `make upgrade'.
This time, the upgrade was *not* uneventful. I immediately noticed
trouble with the active file. Some piece of software didn't seem to
understand that the min/max fields were widened, and I had article
sequences like 345, 346, 34746, 34846 ... 2, 3, 4 ... I quickly pulled
the plug on news by disabling nntpd, but the damage was done.
I tried manually patching the active file and reinstalling the
software, but that didn't work. I would have killed for a utility
that could fix my active file or remove the thousands of duplicate
articles.
Well, by now I was desparate. I contemplated wiping out
/usr/spool/news and starting from scratch. I had two weeks of
articles there, and I didn't want to do that. I decided to try
backing down to 2.11.17. This was not a wise thing to do. In the
process of reversing patches 19 and 18, something got botched. Faced
with the prospect of fetching 2.11.?? and installing N patches, I
decided instead that now might be a good time to try C News. I mean,
things couldn't get worse.
After installing the patches up to Sep14 and reading the docs, I built
and installed C News, and NNTP 1.5.7 since it contained the mods
necessary for batching with C News. Before I plugged it in, though, I
wanted to get my active file and history squared away. Hmmm...
`mkhistory' ... that's just the ticket. After cranking away for 2-3
hours it bombed out when awk bombed out saying "11034@foo.bar... line
too long". It turned out this was one of the articles that I'd
received 50-100 duplicates of, and histdups couldn't handle it.
Okay, well maybe `recovact' would take care of the active file, at
least. Nope, those bogus 5-digit articles would have to go first, as
well as the bogus 2, 3, 4... articles. It only took two hours or so
to dired /usr/spool/news and remove all the bogus articles. I ran
recovact and it seemed to fix the max field, though the min field was
wrong where bogus 2, 3, 4... articles had been created. Oh well, too
bad.
With a reasonable active file, reasonable article numbers, and no
history, I fired up C News and it seemed to work okay.
The next day, I noticed that /usr/spool was 97% full (it usually runs
60-70% with 14 days) so I tried to run expire, via doexpire. It
exited after a few seconds, so I checked errlog. Nothing. It's still
empty. Doesn't C News use errlog? I checked for mail (I have it sent
to `news') and found a message that said "expire problems". Not
terribly informative, that. It seems to be running okay now, I don't
know what the problem was, though. But since the old articles weren't
in history, I ended up removing them using `find ... -mtime +3 ...'.
While I was checking news' mail, I noticed a bunch of messages of the
form:
From ??? Fri Nov 17 10:44:00 1989
Date: Fri, 17 Nov 89 10:43:55 EST
From: root
To: news
ark1 relaynews `bad/627320593' failed, status 1
Another less-than-totally-helpful message. I haven't been able to
figure out what's wrong with these batches or how to get rid of them.
Anyone else know? Sometimes the status is 1005.
I'm not sure I like the way C News' locking works. I've had several
occurrences of processes getting zapped, hung, or suspended, and the
lock files don't always go away. I don't know how that could be
avoided, but B News didn't have that problem.
I was also surprised at first to see articles appearing under soc,
talk, and alt. Until I looked at the Newsgroups: lines I thought C
News was ignoring my sys file, but these articles were cross-posted.
Is there any way to prevent this behavior?
With the exceptions of a growing /usr/spool/news/in.coming/bad
directory and a growing "news" mailbox, everything seems to be running
fine, at last.
Oh, did I forget to mention that what little history I had was wiped
out when I installed the Nov13 patches? What's a few more
duplicates...
BTW, what do I have to do to get NNTP to log? I have the following in
my conf.h:
#define FAKESYSLOG "/usr/lib/news/nntplog"
#define FAKEAPPEND
#define SYSLOG LOG_LOCAL7
#ifdef SYSLOG /* Define LOG if you want copious logging info */
#undef LOG /* undef it if you don't */
#endif /* but you can only have LOG if you have SYSLOG */
Any help would be appreciated.
Configuration: 4.3bsd on VAX 780
C News Nov13
NNTP 1.5.7
Dave Sill (dsill@relay.nswc.navy.mil)dsill@ark1.nswc.navy.mil (Dave Sill) (11/21/89)
Second try since nobody responded. I'd really like to know what the problem is with the bad batches; I'm still hetting error messages by mail. In article <1989Nov17.162448.22026@relay.nswc.navy.mil>, dsill@ark1.nswc.navy.mil (Dave Sill) writes: > Well, it's been a helluva week. > > In the beginning, I was running B News 2.10 and NNTP 1.5, and for the > most part everything worked as expected. But occasionally we'd get an > article posted invalidly to a moderated group which would cause ark1 > to try to respond to the poster by mail. For several days. Hundreds > of attempts. I eventually posted a query here and learned how to stop > this condition. So I said to myself, this must be a result of running > such an old version of News; it's time to switch to B News 2.11.17 > (which was handy). > > The upgrade to 2.11.17 was fairly uneventful. Too uneventful, in > fact, because the aforementioned problem remained. I also started > getting *lots* of duplicate articles. Again, I decided that > upgrading, to 2.11.19, was the solution to my problems. I fetched the > patches from tut, installed them, and built and installed the new > version. Yes, I ran `make upgrade'. > > This time, the upgrade was *not* uneventful. I immediately noticed > trouble with the active file. Some piece of software didn't seem to > understand that the min/max fields were widened, and I had article > sequences like 345, 346, 34746, 34846 ... 2, 3, 4 ... I quickly pulled > the plug on news by disabling nntpd, but the damage was done. > > I tried manually patching the active file and reinstalling the > software, but that didn't work. I would have killed for a utility > that could fix my active file or remove the thousands of duplicate > articles. > > Well, by now I was desparate. I contemplated wiping out > /usr/spool/news and starting from scratch. I had two weeks of > articles there, and I didn't want to do that. I decided to try > backing down to 2.11.17. This was not a wise thing to do. In the > process of reversing patches 19 and 18, something got botched. Faced > with the prospect of fetching 2.11.?? and installing N patches, I > decided instead that now might be a good time to try C News. I mean, > things couldn't get worse. > > After installing the patches up to Sep14 and reading the docs, I built > and installed C News, and NNTP 1.5.7 since it contained the mods > necessary for batching with C News. Before I plugged it in, though, I > wanted to get my active file and history squared away. Hmmm... > `mkhistory' ... that's just the ticket. After cranking away for 2-3 > hours it bombed out when awk bombed out saying "11034@foo.bar... line > too long". It turned out this was one of the articles that I'd > received 50-100 duplicates of, and histdups couldn't handle it. > > Okay, well maybe `recovact' would take care of the active file, at > least. Nope, those bogus 5-digit articles would have to go first, as > well as the bogus 2, 3, 4... articles. It only took two hours or so > to dired /usr/spool/news and remove all the bogus articles. I ran > recovact and it seemed to fix the max field, though the min field was > wrong where bogus 2, 3, 4... articles had been created. Oh well, too > bad. > > With a reasonable active file, reasonable article numbers, and no > history, I fired up C News and it seemed to work okay. > > The next day, I noticed that /usr/spool was 97% full (it usually runs > 60-70% with 14 days) so I tried to run expire, via doexpire. It > exited after a few seconds, so I checked errlog. Nothing. It's still > empty. Doesn't C News use errlog? I checked for mail (I have it sent > to `news') and found a message that said "expire problems". Not > terribly informative, that. It seems to be running okay now, I don't > know what the problem was, though. But since the old articles weren't > in history, I ended up removing them using `find ... -mtime +3 ...'. > > While I was checking news' mail, I noticed a bunch of messages of the > form: > > From ??? Fri Nov 17 10:44:00 1989 > Date: Fri, 17 Nov 89 10:43:55 EST > From: root > To: news > > ark1 relaynews `bad/627320593' failed, status 1 > > Another less-than-totally-helpful message. I haven't been able to > figure out what's wrong with these batches or how to get rid of them. > Anyone else know? Sometimes the status is 1005. > > I'm not sure I like the way C News' locking works. I've had several > occurrences of processes getting zapped, hung, or suspended, and the > lock files don't always go away. I don't know how that could be > avoided, but B News didn't have that problem. > > I was also surprised at first to see articles appearing under soc, > talk, and alt. Until I looked at the Newsgroups: lines I thought C > News was ignoring my sys file, but these articles were cross-posted. > Is there any way to prevent this behavior? > > With the exceptions of a growing /usr/spool/news/in.coming/bad > directory and a growing "news" mailbox, everything seems to be running > fine, at last. > > Oh, did I forget to mention that what little history I had was wiped > out when I installed the Nov13 patches? What's a few more > duplicates... > > BTW, what do I have to do to get NNTP to log? I have the following in > my conf.h: > > #define FAKESYSLOG "/usr/lib/news/nntplog" > #define FAKEAPPEND > > #define SYSLOG LOG_LOCAL7 > > #ifdef SYSLOG /* Define LOG if you want copious logging info */ > #undef LOG /* undef it if you don't */ > #endif /* but you can only have LOG if you have SYSLOG */ > > Any help would be appreciated. > > Configuration: 4.3bsd on VAX 780 > C News Nov13 > NNTP 1.5.7 > > Dave Sill (dsill@relay.nswc.navy.mil) Dave Sill (dsill@relay.nswc.navy.mil)
tale@pawl.rpi.edu (David C Lawrence) (11/22/89)
Dave Sill: > While I was checking news' mail, I noticed a bunch of messages of the > form: > > From ??? Fri Nov 17 10:44:00 1989 > Date: Fri, 17 Nov 89 10:43:55 EST > From: root A root run news system even though news exists? Or is it just a mail alias? > To: news > > ark1 relaynews `bad/627320593' failed, status 1 > > Another less-than-totally-helpful message. I haven't been able to > figure out what's wrong with these batches or how to get rid of them. > Anyone else know? Sometimes the status is 1005. The only time I ever seen error codes from relaynews was when $NEWSARTS/out.going/$TOHOST didn't exist as a directory for an outbound NNTP feed. I think this was returned as "16". [If it's not going to make the directory, could it at least give a little better message about what problem it encountered?] This is besides a couple of times I had to clear blockage because my sys file was calling a script that called relaynews ... duh. More on that later. The status information as known by relaynews is defined in cnews/include/news.h. The bits are OR'ed to provide the exit status, I think, and since no status bit is one, nor could they really get up to 1005, I am a little curious about the answer to this too. Dave -- (setq mail '("tale@pawl.rpi.edu" "tale@ai.mit.edu" "tale@rpitsmts.bitnet"))
geoff@utstat.uucp (Geoff Collyer) (11/22/89)
David C Lawrence: > [If it's not going to make the directory, could it at least give a > little better message about what problem it encountered?] If you are running an up-to-date C news (Nov 13, though the relevant fix was earlier), you always get an error message in $NEWSCTL/errlog when relaynews exits with non-zero status, unless it encountered an error (usually due to memory exhaustion in prelude()) before it could establish that the standard file descriptors were open, or it was killed by a signal. Decoding the bit patterns of relaynews's exit status is not required. We are preparing a list of "commonly-encountered problems". The number one problem people have with relaynews is that their $NEWSCTL/bin/config doesn't match their libcnews/config.c or they have environment variables set to override those values. This is normal during testing or some forms of parallel operation, but it does cause a loss of setuid-privileges to avoid spoofing, so you have to be prepared to compensate, perhaps by fiddling permissions on files owned by "news" or equivalent. -- Geoff Collyer utzoo!utstat!geoff, geoff@utstat.toronto.edu
lamy@cs.toronto.edu (Jean-Francois Lamy) (11/23/89)
geoff@utstat.uucp (Geoff Collyer) writes: >If you are running an up-to-date C news (Nov 13, though the relevant >fix was earlier), you always get an error message in $NEWSCTL/errlog ^^^^^ ^^^^^^^^ Boy, don't I wish there was a supported way to get the logs in the standard place where logs go instead of the control data area... I (nai"vely) think there should be wide agreement for two more magic variables (based on the fact that those are the only two I've seen mentioned with any regularity, in fact I don't remember anyone asking for any other, though I'm sure Henry and Geoff have). $NEWSLOG where to put the logs $NEWSPOOL where to put the incoming/outgoing news spool. I don't think that the spool area belongs logically with article storage, histo(e?)rical reasons notwithstanding. Both of these items seem to be logically distinct part of news processing from control data (NEWSCTL), and from article *storage* (NEWARTS). But whereas handling NEWSSPOOL as a separate area is easy because you can just make two symlinks and forget about it, getting logs to go somewhere standard requires fiddling with the source and is therefore messier. Yah, I even know what the answer is going to be :-( Maybe I should start a petition? Jean-Francois Lamy lamy@ai.utoronto.ca, uunet!ai.utoronto.ca!lamy AI Group, Department of Computer Science, University of Toronto, Canada M5S 1A4
henry@utzoo.uucp (Henry Spencer) (11/23/89)
In article <1989Nov17.162448.22026@relay.nswc.navy.mil> Dave Sill <dsill@relay.nswc.navy.mil> writes: >`mkhistory' ... that's just the ticket. After cranking away for 2-3 >hours it bombed out when awk bombed out saying "11034@foo.bar... line >too long". It turned out this was one of the articles that I'd >received 50-100 duplicates of, and histdups couldn't handle it. Sigh. It's very difficult to deal with arbitrary-length lines unless you're willing to write all your own code and ignore existing tools. The parts of C News that are written in C are pretty much free of limits on line length, but awk and such do have trouble in extreme cases. >The next day, I noticed that /usr/spool was 97% full (it usually runs >60-70% with 14 days) so I tried to run expire, via doexpire. It >exited after a few seconds, so I checked errlog. Nothing. It's still >empty. Doesn't C News use errlog? Relaynews does. Several of the other programs report trouble by mail rather than via errlog. I'm not entirely sure this is the right approach, but it's what's currently done. Reporting via mail does have the advantage that it never runs into permission problems. >I checked for mail (I have it sent >to `news') and found a message that said "expire problems". Not >terribly informative, that... Unless something is severely wrong in your system, it should have been accompanied by a more specific complaint. That message never gets sent (by doexpire) unless there is something else to send. Could your mailer be fouling things up? >While I was checking news' mail, I noticed a bunch of messages of the >form: >... > ark1 relaynews `bad/627320593' failed, status 1 > >Another less-than-totally-helpful message... Unless you are running an out-of-date C News or have serious problems with file ownerships/permissions, such a report should always be accompanied by more specific messages in errlog. >... I haven't been able to >figure out what's wrong with these batches or how to get rid of them. First you need to track down a more specific problem report; then you need to take that and go look at the batches (in in.coming/bad) and find out what's wrong with them. >I'm not sure I like the way C News' locking works. I've had several >occurrences of processes getting zapped, hung, or suspended, and the >lock files don't always go away. I don't know how that could be >avoided, but B News didn't have that problem. It paid the price in other problems. There is *no* portable locking method that dependably removes locks when a process drops dead. And the failure of a locking mechanism is usually a symptom of more severe problems, generally justifying human attention rather than blind removal of the lock. >I was also surprised at first to see articles appearing under soc, >talk, and alt. Until I looked at the Newsgroups: lines I thought C >News was ignoring my sys file, but these articles were cross-posted. >Is there any way to prevent this behavior? Is there a reason why these groups are in your active file when you don't want to receive them? Take them out and C News will stop filing articles under them. Or change the fourth field of the active-file lines to "x" and C News won't even put the articles in "junk". -- A bit of tolerance is worth a | Henry Spencer at U of Toronto Zoology megabyte of flaming. | uunet!attcan!utzoo!henry henry@zoo.toronto.edu
tale@pawl.rpi.edu (David C Lawrence) (11/23/89)
In <89Nov22.130142est.2899@neat.cs.toronto.edu> lamy@cs.toronto.edu
(Jean-Francois Lamy) writes:
JF> Boy, don't I wish there was a supported way to get the logs in the
JF> standard place where logs go instead of the control data area...
I changed this for us pretty easily. There were only two changes I
made because I still wanted logs in NEWSCTL, just one directory
deeper. Once that was done I could put them in /var/log by using a
symlink from NEWSCTL/logs, but I hate having symlinks hanging around.
It also brings up still another edit so that "errlog" and "log" are
slightly better named regarding what they are (err)logging.
The two edits, by the way, are both in relaynews.c in the
redirectlogs() function.
JF> Both of these items seem to be logically distinct part of news
JF> processing from control data (NEWSCTL), and from article *storage*
JF> (NEWARTS). But whereas handling NEWSSPOOL as a separate area is
JF> easy because you can just make two symlinks and forget about it,
JF> getting logs to go somewhere standard requires fiddling with the
JF> source and is therefore messier.
Mmm. If I ran newsdaily here I think it would need an edit to point
to the log files too. I like the arrangement I have now; /usenet is
basically stand-alone on the system, except for other utilities it
calls from /bin. Everything related solely to news, from spooling to
log files to control files and programmes, et al, is found below it.
I can see why other sites would want to push spooling, for example, to
another partition though.
JF> Yah, I even know what the answer is going to be :-( Maybe I should start a
JF> petition?
I'd sign. Flexibility is good. Checking the environment for two more
variables won't make relaynews become an resource hog.
Dave
--
(setq mail '("tale@pawl.rpi.edu" "tale@ai.mit.edu" "tale@rpitsmts.bitnet"))geoff@utstat.uucp (Geoff Collyer) (11/23/89)
> Maybe I should start a petition?
Save your energy. We are not government bureaucrats and we maliciously
ignore petitions. We listen to reasoned arguments. If your arguments
don't convince us, a petition is guaranteed to fail.
--
Geoff Collyer utzoo!utstat!geoff, geoff@utstat.toronto.eduhenry@utzoo.uucp (Henry Spencer) (11/24/89)
In article <89Nov22.130142est.2899@neat.cs.toronto.edu> lamy@cs.toronto.edu (Jean-Francois Lamy) writes: >Boy, don't I wish there was a supported way to get the logs in the standard >place where logs go instead of the control data area... I (nai"vely) think >there should be wide agreement for two more magic variables... Actually, we think there are too many already, although we don't see any easy way to cut down the numbers. There is no limit to how finely some people want to slice things; there comes a time when we simply have to say "sorry, that's your problem, not ours". Obviously, it's a judgement call just when that time comes. One rule of thumb we use is that if people could grit their teeth and live with something under B News, they can probably grit their teeth and live with it under C News. Frankly, we don't see a good technical justification for the split between NEWSCTL and NEWSARTS. They are logically both part of the news database on your system. (There *is* a good reason for the NEWSCTL/NEWSBIN split, since it is not uncommon to have binaries shared among several machines, or possibly even different sets of binaries accessing the same database.) However, there are compelling compatibility reasons for the split, i.e. we don't want to fix all the newsreaders that know about "/usr/lib/news" and "/usr/spool/news", and not all systems have symbolic links. >$NEWSPOOL where to put the incoming/outgoing news spool. I don't think > that the spool area belongs logically with article storage, > histo(e?)rical reasons notwithstanding. This one is a big trickier. The spool areas *aren't* logically part of the database in any strict sense. However, we didn't want to add yet another variable without grave need, and the sort of space reserves needed to handle fluctuating news flow are also reasonably apropos to the problem of fluctuating spooled volume. In the end we invoked the "it was good enough for them under B News" rule and declined to increase the system's complexity for it. -- That's not a joke, that's | Henry Spencer at U of Toronto Zoology NASA. -Nick Szabo | uunet!attcan!utzoo!henry henry@zoo.toronto.edu
henry@utzoo.uucp (Henry Spencer) (11/24/89)
In article <1989Nov23.070330.11698@utstat.uucp> geoff@utstat.uucp (Geoff Collyer) writes: >> Maybe I should start a petition? > >Save your energy. We are not government bureaucrats and we maliciously >ignore petitions. We listen to reasoned arguments. If your arguments >don't convince us, a petition is guaranteed to fail. While Geoff is correct, I should mention one borderline case: when a possible change has been shelved because we perceive a lack of interest, rather than discarded because we've decided against it, indications of widespread interest can make a difference. This particular item wasn't such a case, however. -- That's not a joke, that's | Henry Spencer at U of Toronto Zoology NASA. -Nick Szabo | uunet!attcan!utzoo!henry henry@zoo.toronto.edu
coolidge@brutus.cs.uiuc.edu (John Coolidge) (11/24/89)
henry@utzoo.uucp (Henry Spencer) writes: >Frankly, we don't see a good technical justification for the split between >NEWSCTL and NEWSARTS. They are logically both part of the news database >on your system. (There *is* a good reason for the NEWSCTL/NEWSBIN split, >since it is not uncommon to have binaries shared among several machines, >or possibly even different sets of binaries accessing the same database.) >However, there are compelling compatibility reasons for the split, i.e. we >don't want to fix all the newsreaders that know about "/usr/lib/news" and >"/usr/spool/news", and not all systems have symbolic links. One reason, perhaps spurious (but it's a problem for me, for other reasons), is because of things like cd $NEWSARTS find . (or find $NEWSARTS) that get used in a few places (most noticably mkhistory). Perhaps $NEWSCTL wouldn't break this, but other things do. For instance, I keep nn's database in /usr/spool/news/nn, since it's easier to export it around that way. I may move it to /usr/local/lib/news/nn, since /usr/local/lib/news is now its own partition, but for the moment it's in $NEWSARTS --- and that breaks mkhistory, since a lot of the nn files start with numbers. My solution was to just explicitly list all top-level categories in mkhistory --- ugly, but it fixes the problem. In.coming and out.going cause the same problem --- they're in the same file system as all of the articles (perhaps these can be moved (?) --- I haven't looked the configuration stuff recently). --John -------------------------------------------------------------------------- John L. Coolidge Internet:coolidge@cs.uiuc.edu UUCP:uiucdcs!coolidge Of course I don't speak for the U of I (or anyone else except myself) Copyright 1989 John L. Coolidge. Copying allowed if (and only if) attributed. You may redistribute this article if and only if your recipients may as well. New NNTP connections always available! Send mail if you're interested.
henry@utzoo.uucp (Henry Spencer) (11/25/89)
In article <1989Nov24.062333.4784@brutus.cs.uiuc.edu> coolidge@cs.uiuc.edu writes: > cd $NEWSARTS > find . >(or find $NEWSARTS) that get used in a few places (most noticably >mkhistory)... I keep nn's database in /usr/spool/news/nn... that >breaks mkhistory, since a lot of the nn files start with numbers. >My solution was to just explicitly list all top-level categories... If you've got a modern C News, nothing does "find ." in NEWSARTS; what gets done (taking mkhistory as an example) is: find `ls | egrep -v '\.'` This omits all names with dots in them, which is precisely the set of names which *cannot* be real newsgroup directories. Try naming your nn database directory nn.dbase or something like that instead, and the problem goes away *without* kludges like explicitly listing top-level groups. That's why in.coming and out.going have dots in their names. -- That's not a joke, that's | Henry Spencer at U of Toronto Zoology NASA. -Nick Szabo | uunet!attcan!utzoo!henry henry@zoo.toronto.edu