[news.software.b] Problems with duplicate messages

david@ms.uky.edu (David Herron -- One of the vertebrae) (11/20/88)

Bleah.

We're having problems here, and I don't see what could be causing
the problem.  What's happening is that there are *lots* of duplicate
messages on our system each coming from a different neighbor but
always with the same Message ID.  This is very different from
the normal problem where some site out there munges article IDs
causing duplicate messages.

The software is at v2.11.14, with no local changes, or at least none
of any significance.  We also run nntp v1.5, but without any of the
patches either official or unofficial.  This is all on a Sequent
Symmetry running the latest version of Dynix.  The system has
been running on our Sequent w/o problems until recently....

Something which has been happening is that occasionally the file
system fills up, so I kill of the nntpd's and do

	find /net/spool.news -type f -size 0 -exec rm {} \;
	expire -r
	expire -e 24
	/usr/etc/nntpd

to restart the system.  At least once when this happened another person
here did the actual work of killing nntpd's and cleaning up the system.
He'd have done without the find and expire -r, and just gone with
the other expire.  When he did this the history file disappeared,
I know it did because there was a few pieces of mail saying that it
was gone but that corrective actions fixed it.  So when I saw this
I stopped all news stuff and did expire -r and so forth.  But I haven't
yet done anything to remove the duplicate files that are in the filesystem,
so there are some duplicate entries in the history file.

So first off.  Why did the history file disappear?  And why did it disappear
more than once.

Will removing the duplicate files & rebuilding the database fix things
again so that we're not getting more duplicates?  In other words, is
the software getting confused when it receives back multiple records
from the history file?

There has been some recent noise (er.. messages) along these lines.
One message from Erik Fair saying that if you had some duplicate
entries in history that you were in trouble.  (He gave a short script
for checking, and didn't say anything beyond that).  The other from
someone asking why his history file was disappearing.  (btw, Erik's
message was on nntp-managers a couple of months ago, so maybe I shouldn't
call it recent?).
-- 
<-- David Herron; an MMDF guy                              <david@ms.uky.edu>
<-- ska: David le casse\*'      {rutgers,uunet}!ukma!david, david@UKMA.BITNET
<--
<-- Controlled anarchy -- the essence of the net.