junk runneth over

vanpelt@unisv.UUCP (Mike Van Pelt) (03/16/88)

I've just started getting a news feed, and I've come across a problem
that's probably common to all new sites:  Newsgroups that are not
in the newslib/active file.  All those articles get sent to the junk
newsgroup, which soon has biilyons and biilyons of articles in it.
There doesn't seem to be a way provided to move them, so here's
the script I kludged up to do it.

====beginning of script====
#!/bin/sh
# Shell to create new newsgroups from the Newsgroups: line in every
# article in .../news/junk, remove everything from junk, and repost
# it all in the appropriate new newsgroups.
# 
# ** BUG WARNING ** Doesn't work if there is only one message
#                   in .../news/junk!  Check before running.
#
cd /usr/lib/news
mkdir landfill
cp /usr/spool/news/junk/* landfill
grep \^Newsgroups: landfill/* | cut -f3 -d: | sort | uniq >fixup1
ed - fixup1 <<'EOF'
   1,$s/,/\
 /g
	1,$s/^/inews -C /
	1,$s;$; </dev/null;
	w
	q
EOF
expire -n junk -e 0 -E 0
sh fixup1
/bin/ls landfill >fixup2
ed - fixup2 <<'EOF'
	1,$s;^;inews -x vsi1 -h -p landfill/;
	w
	q
EOF
sh fixup2 
====end of script====

Then after it looks like everything worked,

   rm -r -f landfill
  
The directories for news lib & spooling may need to be changed at your
site.  Also, in the second editor script, "vsi1" should be changed to the
site(s) from which you get your newsfeed.

Note that this script blindly and indescriminantly creates newsgroups
for any and all messages in the junk group.  This may or may not be
desireable.  If there's a better way, I'd love to hear about it!

-- 
       Mike Van Pelt     ..uunet!ubvax!unisv!vanpelt

vanpelt@unisv.UUCP (Mike Van Pelt) (03/18/88)

I found a bug in my previous posting of the 'unjunk' script.  If there
were no cross-posted articles in junk, the global replace for ',' with
newline would fail, and the rest of the ed script would not be executed.
(%$#%@!!)  Here's the corrected script.

====== begin script =======
#!/bin/sh
# Shell to create new newsgroups from the Newsgroups: line in every
# article in ../news/junk, remove everything from junk, and repost
# it all in the appropriate new newsgroups.
# 
# ** WARNING ** Doesn't create the group if there is only one message
#               in /news/junk.
#
cd /usr/lib/news
mkdir landfill
cp /usr/spool/news/junk/* landfill
echo "gag,retch" >fixup1
grep \^Newsgroups: landfill/* | cut -f3 -d: | sort | uniq >>fixup1
ed - fixup1 <<'EOF'
   1,$s/,/\
 /g
	1,2d
	1,$s/^/inews -C /
	1,$s;$; </dev/null;
	w
	q
EOF
expire -n junk -e 0 -E 0
sh fixup1
/bin/ls landfill >fixup2
ed - fixup2 <<'EOF'
	1,$s;^;inews -x vsi1 -h -p landfill/;
	w
	q
EOF
sh fixup2 
====== end script =======

Hope that takes care of it!
-- 
       Mike Van Pelt     ..uunet!ubvax!unisv!vanpelt

dlm@cuuxb.ATT.COM (~XT4111000~Dennis Mumaugh~C31~G18~6023~) (03/19/88)

In article <227@unisv.UUCP> vanpelt@unisv.UUCP (Mike Van Pelt) writes:
># Shell to create new newsgroups from the Newsgroups: line in every
># article in ../news/junk, remove everything from junk, and repost
># it all in the appropriate new newsgroups.
># 

The following is my version of a mvjunk.  It attempts to mv the
articles where they belong and also creat directories and newsgroups.
It will handle multiple newsgroups in a article but may make a second
posting.  It doesn't use inews for the posting since inews will reject
the article.  It doesn't update the active file but leaves a list
around for manual editing:

Use and abuse;  it's free, take it for what it's worth:

------ cut here ---------- cut here -------- cut here ---------- cut here ----
#! /bin/sh
# @(#)mvjunk.sh   (TRW Advanced Technology Facility) STA 2-Jun-1985
#
# Modified 10/22/85-James M. Scardelis, SA, Warner Computer Systems
#       to create newsgroups locally only.
#
# Move news files out of junk and into their proper directories,
# updating the active list and localgroups as we go.  Unknown newsgroups
# are created LOCALLY and the article installed.
#
# @(#)mvjunk.sh   UCS Wed Dec  9 19:07:55 CST 1987
# Modified by Dennis L. Mumaugh for local conditions and made totally
# unrecognizable
#
JUNK=/usenet/spool/junk
NEWSDIR=/usenet/spool
LIB=/usenet/lib	
BIN=/usenet/bin
ACTIVE=$LIB/active
PATH=$PATH:$LIB:$BIN
tmp1=/tmp/news.dirs
cd $JUNK
if test -z "`ls`" 
then 
echo nothing in $JUNK
exit
fi
for j in $JUNK/[01-9]*
do
	echo working on article $j
	for group in `egrep '^Newsgroups:' $j | head -1 | \
		sed 's;Newsgroups:[ 	];;' | \
		sed 's;,; ;g'`
	do
		if grep "$group " $ACTIVE
		then
			echo $group exists in active file!
		else
			echo Creating $group in active file.
			inews -C $group </dev/null
		fi
		GROUPDIR=`echo $group | sed 's;\.;\/;g'`
		cd $NEWSDIR
		if test -d $GROUPDIR ; then
			cd $GROUPDIR
			l=`ls -r|egrep "^[01-9]" | sort -r -n | head -1`
			if test -n "$l" ; then
					if test -f "$l" ; then
						article=`expr $l + 1`
					fi
			else
				article=1
			fi
		else
			newgroup $group
			article=1
		fi
		while test -f "$NEWSDIR/$GROUPDIR/$article" ; 
		do
			# echo directory $NEWSDIR/$GROUPDIR/$article exists!
			article=`expr $article + 1`
		done
		echo posting $j as $NEWSDIR/$GROUPDIR/$article
		ln $j $NEWSDIR/$GROUPDIR/$article
		echo $group: $article  >>$tmp1
	done
	echo removing $j
	/bin/rm -f $j		# But my MOTHER lives in Hackensack!
done
sort -u $tmp1 -o $tmp1
echo The file $tmp1 contains a list of articles and numbers:
echo Use it to edit $ACTIVE.
cat $tmp1
exit
------ cut here ---------- cut here -------- cut here ---------- cut here ----

-- 
=Dennis L. Mumaugh
 Lisle, IL       ...!{attunix,ihnp4,cbosgd,lll-crg}!cuuxb!dlm