[net.news] net overload -- two proposals towards a solution

chuqui@nsc.UUCP (Chuq Von Rospach) (03/10/85)

A week or so ago, I posted an article about the problems we have been
seeing on the net, and tossed out a few ideas on how to go about cleaning
things up. Since that time I've gotten a lot of feedback, mostly through
private mail (fortunately...), that makes me believe the time has come to
seriously consider doing something about the growing problems happening on
the network. 

What follows should be considered a first draft towards finding a series of
steps we can get a consensus to agree upon and implement towards clearing
up some of the problems that we are seeing. These proposals are just that--
preliminary proposals that I want to get feedback from the net on. People
who read my first posting should notice some significant differences
between what I say here and what I said there, differences caused mostly by
the comments and feedback I got from the other people on the net who cared
about the problems enough to take the time to discuss it. 

There are a number of growing problems on the network. I won't pretend to
even try to solve all of them-- some of them are inherent to the
environment of Usenet, and 'fixing' them also means we don't have the same
'Usenet' any more-- it may well be better, but it won't be Usenet. The two
suggestions I have don't really change the things that make Usenet great--
the wide diversity of topics and the ability to discuss them freely.

I'm looking at reducing the so-called signal to noise ratio on the
network-- the number of articles on the net that are useful to people
against the total number of articles on the net. Right now, that number is
fearfully high in some groups. Since 'useful' changes from person to
person, I'll define noise very conservatively: articles posted to
inappropriate newsgroups, articles that duplicate information found in
other posted articles, and articles that improperly use the 'included text'
features of followups by not editing the text sufficiently. The first can
be shown as a request for software to net.sources, the second can be shown
by the 25 or thirty replies you can see to a title request in
net.sf-lovers, and the final can be shown by any article where someone
includes the entire previous article simply to add a one line riposte. I
think we can all agree that these are problems. Not the only ones, but
working on these are a good first step.

To help reduce these kinds of articles on the network, I am going to
suggest two major thrusts. Both are applicable to all usenet sites, both
have the advantage that the major parts of the implementation are
administrative and not technological, so we don't need to worry about sites
that don't upgrade (one can be made more effective with software), and they
can be implemented pretty much independently of whether a site is running A
news, 2.10.2 news, notes, or a persons private hacked up IBM PC. 

The two recommendations are developing a program to improve user education
and a significant restructuring of the topic names on the network. 

			Improving User Education

A big problem we have on the net is that the new user seems to be
essentially on his own. Most Usenet guideline seems to be by folklore-- the
existing news guru sits down with the news neophyte and spends 5 minutes
with the does and don'ts. If we're lucky. The documentation we do have is
terribly out of date, confusing, sometimes contradictory, and not always
available. 

I break this down into the following projects:

    o rewrite the documentation. A year ago I spearheaded a group effort to
    bring the 'Emily Post' document up to date. I looked at it critically
    recently, and I feel that it has aged well over the last year. the
    feedback I have gotten from it has been very positive, and the users
    seem to be helped by it. I think it is time to bring the rest of the
    documentation up to standard. Unless someone else wants to volunteer, I
    am willing to form a mailing list and spearhead an effort to get the
    documentation rewritten-- specifically the document 'how to read the
    network news' but also anything else we decide needs to be done. If you
    are interested in joining this effort, write me a note (in mail!) and
    tell me, and if there is enough interest I'll start the mailing list.

    o rewrite/update the man pages. Many of the manual pages for news
    programs are out of date, incomplete, or (in the case of 2.10.2) simply
    missing. The man pages, especially for the user programs such as
    readnews, vnews, rn, and postnews (and whatever is on the notes end as
    well) need to help the user work with the programs and make the chance
    of pilot error less. [No offense to larry wall, author of rn, but his
    man page is large enough to require a man page... *grin*].

    o improve the user interaction of the programs. None of the programs
    people use to work with the news is particularly helpful. The best of
    the lot is currently rn, which does have a help feature, but which
    doesn't give a lot of detail in the help-- it seems oriented towards
    reminding the guru instead of guiding the novice-- a common Unix
    occurance causing the also common novice screwup. Perhaps what is
    really needed is a 'novice mode' that can help keep newer users from
    making common mistakes and can be disabled when a person feels they can
    keep from making those mistakes on their own. This involves changing
    software, which means it won't be available to as many sites as quickly
    as some of the other solutions, but it can help reduce these problems
    on a long term basis.

    o make the documentation available. We need to find better ways of
    getting information to the user. As you'll see below, I'm suggesting
    that net.announce and net.announce.newusers be moved into the mod.all 
    groups (since they ARE moderated...). mod.newusers is a good first
    step, but I also think we need to look for more aggressive ways of
    making the information that will help users get to those users--
    suggestions here are quite welcome, since I'm not sure I do have any
    good suggestion...

		    Restructuring Newsgroup Topic Names

At the end of this article I've set up a list of what I think all of the
exiting topic names SHOULD be named. What I've done is try to put groups
that fit together logically with each other, try to set up general purpose
top level group names where appropriate, get rid of topics that duplicate
function or conflict with each other, and to try to remove some of the more
obviously misnamed groups and replace them with improved names. What I
HAVEN'T tried to do is get rid of groups, with a few exceptions.

For those that don't already know, I've been a strong supporter of removing
'obsolete' newsgroups-- in fact this whole discussion started because I
viewed removing unused groups as a way to reduce the complexity of the
naming space, thereby making it easier to post to the right place. What I
found, however, based on what people wrote to me in mail and what I found
when I actually started renaming all of the groups was that removing groups
simply didn't make it any better-- in fact in some cases it made it harder
to build what I considered to be a 'good' naming structure. When I analyzed
my complaints, I wasn't really complaining about obsolete groups, but about
a few groups with really badly chosen names (such as net.wobegon). The real
problem in the naming space was that a number of very specific groups were
created, but the general purpose topics that support those groups were
not-- net.wobegon isn't a bad group, but it should have been named
net.radio.wobegon, or net.radio.npr.wobegon to help explain what it is, and
to give a place for topics of more general interest in the same area to
have a home as well.

Traditionally it has been very difficult to create new groups on the net,
mainly because it is also almost impossible to get rid of them again (ask
anyone who lived through the 'wobegon wars' a while back). It is my
feeling that if we have a logical name space and a good choice of general
purpose top level topic names, this becomes a non-issue-- fewer groups will
need to be created because homes will exist for topics in the general
purpose groups, and when a topic IS created it helps to better define what
the general topic it is a subgroup of, so the reasons for getting rid of it
go away. In this light I suggest the following to the network:

    o implement whatever my current list of newsgroups metomorphosises
    into.

    o if a suggestion is made to create a new subtopic to an existing
    top level group, and there seems to be a reasonable amount of support
    for it (reasonable left undefined on purpose) then it is created.
    I don't think existing volume in another newsgroup should really be a
    requirement, but it always helps.

    o if a suggestion is made to create a new top level topic, or a
    subtopic that would hang off a top level topic that currently doesn't
    exist, they are created only if the people who care enough to argue
    about the group can come to the decision that it is named appropriately
    and that it fits within the Usenet naming space unambiguously.

    o subgroups are eligible for removal if they have had no traffic for
    six months AND someone is willing to try to get it removed AND he can
    get people to agree to it. Under no circumstances should we consider
    any form of automated sunset rule for groups, but we should keep the
    options open for getting rid of things that aren't being used that we
    don't think we'll need in the future.

What this means, of course, is more groups. If they are named right, more
groups can be a significant advantage. If they aren't, they make life
worse-- I personally only support the above if we get around to making the
naming space changes.

				Implementation

The implementation of the group renaming is all administrative. 2.10.2
allows us to set up news aliases and have inews rename things as they pass
through sites that support it, so that downstream sites see things posted
to an old name in the new group. We won't be able to get everyone to
support the changes quickly, but we can help remap articles using the
aliases. Trying to do a rearrangment of this severity at once would be
foolhardy, and fail miserably. I suggest we implement the change in phases
over a period of months, slowly bringing the net into a more logical form:

    o make changes in groups of about 10 topics at a time.

    o prior to the change, advertise the upcoming rename in both the group
    itself and mod.announce, and make sure everyone knows WHEN it will
    happen.

    o on the date that it happens, cooperating sites send out simultaneous
    newgroup messages designed to cover the net in a minimal amount of time
    (for example, ihnp4, nsc, seismo, gatech, decvax, mcvax, kaist, cbosgd,
    and someone in australia for a good start). Sites running 2.10.2 or
    newer software also install aliases to forward messages from the old
    group to the new name.

    o after a period of time (probably two weeks or so) rmgroups go out for
    the old group name, with appropriate advertising.

I firmly believe that these suggestions are workable in the current
environment, and my mail leads me to believe the network is ready to try
something of this magnitude to help make the use of the network easier and
better. I'm sure that there are places where the suggestions can be
improved, and I hope that we can all band together and turn this into the
best possible plan. And then DO IT. Comments are more than welcome, but
please send them through mail unless there is an overriding reason to send
them to the net-- I will go through everything I get (and probably discuss
it with you through mail) and excerpts and suggested improvements will show
up as needed, but I don't want to see us make the problem worse by endless
screaming about how to make it better.

chuq
===========
key to new naming space:

all 'net.' references are removed to keep this from being larger than it
already is, but should be assumed to be there. All top level groups have a
paragraph with a justification WHY it is top level, and optionally some
alternate names I considered for the top level. The group names in
parenthesis are the original name of the group in the existing naming space
so you can track down what it was. All editorial comments are surrounded in
brackets. I think this is a good first draft, lets make it better!

Note that this does NOT include regional groups, fa.all, or mod.all. Those
are outside the scope of this discussion, at least right now
==============
the NEW net.all (I am not a net, I am a free man!)
==============

blab
    [ American heritage dictionary: To reveal a secret, especially through
    unreserved talk

    This top level group is for topics that generate a lot of discussion,
    tend to be emotional, and are oriented towards debates on subjects
    between two or more opposing sides]

    other possibilites: edit[orial], disc[ussion], talk

blab.abortion (abortion)
blab.flame (flame)
blab.origins (origins)
blab.philosophy (philosophy)
blab.politics (politics)
blab.politics.theory (politics.theory)
blab.religion (religion)
blab.religion.christian (religion.christian)
blab.religion.jewish (religion.jewish)

consumers (consumers)
    [ no subgroups, but it doesn't fit into any other group well. Perhaps
      groups may attach to it someday, or perhaps someone can suggest where
      it DOES fit-- I don't want to force things and create a new
      ambiguity]

crit
    [criticism/discussion oriented newsgroups, mostly media oriented]

    other possibilities: combined with rec. review

crit.books (books)
crit.books.comics (comics)
crit.books.mag (mag)
crit.books.sf (sf-lovers) [also see crit.movies.sf and crit.tv.sf]
crit.movies (movies)
crit.movies.sf
crit.movies.sw (movies.sw)
crit.tv (tv)
crit.tv.sf
crit.tv.sf.drwho (tv.drwho)
crit.tv.soaps (tv.soaps)
crit.tv.startrek (startrek)

[sf-lovers is really three groups rolled into one, with audiences that
don't quite overlap. The book oriented people get tired of hearing about
movies, the movie oriented people don't care about David Brin, and the
group gets a LOT of traffic, anyway. Splitting this way is a possibility,
but the ARPA connection gets in the way-- maybe crit.sf-lovers and
crit.movies.sf or something is better. comments?]

culture
    [ discussions of cultures and peoples around the world-- cultural
    oriented groups

    I'm not sure I like the group name (MUCH better for the purpose than
    nlang, but I don't think it is the best)

    other possibilities: people soc[ieties]

culture.celts (nlang.celts)
culture.greek (nlang.greek)
culture.india (nlang.india)
culture.poems (poems) [not a great fit]
culture.roots (roots) [not a great fit]

games (games)
    [games both computer oriented and others]

games.emp (games.emp)
games.frp (games.frp)
games.go (games.go)
games.hack (games.hack)
games.pbm (games.pbm)
games.rogue (games.rogue)
games.trivia (games.trivia)
games.video (games.video)

hw
    [computer hardware oriented discussions]

    other possibilities: hard

hw.arch (arch)
hw.arch.works (works) [workstations are both a hardware and software
    issue, and really deal with computer architectures in a specific case]
hw.dcom (dcom)
hw.info-terms (info-terms)
hw.lan (lan)
hw.periphs (periphs)
hw.lsi (lsi)

jobs
    [another lonely top level, because it doesn't fit well elsewhere-- 
     wanted.jobs was considered, but I think it implies resumes more
     than job descriptions ('I want a job', not 'I want a person to fill a
     job opening')]

legal
    [and another lonely top level, one with possibilities-- how about 
    legal.sw, for instance?]

micro (micro)
    [microcomputer based system discussions]

micro.32k (micro.16k)
micro.432 (micro.432)
micro.6809 (micro.6809)
micro.68k (micro.68k)
micro.apple (micro.apple)
micro.atari (micro.atari)
micro.cbm (micro.cbm)
micro.cpm (micro.cpm)
micro.hp (micro.hp)
micro.mac (micro.mac)
micro.pc (micro.pc)
micro.ti (micro.ti)
micro.trs-80 (micro.trs-80)
micro.zx (micro.zx)

misc (misc)
    [the place where everything that doesn't fit in any other group, top
    level or otherwise.]

money
    [ financial discussions]

money.invest (invest)
money.taxes (taxes)

news (news)
    [netnews administration groups-- for the system administrator to
     worry about, mostly]

news.adm (news.adm)
news.b (news.b)
news.config (news.config)
news.group (news.group)
news.mail (mail)
news.mail.headers (mail.headers)
news.mail.msggroup (mail.msggroup)
news.net-people (net-people)
news.newsite (news.newsite)
news.notes (notes)
news.sa (news.sa)
news.stargate (news.stargate)

orgs
    [groups and discussions about organizations]

    other possibilities: groups organ[izations]

orgs.college (college) [not a great fit]
orgs.decus (decus)
orgs.usenix (usenix)

people (social)
    [discussions oriented towards people and how they deal with life]

people.kids (kids)
people.motss (motss)
people.singles (singles)
people.women (women)

plang (lang)
    [programming languages]

plang.ada (lang.ada)
plang.apl (lang.apl)
plang.c (lang.c)
plang.f77 (lang.f77)
plang.forth (lang.forth)
plang.lisp (lang.lisp)
plang.mod2 (lang.mod2)
plang.pascal (lang.pascal)
plang.prolog (lang.prolog)
plang.st80 (lang.st80)

rec (rec)
    [recreations and hobbies]

rec.auto (auto)
rec.aviation (aviation)
rec.bicycle (bicycle)
rec.birds (rec.birds)
rec.boat (rec.boat)
rec.bridge (rec.bridge)
rec.chess (chess)
rec.coins (rec.coins)
rec.cooks (cooks)
rec.cooks.veg (veg)
rec.cooks.wines (wines)
rec.disc (rec.disc)
rec.garden (garden)
rec.ham-radio (ham-radio)
rec.jokes (jokes)
rec.jokes.d (jokes.d)
rec.mcycle (cycle)
rec.music (music)
rec.music.classical (music.classical)
rec.music.folk (music.folk)
rec.music.synth (music.synth)
rec.nude (rec.nude)
rec.pets (pets)
rec.photo (rec.photo)
rec.puzzle (puzzle)
rec.railroad (railroad)
rec.scuba (rec.scuba)
rec.ski (rec.ski)
rec.skydive (rec.skydive)
rec.sport (sport)
rec.sport.baseball (sport.baseball)
rec.sport.football (sport.football)
rec.sport.hockey (sport.hockey)
rec.sport.hoops (sport.hoops)
rec.theater (theater)
rec.travel (travel)
rec.wood (rec.wood)

sci (sci)
    [science oriented topics and discussions]

sci.astro (astro)
sci.astro.expert (astro.expert)
sci.bio (bio)
sci.crypt (crypt)
sci.math (math)
sci.math.stat (math.stat)
sci.math.symbolic (math.symbolic)
sci.med (med)
sci.physics (physics)
sci.space (space)
sci.space.columbia (columbia)

sources (sources)
    [programs and computer software]

    other possibilities: src source

sources.bugs (sources.bugs)
sources.games (sources.games)
sources.mac (sources.mac)

std
    [ another lonely top level group, but standards discussions are
      important enough to give a home to-- we ought to use it, too!]

sw
    [software oriented discussions]

    other possibilities: soft

sw.ai (ai)
sw.cog-eng (cog-eng)
sw.emacs (emacs)
sw.eunice (eunice)
sw.graphics (graphics)
sw.text (text)

tech
    [discussions of non-computer technology]

tech.analog (analog)
tech.audio (audio)
tech.video (video)

unix (unix)
    [unix oriented subjects]

unix.bugs (bugs)
unix.bugs.2bsd (bugs.2bsd)
unix.bugs.4bsd (bugs.4bsd)
unix.bugs.usg (bugs.usg)
unix.bugs.uucp (bugs.uucp)
unix.bugs.v7 (bugs.v7)
unix.wizards (unix-wizards)

wanted (wanted)
    [classified ads]

wanted.forsale (forsale)
wanted.sources (wanted.sources)

-----------------------------------------------------------
Groups that didn't translate into the above naming space. To help people
track down what isn't SOMEWHERE up there, here is a list of all of the
current net.all topics that I didn't translate into the new naming space
and why. 
----------------------------------------------------------

I can't decide where to put these, or whether to get rid of them.
suggestions are welcome.

    cse
    nlang
    research
    rumor
    suicide

Moved somewhere else

    announce -- moved to mod.annouce
    announce.newusers -- moved to mod.newusers

Removed from the network

    followup  -- gone completely
    general -- gone completely
	[ these groups are obsolete-- articles should either be in
	  a specific group, sent to mod.announce, or posted at last
	  resort to net.misc]

    test -- gone completely
	[ useless, except under exceptionally special circumstances, for
	  which mod.test should probably be used]

    usoft -- gone completely
	[ absorbed into sw, I guess (find a good definition for this group!)]

    wobegon -- gone completely
	[ rec.music.folk takes a good part of the reason for this to exist.
	  if it had any volume (it doesn't) I'd suggest rec.radio or 
	  rec.radio.npr]

    women.only --- gone completely
	[failed experiment-- taken over by mail.feminist and women]


-- 
Chuq Von Rospach, National Semiconductor
{cbosgd,fortune,hplabs,ihnp4,seismo}!nsc!chuqui   nsc!chuqui@decwrl.ARPA

Be seeing you!