[net.news] A Thought on Newsgroup Structure

mcq@druxt.UUCP (McQueerRL) (01/28/84)

[]----

In reading suggestions regarding new groups and watching discussions
arise and die in various groups, one notices especially the spasmodic
nature of net discussions.  One article spawns a discussion which
clutters up or enlivens a newsgroup for a while, depending on your
viewpoint.  If it continues for long enough somebody either suggests
net.new-topic-to-get-this-crap-out-of-here-so-I-don't-have-to-wade-through-
it, or net.new-topic-so-I-can-follow-this-discussion-coherently, or if
it's something clearly not deserving of a newsgroup, "OK, enough already!!!".

We need a mechanism to allow more flexible segregation of discussions.
The followup mechanism doesn't seem to be quite enough.  I have a thought
on this.  The proposal may not be feasible for 1001 reasons since I don't
know the internal structure of the news software, but I thought I'd toss it
out for comment anyhow.

It seems to me that since the basic means of control within USENET is the
newsgroup, what we really need is a way to have a more dynamic newsgroup
tree.  There was a discussion on automatic deletion of newsgroups recently,
with points being made on both sides.  What I would suggest is a mechanism
that would allow discussions which result in a temporary increase in volume
to result in a temporary sub-newsgroup to the original.  THESE newsgroups
would expire and their spooling directories removed as the discussion in
them abates.  Let me give a few details.

Put two new header items on articles, FTOPIC and FPOST.  FTOPIC is an
optional field which the original poster of an article can create to
designate a subgroup for subsequent discussion if there needs to be one.
At a given site, the arrival of a followup to an article causes the creation
of the subgroup, if it doesn't already exist, which I will call a followup
group.  The original article and its followup are placed in this subgroup.
Other followups arriving are put there also.  Now you may post to this new
group, unsubscribe to it, or whatever.

The FPOST item is to allow posting to followup groups in a way which won't
conflict with the normal distribution of news.  The newsgroup item of a
posted article contains only the original group, as before.  The FPOST item,
if present, contains the followup group "tail".  Arrival of an article with a
tail should also precipitate the creation of the followup group.  Posting
software should not allow the posting to a followup group that doesn't exist
on the posting site, however.  By handling things this way, the arrival of
a posting at a site previous to the followup article which created its
followup group will cause a minimum of problems.

Followup groups are removed when the articles in them expire, indicating that
the discussion which spawned them has died down.  This also means that
readnews should cleanup lines referring to stale newsgroups in the .newsrc
file.  We probably need an indication in the active newsgroup file to tell
whether a newsgroup is permanent or a followup group.

What we now have is a way for the person whose sense of humor doesn't tend
towards the ridiculous to unsubscribe to net.misc.wombat, or for the person
inclined in that direction to be able to enjoy it undiluted.  Or more
seriously, to keep an eye out for something that might interest you in a
group like net.micro, without having to sift through the usual heavy volume.

Some changes in newsgroup presentation may be needed for readnews, but not
drastic ones, I would guess.  I use my own news reader, so I'm not going to
propose anything for readnews.  If we want to get fancy, we can even maintain
the article which caused the creation of the followup group until the group
expires, and allow the news reader to see the article that started the
discussion on request.

There IS a need for an intelligent decision on the part of the poster of an
original article to determine whether or not the article needs a followup
group designated, and the choice of a sensible name.  I don't think name
collision is a worry - it would probably be rare, and if it happened the
articles probably discuss similar topics anyway - let them get lumped together.

I assume existing versions of news software ignore unrecognized header items,
so that this mechanism could be added to individual sites without causing any
problems in others.  At those sites, the discussions would remain lumped
together in the parent group, as they are now.  You could have the followup
group feature able to be toggled on or off on a site basis, also, to allow
sites worried about resource usage to keep the number of spooling directories
and administrative file sizes down to a minimum.

The basic idea here is that the current newsgroup tree structure remain, with
newsgroups under control of site administrators, however we add a dynamic tree
structure underneath it to accommodate spates of heavy traffic on one topic.
Note that we would also have a good indicator of the need for a new permanent
newsgroup - a followup group which never goes away, indicating permanent
interest (or a non-expiring article, which could be cleaned up).

It's just a thought - maybe somebody wants to kick the pros and cons around
a little.

		Bob McQueer
		decvax!ihnp4!druxt!mcq
			(I MAY be moved from druxt to drutx soon.  If
			 druxt doesn't know about me, try drutx).

sew@minn-ua.UUCP (01/31/84)

#R:druxt:-88400:minn-ua:10900001:000:1456
minn-ua!sew    Jan 30 12:39:00 1984

(A non-blank first line seems to be an obscure net custom)

The "notes" programs already seem to meet your description pretty well.  Notes
presents messages as messages with responses tacked on to them.  Its index shows
the title of the original message and the number of responses.  A message is
thus similar to a subgroup with a bunch of messages in it.  This is as if each
message were its own subgroup.

You can manually select the message which you want to read, then you can step
through the responses in sequence.  This is similar to reading a news file with
a program which simply presents the messages in the sequence received.

If you're using the sequencer, if a message has any new responses then you first
are shown the original message and then the new responses.  This is similar
to using a program which only prints the news received since the last time you
read news.

As for getting rid of stale subjects, "notes" keeps a message and all its
responses if the most recent response is less than a certain age.  I think at
our site a message is deleted if there have been no new responses in two weeks,
but all this is implementation dependent.

There might be some similar programs.  As "notes" shows, modifying usenet news
is not necessary.  It can be (is?) done with the "References" line in a
message header, which contains the message-ID of the original message.


From the analog digits of
Scot E. Wilcoxon
...ihnp4!umn-cs!minn-ua!sew

essick@uiuccsb.UUCP (01/31/84)

#R:druxt:-88400:uiuccsb:3400007:000:2187
uiuccsb!essick    Jan 29 13:06:00 1984

	I think that working on mechanisms to dynamically add and
delete newsgroups (or subgroups) is not the right way to solve
the problem of "there's not enough structure out here". Adding
and deleting newsgroups at this level (followup groups) is 
just another way to map two dimensions (newsgroups and the
discussions within them) into one dimension (even more newsgroups).
This works fine for Mr. A. Square in Flatland, but we're blessed
with being able to see a few more dimensions.  Why not develop
and use a tool that can handle two dimensions?
	At the University of Illinois, we've been using the notesfile
system for 2 years on Unix. Our PLATO system has been using the
notesfile concept for 8 years.  A notesfile contains a time-ordered
(by inception) list of discussions. Each discussion contains a 
base note and a time-ordered list of responses. You get commands
to skip around within a discussion, skip over the rest of boring
discussions plus the usual set of write a response and mail to
the author commands.
	Here are a few things that the notesfile system does that
the news system doesn't:
	@ skip the rest of a discussion
	@ easily see other articles in the same discussion
	@ index pages of discussions

	Notesfiles are handy for other things than reading about Wombats.
We use them for project logs, problem reports, and to help determine
policy.  The fact that we can quickly (one key) go back and see what 
the problem was when a solution comes around makes it very handy.
The archiving facilities are pretty extensive.  How does doing things
like "place all the discussions from this notesfile older than X days
and marked solved into an archive" and then being able to reference
the archive in the same way you reference the active notesfile sound?

	I've managed to get a little off track. I really want to
stress the fact that I think all these extended-newsgroup ideas
are just trying to squeeze two dimensions into one and that
it would be better to use a tool that understood about the extra
dimensions. Why do people prefer screen editors like vi to line
editors like ed?

-- Ray Essick, University of Illinois
-- ihnp4!uiucdcs!essick,  essick.uiuc@rand-relay

mark@cbosgd.UUCP (02/01/84)

I agree with Ray, dynamic creation/removal of newsgroups is not
the solution.  Notesfiles has (apparently) shown that having subdiscussions
marked within newsgroups is sufficient.  News does this as well,
with the References line.  It's just that most of you out there
are running 2.10 or earlier, which pays no attention to this line.

2.11 is under development and sorts by discussion.  There are also
commands being considered to do things like "skip the rest of this
discussion" and "unsubscribe to this discussion".

In doing this, a problem is emerging.  A lot of you "wonderful people"
(-: out there don't use the built in followup command, but instead
post a fresh article.  Often you type in any random subject you can
think of, and of course you don't include a references line.  (I understand
that some of you want a cooling off period, or a line printer interface.
We're working on this but it's a hard problem and suggestions are
welcome.)  When somebody posts a fresh article with a different subject,
there is no way the news system can tell it's really a followup.

The major distinguishing factor, as I understand it, between news and
notes in this regard is that notes forces you to use their one user
interface, so if you want to follow up you must use their command.
News allows several different user interfaces, including delayed things
like the line printer, and so can't force you to use the followup command.
It isn't clear which is better, obviously both have their advantages.

If you have a wonderful idea to ensure that followups are properly marked
as such, please let us know (cbosgd!uucp-news)

	Mark Horton

	Mark

woods@hao.UUCP (Greg Woods) (02/01/84)

   Great. Plug your own program. Notes is nice. But I for one (and I'm
sure also the vast majority of USENET readers who read this article) are
fucking sick and tired of getting articles titled "Orphaned Response"
which we automatically retitle "generic bullshit". If you are going to
plug a system (which may or may not be better than 'readnews'), let's
make people aware of its flaws as well. Find a better way of dealing
with responses whose base note has not yet arrived, and then maybe I'll
join you in touting notes.

	       Greg "prove it first" Woods
-- 
{ucbvax!hplabs | allegra!nbires | decvax!kpno | harpo!seismo | ihnp4!stcvax}
       		        !hao!woods

tjt@kobold.UUCP (02/01/84)

Everybody seems to agree that grouping discussions within a newsgroup is
the right way to organization presentation of news.  It is also clear
that notes has been doing this all along.  How about some feedback from
notes users on how often the followup mechanism gets subverted and how
annoying this is?

Also, I suspect the situation will be self correcting.  Currently,
preserving the Subject: and/or References: is only important to a
minority of people using Usenet (i.e. notesfile sites and 2.11 test
sites).  Therefore, there is little incentive for the rest of us to
prefer using followup commands to posting new articles.  Once 2.11
becomes as widespread as 2.10 is now, the majority of people will
benefit from properly using followup's and will be equally
inconvenienced by new articles which should have been followups.
This should provide the necessary motivation for paying more attention
to the Subject: and References: line.

Note: I assume that the line printer interface would also group
articles by discussions so that line printer users would not be
disenfranchised.
-- 
	Tom Teixeira,  Massachusetts Computer Corporation.  Westford MA
	...!{ihnp4,harpo,decvax}!masscomp!tjt   (617) 692-6200 x275

phil@amd70.UUCP (Phil Ngai) (02/02/84)

I get pretty sick of orphaned response too. Essick's attitude seems
to be that he doesn't care or at least that he's not going to fix
the problem.

I also don't like not being able to read encrypted jokes or automatically
find out about new newsgroups, which are two of many problems with notes.
-- 
Phil Ngai (408) 988-7777 {ucbvax,decwrl,ihnp4,allegra,intelca}!amd70!phil

crl@pur-phy.UUCP (Charles LaBrec) (02/02/84)

We ran notesfiles for about a year in conjuction with B news before I
threw it out the door.  While we liked the screen oriented behavior and
index page, when vnews arrived we decided we liked that better for the
reasons below:

	1) minor point--some of us disliked the 'moded' command set
	2) ver 1.3 on our 2.8 BSD PDP-11/44 constantly screwed up.
		net.sources had to be turned off because large articles
		trashed it completely.  I had to 'adb' some notesfile 
		index about once every other week to clear the NFINVALID
		bit.
	3) My major gripe was, and still is, its basic incompatibility
		with the rest of USENET.  At the time, it interfaced
		to news through an 'A' news interface.  This meant no
		"References:" header for the rest of the net, and losing
		any other header except Article-Id, date, and path, and
		even then notesfiles would lose all of the path except for
		the tailing "site!user"
	4) Its screen display was not very intelligent.  At 300 baud, I
		couldn't stand it.  Also, it was not possible to "fine-tune"
		the position of an article on the screen.  What I mean is
		that both "readnews" (through "more") and "vnews" allow
		you to scroll the screen by one line, thus enabling someone's
		code fragment to fit on a screen.
	5) Other nit-picky things like:  subject limited to 30 (?) or so
		characters,  no subject line in saved notes, not being able
		to save a note w/o a header line and a trailing "-------"
		(makes it a pain to grab things off net.sources and "sh"
		them), REQUIRING a mail router since path info is lost,
		not being able to use a simple "grep" to find something
		somewhere in notes (all notes in a group are kept in a single
		file that is read protected because of read/write access
		controls), and max note size of 64Kb (big problem for
		net.sources).  I could probably go on, if I thought about
		it some more, but these are trivial dislikes.

I guess I'll end this with a list of advantages:
	1) the index page is good, except that it is almost useless when
		trying to scan for a few new responses in a mire of crap,
		since you couldn't tell which notes had new responses.
		(Of course, the sequencer knew, but you didn't want to see
		all the new stuff).
	2) responses are kept together, along with their parent
	3) it takes about 75% the disk space of news
	4) it is possible to limit access to certain groups by user or group.
		The permissions are read/write/answer, where answer allows
		to enter a followup only, and not a new discussion.
	5) each news group can be administered by a normal user, who can
		be given "director" access (a "super-user"-like permission
		list on the specific notesfile)
	6) you can pick the order of presentation of news groups

Charles LaBrec
UUCP:		pur-ee!Physics:crl, purdue!Physics:crl
INTERNET:	crl @ pur-phy.UUCP

sew@minn-ua.UUCP (02/04/84)

#R:druxt:-88400:minn-ua:10900002:000:306
minn-ua!sew    Feb  3 14:39:00 1984

<>

I meant to use "notes" as an example, not as the best.  Aren't there some
other programs which structure the information carried in news files?

Also, does "orphaned response" only affect the site which got the response
before the message, or does notes propagate an additional message around the
net?

essick@uiuccsb.UUCP (02/06/84)

#R:druxt:-88400:uiuccsb:3400008:000:1146
uiuccsb!essick    Feb  5 13:43:00 1984

To answer a few questions ...

re: How often is the notesfile followup mechanism subverted?
	About a dozen times over the last two years for
articles generated at UIUC or the notesfile sites we
feed (total about 12 USENET nodes).
	I think part is that people tend to respond immediately.
Part of it is also that it is easy to get back to a note
you want to respond to.

re: orphan stuff
	The notes code doesn't propogate the foster parent. The
foster parent is used only as a placeholder until the true father
arrives.  Sometimes he never makes it.
	In the notes transfer format, only the base note carries
the title to a discussion. This is why we have the "Orphaned Response"
title.  The parent fills in the correct title when he gets there.

re: interfaces
	PLATO sneaks by with a single interface because all the
terminals are identical.  I designed the Unix version for
CRT's running at least 1200 baud.  No wonder 300 baud hard copy
users don't like it. The "erase-abort" code that is in place now
will flush the remainder of the output when a keypress is detected.
This makes slow CRT users happier.

-- Ray Essick, University of Illinois

tw@hp-pcd.UUCP (02/09/84)

Guy,
You are correct in assuming that most of the win/lose is in the
news gateway software.  Actually the changes to generate legit
news headers were pretty simple, and have the added advantage of
making it unnecessary to run both news and notes on your gateway
machine, since you can now just uux rnews on some remote machine,
same as news does.

I've sent the code back to Ray Essick, so it should make it into
the next release of notes; meanwhile if anyone is dying to change
over, send me mail and I'll see about getting the stuff to you.

Tw Cook - HP Portable Computer Division, Corvallis, OR
{hplabs,harpo,hpfcla}!hp-pcd!tw

donn@hp-dcd.UUCP (02/10/84)

I've been following the notes vs. news wars for a while, and I think
maybe a clarification is in order.  There are two, intimately related
but distinct, versions of notes.  There are the UI versions, directly
done (and supported?) by Ray Essick.  There is also the UCB "version" in
the "contributed software" section.  The UCB version is 99% the Essick
version, but Ray Spickemier has fixed a few of the problems.

It turns out that the problems are exactly those that are complained
about a lot!  He's added a "%" key for decription of rot13 jokes,
and "|" and "^" (analoguous to s and S) for pipes.  He's also cleaned
up the notes creation and monitoring some.  We had one site here that
refused to run notes, but now wouldn't use anything else.  The UCB
version is based on UI 1.3, I'm told; I don't know much about UI 1.6
yet.

We run notes only on all our HP built machines, and have no complaints.

Orphaned responses are in general a symptom of news' occasional
unreliability, not any problem of notes directly;  you see the orphaned
responses in reading news, it just isn't as visible that its an orphan.
(How many times have you thought "I don't know what this is a respnse to".
Usually one blames ones memory; with notes, you know that you aren't
losing your mind, you havn't seen anything about it before!)

Donn Terry
hplabs!hp-dcd!donn

tw@hp-pcd.UUCP (02/12/84)

> /***** hpcvln:net.news / apollo!rees / 11:31 am  Feb 13, 1984*/
> Subject: Re: A Thought on Newsgroup Structure - (not nf)
> 
> The problem is that notes doesn't conform to the usenet standard
> for formatting articles.  It has its own idea of what the headers
> should look like, and it tosses out the "References:" line, while
> inserting a bunch of junk into the body of the article.
> 
> I've heard all the "moving target" arguments, and I sympathize,
> but all you notes fans should get off your soapboxes long enough
> to bring notes up to the standard for message interchange that the
> rest of us have agreed to.
> /* ---------- */

If you'll examine the header for  this  article  (generated  with
notes) you will find that it conforms completely to the "Standard 
for  Interchange  of  USENET  Messages",  and  that it contains a
"References:" line which refers to  your  own  article  [assuming
that  there  isn't  an  'A'  news  site  between  yours and mine,
something not under my control].  You might also notice  that  'a
bunch of junk' was not inserted into the body of the article.  

Tw Cook  -  HP Portable Computer Div, Corvallis OR  -  hp-pcd!tw

rees@apollo.uucp (Jim Rees) (02/13/84)

The problem is that notes doesn't conform to the usenet standard
for formatting articles.  It has its own idea of what the headers
should look like, and it tosses out the "References:" line, while
inserting a bunch of junk into the body of the article.

I've heard all the "moving target" arguments, and I sympathize,
but all you notes fans should get off your soapboxes long enough
to bring notes up to the standard for message interchange that the
rest of us have agreed to.

saj@iuvax.UUCP (02/15/84)

#R:druxt:-88400:iuvax:2800001:000:875
iuvax!apratt    Jan 29 16:16:00 1984

Clearly, your article is one which would deserve the "make this a subgroup if
there are any followups" flag.  But sometimes it isn't all that clear, until
several days after the original note is posted. Then, when it does become
apparent that a subgroup is warranted, *several people* would tack
a subgroup-spawning designator to their postings. Result: six different sub-
group designators for a single subject arriving in the same day. The reverse
problem from identifier clash.

	It does sound like a good idea, but this flaw sprang out at me right
away.

	Could someone please post info on the differences between notes and
news? I use notes, and I am guessing that when I write a "response" to a note,
it is passed as a "followup" to that note. On the other hand, I may be badly
confused. Thanks anyway...

----
					-- Allan Pratt
			...decvax!ihnp4!inuxc!iuvax!apratt

saj@iuvax.UUCP (02/15/84)

#R:druxt:-88400:iuvax:2800002:000:1132
iuvax!apratt    Jan 31 16:59:00 1984

I guess I am not familiar enough with the differences between news and notes.
I wrote based on the "notes" concept (a base note and responses to it, tied
together in two dimensions) because that's really all I've ever dealt with.
I've used the PLATO notesfile system, and UNIX notes, and yet another notes
system, and I guess that's all I really know. The fact that the original
poster used "followup" instead of "response" didn't faze me; it did, however,
leave a wrong impression.
	In any case, the "notes" driver still doesn't do the job. When there
is a protracted discussion on the net, say about waterbeds in net.misc, it
seems logical that there might be a temporary subgroup created to deal with
the sudden volume on a specific topic.  If you want to read it, fine, but
it is much easier to skip over it if you aren't interested.
	Can somebody describe for me the bugs in the "notes" software? A
site here at IU (isrnix) recently gave notes up in favor of news -- to me,
that seems like giving a word processor up in favor of a hammer, chisel,
and stone slab.

----
					-- Allan Pratt
			...decvax!ihnp4!inuxc!iuvax!apratt

guy@rlgvax.UUCP (Guy Harris) (02/17/84)

> If you'll examine the header for  this  article  (generated  with
> notes) you will find that it conforms completely to the "Standard
> for  Interchange  of  USENET  Messages",  and  that it contains a
> "References:" line which refers to  your  own  article  [assuming
> that  there  isn't  an  'A'  news  site  between  yours and mine,
> something not under my control].  You might also notice  that  'a
> bunch of junk' was not inserted into the body of the article.

I suspect what we have here is winning and losing versions of "notes",
with hp-pcd running the former - or, more accurately, winning and losing
versions of the notes/news gateway.  If this is the case, the winning
version should be made available to one and all, and sites currently
using the losing version encouraged to switch.

	Guy Harris
	{seismo,ihnp4,allegra}!rlgvax!guy

msc@qubix.UUCP (Mark Callow) (02/20/84)

-- From Allan Pratt
	Could someone please post info on the differences between notes and
news? I use notes, and I am guessing that when I write a "response" to a note,
it is passed as a "followup" to that note. On the other hand, I may be badly
confused. Thanks anyway...

--------

1.	Look at the subject line of this article.  It has been truncated
	to 40 characters courtesy of "notes".  Also a completely useless
	suffix "- (nf)" has been added.

2.	When you write a "response" to a note it is not passed as a followup
	but as an ordinary article.  The distinguishing feature of a
	followup is a "References:" line in the article header.  The
	notesfiles -> news "gateway" does not provide this extremely
	useful field.
-- 
From the Tardis of Mark Callow
msc@qubix.UUCP,  decwrl!qubix!msc@Berkeley.ARPA
...{decvax,ucbvax,ihnp4}!decwrl!qubix!msc, ...{ittvax,amd70}!qubix!msc