brad@looking.on.ca (Brad Templeton) (10/16/89)
It would be a lot simpler, perhaps, to eliminate all the hierarchies that don't serve some special purpose. (For example, local hierarchies like the "can" in can.politics indicate not just a Canadian group, but a group about Canadian politics.) To do this you eliminate the news "sys" file as a means for feeding sites. Instead you give each site a .newsrc like file, with some special entries indicating which new group creations are passed along. When it comes time to feed, you run a program (which I could write fairly easily) that prepares a list of all articles "unread" by the destination system, which is then fed into the batcher. Add/subtract groups as you like from this .newsrc file. New group creations work more interestingly. When the program encounters a newgroup control message, it checks to see if it is posted to some special pseudo-groups for group creation. We can have as many of these pseudo-groups as you like. What newgroups a site gets then depend on what philosophies of group creation a site follows. For example, you might wish to only create groups that were created according to the guidelines with their votes and all. So such newgroup messages would be issued with a pseudo-group (or codeword if you prefer) indicating that they came from a guidelines discussion. Some sites might be willing to take any newgroup that comes from a trusted benevolent dictator. You could pick me, or Gene, or anybody who did this sort of thing, and you could pick as many of 'em as you liked. Or some sites might be willing to take any group that meets one of the above criteria and also has a 'hierarchy-like' codeword on it. Ie. a site could agree to take only "comp" groups that are approved by the guidelines (or a czar etc.) if they want. That's what anarchy is about. Let more than one solution be used, and let the individual sites pick the one that they think works. ------------- After that a neat trick is possible. You have a program on your system that 'ors' together all the .newsrc files on your system, your client systems, and the .newsrc feeding files of the sites you feed. Each day you send this 'or' to the site that feeds *you*, where it is used to create the feeding .newsrc file for your site. What this means is that the day somebody on your system subscribes to a group, it starts feeding to your system (unless you explicitly prohibit the group.) Likewise, when you, and everybody downstream has unsubscribed to a group, it *stops* feeding to you. Instant fully dynamic distributed network with no waste requiring little human intervention. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
peter@ficc.uu.net (Peter da Silva) (10/16/89)
I like the idea of changing the sys file format to a .newsrc file format,
though I hope wildcards would still work so us lazy system admins can say:
comp.all y
comp.sys.amiga n
comp.sys.atari n
comp.sys.cbm n
comp.sys.ibm n
comp.sys.mac n
comp.sources.amiga n
comp.sources.atari n
comp.sources.mac n
comp.binaries.all n
However adding the automatic .newsrc culling is not going to fly here. We
don't want users determining our feed.
--
Peter da Silva, *NIX support guy @ Ferranti International Controls Corporation.
Biz: peter@ficc.uu.net, +1 713 274 5180. Fun: peter@sugar.hackercorp.com. `-_-'
'U`
Quote: Structured Programming is a discipline -- not a straitjacket.geoff@pmafire.UUCP (Geoff Allen) (10/17/89)
In article <34075@looking.on.ca> brad@looking.on.ca (Brad Templeton) advocates eliminating all the hierarchies (except for limited distribution or other special cases). I don't think that the anarchy that he proposes in his article is a good idea. But Brad has one idea that I like: |When it comes time to feed, you run a program (which I could write fairly |easily) that prepares a list of all articles "unread" by the destination |system, which is then fed into the batcher. Add/subtract groups as you |like from this .newsrc file. [stuff about the virtues of anarchy deleted] |After that a neat trick is possible. You have a program on your system |that 'ors' together all the .newsrc files on your system, your client |systems, and the .newsrc feeding files of the sites you feed. Each |day you send this 'or' to the site that feeds *you*, where it is used |to create the feeding .newsrc file for your site. | |What this means is that the day somebody on your system subscribes to a |group, it starts feeding to your system (unless you explicitly prohibit |the group.) Likewise, when you, and everybody downstream has unsubscribed |to a group, it *stops* feeding to you. | |Instant fully dynamic distributed network with no waste requiring little human |intervention. Interesting... I think this idea's worthy of consideration. Just how tough would something like this be to do? -- Geoff Allen ...{uunet|bigtex}!pmafire!geoff ...ucdavis!egg-id!pmafire!geoff
rob@violet.berkeley.edu (Rob Robertson) (10/17/89)
Brad, your idea would work OK for small leaf sites. The one problem i can think off the top of my head is, what if your feed goes to such a scheme [ your scheme seems to imply that your news feed has a full feed ]. does one's site have a '.newsrc' on your remote node? how would this work with multiple feeds? rob -- william robertson rob@violet.berkeley.edu symbolic links are the GOTO's of filesystems
amanda@intercon.com (Amanda Walker) (10/17/89)
In article <817@pmafire.UUCP>, geoff@pmafire.UUCP (Geoff Allen) writes: [ About Brad's suggestion of using a .newsrc-like sys file ] > Interesting... > > I think this idea's worthy of consideration. Just how tough would > something like this be to do? Well, the news batching module would have to be rewritten, but that's all I can think of off hand (warning: I'm only an amatuer news guru :-)). It is pretty much a logical step in the direction news has been evolving over the past n years. First, it was all or nothing. Then hierarchies and distributions appeared (with alt showing up soon after as a revival of "the good old days."). Next come things like Brad's "newsclip" language and similar ideas. The common thread is that by making the granularity of a newsfeed finer, you can reduce the amount of time you spend getting news you're just going to throw away. This is of limited usefulness to sites like UUNET, Apple, or OSU (which basically don't throw anything away if they can help it :-)), or other sites with large downstream feeds, but for the growing number of small leaf sites (or close to leaf sites), it could be a big win. I know it would be for us, for example. The machine "intercon.com" has a whole (brace yourself) four people reading news. I try and keep track of things that we want and don't want, but it would be very handy to have the feed from uunet adjust itself depending on what's in our four respective .newsrc files, without me getting mail like "can we get rec.birds?" or "can we stop getting comp.sources.amiga?". Now, I admit that this is kind of an extreme case, but combined with a newsrc-based expire (like TMNN can do) which can expire articles when every subscriber has read them, Brad's scheme could make small Usenet sites much more manageable. Another approach, of course, is to buy a T1 link to a big site and just NNTP off of them :-), but not everyone can afford that yet... -- Amanda Walker <amanda@intercon.com> "Tobacco is the only drug in America that will kill you if it's taken as directed." --Dr. C. Everett Koop, former U.S. Surgeon General
baur@venice.SEDD.TRW.COM (Steven L. Baur) (10/17/89)
From article <34075@looking.on.ca>, by brad@looking.on.ca (Brad Templeton): > ... > After that a neat trick is possible. You have a program on your system > that 'ors' together all the .newsrc files on your system ... What about users of alternate news readers which can use something other than .newsrc? (Like vn, which I use). My .newsrc is defined -- NEWSRC=$HOME/.vn-newsrc; export NEWSRC steve (baur@venice.SEDD.TRW.COM)
alan@servax0.essex.ac.uk (Stanier A) (10/17/89)
In article <34075@looking.on.ca> brad@looking.on.ca (Brad Templeton) writes:
?After that a neat trick is possible. You have a program on your system
?that 'ors' together all the .newsrc files on your system, your client
?systems, and the .newsrc feeding files of the sites you feed. Each
?day you send this 'or' to the site that feeds *you*, where it is used
?to create the feeding .newsrc file for your site.
?
?What this means is that the day somebody on your system subscribes to a
?group, it starts feeding to your system (unless you explicitly prohibit
?the group.) Likewise, when you, and everybody downstream has unsubscribed
?to a group, it *stops* feeding to you.
?
?Instant fully dynamic distributed network with no waste requiring little human
?intervention.
I quite like the basic idea, but there seems to be a problem. Say
I subscribe to sci.unicorn, the first reader on my system to do do.
sci.unicorn gets added to our system .newsrc, and next day we are fed
sci.unicorn. But what if our feeding site doesn't get sci.unicorn yet?
Do I not have to wait an extra day until their feeding site starts to
feed them? And if their feeding site doesn't get it ....
Or have I misunderstood?
There is another problem. In the example above, I would only get
soc.unicorn articles from the day I subscribe. At present, if I subscribe
to a new newsgroup, I can read previous articles and understand the
various threads under discussion. The loss of this facility would be
annoying.
--
JANET alan@uk.ac.sx | Internet alan@sx.ac.uk | UUCP ....!mcvax!ukc!sx!alanmerlyn@iwarp.intel.com (Randal Schwartz) (10/18/89)
In article <34075@looking.on.ca>, brad@looking (Brad Templeton) writes: [describing treating neighboring sites as "users" with their own newsrc files...] | After that a neat trick is possible. You have a program on your system | that 'ors' together all the .newsrc files on your system, your client | systems, and the .newsrc feeding files of the sites you feed. Each | day you send this 'or' to the site that feeds *you*, where it is used | to create the feeding .newsrc file for your site. I had to read that twice to make sure it said that... "all the .newsrc files on your system". Uhhh, who ever said that: (a) all the newsreaders available on a system use the same format file, and (b) all the .newsrc files (or whatever they are called!) are available to the "news" userid? For item (a), I offer "gnews" (a GNU Emacs newsreader) which stores my subscribed groups in a file very un-.newsrc-like. For item (b), I offer NNTP, where many of the .newsrc's are on a different system that is not necessarily available to the master host. (I can't even see my "news" system's active file! The only access I have is through NNTP.) Nope. You can invent "UseNet II -- the son of UseNet" and your own transport mechanism and readers to be used on a particular node, but as long as you had somebody using the standard transport mechanism, you can't get away with your proposal. Just another old-time netter, -- /== Randal L. Schwartz, Stonehenge Consulting Services (503)777-0095 ====\ | on contract to Intel's iWarp project, Hillsboro, Oregon, USA, Sol III | | merlyn@iwarp.intel.com ...!uunet!iwarp.intel.com!merlyn | \== Cute Quote: "Welcome to Oregon... Home of the California Raisins!" ==/
ckd@bu-pub.bu.edu (Christopher Davis) (10/18/89)
[About Brad's suggestion to automagically feed only those groups someone's subscribed to:] I see a couple problems with this one. (1) We keep asking people to "please read the group for a while before you post." Let's say I'm the only user on my system using a HAL-9000, and I just found news. I subscribe to comp.sys.hal9000 and *there's NOTHING there*! How am I, Mr./Ms. News Novice, going to know that there will be something there eventually? (2) What about NNTP? My main news system is a machine I don't even have an account on! My newsreading accounts are on 3 different machines run by two different departments...and what if my newsreader doesn't even *use* .newsrc files for whatever reason? I really like the idea but I don't think it's doable as an automagic function. -- Christopher Davis, BU SMG '90 <ckd@bu-pub.bu.edu> <smghy6c@buacca.bitnet> "Technology is dominated by those who manage what they do not understand."
brad@looking.on.ca (Brad Templeton) (10/18/89)
In article <6545@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: >However adding the automatic .newsrc culling is not going to fly here. We >don't want users determining our feed. Who else *do* you want determining your feed? Clearly any such scheme would include a list of patterns of groups that you refuse to subscribe to no matter how many users request it, and a list of patterns of groups that you wish to be fed even if nobody subscribes, so the users only get as much control over the feed as is desired. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
brad@looking.on.ca (Brad Templeton) (10/18/89)
Clearly some sites, backbones if you will, will wish to feed everything regardless of what is read. Such sites can use any mechanism, including existing ones, to do this. If your site gets news from multiple sites, then you have to maintain lists of what groups you want from each site, and which site to request a group from if users start requesting it. But that's no more than you have to do now if you are fed from multiple sites. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
brad@looking.on.ca (Brad Templeton) (10/18/89)
In article <817@pmafire.UUCP> geoff@pmafire.UUCP (Geoff Allen) writes: > > [stuff about the virtues of anarchy deleted] > Don't knock it. It's what USENET really is. (Actually a minarchy, as it relies on laws from the real word to form the basis of society.) You may not like anarchy, but do you like pretend democracy? >Interesting... > >I think this idea's worthy of consideration. Just how tough would >something like this be to do? Not hard, in that I've already done most of it. The newsclip language, which I give away to my customers, can do all the .newsrc feeding stuff and quite a bit more. It's not likely that the whole net would adopt a commercial package, but I might make a give-away stripped down version (or somebody else might) if it is desired to go ahead with this. As for 'oring' .newsrc files, arbitron already does that, it's not hard to do. You need some extra work to get the .newsrc files from nntp clients -- alternately you could have NNTP log all groups requested and use that as the base of what you need. The only real new thing is the control message handling. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
brad@looking.on.ca (Brad Templeton) (10/18/89)
In article <2158@servax0.essex.ac.uk> alan@essex.ac.uk (Stanier A) writes: > > I quite like the basic idea, but there seems to be a problem. Say >I subscribe to sci.unicorn, the first reader on my system to do do. >sci.unicorn gets added to our system .newsrc, and next day we are fed >sci.unicorn. But what if our feeding site doesn't get sci.unicorn yet? >Do I not have to wait an extra day until their feeding site starts to >feed them? And if their feeding site doesn't get it .... > Or have I misunderstood? Yes, this can happen. Hopefully you aren't more than a couple of hops from a feed site. With new groups it's not a problem, as all new groups get fed before they have readers (give 'em a month's grace or so perhaps.) To deal with this, one could speed up the propagation of requests, so that they happen several times a day, or have 'feed me' changes instigate immediate messages to zoom down the line. > > There is another problem. In the example above, I would only get >soc.unicorn articles from the day I subscribe. At present, if I subscribe >to a new newsgroup, I can read previous articles and understand the >various threads under discussion. The loss of this facility would be >annoying. Annoying, but at what cost? If you want to keep every group somebody might want to read some day on your system, then you clearly don't want to use a .newsrc controlled feed mechanism. Barring some sort of server mechanism, where you get groups you don't have via NNTP (which is how I always thought NNTP should work) there is no way around this. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
brad@looking.on.ca (Brad Templeton) (10/18/89)
When I say that you "or" together the .newsrc files (including the .newsrc files of sites you feed) it doesn't have to be exactly a .newsrc file! The only info you really need out of it is the names of the subscribed groups. It's a 2-line sed script to extract that from a .newsrc, and I can't imagine it's a lot more to extract it from any other kind of newsreader record file. The fact that some people don't use .newsrc files is hardly relevant. NNTP is more relevant, of course. Several solutions are possible. One is that clients have to run the 'or'ing program every day, too, from their cron, and transmit it somehow to the server. That's not too hard. Another would be for NNTP to keep a log of the last time anybody requested a message from each group. If it's greater than N, where N is something like 1 month (expiry time?) then consider the group unread by the clients. Not perfect, but what can you do? I see no problem with the program that reads the .newsrc files being run by root. If you want to hide your .newsrc from the program, that's fine, you can put it somewhere it won't look, and not tell anybody. But then you might not get to read the groups you like. That's only fair, for on a system that is trying to reduce the load of feeding and storing groups nobody wants, it is only reasonable that you have to ask somebody to feed the group. If you are really concerned about people being afraid to ask for alt.sex.bondage, or even letting root know that you read it, the sysadmin will just have to put that group in the always-feed list. My mechanism makes newsgroups even more efficient than mailing lists. It means that USENET can grow without problems, for there is a human limit on the number of groups an individual can read. As such, even with 10,000 groups, we would never see a site get more than the few score per person that an individual can read. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
peter@ficc.uu.net (Peter da Silva) (10/19/89)
In article <35283@looking.on.ca> brad@looking.on.ca (Brad Templeton) writes: > In article <6545@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: > >However adding the automatic .newsrc culling is not going to fly here. We > >don't want users determining our feed. > Who else *do* you want determining your feed? The people responsible for the resources that enable us to receive it. [ "clearly", he says, any such scheme would include a kill-list and a required list, ... ] In our case, those lists pretty much define the feed. The gap between them is small enough that micromanagement is meaningless. Your scheme is fine for individuals and small businesses, and I'd love to have such a setup for my home site, but its availability (which is, pardon me, still hypothetical) should not be taken as grounds for abandoning news hierarchies. -- Peter da Silva, *NIX support guy @ Ferranti International Controls Corporation. Biz: peter@ficc.uu.net, +1 713 274 5180. Fun: peter@sugar.hackercorp.com. `-_-' "You can tell when a USENET discussion is getting old when one of the 'U` participants drags out Hitler and the Nazis" -- Richard Sexton
hans@umd5.umd.edu (Hans Breitenlohner) (10/19/89)
In article <34075@looking.on.ca> brad@looking.on.ca (Brad Templeton) writes: ... >After that a neat trick is possible. You have a program on your system >that 'ors' together all the .newsrc files on your system, your client >systems, and the .newsrc feeding files of the sites you feed. Each >day you send this 'or' to the site that feeds *you*, where it is used >to create the feeding .newsrc file for your site. > >What this means is that the day somebody on your system subscribes to a >group, it starts feeding to your system (unless you explicitly prohibit >the group.) Likewise, when you, and everybody downstream has unsubscribed >to a group, it *stops* feeding to you. > >Instant fully dynamic distributed network with no waste requiring little human >intervention. >-- >Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473 This might be doable for those sections of usenet which are tree-structured. For any set of host which are more richly connected it will fail -- once a group makes it into the 'needed' list will stay there forever. A much more complicated scheme would be needed, which keeps track of who needs what, or which ages newsgroup names as they are passed from host to host. This is analogous to the problem of distributing network routing information.
richard@gryphon.COM (Richard Sexton) (10/20/89)
In article <6583@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: > >In our case, those lists pretty much define the feed. The gap between >them is small enough that micromanagement is meaningless. >Your scheme is fine for individuals and small businesses, and I'd love >to have such a setup for my home site, but its availability (which is, >pardon me, still hypothetical) should not be taken as grounds for >abandoning news hierarchies. I dunno Peter. I agree with that this all much ado about nothing. Did you notice that people are able to read and post to alt.sources.amiga even though (*gasp* THE HORROR!) it doesn't even have a meaningful hierarchy ? I think a larger problem is the ``clumping'' of the hierarchies. comp - 149 rec - 76 sci - 24 news - 17 talk - 10 -- @(peter@ficc.uu.net).signature++ richard@gryphon.COM decwrl!gryphon!richard gryphon!richard@elroy.jpl.NASA.GOV
allbery@NCoast.ORG (Brandon S. Allbery) (10/20/89)
As quoted from <35290@looking.on.ca> by brad@looking.on.ca (Brad Templeton): +--------------- | Not hard, in that I've already done most of it. The newsclip language, | which I give away to my customers, can do all the .newsrc feeding stuff | and quite a bit more. It's not likely that the whole net would | adopt a commercial package, but I might make a give-away stripped down | version (or somebody else might) if it is desired to go ahead with this. +--------------- I still plan to do something along these lines. +--------------- | As for 'oring' .newsrc files, arbitron already does that, it's not hard | to do. You need some extra work to get the .newsrc files from nntp | clients -- alternately you could have NNTP log all groups requested | and use that as the base of what you need. +--------------- One problem remains: newsreaders like Gnews and NN do not use the .newsrc format. One of the biggest gripes against NN (still, to my knowledge, unanswered) was that it didn't automatically maintain the .newsrc for the use of arbitron, etc. How do you propose to deal with this? ++Brandon -- Brandon S. Allbery, moderator of comp.sources.misc allbery@NCoast.ORG uunet!hal.cwru.edu!ncoast!allbery ncoast!allbery@hal.cwru.edu bsa@telotech.uucp 161-7070 (MCI), ALLBERY (Delphi), B.ALLBERY (GEnie), comp-sources-misc@backbone [comp.sources.misc-related mail should go ONLY to comp-sources-misc@<backbone>] *Third party vote-collection service: send mail to allbery@uunet.uu.net (ONLY)*
peter@ficc.uu.net (Peter da Silva) (10/20/89)
In article <21100@gryphon.COM> richard@gryphon.COM (Richard Sexton) writes: > Did you notice that people are able to read and post to alt.sources.amiga > even though (*gasp* THE HORROR!) it doesn't even have a meaningful > hierarchy ? Fiddlesticks. It's in 'alt'. So we don't get it at Ferranti. Which is perfectly reasonable. (that's right, Ferranti doesn't get my own group, even when I was the moderator! How about that.) I'm not sure exactly what you're trying to say here, but none of the conclusions I can derive from this comment make much sense. Could you please expand on this... I don't want to demolish the wrong argument. > @(peter@ficc.uu.net).signature++ Glad you like it. -- Peter da Silva, *NIX support guy @ Ferranti International Controls Corporation. Biz: peter@ficc.uu.net, +1 713 274 5180. Fun: peter@sugar.hackercorp.com. `-_-' "You can tell when a USENET discussion is getting old when one of the 'U` participants drags out Hitler and the Nazis" -- Richard Sexton