dg@lakart.UUCP (David Goodenough) (10/18/89)
I've posted this only to news.admin, because the direction it's taking it has left the newsgroup name thread, and become an admin subject. I'm also putting on my +6 ring of fire resistance, because I'm going to speak my mind, and probably offend a lot of people. geoff@pmafire.UUCP (Geoff Allen) sez: > In article <34075@looking.on.ca> brad@looking.on.ca (Brad Templeton) > proposes a scheme for setting up dynamic feeds > > I think this idea's worthy of consideration. Just how tough would > something like this be to do? I'd make one suggestion, partly because I believe this is the way to write software, and partly because it makes this job easier. Break up inews. I sometimes get the feeling inews is too big for it's own good. Inews handles many functions: posting user articles; unbatching incoming news; manipulating control messages; (several functions in this alone) preparing data for "ihave - sendme" runs. What is wrong with making these separate programs. I can't believe that it's disk space: These separate programs might in total be bigger than inews, but they'd be easier to maintain, install and work with. I started out on a PDP11 34A (you know, the ones where each process gets a gallon of ram tops) That approach has stayed with me, as a result I keep 'em small. Back to the separate program issue. If this were done, then addition of Brad's idea would be trivial, and could be added incrementally to an already installed setup. The one question I have is topology - it sounds to me like this relies a bit on an acyclic tree. Well, we don't got that. I'd suggest more a dynamic "ihave - ineed" type setup. I.e. 1. generate the list of newsgroups that you want (the problem here is transmission time - see the note below) 2. generate an "Ihave" list from those newsgroups, this could be done with grep and tr and a shell script 3. send the Ihave / newsgroup list to your first feed (one at a time) 4. your feed builds HIS Ihave list for you, sends it back to you, along with any articles you need (from his point of view) 5. you figure which articles your feed needs from you, and send them off. 6. At this point, if you have multiple feeds move to the next feed, and loop back to step 2. A few issues need to be considered. 1. What happens if another feed calls _YOU_ while this is running - the system _MUST_ be able to run as slave for multiple systems concurrently. I think this may work anyway, since in slave mode you are just processing a request list. Think about UUNET :-/ 2. How do we solve the cyclic tree problem. I get news from both cfisun and mirror, I would consider cfisun to be a feed, but what about mirror. Are they a feed for us or are we a feed for them. Really the problem becomes how to pass newsgroup information "up the tree" - if mirror decide that we feed them and turn round and ask for every newsgroup on their system we're gonna flip out: we only have a 2400 BPS modem for communications. 3. how about a system that is five hops down the tree from a backbone. It would take ten days (assuming one run per day) for a newsgroup appearing in my .newsrc to start getting news: five days for the information to percolate to the top, and five more days for the data to get back down to me. I grant this is not realistic: two hops == four days is more like it, but it's still a while. I agree with Brad - this has to be possible (watch out for D news :-) ), but there are going to have to be some questions answered first. -- dg@lakart.UUCP - David Goodenough +---+ IHS | +-+-+ ....... !harvard!xait!lakart!dg +-+-+ | AKA: dg%lakart.uucp@xait.xerox.com +---+
henry@utzoo.uucp (Henry Spencer) (10/20/89)
In article <725@lakart.UUCP> dg@lakart.UUCP (David Goodenough) writes: >Break up inews. > >I sometimes get the feeling inews is too big for it's own good. Inews >handles many functions: > >posting user articles; >unbatching incoming news; >manipulating control messages; (several functions in this alone) >preparing data for "ihave - sendme" runs. > >What is wrong with making these separate programs... C News has already done most of this. Inews is 100% for posting user articles; relaynews handles incoming traffic from other sites. Control messages are mostly punted to separate programs (mostly shell files) by relaynews. The ihave/sendme stuff is still in relaynews, but some of the complexity has been moved out into the sys file, and the batcher handles actually assembling and transmitting the ihave/sendme messages. So to answer your question, there's nothing much wrong with it and it has already been done (although not in B News). -- A bit of tolerance is worth a | Henry Spencer at U of Toronto Zoology megabyte of flaming. | uunet!attcan!utzoo!henry henry@zoo.toronto.edu
allbery@NCoast.ORG (Brandon S. Allbery) (10/21/89)
As quoted from <725@lakart.UUCP> by dg@lakart.UUCP (David Goodenough): +--------------- | I'd make one suggestion, partly because I believe this is the way to | write software, and partly because it makes this job easier. | Break up inews. +--------------- That's a good idea. So good, in fact, that it's already been done. Run, don't walk, to your nearest comp.sources.unix archive and get C news. ++Brandon -- Brandon S. Allbery, moderator of comp.sources.misc allbery@NCoast.ORG uunet!hal.cwru.edu!ncoast!allbery ncoast!allbery@hal.cwru.edu bsa@telotech.uucp 161-7070 (MCI), ALLBERY (Delphi), B.ALLBERY (GEnie), comp-sources-misc@backbone [comp.sources.misc-related mail should go ONLY to comp-sources-misc@<backbone>] *Third party vote-collection service: send mail to allbery@uunet.uu.net (ONLY)*
gary@sci34hub.UUCP (Gary Heston) (10/21/89)
In article <725@lakart.UUCP>, dg@lakart.UUCP (David Goodenough) writes:
= geoff@pmafire.UUCP (Geoff Allen) sez:
= > In article <34075@looking.on.ca> brad@looking.on.ca (Brad Templeton)
= > proposes a scheme for setting up dynamic feeds
= >
= > I think this idea's worthy of consideration. Just how tough would
= > something like this be to do?
= I'd make one suggestion, partly because I believe this is the way to
= write software, and partly because it makes this job easier.
= Break up inews.
= [ good case made for suggestion ]
= I'd suggest more a dynamic "ihave - ineed" type setup. I.e.
= [ description of how this would work, and potential problems ]
= .... - if mirror decide that we feed them
= and turn round and ask for every newsgroup on their system we're gonna flip
= out: we only have a 2400 BPS modem for communications.
Deciding who feeds who with what should be controllable with a sys file
entry. Limit the newsgroups mirror can demand, or limit the number of
requests a system can make. A modification of the sys file might be
a better approach, a "have_need" file, perhaps. This could include
any limit data needed, and could also establish a prioritized scheme.
Something like "get everything we can from the primary feed, then
call these other feeds in this order to see what else there is".
Of course, 15,000 admins will just tell you to get a TrailBlazer,
but I'm sure you've already considered that possibility.
Potential problem with all this: batching/unbatching time lag.
Once a "ineed" list is sent, the batch must be constructed at the
other system, transferred, unbatched, posted, and scanned by the
dynamic feeder. I don't think this could be done in real-time, it
would probably be necessary to send a "ineed" list, and call back
later for the batched news. Scanning 100MB takes a while....
Even if a database of articles is constructed, it'll take time to
dig thru it and build the batches--not to mention needing space
for it, and maintaining it. You know, this is starting to sound
a lot like a history file...... :-)
= I agree with Brad - this has to be possible (watch out for D news :-) ), but
Right. "Dynamic news". ^
= there are going to have to be some questions answered first.
Starting with a list of volunteers to write all this stuff. Of course,
it can be broken up into pieces, and then it would be easier to
maintain, and smaller..... :-)
= dg@lakart.UUCP - David Goodenough +---+
--
Gary Heston { uunet!gary@sci34hub } System Mismanager
SCI Technology, Inc. OEM Products Department (i.e., computers)
Hestons' First Law: I qualify virtually everything I say.leonard@bucket.UUCP (Leonard Erickson) (10/23/89)
Everyone seems to be complaining that "but that means it'll be X days before I get a group I just subscribed to" under the proposed system. Excuse me? What do you think happens *now* if you want a group that your site isn't currently getting? Hmmm? You ask the newsadmin to get it. If he agrees that it's a good idea he has to ask your feed to get it. This may take *longer* than Brad's idea would. And if your feed doesn't get it, then you move everything up one level, but the delaays at each level *will* be longer! As far as I can tell this "problem" will only occurwhen you subscribe to a group not currently wanted by anybody at your site *or it's leaf nodes*. I'd say this coulxd handled by having the newsreader tell you that your site isn't currently getting that group and that it may take a while before articles arrive. Say by having the process that makes the "master" .newsrc mail a message to the requestor. If implemented properly this could propogate up the chain if you neded to go up more than one link to get the group. That way you'd be kept posted on the status of your request. :-) For that matter, it'd also work for notifying you that a site refused to carry/forward a group. -- Leonard Erickson ...!tektronix!reed!percival!bucket!leonard CIS: [70465,203] "I'm all in favor of keeping dangerous weapons out of the hands of fools. Let's start with typewriters." -- Solomon Short
barrett@Daisy.EE.UND.AC.ZA (Alan P Barrett) (06/02/91)
[Followup-To: news.admin] In article <1991Jun01.041929.8253@looking.on.ca>, brad@looking.on.ca (Brad Templeton) writes: > [...] on my own machine, I am fed exactly the set of current groups > being read on my own site and downstream sites, and this is done > without human intervention. This is what I belive the long term > solution to be. However, until everybody uses a dynamic feeding > scheme of some sort, [...] I don't doubt that dynamic feeding would be useful to many people. But is it really the answer for everybody? It seems to me that, except near the periphery of the net, the term 'downstream' has no real meaning, and this implies that dynamic feeding is not useful except near the periphery of the net. People sometimes want to search through all the articles in all the newsgroups, looking for interesting information. Dynamic feeding introduces undesirable time delays between a user's decision to subscribe to a group and the appearance of any articles in the group. This could be especially irksome for users who read about interesting articles in other groups and want to check them out: instead of just jumping to that group and reading the article, the user has to subscribe to the group, wait for articles to arrive, and later unsubscribe from the group; if the subscription request has to go through several levels of feed sites before it gets to one that already has that group, the delay could be so long that the desired article has already expired fromm that feed site, and will therefore not be seen by the user who wanted it. --apb Alan Barrett, Dept. of Electronic Eng., Univ. of Natal, Durban, South Africa RFC822: barrett@ee.und.ac.za Bang: m2xenix!quagga!undeed!barrett