brad@looking.on.ca (Brad Templeton) (02/26/91)
I have been thinking about setting up a dial-up up to the minute newsfeed. This would be a dial-up uucp feed which gives you all the news that's come in to the second of your dialing in. Now, of course the old one uux per article system did that, but now with batching you only get news as of the last time you batched. And with non-uucp dialups it would not be hard to arrange such a thing. But the idea system would work with uucp. To do this, you of course need a command that will output the new news batches. Most batchers can be made to do this, and dynafeed (which is what I will use) does it easily as well. One idea is to use named pipes. A named pipe would exist on the system, and a daemon would be sitting waiting to open and write to that pipe. It would write a news batch to that pipe as soon as it unblocks. One would then arrange to uux an rnews of the named pipe file. It does not matter if there are multiple uux's queued up, as the latter ones will just do null transfers, albeit wasting a bit of CPU. Then the system calls in, and tries to copy over the 'pipe' file. It gets an up-to-the-minute news batch, probably all as one big batch. This sounds ideal, but there's too much that can go wrong. If the daemon process isn't around writing to the pipe (should it die unexpectedly) the uucp will hang waiting to open the named pipe. And of course, it is hard to figure what to do if the phone call aborts mid-file. Do broken pipe signals get sent on named pipes? It does not seem so on my system but perhaps I have not fully checked. If they do, you can handle this one. Sending all the news in one batch can be risky if there's a long blockage that causes a mega-batch. Batchers all have a batching limit and of course you could arrange to uux the pipe several times. ---- Another idea is to change uucp's shell from uucico to a wrap-around program that starts up the batcher in the background and then calls uucico. Only problem is that if uucico is too fast, it will quickly decide there is no work to do and hang up before the first batch is queued by the batcher. One could delay the start of uucico until the first batch has definitely made it, but this is not a great idea, and wastes phone time. Of course, many solutions are possible with the source to uucico and custom patches. I don't have that, so am interested in more general alternatives. Any ideas? -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
zeeff@b-tech.ann-arbor.mi.us (Jon Zeeff) (02/26/91)
>I have been thinking about setting up a dial-up up to the minute >newsfeed. This would be a dial-up uucp feed which gives you all the >news that's come in to the second of your dialing in. > >Another idea is to change uucp's shell from uucico to a wrap-around program >that starts up the batcher in the background and then calls uucico. Only Always spool up 1MB (or whatever) of news for a site. Then make sure that you batch up news (with a check to not batch up more if there is already 1MB spooled) more often than it takes to send 1MB. A site calls in and doesn't finish until the batcher finds nothing left to send. This gives you on-the-fly news batching, never taking more than 1MB per site and always sending all the news there is. I run something like this and it works fine. -- Jon Zeeff (NIC handle JZ) zeeff@b-tech.ann-arbor.mi.us
brad@looking.on.ca (Brad Templeton) (02/27/91)
You can't always spool up 1MB if the site is getting a non-full feed. And few phone sites are these days. I want a fast, small feed. As you might guess, I send out news that is time-critical, but this would apply to any hierarchy. It is a waste to use small batches, and a waste to send only what was batched half an hour ago. You can't time batching and calls because calls vary in time due to failures, busy signals etc. You can't even full time batching and outgoing calls, but that is easier. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
david@bacchus.esa.oz.au (David Burren [Athos]) (02/27/91)
In <1991Feb26.024320.24820@looking.on.ca> brad@looking.on.ca (Brad Templeton) writes: >I have been thinking about setting up a dial-up up to the minute >newsfeed. This would be a dial-up uucp feed which gives you all the >news that's come in to the second of your dialing in. >One idea is to use named pipes. >This sounds ideal, but there's too much that can go wrong. I'll agree with part of that. It doesn't sound ideal, precisely because there is too much to go wrong and the design seems overly complicated. Of course, feel free to correct me. >And of course, it is hard >to figure what to do if the phone call aborts mid-file. Do broken pipe >signals get sent on named pipes? It does not seem so on my system but perhaps >I have not fully checked. If they do, you can handle this one. Bzzt. Seems very system-specific.... >Sending all the news in one batch can be risky if there's a long blockage >that causes a mega-batch. News batches are usually 50 or 100 kbytes for a reason. Most (hehe :-) UUCPs do not have the ability to restart aborted transfers. I cannot see any reason to increase the batch size that outweighs the problems introduced. >Batchers all have a batching limit and of course >you could arrange to uux the pipe several times. After reading the end of the file, UUCP handshakes are required to determine that it was received correctly. Thus your pipe batcher would have to wait for an acknowledgement from UUCP before updating the list of articles to batch. This would require modifications to the UUCP code. That's fine if calling sites see the new setup as a normal batched-news feed, as all the changes are at your end. That said, I'd prefer to see (if this whole scheme is really necessary) a solution that was more general, and easily implemented across a variety of systems. >Another idea is to change uucp's shell from uucico to a wrap-around program >that starts up the batcher in the background and then calls uucico. Only >problem is that if uucico is too fast, it will quickly decide there is no >work to do and hang up before the first batch is queued by the batcher. One >could delay the start of uucico until the first batch has definitely made >it, but this is not a great idea, and wastes phone time. >Of course, many solutions are possible with the source to uucico and >custom patches. I don't have that, so am interested in more general >alternatives. How do you propose to cope with news that comes in while the batching and transfer is in progress? This boils down to how often the is site calling up... If it's going to call again soon, there shouldn't be a problem. If it only calls sporadically, why not batch news for them on an hourly (or an arbitrary N minutes) basis. If they miss out on news that came in during the last n (<N) minutes, is that a major problem? Also, some sites expire news after batching it, especially if they're running short of disk space. If you're suggesting leaving off batching until the last minute then I'd suggest that assumption may be hazardous. Batch news regularly for them. If up-to-date news is a problem, I'd suggest using your scheme of spawning a background batcher when the site dials up. While the batcher is running, UUCP will still be online transferring the other batches. >Any ideas? Why do you want this scheme? Is it because you don't want to keep batched news around for lots of sites, waiting until they call, and chewing up disk space? You could modify the above suggestion so that the periodic batcher only makes one batch (none if one is already there). Hope this feedback is of use... - David Burren (david@bacchus.oz.au) (athos@eyrie.oz.au)
agod@agsc.sub.org (Andreas Godzina) (02/28/91)
In <1991Feb26.024320.24820@looking.on.ca> brad@looking.on.ca (Brad Templeton) writes: >Another idea is to change uucp's shell from uucico to a wrap-around program >that starts up the batcher in the background and then calls uucico. Only I had the same idea and realized it this way: I periodically run a modified version of sendbatches which processes only 2 batches so that there is always some work. If these 2 batches were to small, my sendbatches will process more batches until du reports that the spooldirectory is filled with more bytes than chosen with the size-field of $NEWSCTL/batchparms. When a machine logs in, I start a small program in the background which delivers batched-smtp requests and starts the real sendbatches for this machine, running in a loop. It's far from ideal, but seems to work. -- Andreas Godzina Ginsterweg 14 [D-]W-3014 Laatzen 1 voice +49 511 824333
brad@looking.on.ca (Brad Templeton) (03/02/91)
There are several reasons for wanting an up to the minute feed. As suggested, with an up-to-the-minute feed scheme you would never waste any spool space on news batches. Ideally the site would receive news right from the batching process, and news would never be batched. Secondly, it just seems inherently wrong for a site to call up and not get all the news that it should get because the last batch ran 30 minutes ago and a whole bunch of news arrived 15 minutes ago. Thirdly, while my case is a special case, I am sending out timely news that you want to get up to the minute. -- Brad Templeton, ClariNet Communications Corp. -- Waterloo, Ontario 519/884-7473
urlichs@smurf.sub.org (Matthias Urlichs) (03/07/91)
In news.software.b, article <1991Mar02.071733.23588@looking.on.ca>,
brad@looking.on.ca (Brad Templeton) writes:
< There are several reasons for wanting an up to the minute feed.
[.. three reasons deleted ..]
Fourthly, not batching means that cancelled articles don't get batched and
transmitted. Good idea if you ask me.
One solution would be a specialized user-level "NFS" server which could
offer a file for batching. The advertised size for that file won't matter
because uucico just looks for EOF when using the g or f protocol. Removal of
the file would indicate successful transfer, and if the server has more it
could just queue another batch. Depending on the UUCP used you may get some
superfluous master<->slave turnarounds, but that can be worked around by
estimating how many batch jobs are needed and creating them at startup.
This might not work with all machines (you'll need at least NFS and a
reasonably intelligent file system for concurrency), but it should work
without risking data loss.
--
Matthias Urlichs -- urlichs@smurf.sub.org -- urlichs@smurf.ira.uka.de /(o\
Humboldtstrasse 7 - 7500 Karlsruhe 1 - FRG -- +49+721+621127(0700-2330) \o)/