ahby@meccts.UUCP (02/25/87)
Okay, here we go... Last summer I was at the Atlanta USENIX. There I met Mel Pleasant, Mark Horton, and other people like that. As part of the discussion, I mentioned that we had a program here in Minnesota that allows a site to feed N downstream sites while only using the spool space for 1 down stream feed. Everybody said "really? send it out!" Well, I always meant to, but the code is kind of messy, and the solution isn't exactly perfect - let me summarize how it works. News 2.10.3 and beyond has a facility called MULTICAST. This allows sites in Austrailia to send news to each other in the manner described above. However, these was no way supplied with the distribution for doing this on UUCP sites. Our solution was first to write a program called uucast. Uucast uses link() and a little spit and glue to create uucp job files that are destined for a number of systems, as well as associating a command with the file: uucast file command site1 site2 ... Uucast seems to work with HDB, Sys III, Sys V, BSD 4.2, BSD 4.3, Ultrix versions 1.1 and 1.2, and a few different versions of Xenix. However, uucast's method of generating sequence numbers is a kludge, and I felt it was inelegant. This is actually the whole reason I have been sitting on the news broadcasting system as long as I have. It works, but it's ugly. Uucast uses it's process ID as a seed, and generates sequential UUCP Sequence numbers from that. The next phase of broadcasting was the program multibatch. Multibatch looks at a batch file generated by news 2.10.3 or later with the MULTICAST option enabled and 'M' settings in the sys file. It scans this file for articles that are destined for the same set of systems, and builds a traditional batch file for each of these sets. Then for each set it calls a shell script called multisend. Multisend is a close cousin of sendbatch, except it is designed with broadcasting in mind. It also cannot handle ihave/sendme protocols (for obvious reasons). Anyway, multisend takes this traditional batch file and feeds it a piece at a time to the news batch program. It then takes these generated batches, compresses them if requested, and gives them to uucast along with the set of systems indicated for that set of articles. The upshot of all this is that if an article is destined for 5 downstream sites, it is only batched once, only compressed once, and only takes one slice of the spool device (well, two slices if you count the original). You're probably asking yourself "How well does this work?" Well, Minnesota is an exception to just about every Usenet rule, I know. However, my site has 2.3 meg of spool space maximum for traffic (news is kept on another device) and we full feed 4 sites and partial feed 4 others. We very rarely have a problem, and when we do we would have had the same problem regardless of how many sites we feed. Once you have solved the disk space problem, news becomes a problem of dollars and modem bandwidth. A 1200 baud modem can perform a typical compressed batched feed in about 2.2 hours per night. Our modems are pretty busy a lot of the time :-). I think it is high time that I got off my butt and published this software. I will be putting it all together and sending it out to mod.sources tonight if at all possible. It has been running on 5 sites for about 6 months, so I hope all of the bugs are out of it. All I can say is that I'm sorry I took so long about this. -- Shane P. McCarron UUCP ihnp4!meccts!ahby, ahby@MECC.COM MECC Technical Services ATT (612) 481-3589 "Character is what you are in the dark!"