jonathan@cs.pitt.edu (Jonathan Eunice) (08/22/90)
How efficient is/are the protocol(s) used by uucp? How do they compare with zmodem? For some reason (not based on anything resembling knowledge), I have the impression that uucp uses crufty, old, slow protocol(s). Am I in left field? In a related matter, how difficult would it be to replace uucp's protocol(s) with zmodem or a similar fast protocol? Or with a slower one (!), or a compressed one, or whatever? Ie, is uucp bound to its current protocol(s)?
jeh@dcs.simpact.com (08/23/90)
In article <8464@pitt.UUCP>, jonathan@cs.pitt.edu (Jonathan Eunice) writes: > How efficient is/are the protocol(s) used by uucp? How do they compare > with zmodem? For some reason (not based on anything resembling > knowledge), I have the impression that uucp uses crufty, old, slow > protocol(s). Am I in left field? Yes, you're in left field. The worldwide de facto standard for uucp dialup with 'g' protocol uses packets with 64-byte data fields and six-byte headers, thus achieving about 90% utilization of the available bandwidth. Many modern uucps can be configured to support larger packets, up to 4096 bytes, which squeezes the "overhead" down to almost nothing, but at the expense of MUCH greater retransmission cost when errors occur. Some sites with reliable connections run with 1K packets, which seems to be a good compromise. > In a related matter, how difficult would it be to replace uucp's > protocol(s) with zmodem or a similar fast protocol? Or with a slower > one (!), or a compressed one, or whatever? Ie, is uucp bound to its > current protocol(s)? There are at least three sides to the "how difficult" question. Technically: There is a "protocol negotiation" sequence in the uucp startup, so it is possible to have a uucico that supports both the standard 'g' protocol and, say, zmodem, and which will use the appropriate protocol for each host to which it connects. Legally: I think there are some copyright/licensing issues around the Zmodem protocol. Practically: There are tens of thousands (at least) of uucp sites out there. The chances of getting a significant number of them to use a new uucico just because it supports zmodem are slim to none. Now, if you came up with a "magic bullet" that offered double or triple the performance of current uucps, that would be another matter. But don't think "compression", because news batches (which are the bulk of the traffic) are already compressed, so further compression will hurt rather than help. One of zmodem's features, the ability to restart an interrupted file transfer from where it left off rather than having to start all over again, is really nifty for the PC user who is downloading tens of megabytes from BBSs at 2400 bps, but would be of limited use for most uucp traffic. At my site and the other sites I've looked at, at least, the great bulk of uucp traffic consists of news batches which are already compressed and which are already broken up into files of about 50 Kbytes each. These are moved via uucp through Trailblazer modems at 1200 to 1400 char/sec; thus the typical large file we deal with only takes 30 to 40 seconds to transfer. If the call dies before completion, on the next call we restart with that same file. It just isn't worth the hassle of putting up a new protocol to save less than a minute on the occasional interrupted call. --- Jamie Hanrahan, Simpact Associates, San Diego CA Chair, VMSnet [DECUS uucp] and Internals Working Groups, DECUS VAX Systems SIG Internet: jeh@dcs.simpact.com, or if that fails, jeh@crash.cts.com Uucp: ...{crash,scubed,decwrl}!simpact!jeh
petrilli@walt.cc.utexas.edu (Chris Petrilli) (08/23/90)
In article <1556.26d282ea@dcs.simpact.com> jeh@dcs.simpact.com writes: >In article <8464@pitt.UUCP>, jonathan@cs.pitt.edu (Jonathan Eunice) writes: >> How efficient is/are the protocol(s) used by uucp? How do they compare >> with zmodem? For some reason (not based on anything resembling >> knowledge), I have the impression that uucp uses crufty, old, slow >> protocol(s). Am I in left field? > >Yes, you're in left field. > >The worldwide de facto standard for uucp dialup with 'g' protocol uses packets >with 64-byte data fields and six-byte headers, thus achieving about 90% >utilization of the available bandwidth. Many modern uucps can be configured to >support larger packets, up to 4096 bytes, which squeezes the "overhead" down to >almost nothing, but at the expense of MUCH greater retransmission cost when >errors occur. Some sites with reliable connections run with 1K packets, which >seems to be a good compromise. > One thing that was forgotten is the availability and use of the Telebit Trailblazer modem. The Trailblazer implements UUCP g-protocol in hardware, and strips all incoming framing information, and adds it when it gets there. My timings/measurements, seem to say it is about 99% efficient, which means at "9600bps" (really 11K on a trailblazer, but for example), you would get about 9504cps without the Trailblazer, but with it you would get 9598cps or so... it helps alot on long distance transmissions. + Chris Petrilli "Opinons represented here | University of Texas at Austin do not necessarily | INTERNET: petrilli@ccwf.cc.utexas.edu represent those of a sane | SNAILMAIL: 429 Brady Lane, Austin, Texas, 78746 person. Take them as + PHONE: +1 512 327 0986 simply that."
fmayhar@hermes.ladc.bull.com (Frank Mayhar) (08/24/90)
In article <1556.26d282ea@dcs.simpact.com>, jeh@dcs.simpact.com writes: |> In article <8464@pitt.UUCP>, jonathan@cs.pitt.edu (Jonathan Eunice) writes: |> > How efficient is/are the protocol(s) used by uucp? How do they compare |> > with zmodem? For some reason (not based on anything resembling |> > knowledge), I have the impression that uucp uses crufty, old, slow |> > protocol(s). Am I in left field? |> [...] |> There are at least three sides to the "how difficult" question. |> [...] |> Legally: I think there are some copyright/licensing issues around the Zmodem |> protocol. Not true, actually. Chuck Forsberg developed the Zmodem protocol as part of a contract with Telenet, for use over packet-switched networks. The protocol was then contributed to the public domain. At least, that's my understanding. I'm sure Chuck will correct me if I'm wrong. |> Practically: There are tens of thousands (at least) of uucp sites out there. |> The chances of getting a significant number of them to use a new uucico just |> because it supports zmodem are slim to none. That depends. If you managed to get the support of some vendors, and they started bundling your uucp with their Unix systems (and others, for that matter), it could eventually catch on. It _would_ be a slow process, though. The other side of the coin, of course, is that those of us that _want_ to use Zmodem protocol on our uucp transfers could do so. I mean, who really cares about those tens of thousands of sites, if _your_ site, and the sites you have uucp connections with, have the capabilities you need. As far as Trailblazers supporting the g protocol, Zmodem would probably not be an improvement, as it's my understanding that the Trailblazer fakes the g protocol to the local host, and the real transfer takes place using PEP. Of course, there are still _lots_ of 2400 baud modems out there, doing a lot of these uucp transfers, and they would benefit immensely from Zmodem, IMHO. -- Frank Mayhar fmayhar@hermes.ladc.bull.com (..!{uunet,hacgate}!ladcgw!fmayhar) Bull HN Information Systems Inc. Los Angeles Development Center 5250 W. Century Blvd., LA, CA 90045 Phone: (213) 216-6241
david@actsn.fay.ar.us (David Summers) (08/24/90)
In article <8464@pitt.UUCP>, jonathan@cs.pitt.edu (Jonathan Eunice) writes: > [ ... deleted ... ] > In a related matter, how difficult would it be to replace uucp's > protocol(s) with zmodem or a similar fast protocol? Or with a slower > one (!), or a compressed one, or whatever? Ie, is uucp bound to its > current protocol(s)? Replacing the protocol with your own is *fairly* easy. When we used to have a source license for Unix I played with adding my own (bi-directional) file transfer to UUCP. The source code for HDB is very well layed out and I was even able to fool UUCP into doing bi-directional file transfers (using my own protocol). It had to start up a total of 8 processes per machine to handle things correctly (and elegantly) but it seemed to work. Now if I could only get the error recovery to work 100% of the time! (sigh). If anyone is interested in discussing error recovery on a simple (bi-directional) file transfer protocol then I would very much like to talk things over with you. - David Summers - (david@actsn.fay.ar.us) -- I'm sick and tired of this machine, I wish that I could sell it. It never does just what I want, but only what I tell it! - David Summers (david@actsn.fay.ar.us)
tneff@bfmny0.BFM.COM (Tom Neff) (08/24/90)
In article <1990Aug24.155109.3373@ism.isc.com> johnan@mchale.ism.isc.com (John Antypas) writes: >There is one issue that hasn't been addressed here. A new UUCP >protocol such as "z" for zmodem, would allow use of UUCP over X.25 >networks such as Telenet where straight "g" protocol shows its age. Chuck Forsberg has put years of work into ZMODEM, adapting it for hostile environments and making it very robust. The savings over 'g' may be marginal under *ideal* circumstances, but ZMODEM's charm is that when things get bad it keeps chugging through (with decent throughput) while the others fall by the wayside. I think it's a natural for inclusion in the UUCP protocol suite. The ideal approach would be for one or more value adders like Interactive to add ZMODEM to their UUCICO implementations. Any vendor who did this would have a competitive advantage to offer buyers. On the other hand, since ZMODEM code is widely available and running in UNIX environments today (the rz/sz suite) perhaps some source licensee could simply integrate it into the SV/386 R4 UUCICO and publish the patch. Assuming AT&T doesn't want to get into the act. Ah, they call me a dreamer... -- Perestroika: could \O\ Tom Neff it happen here? \O\ uunet.bfm.com!bfmny0!tneff
les@chinet.chi.il.us (Leslie Mikesell) (08/24/90)
In article <1556.26d282ea@dcs.simpact.com> jeh@dcs.simpact.com writes: >In article <8464@pitt.UUCP>, jonathan@cs.pitt.edu (Jonathan Eunice) writes: >> I have the impression that uucp uses crufty, old, slow >> protocol(s). Am I in left field? >Yes, you're in left field. >The worldwide de facto standard for uucp dialup with 'g' protocol uses packets >with 64-byte data fields and six-byte headers, thus achieving about 90% >utilization of the available bandwidth. This is true for direct connections with "real" full duplex modems where the turn-around time for the ack packets is insignificant. However, many connections currently happen over packet-switched networks, satellite links, and/or modems that pretend to be full duplex but actually take some time to switch directions. Then you can end up waiting for the ack's and reducing throughput. On a network where there is a per-packet fee, the ack's and the fact that you often send unfilled network packets can be expensive. Older g's allow a 3-packet window that can be sent before the ack's get back, newer ones allow 7 which is the limit for the g protocol field that specifies the number. Some real-world figures for satellite xfers (from todays xferstats on another machine) show about 120 cps over a 2400 baud connection to a machine with the 7-packet window, 90 cps to a machine with a window of 3. Obviously there is some room for improvement here. On top of this, there is the non-windowed per-file handshake and the fact that mail messages carry along a second small file for the uuxqt command. Increasing the packet size or window wouldn't help much with mail transfers, since the files often don't fill the existing window. Note: this applies to trailblazers as well, although you have to look at real connect time to see it - the xferstats don't count the per-file handshake time. >Technically: There is a "protocol negotiation" sequence in the uucp startup, >so it is possible to have a uucico that supports both the standard 'g' >protocol and, say, zmodem, and which will use the appropriate protocol for >each host to which it connects. The big hangup here is that you need a source license to begin with, and then you can't distribute copies. Does anyone have a HDB compatible uucico that can be freely distributed? >Practically: There are tens of thousands (at least) of uucp sites out there. >The chances of getting a significant number of them to use a new uucico just >because it supports zmodem are slim to none. The only ones that matter are the ones that cost you money for your connections. >Now, if you came up with a "magic bullet" that offered double or triple the >performance of current uucps, that would be another matter. But don't think >"compression", because news batches (which are the bulk of the traffic) are >already compressed, so further compression will hurt rather than help. My figures show that it would be possible to double the speed of the 7-window version and almost triple the 3-window version when using a satellite link. Eliminating the acks (except per-file) could cut the cost in half if you pay per network packet. I'd like to see a protocol where you could specify a minimum and maximum packet and window size per host, device and dialer. The highest maximum for the connection would then be used a starting point for negotiation with the remote, and then the sizes would dynamically adjust downward in response to errors with appropriate logging so the setup could be changed if needed. Also, file boundaries should stream through the window. Les Mikesell les@chinet.chi.il.us
johnan@mchale.ism.isc.com (John Antypas) (08/24/90)
There is one issue that hasn't been addressed here. A new UUCP protocol such as "z" for zmodem, would allow use of UUCP over X.25 networks such as Telenet where straight "g" protocol shows its age. Any ideas? John Antypas / Interactive Systems Corp. uucp: ...!uunet!ism.isc.com!johnan Internet: johnan@ism.isc.com All statements above responsability of the author.
david@twg.com (David S. Herron) (08/24/90)
The protocols that UUCP uses can be swapped out -- though you'd have a pretty steep obstacle to climb to get AT&T to put some random piece of (3rd party) software into UUCP. The `g' protocol gets about 80-90% efficiency in pushing bits around at the line speeds with which I am familiar. (300 baud up to 9600 baud) I see little reason to switch away from what I have now, especially since it would greatly limit who I can talk to. Zmodem has, on the other hand, an interesting feature in that it can restart failed transmissions in the middle. My work-around for the lack of that feature is to simply split files before uucp'ing them. But I don't always have to do that -- I've UUCP'd 30-meg+ files over trailblazers before and had 'em work ;-).. -- <- David Herron, an MMDF & WIN/MHS guy, <david@twg.com> <- Formerly: David Herron -- NonResident E-Mail Hack <david@ms.uky.edu> <- <- Sign me up for one "I survived Jaka's Story" T-shirt!
greg@cheers.Bungi.COM (Greg Onufer) (08/25/90)
jeh@dcs.simpact.com writes: > [....] >One of zmodem's features, the ability to restart an interrupted file transfer >from where it left off rather than having to start all over again, is really >nifty for the PC user who is downloading tens of megabytes from BBSs at 2400 >bps, but would be of limited use for most uucp traffic. At my site and the > [....] And, according to the S5R4 docs, there is a "Checkpointing" feature in S5R4 BNU that isn't dependent on the protocol used. If a file transfer dies in the middle, it can be restarted where it left off. The catch is that both systems have to be running S5R4 UUCP (of course). Cheers!greg
syd@DSI.COM (Syd Weinstein) (08/25/90)
>>Technically: There is a "protocol negotiation" sequence in the uucp startup, >>so it is possible to have a uucico that supports both the standard 'g' >>protocol and, say, zmodem, and which will use the appropriate protocol for >>each host to which it connects. What everyone is forgetting here, is that the protocol is only used for sending the file itself. As was stated, there is still the undrelying problem of sending small files, such as mail messages. uucico is split into two functions, one find the requests to send across, and the other actually does the data transfer. Its the data transfer that uses the protocol negotiated. Thus for a mail send, uucico doesn't know its mail. It just sees a request so send 2 files, a 'command file' which is the remote execution of the rmail command, and the 'data file'. Both are just a simple file transfer request to uucico. Thus there is still the time required to send these small files that will not be helped by changing the protocol. A new protocol will only handle the actual sending of a single request itself. Talking about zmodem's restart ability is useless, as uucico will retransmit the entire fill, (again using zmodem protocol if that is the negotiation). Yes, the newest HDB from AT&T can restart a file in the middle, but older ones cannot, and even with Zmodem, they won't even realize they need to. Zmodem, as viewed by itself, is a complete transfer system, so it knows what to do. The protocol is only a small section of uucico, and must be viewed in that context. If you are a mail site, and sending lots of small messages for mail, then you can improve your throughput by switching to a different mail protocol, such as BSMTP, still sent via uucp, instead of rmail. If its news, then selecting correct batch sized will allow for restarts effectively. If its a packet link, then just a protocol with a larger window and selective reject would be needed. Again, its only the transfer of a single file that can be improved by changing the 'protocol' module within uucico. You would need a whole different concept as a file transfer device to improve more than that, and then it wouldn't be uucico. -- ===================================================================== Sydney S. Weinstein, CDP, CCP Elm Coordinator Datacomp Systems, Inc. Voice: (215) 947-9900 syd@DSI.COM or dsinc!syd FAX: (215) 938-0235
jeh@dcs.simpact.com (08/26/90)
In article <1990Aug24.152938.7413@chinet.chi.il.us>, les@chinet.chi.il.us (Leslie Mikesell) writes: > This is true for direct connections with "real" full duplex modems where the > turn-around time for the ack packets is insignificant. However, many > connections currently happen over packet-switched networks, satellite links, > and/or modems that pretend to be full duplex but actually take some time > to switch directions. Then you can end up waiting for the ack's and > reducing throughput. These links provide an error-free path, no? There is already the "f" protocol which is intended for use through mostly-error-free, flow-controlled links. It sends a single checksum at the end of the entire file, relying on the network to get the data through and relying on the fact that while the RS232 (or whatever) connection to your local modem may not be perfect it is quite close. > Older g's allow a 3-packet window that can be sent > before the ack's get back, newer ones allow 7 which is the limit for > the g protocol field that specifies the number. You could hack 'g' fairly readily to allow larger fields for the packet number. While you're at it you'd better add selective retransmit, aka receive window size > 1, to reduce the cost of retransmission when errors occur. > On top of this, there is the non-windowed > per-file handshake and the fact that mail messages carry along a second > small file for the uuxqt command. Increasing the packet size or window > wouldn't help much with mail transfers, since the files often don't > fill the existing window. Note: this applies to trailblazers as well, > although you have to look at real connect time to see it - the xferstats > don't count the per-file handshake time. yup! > The big hangup here is that you need a source license to begin with, and > then you can't distribute copies. Does anyone have a HDB compatible > uucico that can be freely distributed? well, how "compatible" does it have to be? DECUS uucp uses old-style spool file names and a single spool directory but talks to HDB systems just fine. It also has the best login and dialer scripting capability of any uucp arround (not just my opinion). Of course it's written for VMS, but someone told me they ported the g protocol module to MS-DOS in just two days and are getting first-class performance with it, so that can't be too much of a drawback. "Freely distributed"? It used to be based on gnuucp, but the gnu folks tell me that their version has diverged wildly from what they originally sent me, and our version has too (I completely threw away their 'g' implementation and did a whole new one that does windowing, for a start), so I have declared it to be in the public domain. We are going to go to neighbor-system-specific spool directories for the next release. (Right, Mark? :-) HDB-style permissions files are so bizarre that we'll be doing something different. > I'd like to see a protocol where you could specify a minimum and > maximum packet and window size per host, device and dialer. we support this in 'g' via an options field in the entries in the systems file. > Also, file boundaries should stream through the window. This is clearly impossible for "pull-mode" ("I want to receive..."), since the slave can't very well start sending a file until it receives your request for it! But the vast majority of traffic (there I go again) is push- mode, and it would certainly be possible to start sending the file immediately after sending the "I want to send you this file" command, without waiting for a yea or nay. You could assign a simple id code to each file sent during a call; this would be sent in the S command and also in the header for each data block for the file. If the receiving system rejects an S command it would simply drop any data packets containing the same id code. I agree with you about the performance hit for small files, and doing something about it at uucp level sounds like a hell of a good idea. I've seen too many small mail messages (and X files for both mail and news) go through my trailblazer with an effective throughput in the real of 100 char/sec! (Our throughput reporting DOES take the beginning- and end-of-file overhead time into account. 50 Kbyte news batches do hit it up at around 1200-1400.) Unfortunately, in order to use this with TrailBlazer modems, we'd have to abandon their fabled 'g' protocol spoofing. a TB in g-spoof mode not only "knows about" the data link level, it looks for and interprets the "messages" (S, SY, CY, etc.) that mark the beginning and end of a file transfer. It has to, because it has to know to flush its buffers and prepare for line turnaround when the sending system sends the last packet of a file. Another way to fix it would be (for mail) to go to BSMTP, and (for news) to hack uuxqt to allow multiple commands per X file. Then you could send all of your 50 Kbyte news batches and just one (still fairly small) X file. --- Jamie Hanrahan, Simpact Associates, San Diego CA Chair, VMSnet [DECUS uucp] and Internals Working Groups, DECUS VAX Systems SIG Internet: jeh@dcs.simpact.com, or if that fails, jeh@crash.cts.com Uucp: ...{crash,scubed,decwrl}!simpact!jeh
news@m2xenix.psg.com (Randy Bush) (08/26/90)
In <1556.26d282ea@dcs.simpact.com> jeh@dcs.simpact.com writes: >In article <8464@pitt.UUCP>, jonathan@cs.pitt.edu (Jonathan Eunice) writes: >> How efficient is/are the protocol(s) used by uucp? How do they compare >> with zmodem? For some reason (not based on anything resembling >> knowledge), I have the impression that uucp uses crufty, old, slow >> protocol(s). Am I in left field? > Yes, you're in left field. Well, being out there sure has not obscured his vision, then. For those of us moving significant amounts of _mail_ with uucp-g, let me tell you it stinks. The inter-file handshaking eats it to death. While I get intra-file rates of 1000cps, for a couple of hundred mail messages, the effective overall rate is under 200cps. Some of us HDB smail3ers are hacking the stuff from BSMTP, but few sites (e.g. uunet) use it, so it is not of significant help. Also note that, while zmodem would be helpful with news on noisy lines (zmodem's restart is hot), it would not be of significant help with the many small mail files. -- ..!{uunet,qiclab,intelhf}!m2xenix!news
peter@ficc.ferranti.com (Peter da Silva) (08/26/90)
In article <1565.26d6609c@dcs.simpact.com> jeh@dcs.simpact.com writes: > "Freely distributed"? It used to be based on gnuucp, but the gnu folks tell > me that their version has diverged wildly from what they originally sent me, > and our version has too, so I have declared it to be in the public domain. Um, I was under the impression that this wasn't possible. Do you have the FSF's approval for this? -- Peter da Silva. `-_-' +1 713 274 5180. 'U` peter@ferranti.com
vixie@decwrl.dec.com (Paul A Vixie) (08/26/90)
I've got IDA sendmail, which is able to generate BSMTP scripts. Decwrl lets incoming UUCP requests execute the "bsmtp" command, and that all works fine. I need one more piece to get rid of the inter-file handshake delay: If I generate a BSMTP file, it will have in it exactly one mail transaction. This is the same as for "rmail". What's needed is a way to get many of these concat'd together into a single UUCP request, and since they can be uux'd at different times (up to the uucico interval), sendmail is not the place to put the concat logic. The only way I can think of to solve this is to put the bsmtp scripts into some non-UUCP place and concat them with a perl or sh script run from cron. This script would concat everything in, say, /var/spool/bsmtp/uunet/ into one big file and then uux that to uunet!bsmtp. Anybody else done this? Did it help at all? What was the value for N? -- Paul Vixie DEC Western Research Lab <vixie@wrl.dec.com> Palo Alto, California ...!decwrl!vixie
david@twg.com (David S. Herron) (08/27/90)
In article <VIXIE.90Aug26155329@volition.pa.dec.com> vixie@decwrl.dec.com (Paul A Vixie) writes: >I've got IDA sendmail, which is able to generate BSMTP scripts. Decwrl >lets incoming UUCP requests execute the "bsmtp" command, and that all >works fine. I need one more piece to get rid of the inter-file handshake >delay: > >If I generate a BSMTP file, it will have in it exactly one mail transaction. >This is the same as for "rmail". What's needed is a way to get many of >these concat'd together into a single UUCP request, and since they can >be uux'd at different times (up to the uucico interval), sendmail is not >the place to put the concat logic. > >The only way I can think of to solve this is to put the bsmtp scripts into >some non-UUCP place and concat them with a perl or sh script run from cron. >This script would concat everything in, say, /var/spool/bsmtp/uunet/ into >one big file and then uux that to uunet!bsmtp. > >Anybody else done this? Did it help at all? What was the value for N? Paul, You've figured this out pretty much correctly. Some guys at NCR in San Diego worked up something a few years ago did a really low-level & non-portable thing. It figured out what the file name in /usr/spool/uucp for the outgoing batch was and wrote over the end of the BSMTP file with a new BSMTP message piece. What they wanted to avoid was queueing delays -- if their UUCP neighbor called up they wanted to make sure that all of the mail for that neighbor was in the UUCP, rather than some BSMTP queue. I would do pretty much what you suggest. Though since I use MMDF this is a bit easier since the `deliver' for the channel can be run at times I select (using cron) and it would take care of sweeping the queue. I wouldn't mind losing a few times when the neighbor polls me -- just to avoid the low-level hacking around with the queue'd data files. It really "ought" to help out a lot. I haven't done it, despite all the times I've said it would help a lot. I haven't ever felt I had the time to develop it any farther than I have.. Definitely not now. -- <- David Herron, an MMDF & WIN/MHS guy, <david@twg.com> <- Formerly: David Herron -- NonResident E-Mail Hack <david@ms.uky.edu> <- <- Sign me up for one "I survived Jaka's Story" T-shirt!
vixie@decwrl.dec.com (Paul A Vixie) (08/27/90)
Dave, you wrote: # the following line should be changed to reflect the # organization of your system. /usr/bin/compress -d | /usr/bin/rsmtp exit 0 I use: exec /usr/bin/compress -d | /usr/bin/rsmtp (actually my paths are different but you get the idea.) This keeps an extra (useless) sh process from hanging around. >> Nothing to it. smail3 is simple, configurable, works on nearly >> all systems, and doesn't have a sendmail.cf file! What more could >> you want :-) I'd quibble over "simple". Some of those files you quoted look pretty daunting, and I note with terror that there are a lot more of them than in good 'ol (or is that bad 'ol) sendmail. Anyone know when smail3 is coming out of beta-test, OFFICIALLY ? -- Paul Vixie DEC Western Research Lab <vixie@wrl.dec.com> Palo Alto, California ...!decwrl!vixie
dlr@daver.bungi.com (Dave Rand) (08/27/90)
In article <VIXIE.90Aug26155329@volition.pa.dec.com> vixie@decwrl.dec.com (Paul A Vixie) writes: >If I generate a BSMTP file, it will have in it exactly one mail transaction. >This is the same as for "rmail". What's needed is a way to get many of >these concat'd together into a single UUCP request, and since they can >be uux'd at different times (up to the uucico interval), sendmail is not >the place to put the concat logic. smail3 does this right now. The transport to use is "batch_smtp". This places all outgoing messages into a special directory (user configurable), which can be queued for transmission later. Further, and better, you can compress these mail batches, for reduced size (really helps, folks!). I run this to 4 sites now, and have been running it for about 6 months now. Works great, and significantly reduces the amount of transfer time for mail on Trailblazers. Now, if we can only convince UUNET to allow this... Anyway, here's how you set it up for smail3. First, you need to add a new transport to the smail configuration file. Edit your /usr/local/lib/smail/transports (or wherever you have located the smail3 configuration files), and add: # accumulate messages into a directory on a per-host basis batch_smtp: # the appendfile driver can also accumulate in directories driver=appendfile, hbsmtp; # half-baked BSMTP, no HELO or QUIT # files whose names begin with `q' will be placed here: dir=/usr/spool/smail/outq/${lc:host}, user=uucp, # files will be owned by this user mode=0600, # and unreadable by other users Feel free to change the directory as appropriate. Now, you have to have a way for smail3 to know that a specific host should use this transport. This is done using a methods file, although I suppose you could hard-code it, if you wanted to. Edit your /usr/local/lib/smail/routers and change all the transport = xxx to "method = methodfilename". I use the name "uucp" for uucp-type transports, and "smtp" for smtp-type tranports. If you are running a uucp-only site, they perhaps my routers file will do the right thing for you; here it is: # /usr/local/lib/smail/routers # This file defines the configuration of the router subsystem as # compiled into the smail binary. By modifying the source files # conf/EDITME, src/config.h or src/default.c the actual internal # configuration can be changed. Thus, this should be matched up # against thes files before assuming this is completely correct. # paths - route using a paths file, like that produced by the pathalias program paths: driver = pathalias, # general-use paths router method = uucp; # transport = uux; # for matches, deliver over UUCP file = smap, # sorted file containing path info proto = lsearch, # use a linear search (small file) optional, # ignore if the file does not exist domain = uucp # strip ending ".uucp" before searching paths: driver = pathalias, # general-use paths router method = uucp; # transport = uux; # for matches, deliver over UUCP file = paths, # sorted file containing path info proto = bsearch, # use a binary search optional, # ignore if the file does not exist domain = uucp # strip ending ".uucp" before searching # uucp_neighbors - match neighbors accessible over UUCP uucp_neighbors: driver = uuname, # use a program which returns neighbors method = uucp; # transport = uux; cmd = /usr/bin/uuname, # specifically, use the uuname program domain = uucp # smart_host - a partically specified smarthost director # # If the config file attribute smart_path is defined as a path from the # local host to a remote host, then hostnames not matched otherwise will # be sent off to the stated remote host. The config file attribute # smart_transport can be used to specify a different transport. # # If the smart_path attribute is not defined, this router is ignored. smart_host: driver = smarthost, # special-case driver transport = uux # by default deliver over UUCP ----------------------------------------------------------------------- Note that instead of "transport = uux", I have "method = uucp". This tells smail3 to look in the /usr/local/lib/smail/methods/uucp file to see what type of transport it should use for each host. Here is what I use: apt uusmtp cheers batch_smtp chips demand_uusmtp dlb batch_smtp indetech uusmtp interex uusmtp kcdev batch_smtp lynx batch_smtp mips demand news demand_uusmtp sun demand tscs uusmtp uop uusmtp uunet biguux vw26 demand_uusmtp wombat demand * uux ----------------------------------------------------------------- Note that the last line of the file is "* uux". This is required, since smail3 must have some way of getting to a host that is not listed. The "*" matches any host name not previously matched. Anyway, that's it! You are now set up such that outgoing mail messages for the site(s) in question end up in a special spool directory. You can just cat these files together and compress them, feed the result to uux, and you are done. I do this once per hour on my site. I also have another neat trick though - when those sites that run compressed smtp mail log in, I queue all the mail waiting for them before execing uucico. This is all done with a single do-it-all script: #!/bin/sh # @(#)cbsmtp.sh 1.1 7/10/88 02:00:54 . /etc/TIMEZONE # deliver messages accumlated into subdirectories of the # outq spool directory. Subdirectory names are based on # the actual hostnames involved: OUTQ=/usr/spool/smail/outq UUX=/usr/bin/uux LOCALHOST=daver.bungi.com COMPRESS=/usr/bin/compress cd $OUTQ NAME=x case "$LOGNAME" in Ucheers) NAME=cheers ;; Ulynx) NAME=lynx ;; Udlb) NAME=dlb ;; Ukcdev) NAME=kcdev ;; esac if [ "$NAME" != "x" ] then cd $NAME list=`echo q*` # get the list of message files if [ "$list" = "q*" ]; then # no messages were found exit 0 # leave sub-shell fi # compress all of the files, adding HELO and QUIT commands (echo "HELO $LOCALHOST" cat $list echo QUIT) | $COMPRESS | $UUX -r - $NAME!rcsmtp exit 0 else # loop through all of the subdirectories for i in *; do ( cd $i list=`echo q*` # get the list of message files if [ "$list" = "q*" ]; then # no messages were found exit 0 # leave sub-shell fi # compress all of the files, adding HELO and QUIT commands (echo "HELO $LOCALHOST" cat $list echo QUIT) | $COMPRESS | $UUX -r - $i!rcsmtp rm $list ); done fi exit 0 -------------------------------------------------------------- And that does it for outbound. Incoming compressed smtp mail is handled by another simple shell script "rcsmtp": #!/bin/sh # @(#)rcsmtp.sh 1.1 G% 02:01:01 # Receive compressed batches of SMTP commands and send them # to smail. # the following line should be changed to reflect the # organization of your system. /usr/bin/compress -d | /usr/bin/rsmtp exit 0 ---------------------------------------------------------------- Nothing to it. smail3 is simple, configurable, works on nearly all systems, and doesn't have a sendmail.cf file! What more could you want :-) Drop me mail if you have questions... -- Dave Rand {pyramid|mips|bct|vsi1}!daver!dlr Internet: dlr@daver.bungi.com
vixie@decwrl.dec.com (Paul A Vixie) (08/27/90)
vix> Anyone know when smail3 is coming out of beta-test, OFFICIALLY ?
dlr> Who can tell? Source is on uunet now, and I've been running it here
dlr> for months (years?). Ferget the beta test, and go with it. It works well.
Usually when I run a beta test of something, along comes the official
version some months or years later with a few fundamental architectural
changes in it such that my config files, local source patches, and brain-
wave patterns are all subtly "wrong". Just ask anybody who ran my cron
when it was in beta :-)...
dlr> Give it a try; what have you got to lose but a
dlr> few hours of your time?
It's taken years to get the subtlties "right" in our sendmail setup. We
have a a ring of eight mail queues such that messages get moved "outward"
as they get old ("old" starts at an hour) with fewer delivery attempts
as age increases; we run 27 dequeuing agents (sendmail -q) in parallel;
we have a 700-line sendmail.cf with hooks and tweaks from every corner
of the universe that has the look and smell of 80-year old bleu cheese.
The bright side is that we're only forwarding 14,000 messages a day. It
doubled every year for the last four years, and we weren't sure what kind
of architecture we'd use to handle 28,000 messages a day. I think we're
max'ed out on internal network bandwidth; there are 50,000 DECnet machines
using us for internet forwarding, but we've only got five 56KB lines into
that side of our universe so they can only throw so many bits per day.
There's no way in the world that we could run a different mailer in only
a few hours. Given the number of address tweaks we do and the number of
messages we forward, new mailers need to be run in sidelined test machines
for N days or weeks before we can put them into production. I tend to
hack sendmail.cf "live", but it's a bit like running adb on a running
kernel -- nobody else around here will try it :-).
--
Paul Vixie
DEC Western Research Lab <vixie@wrl.dec.com>
Palo Alto, California ...!decwrl!vixiedlr@daver.bungi.com (Dave Rand) (08/27/90)
In article <VIXIE.90Aug26205245@volition.pa.dec.com> vixie@decwrl.dec.com (Paul A Vixie) writes: >>> dlr@daver.bungi.com (Dave Rand) writes: >>> Nothing to it. smail3 is simple, configurable, works on nearly >>> all systems, and doesn't have a sendmail.cf file! What more could >>> you want :-) > >I'd quibble over "simple". Some of those files you quoted look pretty >daunting, and I note with terror that there are a lot more of them than >in good 'ol (or is that bad 'ol) sendmail. At most, you have to deal with 4 files: 1. directors, which tells local mail how to get delivered; 2. routers, which tells non-local mail how to get routed; 3. transports, which tells smail how to get mail out of smail; 4. and a methods file, which tells for a given machine name which transport to use to best get the mail there. Of course, the advantages are that smail configuration files are human-readable :-), and ALL ARE OPTIONAL. If you don't have them, smail will default to "sensible" values. Sensible, in this case, means values that make sense based on the configuration options you specified when smail3 was compiled (it won't try for a DNS resolution on a UUCP-only machine, for example). On various machines that I administer, only the externally-visable machine(s) have any of these files. The internal machines generally stick with the compiled-in defaults, yet still offer all of the neato-kean features (thanks to smart-hosting)! > >Anyone know when smail3 is coming out of beta-test, OFFICIALLY ? Who can tell? Source is on uunet now, and I've been running it here for months (years?). Ferget the beta test, and go with it. It works well. Like I've said before - I really like smail3. It makes my job easier, because it does the "right thing" out of the box. I can actually understand how the configuration files work (yeah, I know - people can read sendmail.cf as well, but I'm just not smart enough). For uucp-only sites, it offers many of the advantages of sendmail (.forward files with command pipes, RFC-822, etc), but without the pain. Give it a try; what have you got to lose but a few hours of your time? -- Dave Rand {pyramid|mips|bct|vsi1}!daver!dlr Internet: dlr@daver.bungi.com
dlr@daver.bungi.com (Dave Rand) (08/27/90)
In article <VIXIE.90Aug26233735@volition.pa.dec.com> vixie@decwrl.dec.com (Paul A Vixie) writes: >The bright side is that we're only forwarding 14,000 messages a day. > >There's no way in the world that we could run a different mailer in only >a few hours. Given the number of address tweaks we do and the number of >messages we forward, new mailers need to be run in sidelined test machines >for N days or weeks before we can put them into production. I tend to >hack sendmail.cf "live", but it's a bit like running adb on a running >kernel -- nobody else around here will try it :-). And that, friends, is what separates the Real Computers from the toys. daver.bungi.com only deals with about 4,500 messages per week! My hat is off (again) to Paul for: 1. Being able to administer such a system. 2. Putting up with people like me that make sweeping assumptions. Thanks, Paul. (a 700 line sendmail.cf? <shudder>) What I should have said is that for small sites, getting smail3 running is very easy. It also ports very easily to System V environments. If you are in such a catagory, please give smail3 a try. -- Dave Rand {pyramid|mips|bct|vsi1}!daver!dlr Internet: dlr@daver.bungi.com
tron@veritas.uucp (Ronald S. Karr) (08/28/90)
In article <VIXIE.90Aug26205245@volition.pa.dec.com> vixie@decwrl.dec.com (Paul A Vixie) writes: >Anyone know when smail3 is coming out of beta-test, OFFICIALLY ? Anyone can get it now. It is on uunet definitely, and have heard that it is at the OSU archive site, for annonymous uucp, as well. It hasn't been posted because of some design problems that are likely to be a problem for naive users, and partially because I have entirely insufficient available time for fielding questions or complaints from large numbers of users. However, I won't really be happy with it until the design problems are resolved, so you can probably still think of it as a generally available beta release, rather than an officially available general release. -- tron |-<=>-| ARPAnet: veritas!tron@apple.com tron@veritas.com UUCPnet: {amdahl,apple,pyramid}!veritas!tron
honey@doom.ifs.umich.edu (Peter Honeyman) (08/29/90)
jeh@dcs.simpact.com writes: >a TB in g-spoof mode not only > "knows about" the data link level, it looks for and interprets the "messages" > (S, SY, CY, etc.) that mark the beginning and end of a file transfer. It has > to, because it has to know to flush its buffers and prepare for line turnaround > when the sending system sends the last packet of a file. no, no, no, the trailblazer spoof code doesn't know anything about the format of uucp's upper layer protocol. peter
les@chinet.chi.il.us (Leslie Mikesell) (08/29/90)
In article <1990Aug25.144323.10811@DSI.COM> syd@DSI.COM writes: > >>>Technically: There is a "protocol negotiation" sequence in the uucp startup, >A new protocol will only handle the actual sending of a single request >itself. If you just drop a new protocol into the existing routines, that would be true, but it would also be possible after the protocol negotiation to drop into an entirely new mode. >If you are a mail site, and sending lots of small messages for mail, >then you can improve your throughput by switching to a different >mail protocol, such as BSMTP, still sent via uucp, instead of rmail. But only if you are willing to delay mail while your mailer attempts to bundle requests. A single message sent BSMTP will still cause 2 files to be sent via uucp. >If its a packet link, then just a protocol with a larger window and >selective reject would be needed. Again, its only the transfer of >a single file that can be improved by changing the 'protocol' module >within uucico. You would need a whole different concept as a file >transfer device to improve more than that, and then it wouldn't >be uucico. No, it would still be uucico as long as it can negotiate and use 'g' protocol. It would be much more complicated to stream files, though since you would need to delay deleting the request from the sender's queue until the ack comes back, which might cause a job to be resent later accidentally if the connection is broken at the wrong moment. Saving status between calls with a startup negotiation that includes "last packet received" could avoid that problem, though. Les Mikesell les@chinet.chi.il.us
les@chinet.chi.il.us (Leslie Mikesell) (08/29/90)
In article <1565.26d6609c@dcs.simpact.com> jeh@dcs.simpact.com writes: >These links provide an error-free path, no? There is already the "f" protocol >which is intended for use through mostly-error-free, flow-controlled links. It >sends a single checksum at the end of the entire file, relying on the network >to get the data through and relying on the fact that while the RS232 (or >whatever) connection to your local modem may not be perfect it is quite close. I believe that "f" protocol is for 7-bit links and thus adds a lot of overhead, and not everyone has it. "e" protocol works if the link is, in fact, error free (i.e. a network with end to end error control), but again, not everyone has it. >> The big hangup here is that you need a source license to begin with, and >> then you can't distribute copies. Does anyone have a HDB compatible >> uucico that can be freely distributed? >well, how "compatible" does it have to be? DECUS uucp uses old-style spool >file names and a single spool directory but talks to HDB systems just fine. It >also has the best login and dialer scripting capability of any uucp arround >(not just my opinion). I had in mind just dropping a new uucico into a standard HDB program set. Uucp, uux, uuxqt all work just fine. The only thing lacking in the dialer scripts (since \M,\m was added) is the ability to adjust speeds to match up with a modem that sync's with the answering end, and perhaps a "reset" script. > .... so I have declared it to be in the public domain. How can I get a copy? >We are going to go to neighbor-system-specific spool directories for the next >release. (Right, Mark? :-) HDB-style permissions files are so bizarre that >we'll be doing something different. Hmm, nothing wrong with HDB permissions here except that a syntax error in the file will make uucico silently fail. Perhaps under VMS you can do something better than the LOGNAME/MACHINE setup that is needed under unix. >> I'd like to see a protocol where you could specify a minimum and >> maximum packet and window size per host, device and dialer. >we support this in 'g' via an options field in the entries in the systems file. You need it for the devices file also, since you may have many choices for connections to the same host. The differences in dialers could be accomodated by multiple entries for the same device. >Unfortunately, in order to use this with TrailBlazer modems, we'd have to >abandon their fabled 'g' protocol spoofing. a TB in g-spoof mode not only >"knows about" the data link level, it looks for and interprets the "messages" >(S, SY, CY, etc.) that mark the beginning and end of a file transfer. It has >to, because it has to know to flush its buffers and prepare for line turnaround >when the sending system sends the last packet of a file. Or you could just spoof the spoofing. Pretend you were sending one big file with the normal 'g' protocol (perhaps making it look like a cpio archive of all the files you want to send) and let the other end extract them back out. This presents the problems of refusing transmissions (who cares?) and restarting after an interruption (possible, but perhaps not worth the trouble). Les Mikesell les@chinet.chi.il.us
chip@tct.uucp (Chip Salzenberg) (08/31/90)
According to vixie@decwrl.dec.com (Paul A Vixie): >we run 27 dequeuing agents (sendmail -q) in parallel; >we have a 700-line sendmail.cf ... > >The bright side is that we're only forwarding 14,000 messages a day. Exhibit A: a REAL POSTMASTER[tm]. >There's no way in the world that we could run a different mailer in only >a few hours. Granted. But when decwrl.dec.com converts to Smail 3, then I'll know it's *really* arrived. :-) -- Chip Salzenberg at Teltronics/TCT <chip@tct.uucp>, <uunet!pdn!tct!chip>
vixie@wrl.dec.com (Paul Vixie) (09/01/90)
In article <26DD637F.39E1@tct.uucp> chip@tct.uucp (Chip Salzenberg) writes:
# Exhibit A: a REAL POSTMASTER[tm].
Don't be too impressed -- postmaster@uunet.uu.net is a REAL job.
--
Paul Vixie
DEC Western Research Lab <vixie@wrl.dec.com>
Palo Alto, California ...!decwrl!vixierick@uunet.UU.NET (Rick Adams) (09/05/90)
In article <1990Sep1.060403.15071@wrl.dec.com>, vixie@wrl.dec.com (Paul Vixie) writes: > In article <26DD637F.39E1@tct.uucp> chip@tct.uucp (Chip Salzenberg) writes: > # Exhibit A: a REAL POSTMASTER[tm]. > > Don't be too impressed -- postmaster@uunet.uu.net is a REAL job. Actually, it's THREE real jobs (or at least I pay 3 real people to do it. They would probably claim that it's more than 3 real jobs...) --rick