david@ms.uky.edu (David Herron -- One of the vertebrae) (06/08/88)
In article <10267@ncc.Nexus.CA> lyndon@ncc.nexus.ca (Lyndon Nerenberg) writes: >In article <1787@van-bc.UUCP> sl@wimsey.bc.ca (10 Stuart Lynne) writes: >>Something else which could (should) be done is to get BSMTP implemented at >>UUNET. >I have the bsmtp code sitting here. If Rick's willing to install >it then so am I. Do I hear Stuart volunteering to teach smail >how to grok the bsmtp mailer? Well now, there's finally some interest in doing this :-), a year after I posted that code you've got sitting there. Anyway, let me toss out a few thoughts which'll have to be overcome. Getting my bsmtp program to interface to smail would be trivial, merely follow the outline I wrote into domail() (or whatever that routine was named) to call /bin/smail with the right arguments. Easy stuff ... Now, a couple of problems. Queueing. To take advantage of the batching in bsmtp you want to have as many mail messages in the batch as you can manage. But neither smail or sendmail will handle more than one message at a time, will they? (My memory on them is a little poor, but I don't remember anything for handling more than one message at any one moment). Under MMDF I would write a channel program derived from the SMTP channel. It'd look through the queue, pick up all the messages headed for a particular host, bundle them up and send them out. I'd also schedule the daemon to run (say) once an hour to increase the chances of having >1 message in a bundle. Under sendmail or smail how would one have a good chance of having more than one message per bundle? If you end up having only one message per bundle then there's no decrease in the number of files sent, and even a loss because of more data being sent -- the BSMTP wrapper. Routing. Yes, you can say that uunet does BSMTP (assuming that you get Rick to implement it). Now, what do you do after that? You'll have other sites with BSMTP capabilities also. Other places might want to send BSMTP to remote sites. How do you tell the world that you can do BSMTP and what domains you're willing to handle gatewaying? I'm not saying this very well so I hope I'm getting through anyway. I'm curious about something. There's no error recovery in the BSMTP program. Has anybody had any problems because of this? Should there be error recovery? Interesting coincidence. Just today I started work on an "obsmtp" counterpart which'll include a restructuring of the source for "ibsmtp". (Basically to spin off most of ibsmtp into a utils file which'll also be used in obsmtp). What should the arguments for obsmtp be? At the moment I'm using: obsmtp [ -f from ] [ -h local-host ] [ -t to ] ... [ -c cmd ] But don't really see how to make that do multiple things properly. My immediate application can't use multiple things in one batch because I don't badly need it for sending mail across BITNET. Besides, I'm not sure if the Crosswell mailer will decode a bsmtp file that has multiple messages -- it ought to. Anyway, another option would be something like: ( echo HELO local-host for f in ${files}; do obsmtp -f from -t to-list done echo QUIT ) | command Which could work out of the same program I described above -- assuming proper actions when options aren't specified. Thoughts? -- <---- David Herron -- The E-Mail guy <david@ms.uky.edu> <---- s.k.a.: David le casse\*' {rutgers,uunet}!ukma!david, david@UKMA.BITNET <---- But if you saw me on the road, would you know who I am? <---- That's why I wear a hat.
sl@van-bc.UUCP (pri=-10 Stuart Lynne) (06/08/88)
In article <10267@ncc.Nexus.CA> lyndon@ncc.nexus.ca (Lyndon Nerenberg) writes: ][ This doesn't have anything to do with modems any more. Followups ] are in comp.mail.uucp ] ]In article <1787@van-bc.UUCP> sl@wimsey.bc.ca (Stuart Lynne) writes: ]>Something else which could (should) be done is to get BSMTP implemented at ]>UUNET. ]>This would reduce costs *MORE* than spending money on faster drives. (Of ]>course money would have to be spent to pay someone at UUNET to implement ]>this, I'm sure they don't work there just for the fun of it, but I think ]>this would provide more bang for the buck!) ]I have the bsmtp code sitting here. If Rick's willing to install ]it then so am I. Do I hear Stuart volunteering to teach smail ]how to grok the bsmtp mailer? Sure! I guess it's about time us Canucks did something to show that we are worth keeping around. Besides I always find time to do things that save money. Especially when it's my money :-) -- Stuart.Lynne@wimsey.bc.ca {ubc-cs,uunet}!van-bc!sl Vancouver,BC,604-937-7532
lyndon@ncc.Nexus.CA (Lyndon Nerenberg) (06/09/88)
In article <9593@g.ms.uky.edu> david@ms.uky.edu (David Herron -- One of the vertebrae) writes: [ ... ] >Queueing. To take advantage of the batching in bsmtp you want to >have as many mail messages in the batch as you can manage. But >neither smail or sendmail will handle more than one message at >a time, will they? (My memory on them is a little poor, but I >don't remember anything for handling more than one message at >any one moment). In this case, smail has a lot more brains than sendmail - see my comments later... >Under MMDF I would write a channel program derived from the SMTP >channel. It'd look through the queue, pick up all the messages >headed for a particular host, bundle them up and send them out. >I'd also schedule the daemon to run (say) once an hour to increase >the chances of having >1 message in a bundle. Exactly the plan I have [ see later... ] >Under sendmail or smail how would one have a good chance of having >more than one message per bundle? If you end up having only one message >per bundle then there's no decrease in the number of files sent, >and even a loss because of more data being sent -- the BSMTP wrapper. This is a big problem with (our) sendmail right now, anyway. The solution I see is to redefine the mailers into sendmail classes based on the "transport" layer. What this means is we have to massage the output from pathalias a bit. As an example, about 80% of our mail wants to route via uunet. Unfortunately, sendmail doesn't know this. It assumes anything not on the Ethernet should get punted to a "relay" host. What I have in mind is to take the output from pathalias and sub-divide it, based on who the first host in the path is. If, after generating my paths database, I extract all the paths that route via uunet as the first hop, I can take those site names and place them into a file such that sendmail can understand that they are a class unto themselves: FB/usr/local/lib/uunet-bsmtp Any host found in this database would get routed to the "bsmtp" mailer, in preference to the generic "smail" mailer. The bsmtp mailer would queue into a directory which would be batched up just prior to my hourly UUCP polling rounds. Obviously you would need a seperate class for each site you talk to via bsmtp... (even *I* don't have that many direct uucp/mail links (yet :-) ) The second option is to build a table that tells smail how to get to the first host in its route. This would save a lot of CPU cycles in comparison to the first solution. >Routing. Yes, you can say that uunet does BSMTP (assuming that >you get Rick to implement it). Now, what do you do after that? >You'll have other sites with BSMTP capabilities also. Other >places might want to send BSMTP to remote sites. How do you >tell the world that you can do BSMTP and what domains you're >willing to handle gatewaying? I'm not saying this very well >so I hope I'm getting through anyway. Who you route to, and how you get the mail there, isn't something fred@foo shouldn't have to worry about. Depending on the phase of the moon, I might decide to route via uux, tcp, bsmtp, or tin can and wax string. Once the message is on my box, all I have to do is get it to the next node as specified in the envelope (without munging the header info). Whether it's via bsmtp is not important. The routing information is already provided by the Internet nameservers, and the UUCP map info. >I'm curious about something. There's no error recovery in the >BSMTP program. Has anybody had any problems because of this? >Should there be error recovery? This isn't any different from the current uux batching system. > [ ... ] What should the arguments >for obsmtp be? At the moment I'm using: > > obsmtp [ -f from ] [ -h local-host ] [ -t to ] ... [ -c cmd ] I think all you could reasonably pass along outside of the envelope context is the sending hosts name, and the transport protocol. >But don't really see how to make that do multiple things properly. >My immediate application can't use multiple things in one batch >because I don't badly need it for sending mail across BITNET. >Besides, I'm not sure if the Crosswell mailer will decode a bsmtp >file that has multiple messages -- it ought to. So let's fix Crosswell :-) All we're doing is cloning bnews' sendbatch. [ Call me when you get your Trailblazers, Dave ] -- {alberta,utzoo,uunet}!ncc!lyndon lyndon@Nexus.CA
sl@van-bc.UUCP (pri=-10 Stuart Lynne) (06/10/88)
In article <9593@g.ms.uky.edu> david@ms.uky.edu (David Herron -- One of the vertebrae) writes: >In article <10267@ncc.Nexus.CA> lyndon@ncc.nexus.ca (Lyndon Nerenberg) writes: >>In article <1787@van-bc.UUCP> sl@wimsey.bc.ca (10 Stuart Lynne) writes: >>>Something else which could (should) be done is to get BSMTP implemented at >>>UUNET. >Getting my bsmtp program to interface to smail would be trivial, >merely follow the outline I wrote into domail() (or whatever that >routine was named) to call /bin/smail with the right arguments. >Easy stuff ... Actually it's easier than that. Just have smail use a delivery program which can differentiate between sites that can receive BSMTP bundles and ones that don't. Messages to the former get routed to a mailer that creates the BSMTP message, to the latter get sent to uux rmail!... >Queueing. To take advantage of the batching in bsmtp you want to >have as many mail messages in the batch as you can manage. But >neither smail or sendmail will handle more than one message at >a time, will they? (My memory on them is a little poor, but I >don't remember anything for handling more than one message at >any one moment). What you have to do is to append the output of the BSMTP program to the end of an outbound mailbag, one per site that you talk to. When the size of that file exceeds a certain size (say 50k) you dispatch the mailbag to the remote site via rmail. Eg: bsmtp ..... >> remote.mailbag ... uux remote!bsmtp < remote.mailbag There are two problems. First contention for the mailbag file. You can't do the rmail with the actual mailbag file. You first have to lock it, rename it, create a new mailbag file, rmail it, and delete it. The second problem is that when a site calls in he probably wants to get all of his messages, including the ones in the mailbag. So during his login you have to check to see if the mailbag file is non empty, lock it, rename it, create an empty one, mail it, and continue. The problem here is contention for the locks might increase the time to get uucico running, causing the occasional timeout. This might be easily solved by having the script emit a buzzword after it's finished - i.e. "OK" - which the other ends uucico login script can look for - i.e. in: myname ord: mypasswd OK--OK--OK. >Routing. Yes, you can say that uunet does BSMTP (assuming that >you get Rick to implement it). Now, what do you do after that? >You'll have other sites with BSMTP capabilities also. Other I don't think you tell the world. BSMTP mail bundles simply replace smaller single message files on a single hop. So if you know that uunet can receive BSMTP mail bundles (for example by mailing them to uunet!bsmtp) you simply place there name in a file listing the hosts that you talk to that can receive BSMTP bundles. Similarily if you can receive BSMTP bundles and you know that uunet can send BSMTP bundles you inform uunet!postmaster that you can. He then does the same for you. Now when mail arrives at uunet it is sent to you via BSMTP instead of a single message at a time. The bottom line is that you have to inform the sites that talk to you, no one else is involved. >I'm curious about something. There's no error recovery in the >BSMTP program. Has anybody had any problems because of this? >Should there be error recovery? It should be at least as robust as rmail is now. It's just that if a set of files to tell a remote uuxqt to do an rmail gets trashed you only loose one message, now you loose the whole bundle. Don't think this will be a big problem. -- Stuart.Lynne@wimsey.bc.ca {ubc-cs,uunet}!van-bc!sl Vancouver,BC,604-937-7532
matt@ncr-sd.SanDiego.NCR.COM (Matt Costello) (06/18/88)
In article <9593@g.ms.uky.edu> david@ms.uky.edu (David Herron -- One of the vertebrae) writes: >Well now, there's finally some interest in doing this :-), a year >after I posted that code you've got sitting there. Anyway, let me >toss out a few thoughts which'll have to be overcome. I wrote two programs (bsmtp-send and bsmtp-rcv) to do Batched SMTP over UUCP communication links last year. Unfortunately I did them on company time so they are owned by NCR. I might be able to get permission to distribute them if necessary, or it should be easy to reverse-engineer them from the manual pages. I'll mail the manual pages upon request. I'm going to talk about how I implemented bsmtp-send, as bsmtp-rcv is trivial (with our mail router). It is not always pretty, but it works. This implementation is specific to HDB UUCP, although it could be modified for other flavors of UUCP. >Queueing. To take advantage of the batching in bsmtp you want to >have as many mail messages in the batch as you can manage. But >neither smail or sendmail will handle more than one message at >a time, will they? True. The solution is to let UUCP do it. This also presents the minimum delay since no mail is off hiding elsewhere. My solution is to keep a link to the transmitted data file and then append new mail messages to the queued data file whenever possible. This is very easy except for finding the queued data file in the first place; this is the part specific to Honey DanBer UUCP. Appending to the BSMTP batch involves opening a link to the queued batch and then appending from the trailing "QUIT" in the file which requires that the last 5-7 bytes of the file be overwritten. The links to the spooled data files are kept in the directory /usr/spool/uucp/.bsmtp with the system name as the file name. As long as the link count of the file is still two it means that the BSMTP batch has not been transmitted, although the UUCP lock file is checked to make sure that it does try to modify the data file while uucico is reading it. If it is not able to append to the existing batch it creates and queues a new one. When bsmtp-send creates a new batch it spools it using uux and then scans the UUCP spool directory for that system looking for it. The file must pass 3 criteria: the file name must start with a 'D', it must have the right date, and it must end with QUIT. The size of the file is not checked because it is passed to uux in a pipe and the length is not known. When the file is found it is linked into /usr/spool/uucp/.bsmtp for use by the next invocation of bsmtp-send. >Routing. Yes, you can say that uunet does BSMTP (assuming that >you get Rick to implement it). Now, what do you do after that? >You'll have other sites with BSMTP capabilities also. Other >places might want to send BSMTP to remote sites. How do you >tell the world that you can do BSMTP and what domains you're >willing to handle gatewaying? I'm not saying this very well >so I hope I'm getting through anyway. Only our neighbors need to know that we handle BSMTP. I prefer to think of BSMTP as a protocol layer on top of UUCP. Any neighbors of ncr-sd are welcome to send us BSMTP batches. See! I've just announced it. -- Matt Costello <matt.costello@SanDiego.NCR.COM> +1 619 485 2926 <matt.costello%SanDiego.NCR.COM@Relay.CS.NET> {ucsd,cbosgd,pyramid,nosc.ARPA}!ncr-sd!matt
honey@umix.cc.umich.edu (Peter Honeyman) (06/18/88)
here's an idea, inspired by matt's note.
# usage bmail host. input from pipe.
JOBDIR=/usr/spool/uucppublic/bsmtp
# check args, grab host name.
if [ $# != 1 ]; then
echo usage: `basename $0` host 2>&1
exit 1
fi
host=$1
if [ ! -d $JOBDIR/$host ]; then
mkdir $JOBDIR/$host # should check error status
fi
# make a uucp lock for $host.
uucplock $host $$
newbatchfile=$?
# clean old spooled stuff.
bsmtpclean $host
# get name of batch file.
if [ $newbatchfile = 0 ]; then
suffix=`oldbatchfile host`
else
suffix=`newbatchfile host`
fi
batchfile=batch.$suffix
statfile=stat.$suffix
controlfile=control.$suffix
# dump new mail into the batch file. (from stdin.)
addtobatch $batchfile
# if new, spool it. store the jobid for later reference.
if [ $newbatchfile != 0 ]; then
uux -s $JOBDIR/$host/$statfile host!bsmtpunpack '<' !$batchfile
if [ $? != 0 ]; then
exit 101
fi
echo $batchfile $statfile > $controlfile
fi
exit 0
bsmtpclean, uucplock, uucpunlock, oldbatchfile, newbatchfile, and
addtobatch are not hard -- one or more of them are appended. (these
too are just sketches, but they should get the idea across.)
i haven't thought this through entirely, so access protection is not
addressed here. but some combination of ruid, euid, file modes, and
PERMISSIONS entries are needed to flesh this out.
peter
# usage: bsmtpclean host
JOBDIR=/usr/spool/uucppublic/bsmtp
# check args, grab host name.
if [ $# != 1 ]; then
echo usage: `basename $0` host 2>&1
exit 1
fi
host=$1
cd $JOBDIR/$host # should check for error
# clean up.
for controlfile in $control.*; do
set `cat $i`
batchfile=$1
statfile=$2
if [ grep -s 'exited normally' $statfile ]; then
rm -f $batchfile $statfile $controlfile
fi
done
exit
# usage: uucplock host pid
# make a uucp lock for host. return 0 if successful,
# return 1 if a lock already exists.
LOCKDIR=/usr/spool/locks
# check args, grab host name.
if [ $# != 1 ]; then
echo usage: `basename $0` host 2>&1
exit 1
fi
host=$1
pid=$2
# make a lock if one does not exist. uucp will eventually clean up.
# this can be improved to override a dead lock.
cd $LOCKDIR
locklink=bsmtp-lock.$$
trap "rm -f $locklink" 0
echo $pid | awk '{printf("%10d\n", $1)}' > $locklink
lockfile=`echo LCK..$host|cut -c1-14` # or colrm 15
ln $locklink $lockfile
if [ $? != 0 ]; then
exit 101
fi
exit 0
exit $? # redundanttron@amdahl.uts.amdahl.com (Ronald S. Karr) (06/18/88)
In article <1787@van-bc.UUCP> sl@wimsey.bc.ca (10 Stuart Lynne) writes: >Something else which could (should) be done is to get BSMTP implemented at >UUNET. The Smail3.1 mailer, currently in Alpha test, handles a batched form of SMTP. It allows a site to a create a transport that sends a message formatted as: HELO site.domain MAIL FROM:<sender> RCPT TO:<recipient> ... DATA From: user (name) Subject: ... To: whomever body... . QUIT Smail3.1 can then send this over UUCP using a command such as: uux - host!rsmtp or: uux - host!rmail -bS Error recover on the recipient side is handled by returning mail to the sender identified in the "RCPT TO:" command. Paritially formed transactions (without the HELO and QUIT) can also be accumlated in a file or directory and periodically delivered in one one large batch of SMTP commands with one UUCP transaction. This method of delivery, or any number of other methods, can be selected on a per-host basis. Other features in Smail3.1 are: * Multiple alias files (using YP, DBM, sorted or unsorted ASCII files). * Multiple path files. * Networking with SMTP over TCP/IP, wherever the socket calls exist. * Mailing list directories (create a mailing list merely by creating a file in these directories--great with sticky directories). * The ability to call arbitrary programs to calculate paths. * Fairly simple site configuration to add new forms for aliasing/forwarding, host routing and delivery. * Standard hooks for adding new C routines to perform aliasing, host routing, delivery, or database lookup. * Adequate security to allow users to specify shell commands and mailbox files in alias and .forward files. * Command line compatibility with sendmail, allowing sendmail to be entirely replaced simply by copying smail3.1 to /usr/lib/sendmail. The following is an example of how one would batch together messages to be sent using batched SMTP in one UUCP transaction: First of all, create an entry in a routing configuration file to define a paths file: # define a paths file created by the pathalias program paths: driver=pathalias, # this file maps hosts to transports: method=/usr/lib/smail/methods/uucp-transports; # the paths file is stored as a DBM database: file=/usr/lib/smail/paths, proto=dbm Then, create the file /usr/lib/smail/methods/uucp-transports to state which hosts should receive mail over batched SMTP: # map hosts to transports uunet batched_smtp cbosgd batched_smtp * uux # all other hosts are delivered to remote rmail Then, entries are added to a transport description file: # deliver using uux-->rmail uux: driver=pipe, # deliver by calling a program max_chars=200, # at most 200 chars in command line max_addrs=5, # up to 5 addresses per call uucp, # limit envelope addresses to UUCP forms from; # expects a From_ line # deliver using the following command: cmd="/usr/bin/uux - -r -g$grade $host!rmail $(($user)$)" # accumulate batched SMTP commands in a file based on host: batched_smtp: driver=appendfile, # deliver by appending to a file -max_chars, # no limit on chars of address per invocation -max_addrs, # no limit on addresses per invocation -hbsmtp; # use batched SMTP without HELO and QUIT # deliver to a file based on the name of the host: file=/usr/spool/smail/smtp_outq/$host Then, to send these files periodically to remote hosts, use a script such as the following (executed from cron): #!/bin/sh cd /usr/spool/smail/smtp_outq for host in *; do # move to prevent inconsistent copy of file mv $host .$host # if one more transaction being added, allow to complete sleep 10 # send to remote host: uux - -r $host!rsmtp < .$host rm -f .$host done exit 0 Anybody interested in more information on smail (such as the current set of man pages) can send mail to: info-smail-request@uts.amdahl.com {decwrl,sun,uunet}!amdahl!info-smail-request to get on the smail3.1 information mailing list. The Smail3.1 mailer is related to the Smail2.5 mailer both by name and through discussions with Larry Auton and Mark Horton at the last summer USENIX (1987). It is only slightly related by code. The hope is that this will replace the previous version of smail at some time in the future. Hopefully, if Larry and Mark haven't become too bored waiting for significant progress this will still occur. The authors of Smail3.1 are Ronald S. Karr (myself) and Landon Curt Noll. Both of us are employees at Amdahl Corp. of Sunnyvale CA. -- tron |-<=>-| ARPAnet: amdahl!tron@Sun.COM tron@uts.amdahl.com UUCPnet: {decwrl,sun,uunet}!amdahl!tron [views above shouldn't be viewed as Amdahl views, or as views from Amdahl, or as Amdahl views views, or as views by Mr. Amdahl, or as views from his house]
david@ms.uky.edu (David Herron -- One of the vertebrae) (06/24/88)
In article <1793@van-bc.UUCP> sl@van-bc.UUCP (pri=-10 Stuart Lynne) writes: >In article <9593@g.ms.uky.edu> david@ms.uky.edu (David Herron -- One of the vertebrae) writes: >>In article <10267@ncc.Nexus.CA> lyndon@ncc.nexus.ca (Lyndon Nerenberg) writes: >>>In article <1787@van-bc.UUCP> sl@wimsey.bc.ca (10 Stuart Lynne) writes: >>>>Something else which could (should) be done is to get BSMTP implemented at >>>>UUNET. >>Getting my bsmtp program to interface to smail would be trivial, >>merely follow the outline I wrote into domail() (or whatever that >>routine was named) to call /bin/smail with the right arguments. >>Easy stuff ... >Actually it's easier than that. Just have smail use a delivery program which >can differentiate between sites that can receive BSMTP bundles and ones that >don't. Messages to the former get routed to a mailer that creates the BSMTP >message, to the latter get sent to uux rmail!... We're talking about two different halves of the problem. For some reason I was only talking about the receiving site and you're talking about the sending site. >>Queueing. To take advantage of the batching in bsmtp you want to >>have as many mail messages in the batch as you can manage. But >>neither smail or sendmail will handle more than one message at >>a time, will they? (My memory on them is a little poor, but I >>don't remember anything for handling more than one message at >>any one moment). >What you have to do is to append the output of the BSMTP program to the end >of an outbound mailbag, one per site that you talk to. When the size of that >file exceeds a certain size (say 50k) you dispatch the mailbag to the remote >site via rmail. Eg: Yeah, after some thought that's my basic plan also > bsmtp ..... >> remote.mailbag > ... > uux remote!bsmtp < remote.mailbag um, rfc976 already allows for uux remote!rmail path!b-smtp uux remote!rmail b-smtp So let's use that scheme, please? >>[Some things I said about routing issues, but I accidentally deleted ...] >I don't think you tell the world. BSMTP mail bundles simply replace smaller >single message files on a single hop. So if you know that uunet can receive >BSMTP mail bundles (for example by mailing them to uunet!bsmtp) you simply >place there name in a file listing the hosts that you talk to that can >receive BSMTP bundles. Well, that's fine and dandy if you only want to do BSMTP between two neighbors. However the stuff in rfc976 allows for BSMTP to be sent to some place far far away, across many hops. It would be nice to have some organized way of doing so. -- <---- David Herron -- The E-Mail guy <david@ms.uky.edu> <---- s.k.a.: David le casse\*' {rutgers,uunet}!ukma!david, david@UKMA.BITNET <---- A proud supporter of the Marcel Marceau Foundation <------ Because a mime is a terrible thing to waste