Sam_Coleman.NSD@LCCMAIL.OCF.LLNL.GOV (Sam Coleman) (12/13/90)
Subject: Time:8:39 AM
OFFICE MEMO FTP enhancements Date:12/5/90
We at LLNL are looking at requirements for future file transport between
various Unix machines. We use FTP now and we've looked at Background FTP. In
our environment, these seem to fall short in several areas that I'll describe
below. I'd appreciate your corrections and additions.
We'd like to stick with "standard" interfaces, maybe enhancing FTP and/or BFTP.
I'm wondering if anyone has encountered similar problems, developed better
solutions, or is interested in working with us to develop something better.
I haven't heard anything lately concerning the RFC for BFTP. Is anything
happening there? The chair of IETF (Internet Engineering Task Force), Phill
Gross, is not aware of any active FTP development. Would it be reasonable to
form a working group under IETF to enhance FTP and/or BFTP? Would an enhanced
FTP protocol make it easier to produce standard higher-level command or
graphical interfaces like HyperFTP and FTA?
Sam Coleman
Lawrence Livermore National Lab
coleman@llnl.gov
Some comments on FTP and BFTP
-----------------------------
Passwords - Both FTP and BFTP require clear-text passwords. Our security folks
won't allow clear-text passwords to be stored in files. Is Kerberos a solution?
Persistence - FTP has none; if a transfer fails, the user has to retry.
Background FTP has a simple retry mechanism.
Performance - File transfers go through the FTP client and TCP/IP. Is there a
faster way to do the actual data transfers?
Notification - BFTP sends mail to the client when a transfer completes. Is this
the best way to handle notification in production environments?
Synchronous operation - FTP does one thing at a time. The user can give
BFTP multiple jobs, but BFTP seems to plug through these synchronously if they
are invoked using wild cards. Thus you can't take advantage of parallelism that
might be available in the file system (e.g. mounting tapes concurrently).
Flow control - FTP servers generally have no mechanisms to inhibit file
transfers during heavy load; file movement slows to a crawl as the load on the
server increases, then they reject log-ons, and then connections are refused
(maybe it's a good thing that FTP is synchronous?). We would like to put flow
control in the FTP server to delay file transfers when necessary, but allow
log-ons and short commands (like ls). Perhaps priority could be given to small
and/or fast transfers?
Tape files - FTP doesn't handle well the case where there is a delay accessing
a file, e.g. when the file is on tape.
Should there be a way to configure FTP? A .FTPRC file to allow the user to
specify his own defaults.
Error reporting in some clients could be improved.
Clients - there are differences between various FTP clients. Perhaps a working
group could help standardize them?
Missing functions - Some useful functions, like chmod, are provided by Unicos
FTP but not by other clients. When FTP is the primary interface to an archive,
it would be nice to have more complete file system functionality.
Wild cards - FTP provides MGET, MPUT, and MDEL (why not GET *, etc?), but
functions like chmod don't support wild cards. I would suggest wider and more
consistent semantics. Should the server prompt the user rather than the
client?libes@cme.nist.gov (Don Libes) (12/18/90)
In article <9012160626.AA14268@ucbvax.Berkeley.EDU> Sam_Coleman.NSD@LCCMAIL.OCF.LLNL.GOV (Sam Coleman) writes: > We use FTP now and we've looked at Background FTP. In >our environment, these seem to fall short in several areas that I'll describe >below. I'd appreciate your corrections and additions. >I'm wondering if anyone has encountered similar problems, developed better >solutions, or is interested in working with us to develop something better. Rather than trying to create a BFTP with a gazillion options, and not knowing if you've solved everyone's special problem, you might try using an intelligent front-end like 'expect'. This provides people with an open-ended way of controlling ftp without having to rewrite anything. (In fact, ftp is used in the expect man page as an example.) Here are some specific comments on how expect address your requirements. >Passwords - Both FTP and BFTP require clear-text passwords. Our >security folks won't allow clear-text passwords to be stored in files. >Is Kerberos a solution? Let expect prompt the user for the passwords immediately. (This is also described in the man page.) Expect remembers them, and uses them when it comes time to connect to the remote system later. >Persistence - FTP has none; if a transfer fails, the user has to retry. >Background FTP has a simple retry mechanism. Expect can retry however you want. Every minute, hour, just once, twice, only during wee hours, whatever. >Notification - BFTP sends mail to the client when a transfer completes. >Is this the best way to handle notification in production environments? Expect can notify you any way you want. Mail, write, reboot, whatever. >Synchronous operation - FTP does one thing at a time. The user can give >BFTP multiple jobs, but BFTP seems to plug through these synchronously if >they are invoked using wild cards. Thus you can't take advantage of >parallelism that might be available in the file system (e.g. mounting >tapes concurrently). Expect can run multiple ftps simultaneously (on different machines, for that matter). And it can notice whenever one completes and request the next file immediately. >Flow control - FTP servers generally have no mechanisms to inhibit file >transfers during heavy load; file movement slows to a crawl as the load on the >server increases, then they reject log-ons, and then connections are refused >(maybe it's a good thing that FTP is synchronous?). We would like to put flow >control in the FTP server to delay file transfers when necessary, but allow >log-ons and short commands (like ls). Perhaps priority could be given to >small and/or fast transfers? If you have another command (like ping for network load) or there is some port that you can telnet to which reports the load, you can have expect check that first. You can order files by size. >Tape files - FTP doesn't handle well the case where there is a delay accessing >a file, e.g. when the file is on tape. FTP needs to be rewritten for this. Expect can't help you here. >Should there be a way to configure FTP? A .FTPRC file to allow the user to >specify his own defaults. Trivial. Have your expect script source your .ftprc file. >Error reporting in some clients could be improved. Using expect, you can translate error messages, or report any or all of the session verbatim. >Wild cards - FTP provides MGET, MPUT, and MDEL (why not GET *, etc?), >but functions like chmod don't support wild cards. You can emulate this. The rest of your issues (faster transmission, complete file system functionality) require ftp and protocol to be rewritten. I don't have scripts that support all the features you have asked for, but they are easy to write. And, like I said earlier, you're never going to be able to think of everyone's need. For example, one of the sample expect scripts in the distributions attempts to connect as "libes" and only if that fails does it try to connect as "anonymous". Another neat thing capability would be to name a file, and let the script figure out what directory it is in (possibly by looking in an "ls-lR" file if it finds one.) Or make it work with McGill University's archie system! The script would telnet to mcgill, find out what ftp sites have the file and what the full directory paths are, and then ftp it. If one site is down, it could go on to the next one. (I think you'll be surprised how short this script will be.) I know some people like dedicated tools such as BFTP. But I suspect that if it really addresses all the issues you've raised, it's no longer going to be simple to use as what we all associate with FTP, and may not satisfy as many people as originally intended. In any case, there will always be a tradeoff between simplicity and completeness. Meanwhile, in the year that its going to take to add all the features you asked for, try using expect. Don Libes libes@cme.nist.gov ...!uunet!cme-durer!libes