ziegast@eng.umd.edu (Eric W. Ziegast) (05/15/91)
Here's a good project for someone to try if it hasn't been done already... Don't you wish there was an anonymous rcp protocol? Tonight I just downloadd the InterViews distribution from stanford using ftp. They put alot of stuff in sub-directories especially things like compiled code. It would have been easier for me if I could just anon-rcp -r interviews.stanford.edu:dist /software/interviews Instead I spent lots of time using mkdirs, cd's, lcd's, mgets , etc. to get all of it's subdirectories transferred to my machine. Do you suppose someone could create a program to sign on as ftp and recursively go through some specified directory and transfer the whole thing in one non-interactive command? I've seen some feeds about a chat2.pl (?) that could help do such non-interactive things and I wonder if anyone has thought about doing this. The only alternative is an extension to the ftp protocol to allow recursive transfers. +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-+ | Eric W. Ziegast Internet: ziegast@eng.umd.edu | | Univ. of Merryland Phonenet: Eric@[301.405.3689] | +=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-+
clipper@csd.uwo.ca (Khun Yee Fung) (05/15/91)
Sorry if this does not have a lot to do with Perl.
>>>>> On 15 May 91 00:51:36 GMT, ziegast@eng.umd.edu (Eric W. Ziegast) said:
Eric> Do you suppose someone could create a program to sign on as ftp and
Eric> recursively go through some specified directory and transfer the
Eric> whole thing in one non-interactive command?
I have been thinking about this for a long time. All the solutions I
could think of are all basically hacks. The biggest problem is the
multitude of operating systems out there. All with different ways of
organizing directories and files. If you assume only Unix systems,
then I have encountered three very different ftpds. One is the
pseudo-standard vanilla ftpd from Berkeley, one is like the ftpd of
the emsworth node of CMU, and the other one I forgot where I saw but
it has brief explanations on the same lines as the file and directory
entries. You can of course write a program just to handle the
pseudo-standard ftpds. There is already one such program, a bourne
shell script. I once had a copy. If that is all you want, find that
program.
Suppose the ftp protocol allows recursive transfering. There is still
a problem: some operating systems do not have a tree-like file system.
Too bad. If you define how the file system should look like in ftp,
think about millions of people who use ftp to transfer files to and
from different machines. They surely do not want to have a different
look to their directories and files when they log in as a user and log
in via ftp. So this solution is out as anonymous ftp is only one small
application of ftp. I personally think that the purposes of ftp and
anonymous ftp are very different. The only solution is then to define
a protocol specifically for anonymous ftp. In this way, you can
include all the features (e.g. a line of description for each file or
directory) you want without worrying about other ftp applications. I
somehow don't think the networking gods will be interested in this. It
is up to us, anonymous ftp users, to derive our own protocol and make
it the de facto standard, I guess.
I wrote a ftp client long ago and posted it here a few weeks ago. All
the basic stuff is there and you can surely modify it to recursively
get all the files in a directory tree. This program allows you to
specify a command file even now. It will also let you change your mind
half way, specify a set of commands, get rid of the control terminal,
and send the whole thing to background.
Khun Yee
P.S. If you are interested, I have a draft anonymous ftp protocol I
wrote last year. Not many people cared.
--
Name: Khun Yee Fung Email: clipper@csd.uwo.ca
Paper mail: Department of Computer Science, Middlesex College
The University of Western Ontario, London, Ontario, N6A 5B7 CANADAcristy@eplrx7.uucp (John Cristy) (05/15/91)
In article <1991May15.005136.20447@eng.umd.edu> ziegast@eng.umd.edu (Eric W. Ziegast) writes: > >Do you suppose someone could create a program to sign on as ftp and >recursively go through some specified directory and transfer the >whole thing in one non-interactive command? > Fetch contrib/ImageMagick.tar.Z on export.lcs.mit.edu. You will find XTP in the ImageMagick/xtp directory. DESCRIPTION Xtp is a utility for retrieving, listing, or printing files from a remote network site, or sending files to a remote network site. Xtp performs most of the same functions as the ftp program, but does not require any interactive commands. You simply specify the file transfer task on the command line and xtp performs the task automatically. EXAMPLES To retrieve file display.tar.Z from host wizard.dupont.com, use: xtp -binary -retrieve display.tar.Z wizard.dupont.com To retrieve all the files from directory public/documents from host wizard.dupont.com, use: xtp -binary -retrieve documents/ wizard.dupont.com public cristy@dupont.com -- The UUCP Mailer
russell@ccu1.aukuni.ac.nz (Russell J Fulton;ccc032u) (05/16/91)
ziegast@eng.umd.edu (Eric W. Ziegast) writes: >Here's a good project for someone to try if it hasn't been done >already... >Don't you wish there was an anonymous rcp protocol? Tonight I just >downloadd the InterViews distribution from stanford using ftp. >They put alot of stuff in sub-directories especially things like >compiled code. It would have been easier for me if I could just I have a package called rftp that I got of this list a while back. It was written by Mike Ferrara <mikef@hpsadle.hp.com>. The r is for recursive it is written in perl and requires sockets. I suggest you contact Mike. Russell BTW Does anybody archive *all* the code that is posted to this group? -- Russell Fulton, Computer Center, University of Auckland, New Zealand. <rj_fulton@aukuni.ac.nz>
tchrist@convex.COM (Tom Christiansen) (05/17/91)
From the keyboard of russell@ccu1.aukuni.ac.nz (Russell J Fulton;ccc032u):
:BTW Does anybody archive *all* the code that is posted to this group?
Barring news glitches, I have every article posted except for a few
in the beginning. So does Larry, I believe. But they aren't in
any decent retrieval system (I use MH, but it's far too slow for
complex retrievals.) From the FAQ:
5) Are archives of comp.lang.perl available?
Yes, although they're poorly organized. You can get them from
the host betwixt.cs.caltech.edu (131.215.128.4) in the directory
/pub/comp.lang.perl. Perhaps by next month you'll be able to
get them from uunet as well. It contains these things:
comp.lang.perl.tar.Z -- the 5M tarchive in MH/news format
archives/ -- the unpacked 5M tarchive
unviewed/ -- new comp.lang.perl messages since 4-Feb or 5-Feb.
These are currently stored in news- or MH-style format; there are
subdirectories named things like "arrays", "programs", "taint", and
"emacs". Unfortunately, only the first ~1600 or so messages have been
so categorized, and we're now up to almost 5000. Furthermore, even
this categorization was haphazardly done and contains errors.
A more sophisticated query and retrieval mechanism is desirable.
Preferably one that allows you to retrieve article using a fast-access
indices, keyed on at least author, date, subject, thread (as in "trn")
and probably keywords. Right now, the MH pick command works for this,
but it is very slow to select on 5000 articles.
If you're serious about this, your best bet is probably to retrieve
the compressed tarchive and play with what you get. Any suggestions
how to better sort this all out are extremely welcome.
--tom
--
Tom Christiansen tchrist@convex.com convex!tchrist
"So much mail, so little time." csu@alembic.acs.com (Dave Mack) (05/19/91)
In article <1991May15.005136.20447@eng.umd.edu> ziegast@eng.umd.edu (Eric W. Ziegast) writes: >Here's a good project for someone to try if it hasn't been done >already... > >Don't you wish there was an anonymous rcp protocol? Tonight I just >downloadd the InterViews distribution from stanford using ftp. >They put alot of stuff in sub-directories especially things like >compiled code. It would have been easier for me if I could just > > anon-rcp -r interviews.stanford.edu:dist /software/interviews > >Instead I spent lots of time using mkdirs, cd's, lcd's, mgets , etc. >to get all of it's subdirectories transferred to my machine. > >Do you suppose someone could create a program to sign on as ftp and >recursively go through some specified directory and transfer the >whole thing in one non-interactive command? Someone has. This (almost) works like a champ, the one flaw being that the logic for placing the ftp'd hierarchy on the receiving machine is slightly brain damaged. I put an alias in my .cshrc: alias anonrftp rftp -uftp:mack2@mdsol1.mdc.com then running "anonrftp -sdir/subdir -g host.subdom.domain" will put everything in dir/subdir in my current working directory. My thanks to Mike Ferrara for making this available. Here's the shar file: Path: wuarchive!texbell!cs.utexas.edu!uunet!allbery From: mikef@hpsadle.hp.com (Mike Ferrara) Newsgroups: comp.sources.misc Subject: v13i026: recursive ftp'er in Perl Message-ID: <92118@uunet.UU.NET> Date: 5 Jun 90 23:26:50 GMT Sender: allbery@uunet.UU.NET Lines: 683 Approved: allbery@uunet.UU.NET (Brandon S. Allbery - comp.sources.misc) Posting-number: Volume 13, Issue 26 Submitted-by: mikef@hpsadle.hp.com (Mike Ferrara) Archive-name: rftp/part01 Here's a recursive ftp'er for grabbing, putting, and listing whole file trees via ftp. It uses perl to drive ftp, so in order to use it you need perl. There are two versions, rftp and rftp.nosockets. rftp.nosockets is much smaller, slower and has many fewer features. It was my proof of concept code, but it works without perl sockets. That's why I put it in the shar. For more info, see the code, or the man page. You may also need to fool with the #! line at the beginning, and make sure you've done a "makelib" (from perl distribution) on /usr/include/sys/sockets.h. This has been tested on perl 3 patchlevel 18. Mike Ferrara M/S 2LRR HP Signal Analysis Div R&D 1212 Valley House Drive Rohnert Park, CA 94928 (707) 794-4479 mikef%hpsadle@hp-sde.sde.hp.com mikef@hpsadle.hp.com --------------------------cut here----------------------------- # This is a shell archive. Remove anything before this line, # then unpack it by saving it in a file and typing "sh file". # # Wrapped by Mike Ferrara <mikef@hpsadle> on Tue Jun 5 10:41:15 1990 # # This archive contains: # rftp # LANG=""; export LANG PATH=/bin:/usr/bin:$PATH; export PATH echo mkdir - rftp mkdir rftp echo x - rftp/rftp cat >rftp/rftp <<'@EOF' #!/usr/local/bin/perl #Recursively decend via anon FTP and either get a listing #or retrieve the tree. # Usage: # rftp [options] host [list-file] # Options --- # [-s<source_dir>] Specify the root for transfer on remote host (default "/") # [-d<dest_dir>] Specify the root for transfer on local host (default ".") # [-l] Just a listing, thank you. # [-a] ASCII mode transfers (default: BIN) # [-g] get files # [-p] put files # [-u<user>:<passwd>] Specify a userid and passwd # [-b] debug mode # # return value is 0 if ok. # 1 if an error occurred during transfer. # 2 if login failed. # # # Mail bugs or comments to: # # Mike Ferrara M/S 2LRR # HP Signal Analysis Div R&D # 1212 Valley House Drive # Rohnert Park, CA 94928 # (707) 794-4479 # mikef%hpsadle@hp-sde.sde.hp.com # mikef@hpsadle.hp.com #main $cd=""; $dirs[1]='/'; $source='.'; $dest='.'; $ftpin="/tmp/ftpin$$"; $ftpout="/tmp/ftpout$$"; $listing=1; $bin=1; $reader=0; $writer=0; $user='anonymous'; $passwd=`hostname`; $debug=0; #Setup signal handler $SIG{'INT'}='cleanup'; $SIG{'HUP'}='cleanup'; $SIG{'QUIT'}='cleanup'; $SIG{'TERM'}='cleanup'; &parseopts; if (!($putting)&&!($listing)) { if (-d $dest) { } else { system("mkdir -p $dest"); } chdir $dest; } &setupcomm; if ($writer) { open(LSOUT,">$lsout"); open(FTPIN,"| ftp -i -v -n 1>$ftpout 2>&1"); select(FTPIN);$|=1;select(stdout); &sendftp ("open $host"); &sendftp ("user $user $passwd"); while (1) { #Read from the socket to see if login worked. $_=<NS>; last if (/^230\s/); exit(2) if (/^530\s/); } if ($bin==1) { &sendftp("bin"); } undef($lastdirectory); if (!$putting) { &recurse; } if ($putting) { &putfiles; } &sendftp("quit"); close(FTPIN); &cleanup; } if ($reader) { &readloop; } sub putfiles { &sendftp("bin") if ($bin); open(FIND,"find $source -print |"); while ($_=<FIND>) { chop; $destfile="$dest/$_"; $destfile=~s,/\./,/,g; $destfile=~s,//,/,g; $destfile=~s,\.$,,; $destfile=~s,/$,,; $srcfile="$source/$_"; $srcfile=~s,/\./,/,g; $srcfile=~s,//,/,g; if (-f $_) { &sendftp("put $srcfile $destfile"); &readsock; next; } if (-d $_) { &sendftp("mkdir $destfile"); } } close(FIND); } sub parseopts { &Getopts('abs:d:lu:pg'); $host=shift(@ARGV); if (! defined($host)) { die "I need you to tell me the hostname!"; } if (defined($opt_s)) { $source=$opt_s; } if (defined($opt_d)) { $dest=$opt_d; } if ($opt_a==1) { $bin=0; } if ($opt_l) { $listing=1; $bin=0; $lsout=shift(@ARGV); } if (defined($lsout)) { } else { $lsout='-'; } if (defined($opt_u)) { ($user,$passwd)=split(":",$opt_u); } if (defined($opt_g)) { $listing=0; $putting=0; die "What do you want to do? put OR get?" if (defined($opt_p)); } if (defined($opt_p)) { $listing=0; $putting=1; die "What do you want to do? put OR get?" if (defined($opt_g)); } $debug=1 if (defined($opt_b)); } # getopts.pl - a better getopt.pl # Usage: # do Getopts('a:bc'); # -a takes arg. -b & -c not. Sets opt_* as a # # side effect. sub Getopts { local($argumentative) = @_; local(@args,$_,$first,$rest,$errs); local($[) = 0; @args = split( / */, $argumentative ); while(($_ = $ARGV[0]) =~ /^-(.)(.*)/) { ($first,$rest) = ($1,$2); $pos = index($argumentative,$first); if($pos >= $[) { if($args[$pos+1] eq ':') { shift(@ARGV); if($rest eq '') { $rest = shift(@ARGV); } eval "\$opt_$first = \$rest;"; } else { eval "\$opt_$first = 1"; if($rest eq '') { shift(@ARGV); } else { $ARGV[0] = "-$rest"; } } } else { print STDERR "Unknown option: $first\n"; ++$errs; if($rest ne '') { $ARGV[0] = "-$rest"; } else { shift(@ARGV); } } } $errs == 0; } # # readloop -- keep reading stuff from the $ftpout file and # stuffing it over the socket. # sub readloop { while (1) { if (-f $ftpout) { open (FTPOUT,$ftpout); while (1) { $_=<FTPOUT>; if (/^221\sGoodbye/) { last; } print(S $_); } exit(0); } } } # # Workhorse subroutine, gets a whole directory or listing # of a whole directory. It also forms the list of directories # below the current one. # sub readsock { $i=1; $n=0; while (1) { $_=<NS>; if ($listing==1) { print (LSOUT $_) if (!/^\s?$/ && !/^[0-9]*\s/); } if (/^d/) { chop; split; $dirs[$i]=pop(@_); $i=$i+1; } if (/^-/) { chop; split; $fname[$n]=pop(@_); $n=$n+1; } if (/^226\s/) { last; } if ((/^5[0-9][0-9]\s/)&&(!/^5[0-9][0-9]\sbytes/i)) { print (stderr "A fatal error occurred during transfer:"); print (stderr $_); exit(1); } } } # # Do the recursion, using getdir as the workhorse. # sub recurse { local(@dirlist)=@dirs; local($currentparent)=shift(@dirlist); while (defined($child=shift(@dirlist))) { $cd="$source/$currentparent/$child"; undef @dirs; $cd=~s,//,/,g; $cd=~s,//,/,g; $cd=~s,/$,,; if (($cd EQ $lastdirectory) && ($lastdirectory NE "")) { die "OOOPS! I'm looping!!"; } &sendftp("dir $cd"); # print ("dir $cd\n"); if ($listing==1) { print(LSOUT "\n$cd:\n"); } &readsock; if ($listing == 0) { $ddir="$dest/$currentparent/$child"; $ddir=~s,//,/,g; $ddir=~s,//,/,g; system("mkdir -p $ddir"); while (defined($file=shift(@fname))){ &sendftp("get $cd/$file $ddir/$file"); &readsock; } } $lastdirectory=$cd; $dirs[0]="$currentparent/$child"; &recurse; } } # # Delete the temporary files, close the output, and leave # sub cleanup { unlink($ftpout); kill 15,$childpid; close(LSOUT); exit(0); } sub sendftp { $line=@_[0]; $line="$line\n" if (!($line=~m/\n$/)); print (STDERR "$line") if ($debug); print (FTPIN $line); } # # Setup socket based communication between the child and parent # and fork # sub setupcomm { do 'sys/socket.h' || die "Can't do sys/socket.h"; $port=$$; $sockaddr='S n a4 x8'; chop($hostname=`hostname`); ($name,$aliases,$proto)=getprotobyname('tcp'); ($name,$aliases,$type,$len,$thisaddr)=gethostbyname($hostname); if ($childpid == fork) { $reader=1; #The child reads FTP output, client sleep 3; $client=pack($sockaddr,&AF_INET,0,$thisaddr); $server=pack($sockaddr,&AF_INET,$port,$thisaddr); socket(S,&PF_INET,&SOCK_STREAM,$proto) || die "socket: $!"; bind(S,$client) || die "bind: $!"; connect(S,$server) || die "connect: $!"; select(S);$|=1;select(stdout); } else { $writer=1; #The parent writes FTP input and effects transfers., server. $server=pack($sockaddr,&AF_INET,$port,"\0\0\0\0"); select(NS);$|=1;select(stdout); socket(S,&PF_INET,&SOCK_STREAM,$proto) || die "socket: $!"; bind(S,$server) || die "bind: $!"; listen(S,5) || die "connect: $!"; select(S);$|=1;select(stdout); loop: ($addr=accept(NS,S)) || goto loop; ($af,$port,$inetaddr)=unpack($sockaddr,$addr); } } @EOF chmod 755 rftp/rftp echo x - rftp/rftp.1 cat >rftp/rftp.1 <<'@EOF' .TH RFTP 1 .ds )H .ds ]W March 1990 .SH NAME rftp \- recursively ftp on a remote machine .SH SYNOPSIS .B rftp [ .I options ] .I host [ .I listing-file ] .SH DESCRIPTION .I Rftp\^ Drives ftp to recursively descend file trees, either for listings, or to get or put a bunch of files keeping the tree structure intact. Rftp is written in perl and requires that berkeley sockets be wired into it. This generally requires that a "makelib" be run on "/usr/include/sys/sockets.h" before running rftp. Rftp works as follows: 1) fork a copy of itself, and setup socket based IPC between parent and child. 2) start ftp, with output redirected to /tmp/ftpout$$. 3) poke stdin of ftp while child reads /tmp/ftpout$$, and stuffs the output back to the parent over the socket. .SS Options .PP There are several options: .TP .B [\-s source-dir] specifies the source directory for the transfer. Default is "." .TP .B [\-d dest-dir] specifies the destination directory for the transfer. Default is "." .TP .B [\-a] force transfer mode to ascii. Default is binary (image). .TP .B [\-l] for directory listing only. This is the default action. Listings will go to stdout or to "listing-file" if specified. .TP .B [\-u username:passwd] specify a user name and password for logging in. Default is user "anonymous" password `hostname`. .TP .B [-g] get files from remote host. .TP .B [\-p] send (put) files to remote host. .TP .SH DIAGNOSTICS The program tries to clue you in to common failures. Source is available. Return codes are as follows: 0 for normal exit, 1 for a failure during a transfer, and 2 for a login failure .SH EXAMPLES This command gets a directory listing from machine "jupiter" and writes it in file "xyzzy". .IP rftp jupiter xyzzy .TP This command retrieves the tree headed at "pub/pub" into "/users/foo/xx" on machine "jupiter" .IP rftp -spub/pub -d/users/foo/xx -g jupiter .SH AUTHOR .I rftp was developed by Mike Ferrara of Hewlett-Packard (Signal Analysis Division) mikef%hpsad@hp-sde.sde.hp.com or mikef@hpsadle.hp.com. .PP .SH "STANDARDS CONFORMANCE" .IR None. @EOF chmod 666 rftp/rftp.1 echo x - rftp/rftp.nosockets cat >rftp/rftp.nosockets <<'@EOF' #!/usr/local/bin/perl #Recursively decend via anon FTP and either get a listing #or retrieve the tree. # Usage: # rftp [options] host [list-file] # Options --- # [-s<source_dir>] Specify the root for transfer on remote host (default "/") # [-d<dest_dir>] Specify the root for transfer on local host (default ".") # [-l] Just a listing, thank you. # [-a] ASCII mode transfers (default: BIN) # ;# getopts.pl - a better getopt.pl ;# Usage: ;# do Getopts('a:bc'); # -a takes arg. -b & -c not. Sets opt_* as a ;# # side effect. sub Getopts { local($argumentative) = @_; local(@args,$_,$first,$rest,$errs); local($[) = 0; @args = split( / */, $argumentative ); while(($_ = $ARGV[0]) =~ /^-(.)(.*)/) { ($first,$rest) = ($1,$2); $pos = index($argumentative,$first); if($pos >= $[) { if($args[$pos+1] eq ':') { shift(@ARGV); if($rest eq '') { $rest = shift(@ARGV); } eval "\$opt_$first = \$rest;"; } else { eval "\$opt_$first = 1"; if($rest eq '') { shift(@ARGV); } else { $ARGV[0] = "-$rest"; } } } else { print STDERR "Unknown option: $first\n"; ++$errs; if($rest ne '') { $ARGV[0] = "-$rest"; } else { shift(@ARGV); } } } $errs == 0; } 1; sub getdir { $i=1; undef @dirs; $cd=~s,//,/,g; $mcd=$cd; $mcd=~s,^/,,; if ($cd EQ $lastdirectory) { die "OOOPS! I'm looping!!"; } open(FTPIN,">$ftpin"); print(FTPIN "open $host \n"); print(FTPIN "user anonymous xx\n"); print(FTPIN "cd /\n"); if ($bin==1) { print(FTPIN "bin \n"); } print(FTPIN "dir $cd\n"); if ($listing == 0) { system("mkdir -p $dest/$cd"); print (FTPIN "mget $mcd/* \n"); } print(FTPIN "quit \n"); close(FTPIN); # system("ftp -i -n <$ftpin 1>$ftpout 2>&1"); system("ftp -i -n <$ftpin 1>$ftpout "); open(FTPOUT,"$ftpout"); if ($listing==1) { print(LSOUT "\n$cd:\n"); } $dirs[0]=$cd; while($_=<FTPOUT>) { if ($listing==1) { print (LSOUT $_) if (!/^\s?$/); } if (/^\s?d/) { chop; split; $dirs[$i]=pop(@_); $i=$i+1; } } close(FTPOUT); $lastdirectory=$cd; } sub recurse { local(@dirlist)=@dirs; local($currentparent)=shift(@dirlist); while (defined($child=shift(@dirlist))) { $cd="$currentparent/$child"; &getdir; &recurse; } } sub cleanup { unlink($ftpin,$ftpout); } #main $cd=""; $dirs[1]='/'; $dest='.'; $ftpin="/tmp/ftpin$$"; $ftpout="/tmp/ftpout$$"; $listing=0; $bin=1; $SIG{'INT'}='cleanup'; $SIG{'HUP'}='cleanup'; $SIG{'QUIT'}='cleanup'; $SIG{'TERM'}='cleanup'; &Getopts('as:d:l'); $host=shift(@ARGV); if (! defined($host)) { die "I need you to tell me the hostname!"; } if (defined($opt_s)) { $dirs[1]=$opt_s; } if (defined($opt_d)) { $dest=$opt_d; } if ($opt_a==1) { $bin=0; } if (-d $dest) { } else { mkdir($dest,0755) || die "$dest already exists and is not a directory!"; } if ($opt_l) { $listing=1; $bin=0; $lsout=shift(@ARGV); } if (defined($lsout)) { } else { $lsout='-'; } open(LSOUT,">$lsout"); chdir $dest; &recurse; &cleanup; close(LSOUT); exit(0); @EOF chmod 755 rftp/rftp.nosockets chmod 755 rftp exit 0