michael@fe2o3.UUCP (Michael Katzmann) (08/15/90)
I have a task that requires the ability to fork off another programme but to supply it's standard input and output. The SysV manuals describe FILE *popen( command, type ) char *command, *type; which execs the command a la "system()" and creates a pipe. "type" and be "r" if you want to read from the standard output of "command", or "w" if you want to write to standard input. However there doesn't seem to be any way to use this routine to do both similtaneously. What is the usual way to to this? Important points: The exec'ed command must run asynchronously (obvious if the parent is supplying input.) The child process id must be available to the parent. (so that it can be killed if necessary) Any ideas would be appreciated. --------------------------------------------------------------------- email to UUCP: uunet!mimsy!{arinc,fe2o3}!vk2bea!michael _ _ _ _ Amateur | VK2BEA (Australia) ' ) ) ) / // Radio | G4NYV (United Kingdom) / / / o _. /_ __. _ // Stations| NV3Z (United States) / ' (_<_(__/ /_(_/|_</_</_ Michael Katzmann Broadcast Sports Technology. 2135 Espey Ct. #4 Crofton Md. 21114 USA Ph: +1 301 721 5151
bogatko@lzga.ATT.COM (George Bogatko) (08/15/90)
In article <377@fe2o3.UUCP>, michael@fe2o3.UUCP (Michael Katzmann) writes: > > I have a task that requires the ability to fork off another programme but > to supply it's standard input and output. . . . > However there doesn't seem to be any [way to get popen] to do both > similtaneously. > > > Any ideas would be appreciated. I wrote new routines to do it called fpipe.[ch...], and I've posted them in alt.sources. If you can't get to alt.sources, write me privately, and we'll try to get it to you. GB
bogatko@lzga.ATT.COM (George Bogatko) (08/15/90)
HI: This posting is in response to: "michael@fe2o3.UUCP (Michael Katzmann @ Rusty's BSD machine at home)" found in comp.unix.questions > I have a task that requires the ability to fork off another programme but > to supply it's standard input and output. . . . > However there doesn't seem to be any way to use popen() to do both > similtaneously. > What is the usual way to to this? ********** Popen is expensive. It requires a shell for 'system' and then another shell to run your request in. On large systems with many users, this is death. I did it another way, which doesn't require any sub-shells. However It will not handle complicated piping schemes, which 'popen()' does. GB ********** CUT HERE ********** CUT HERE ********** CUT HERE ********** CUT HERE #! /bin/sh # This is a shell archive, meaning: # 1. Remove everything above the #! /bin/sh line. # 2. Save the resulting text in a file. # 3. Execute the file with /bin/sh (not csh) to create: # makefile # fcopy.c # fpipe.c # fpipe.h # main.c # tst.c # This archive created: Tue Feb 27 13:44:14 1990 export PATH; PATH=/bin:/usr/bin:$PATH if test -f 'makefile' then echo shar: "will not over-write existing file 'makefile'" else cat << \SHAR_EOF > 'makefile' CFLAGS=-I. LIBS= OBJS=\ main.o \ fpipe.o \ fcopy.o OUTPUT=\ main $(OUTPUT): $(OBJS) cc -o $(OUTPUT) $(OBJS) $(LIBS) SHAR_EOF fi if test -f 'fcopy.c' then echo shar: "will not over-write existing file 'fcopy.c'" else cat << \SHAR_EOF > 'fcopy.c' /****************************************************************************** * * * fcopy.c * * * ******************************************************************************/ /*-------------------------- INITIAL CODING DATE ----------------------------- Thu Nov 10 10:56:37 EST 1988 by George M. Bogatko -------------------------------- HEADER FILES -----------------------------*/ #include <stdio.h> #include <fpipe.h> #include <fcntl.h> /*------------------ TYPEDEF'S, DEFINES, STRUCTURE DEF'S ------------------*/ /*---------------- IMPORTED GLOBAL VARIABLE/FUNCTION DEF'S ----------------*/ extern char *fgets(); /*---------------- EXPORTED GLOBAL VARIABLE/FUNCTION DEF'S ----------------*/ /*---------------- INTERNAL GLOBAL VARIABLE/FUNCTION DEF'S ----------------*/ #ident "%W% %G% - George M. Bogatko -" /*----------------------------------------------------------------------------- SYNOPSIS: fcopy(in_fd, out_fname, flag) DESCRIPTION: FCOPY copies lines from an IN_FD (maybe a pipe) to a standard file. The arguments are: in_fd - a 'FILE *' type file descriptor out_fname - The file you want to write to flag - Whether your want the READ to be BLOCK or NON_BLOCK with BLOCK, the fgets will block (in the case of pipes, a block occurs if there is nothing in the pipe to read. with NON_BLOCK, the first READ is BLOCKED, (to allow pipe synchronization) and subsequent reads are NON_BLOCK. RETURN: 0 on success -1 on error (use perror()) =============================================================================*/ int fcopy(in_fp, outname, blockflag) FILE *in_fp; char *outname; int blockflag; { FILE *out_fp; char buf[BUFSIZ]; enum {first, next} time = first; if( (out_fp = fopen(outname, "w")) == (FILE *)NULL ) return -1; while( fgets(buf, BUFSIZ, in_fp) != (char *)NULL ) { if( time == first ) { if( blockflag == NO_BLOCK ) { if(fcntl( fileno(in_fp), F_SETFL, O_NDELAY ) == -1 ) { perror(""); fclose(out_fp); return -1; } } time = next; } fputs(buf, out_fp); } fclose(out_fp); return 0; } SHAR_EOF fi if test -f 'fpipe.c' then echo shar: "will not over-write existing file 'fpipe.c'" else cat << \SHAR_EOF > 'fpipe.c' /****************************************************************************** * * * fpipe.c * * * ******************************************************************************/ /*-------------------------- INITIAL CODING DATE ----------------------------- Thu Nov 10 10:30:22 EST 1988 by George M. Bogatko -------------------------------- HEADER FILES -----------------------------*/ #include <stdio.h> #include <fpipe.h> #include <fcntl.h> #include <varargs.h> /*------------------ TYPEDEF'S, DEFINES, STRUCTURE DEF'S ------------------*/ /*---------------- IMPORTED GLOBAL VARIABLE/FUNCTION DEF'S ----------------*/ /*---------------- EXPORTED GLOBAL VARIABLE/FUNCTION DEF'S ----------------*/ /*---------------- INTERNAL GLOBAL VARIABLE/FUNCTION DEF'S ----------------*/ #ident "%W% %G% - George M. Bogatko -" /*----------------------------------------------------------------------------- SYNOPSIS: int fpipe(flag, arg_array, program, arg1, arg2, ... argn, 0 ); DESCRIPTION: FPIPE creates a two way pipe between a calling process and a child process. That child process should expect to read from STANDARD IN and write to STANDARD OUT. The arguments are: flag - one of STD_ERR_SCR or STD_ERR_NUL. STD_ERR_SCR will put STANDARD ERR to the screen, STD_ERR_NUL will put STANDARD ERR to "/dev/null" arg_array - an array of 2 'FILE *' type pointers EX: FILE *arg_array[2]; When the function returns, arg_array[0] will contain the READING end of the pipe connected to the process's standard output, and arg_array[1] will contain the WRITING end of the pipe connected to the process's standard input. program - the full path of the program you want to invoke. arg1 ... argN - all the arguments to your program 0 - the termination point of the arg1 ... argN sequence. RETURN: the PROCESS ID of the created child on success, -1 on any error (use "perror()" to find out what happened). CAVEATS: The LAST argument MUST be 0 =============================================================================*/ int fpipe(flag, arg_array, va_alist) int flag; FILE *arg_array[]; va_dcl { char **args; int in_pfd[2]; int out_pfd[2]; FILE *read_fp, *write_fp; char *file; int i; int proc_id = 0; if( pipe(in_pfd) == (-1) ) return(-1); if( pipe(out_pfd) == (-1) ) return(-1); args = (char **)&va_alist; /* won't work on Pyramid */ switch( proc_id = fork() ) { case -1: return(-1); case 0: close(0); dup(out_pfd[0]); close(1); dup(in_pfd[1]); if( flag == STD_ERR_NUL ) { close(2); if(open("/dev/null", O_WRONLY) == -1) return(-1); } close(in_pfd[0]); close(in_pfd[1]); close(out_pfd[0]); close(out_pfd[1]); execv(args[0], args); return(-1); default: close(in_pfd[1]); close(out_pfd[0]); if( (read_fp = fdopen( in_pfd[0], "r" )) == (FILE *)NULL ) return(-1); if( (write_fp = fdopen( out_pfd[1], "w" )) == (FILE *)NULL ) return(-1); arg_array[0] = read_fp; arg_array[1] = write_fp; return(proc_id); } } SHAR_EOF fi if test -f 'fpipe.h' then echo shar: "will not over-write existing file 'fpipe.h'" else cat << \SHAR_EOF > 'fpipe.h' /****************************************************************************** * * * fpipe.h * * * ******************************************************************************/ /*-------------------------- INITIAL CODING DATE ----------------------------- Thu Nov 10 11:01:32 EST 1988 by George M. Bogatko -------------------------------- HEADER FILES -----------------------------*/ /*------------------ TYPEDEF'S, DEFINES, STRUCTURE DEF'S ------------------*/ #define STD_ERR_SCR 0 #define STD_ERR_NUL 1 #define NO_BLOCK 0 #define BLOCK 1 #ident "%W% %G% - George M. Bogatko -" /*----------------------------------------------------------------------------- SYNOPSIS: #include <fpipe.h> DESCRIPTION: This includes #define's used by FCOPY and FPIPE CAVEATS: =============================================================================*/ SHAR_EOF fi if test -f 'main.c' then echo shar: "will not over-write existing file 'main.c'" else cat << \SHAR_EOF > 'main.c' #include <stdio.h> #include <fcntl.h> #include <fpipe.h> #include <signal.h> #define READ_END 0 #define WRITE_END 1 extern int fpipe(); extern char *fgets(); char buf[BUFSIZ]; main() { FILE *args[2]; enum {first, next} time = first; int pid; /* if( (pid = fpipe(STD_ERR_SCR, args, "./tst",0)) == -1 ) */ if( (pid = fpipe(STD_ERR_NUL, args, "/usr/bin/mailx", "-u", "gb", "-f","$HOME/mbox",0)) == -1 ) { puts("AARRRRRGGGG!!!!"); exit(-1); } for(;;) { fcopy(args[READ_END], "/dev/tty", NO_BLOCK); printf("enter command\n"); gets(buf); if( *buf == 27 ) break; fprintf(args[WRITE_END], "%s\n", buf); fflush(args[WRITE_END]); } /* * closing the file pointers also sends 'CTRL-D' (EOF) to the child * process. In the case of 'mailx' this will kill it. */ fcopy(args[READ_END], "/dev/tty", NO_BLOCK); fclose(args[READ_END]); fclose(args[WRITE_END]); return 0; } SHAR_EOF fi if test -f 'tst.c' then echo shar: "will not over-write existing file 'tst.c'" else cat << \SHAR_EOF > 'tst.c' #include <stdio.h> main() { fprintf(stdout, "THIS TO STANDARD OUT\n"); fprintf(stderr, "THIS TO STANDARD ERR\n"); } SHAR_EOF fi exit 0 # End of shell archive
gwyn@smoke.BRL.MIL (Doug Gwyn) (08/15/90)
In article <377@fe2o3.UUCP> michael@fe2o3.UUCP (Michael Katzmann) writes: >which execs the command a la "system()" and creates a pipe. "type" and be >"r" if you want to read from the standard output of "command", or "w" if >you want to write to standard input. However there doesn't seem to be any >way to use this routine to do both similtaneously. >What is the usual way to to this? This should probably be added the the FAQ list. The problem with trying to pipe both input and output to an arbitrary slave process is that deadlock can occur, if both processes are waiting for not-yet-generated input at the same time. Deadlock can be avoided only by having BOTH sides follow a strict deadlock-free protocol, but since that requires cooperation from the processes it is inappropriate for a popen()-like library function.
mayne@VSSERV.SCRI.FSU.EDU (William (Bill) Mayne) (08/17/90)
In article <377@fe2o3.UUCP> michael@fe2o3.UUCP (Michael Katzmann) writes: > >I have a task that requires the ability to fork off another programme but >to supply it's standard input and output. The SysV manuals describe > > FILE *popen( command, type ) > char *command, *type; > >which execs the command a la "system()" and creates a pipe. "type" and be >"r" if you want to read from the standard output of "command", or "w" if >you want to write to standard input. However there doesn't seem to be any >way to use this routine to do both similtaneously. > >What is the usual way to to this? > >Important points: The exec'ed command must run asynchronously (obvious if > the parent is supplying input.) > > The child process id must be available to the parent. > (so that it can be killed if necessary) > > >Any ideas would be appreciated. > > >--------------------------------------------------------------------- >email to >UUCP: uunet!mimsy!{arinc,fe2o3}!vk2bea!michael > _ _ _ _ > Amateur | VK2BEA (Australia) ' ) ) ) / // > Radio | G4NYV (United Kingdom) / / / o _. /_ __. _ // > Stations| NV3Z (United States) / ' (_<_(__/ /_(_/|_</_</_ > >Michael Katzmann >Broadcast Sports Technology. >2135 Espey Ct. #4 >Crofton Md. 21114 USA > >Ph: +1 301 721 5151 Although, as a response on the net indicates, there can be problems connecting both stdin and stdout from a child if you aren't very careful about protocols there are cases where it is useful. As you have discovered popen() won't do it, though. I have waited to see if a detailed answer was provided by others. Since I haven't seen one I will post part of a program I wrote which does this. The key is that the child inherits stdin and stdout from the parent, so the parent changes his own stdin and stdout to pipes before spawning the child, using dup() to change file desciptors. The purpose of this program is to run all permutations of given string through the program used by the standard spell script looking for all the words which can be formed from the letters in the input, as when solving word puzzles such as "Jumbles" found in many newspapers. I have omitted the details and tried to leave everything necessary to (1) see how to do the i/o redirection you wanted to do and (2) see the whole structure of the process tree in which this was used in a realistic (or at least fun) application. The permutation generator created the list of permutations in sorted order since this was much faster than creating the whole list and then sorting, even if space permitted. It also actually generated two separate streams of permutations since that was actually more efficient than splitting one up and avoided possible deadlock problems. I hope the editting jobs leaves enough for you to see what is going on without getting lost in details. Incidentally I tried to email the whole thing but got "host not known". Can experienced netters suggest how I might be able to use the routing show at the top of a post to reply even when my local host doesn't know the final destination? /* anagram.c - extracted fragments to show redirection */ #include <string.h> #include <stdio.h> #ifndef SPELLPROG #define SPELLPROG "/usr/bin/spellout" #endif #ifndef DICT #define DICT "/usr/dict/hlista" #endif /* Omitting some irrelevant stuff here */ main(argc,argv,envp) int argc; char **argv, **envp; { int pipe1[1], pipe2[2]; char word[21]; int i; #define TOCHILD pipe1[1] #define FROMCHILD pipe2[0] #define TOPARENT pipe2[1] #define FROMPARENT pipe1[0] /* Stuff omitted */ pipe(pipe1); pipe(pipe2); if (fork()==0) { if (fork()==0) { /* Child to invoke spell program in a pipeline */ /* Close unused pipes */ close (TOCHILD); close (FROMCHILD); /* Change stdin and stdout to pipe */ close(0); dup(FROMPARENT); close(1); dup(TOPARENT); /* Invoke spell command */ execl(SPELLPROG,"spellout",DICT,(char *)0); /* No return to here. For completeness make parent terminate */ } else { /* Process to send distinct permutations to spelling checker */ /* Close unused pipes */ close(FROMCHILD); close(TOPARENT); close(FROMPARENT); /* Switch stdout to TOCHILD pipe */ close(1); dup(TOCHILD); /* Send all distinct permutations to spell child */ /* by writing them to stdout - details omitted */ } } else { /* Root process invokes permute() to display accepted permutations */ /* Close unused pipes */ close(TOCHILD); close(TOPARENT); close(FROMPARENT); /* Change stdin to pipe from speller */ close(0); dup(FROMCHILD); /* Generate permutations in sorted order; Read misspellings from pipe; Print everything not on misspell list; - details omitted */ } }
tchrist@convex.COM (Tom Christiansen) (08/17/90)
Hmm.. I think I posted this before, but it may have been five years ago. This is for systems with socketpair(), and you should read and consider the caveat section very carefully. Also, contains non-ANSI cpp abuse, if that bugs you; what can I say? I was young and foolish. :-) --tom #! /bin/sh # This is a shell archive, meaning: # 1. Remove everything above the #! /bin/sh line. # 2. Save the resulting text in a file. # 3. Execute the file with /bin/sh (not csh) to create: # process.c # This archive created: Thu Aug 16 13:17:01 1990 export PATH; PATH=/bin:/usr/bin:$PATH echo shar: "extracting 'process.c'" '(4801 characters)' if test -f 'process.c' then echo shar: "will not over-write existing file 'process.c'" else sed 's/^ X//' << \SHAR_EOF > 'process.c' X/* process.c X * X * written by tom christiansen on Wed May 22 15:02:19 CDT 1985 to open X * a socket pair and let the user read and write from both sides. X * return -1 on error, otherwise put the correct file pointers X * into *input and *output. X * X * CAVEAT UTILITOR: X * you will block forever if one of the sides of the X * pipes blocks because of too much in the buffer. X */ X X#include <stdio.h> X#include <signal.h> X#include <sys/param.h> X#include <sys/wait.h> X#include <sys/socket.h> X X#define READ 0 X#define WRITE 1 X#define CHILD 0 X#define PARENT 1 X X/* X * define QUIET if you don't want error messages X * to print out if something blows up due to test X * macros. this assumes you will perror() it yourself X * when the routine returns. X */ X#ifdef QUIET X# define announce(x) /* nothing at all */ X#else X# define announce(x) perror(x) X#endif QUIET X X/* X * first some macros to avoid lots of typing and ugly X * code. X */ X#define test0(x) \ X if (!(x)) { \ X announce("process: x"); \ X return -1; \ X } X X#define test(x) \ X if ( (x) < 0 ) { \ X announce("process: x"); \ X return -1; \ X } X Xchar FD_READ[] = "r"; Xchar FD_WRITE[] = "w"; X X/* X * first a static array to hold the pid of X * the process we create so we can wait for X * it to die when we close up. there is enough X * room for all possible file descriptors (NOFILE) X * so this function may be called repeatedly by X * a program with no ill effects. X */ Xstatic int pids[NOFILE]; X X/***************************************************************** X * X * process - a function to do what popen does, but to X * give you back file pointers for read as X * well as for write. X * X *****************************************************************/ X Xint Xprocess(cmd,input,output) X char *cmd; X FILE **input,**output; X{ X int sock[2]; X register pid; X X X/* X * use connected socket pair so we can do reads and X * writes on these things. if we used a pipe() call, X * it would be unidirectional and we would have to X * make two calls to get 4 file descriptors. X */ X test(socketpair(AF_UNIX,SOCK_STREAM,0,sock)); X X/* X * fork for the child command. don't bother doing X * a real fork, since we're just going to exec anyway, X * so borrow the parent's pages with a vfork. X */ X if((pid = vfork()) == CHILD) { X X/* X * close old stdin and stdout to make room for socket X * descriptors. X */ X test(close(READ)); X test(close(WRITE)); X X/* X * the kid will use the CHILD end. connect both his stdin X * and stdout to the CHILD socket, which is fine because X * unix domain sockets are bidirectional. X */ X test(dup2(sock[CHILD], WRITE)); X test(dup2(sock[CHILD], READ)); X X/* X * don't need these anymore, and if we don't get rid of them, we X * won't be happy later on, because the process doesn't seem to die. X */ X test(close(sock[CHILD])); X test(close(sock[PARENT])); X/* X * now do the command. use the csh for tilda's sake. X */ X execl("/bin/csh", "csh", "-fc", cmd, 0); X perror("process kid: execl"); X _exit(1); X } X X/* X * -1 pid means we couldn't fork. X */ X if(pid == -1) { X perror("process: vfork"); X (void) close(sock[CHILD]); X (void) close(sock[PARENT]); X return -1; X } X X/* X * otherwise, we are the parent and healthy; X * remember the kid pid for a later close. X */ X pids[sock[PARENT]] = pid; X X/* X * connect up the user's input and output file X * pointers to our end of the socket connection. X * give him one for read and one for write. X */ X test0(*input = fdopen(sock[PARENT],FD_READ)); X test0(*output = fdopen(sock[PARENT],FD_WRITE)); X X test(close(sock[CHILD])); X X return 0; X} X X X/**************************************************************** X * close up the passed file and wait for the X * child to die. X ***************************************************************/ X X#undef test X#define test(x) \ X if ( (x) < 0 ) { \ X announce("process_close: x"); \ X return -1; \ X } X X/* X * don't need them both since they are the X * same thing X */ Xint Xprocess_close(input) XFILE *input; X{ X register f,r, (*hstat)(), (*istat)(), (*qstat)(); X int status; X X f = fileno(input); X test(fclose(input)); X/* X * don't need to close also output, as it is the same X */ X X/* X * protect ourselves from unfriendly signals while X * waiting for child's death. X */ X istat = signal(SIGINT, SIG_IGN); X qstat = signal(SIGQUIT, SIG_IGN); X hstat = signal(SIGHUP, SIG_IGN); X X/* X * wait for the child to die, keeping track of status. X * we saved the kid pid in the pids[] array when we X * first did the process call. X */ X while((r = wait((union wait *)&status)) != pids[f] && r == -1) X ; X if(r == -1) X status = -1; X X/* X * restore old sig values. X */ X (void) signal(SIGINT, istat); X (void) signal(SIGQUIT, qstat); X (void) signal(SIGHUP, hstat); X X return(status); X X} X X X/* lint output: Xprocess.c: X*/ X X/* X * yes, folks, unlike the kernel, this file lints clean! X */ SHAR_EOF if test 4801 -ne "`wc -c < 'process.c'`" then echo shar: "error transmitting 'process.c'" '(should have been 4801 characters)' fi chmod 644 'process.c' fi exit 0 # End of shell archive -- Tom Christiansen {uunet,uiucdcs,sun}!convex!tchrist Convex Computer Corporation tchrist@convex.COM "UNIX was never designed to keep people from doing stupid things, because that policy would also keep them from doing clever things." [gwyn]
maart@cs.vu.nl (Maarten Litmaath) (08/17/90)
In article <410@sun13.scri.fsu.edu>, mayne@VSSERV.SCRI.FSU.EDU (William (Bill) Mayne) writes: )... ) /* Close unused pipes */ ) close (TOCHILD); ) close (FROMCHILD); ) /* Change stdin and stdout to pipe */ ) close(0); ) dup(FROMPARENT); close(FROMPARENT); /* !!! */ ) close(1); ) dup(TOPARENT); close(TOPARENT); /* !!! */ I've deleted the rest of the code, but there were still a few other places where calls to close() should be added. Always close every unused file descriptor (esp. when dealing with pipes), else you will get bitten one day; example scenario: - the child exits - the parent reads from the pipe to the child - the parent still has the write side of the pipe open - the read hangs... -- "UNIX was never designed to keep people from doing stupid things, because that policy would also keep them from doing clever things." (Doug Gwyn)
guy@auspex.auspex.com (Guy Harris) (08/23/90)
>Popen is expensive. It requires a shell for 'system' and then >another shell to run your request in. It doesn't use "system()" on any UNIX implementation I've used (V7, BSD, S3, S5, etc.). It just runs *one* shell, which runs your command.
bogatko@lzga.ATT.COM (George Bogatko) (08/23/90)
In article <3939@auspex.auspex.com>, guy@auspex.auspex.com (Guy Harris) writes: > >Popen is expensive. It requires a shell for 'system' and then > >another shell to run your request in. > > It doesn't use "system()" on any UNIX implementation I've used (V7, BSD, > S3, S5, etc.). It just runs *one* shell, which runs your command. Absolutly right. I goofed. What we saw was an explosion of 'exec' calls when people used popen instead of directly calling 'fork' and 'exec'. One exec for the shell, and another exec for the program, if it didn't involve a pipeline. Sorry, GB
bogatko@lzga.ATT.COM (George Bogatko) (08/23/90)
In article <3939@auspex.auspex.com>, guy@auspex.auspex.com (Guy Harris) writes: > >Popen is expensive. It requires a shell for 'system' and then > >another shell to run your request in. > > It doesn't use "system()" on any UNIX implementation I've used (V7, BSD, > S3, S5, etc.). It just runs *one* shell, which runs your command. Absolutly right. I goofed. What we saw was an explosion of fork/exec combinations when the application made heavy use of 'popen' instead of directly using fork/exec. I should have said: "Popen is expensive. It requires a fork/exec to invoke the shell and then another fork/exec to run your request." Crow is not digestable, by the way. GB.