al@unhd.unh.edu (Anthony Lapadula) (04/16/91)
I'll soon be writing a script (most likely in csh) that has at least one critical section (updating a file). I need to ensure that only one executing copy of the script actually accesses the file at any given time. I had thought that creating a lock file -- and checking for its presence when starting up -- would be a reasonable way to solve the problem. I can't seem to figure out how to do it (short of invoking a C program that invokes open(2) with O_EXCL). We're trying to keep the script machine-independent, so the open(2) solution is out. I looked at mktemp(1), but it also seems insufficient. Lastly, does it matter that we're using NFS? Any and all replies welcome. Thanks in advance. -- Anthony (uunet!unhd!al, al@cs.unh.edu) Lapadula
tchrist@convex.COM (Tom Christiansen) (04/16/91)
From the keyboard of al@unhd.unh.edu (Anthony Lapadula): :I'll soon be writing a script (most likely in csh) that has at least :one critical section (updating a file). I need to ensure that only :one executing copy of the script actually accesses the file at any :given time. : :I had thought that creating a lock file -- and checking for its presence :when starting up -- would be a reasonable way to solve the problem. :I can't seem to figure out how to do it (short of invoking a C program :that invokes open(2) with O_EXCL). We're trying to keep the script :machine-independent, so the open(2) solution is out. : :I looked at mktemp(1), but it also seems insufficient. : :Lastly, does it matter that we're using NFS? I would strongly counsel against using csh. Use a Bourne-compatible shell for shell programming. Yes, it makes a difference whether you're using NFS: old assumptions about idempotency of operations (like create or unlink or O_EXCL) are no longer valid; you have to go through the lockdaemon_from_hell to make these things work. Of course, not all hosts have a lockdaemon, so so much for machine-independence. You could use fcntl() so it works both w/ and w/o a lockd, but old BSD hosts don't do file-locking that way. And anyway, you can't get to fcntl() from a shell script anyway. You *can* get at these functions from perl scripts, and you still get machine-independence because you don't have to recompile. You might even use #ifdef __convex__ or whatever if you really want. --tom
rhartman@thestepchild.sgi.com (Robert Hartman) (04/17/91)
In article <1991Apr15.205654.26253@unhd.unh.edu> al@unhd.unh.edu (Anthony Lapadula) writes: >I'll soon be writing a script (most likely in csh) that has at least >one critical section ... > >I had thought that creating a lock file -- and checking for its presence >when starting up -- would be a reasonable way to solve the problem. > >-- Anthony (uunet!unhd!al, al@cs.unh.edu) Lapadula Seems to me you could make this work in csh as follows: set noclobber (echo "${user}: $$ `date`" > $file.lock) >& /dev/null if ($status == 0) then if ($$ == "`awk '{ print $2 }' $file.lock`") then # enter critical section # ... # exit critical section else echo "$file in use by `cat $file.lock`" exit 1 endif else echo "$file in use by `cat $file.lock`" exit 1 endif unset noclobber Noclobber insures that if the file exists your attempt to overwrite it wil fail. The redirect to /dev/null is a standard way to peel off the standard error message (the "else" statements are more meaningful). I suppose that there are more efficient comparisons than ($status == 0), but that one is quite clear. The nested if double-checks to make sure that the current process actually holds the lock. This is probably overkill, but it's easy enough to comment out or delete. Can't see how NFS could matter here. -r
ronald@robobar.co.uk (Ronald S H Khoo) (04/17/91)
tchrist@convex.COM (Tom Christiansen) writes: > I would strongly counsel against using csh. Use a Bourne-compatible shell. Strongly seconded. > Yes, it makes a difference whether you're > using NFS: old assumptions about idempotency of operations (like create > or unlink or O_EXCL) are no longer valid; Actually, I've seen O_EXCL fail on local filesystems too. Never trust O_EXCL, it's a flag from hell. > lockdaemon_from_hell to make these things work. > And anyway, you can't get > to fcntl() from a shell script anyway. Surely NFS can't possibly break link(2) based locks ? I fail to see how *any* Unixoid filesystem can possibly survive AT ALL if link() doesn't work. If it's OK, the original poster can achieve locking at the shell level using the same techniques as C News does. C News provides the necessary program (/etc/link lookalike) for those who haven't got it, and there's even a tutorial on how to do a simple form of reliable locking *portably* under all kinds of Unix. I think the original poster should at least grab pub/c-news/c-news.tar.Z from ftp.cs.toronto.edu and read the file notebook/newslock in there. It's not actually particularly news-specific, more a general purpose simple locking mechanism that *works*, although it does have its shortcomings. -- Ronald Khoo <ronald@robobar.co.uk> +44 81 991 1142 (O) +44 71 229 7741 (H)
jik@athena.mit.edu (Jonathan I. Kamens) (04/17/91)
In article <1991Apr16.175347.1082@odin.corp.sgi.com>, rhartman@thestepchild.sgi.com (Robert Hartman) writes: 1|> set noclobber 2|> (echo "${user}: $$ `date`" > $file.lock) >& /dev/null 3|> if ($status == 0) then 4|> if ($$ == "`awk '{ print $2 }' $file.lock`") then 5|> # enter critical section 6|> # ... 7|> # exit critical section 8|> else 9|> echo "$file in use by `cat $file.lock`" 10|> exit 1 11|> endif 12|> else 13|> echo "$file in use by `cat $file.lock`" 14|> exit 1 15|> endif 16|> unset noclobber |> |> Can't see how NFS could matter here. Let's assume that two processes are trying to get a lock in this way at the same time over NFS. There is no way in the NFS protocol to open a file for write if it does not exist in an atomic operation. Therefore, NFS client kernels first have to check with the server to see if the file exists, and send a request to create it if it doesn't. So here's what could happen: 1) Process A and Process B ask the NFS server if the file exists, at about the same time, at line 2 above. 2) Because they both ask at the same time, before either of them has time to create the file, the NFS server tells both of them that the file doesn't exist. 3) They then both send requests to the server to create and write to the file. All of their requests are received and carried out, although the order is completely race-dependent. 4) Process A manages to write to the file and read it back at line 4 before Process B's write requests to the server are carried out. So Process A thinks it has the lock and does the critical code. 5) A very little while later, Process B's write request goes through, and then it reads at line 4 and discovers that it has the lock, and does the critical code. So, as you can see, both processes are doing the critical code at the same time. That's why NFS "could matter here." The only way to do locks under NFS is to use an NFS lock manager (gag choke wheeze). You're even more likely to get into race conditions with multiple processes thinking they hold the lock if you remove lines 4 and 8-11 as you suggested when you said: |> The nested if double-checks to make sure |> that the current process actually holds the lock. This is probably |> overkill, but it's easy enough to comment out or delete. -- Jonathan Kamens USnail: MIT Project Athena 11 Ashford Terrace jik@Athena.MIT.EDU Allston, MA 02134 Office: 617-253-8085 Home: 617-782-0710
rhartman@thestepchild.sgi.com (Robert Hartman) (04/18/91)
In article <1991Apr17.102654.4476@athena.mit.edu> jik@athena.mit.edu (Jonathan I. Kamens) writes: > >Let's assume that two processes are trying to get a lock in this way at >the same time over NFS. There is no way in the NFS protocol to open a file >for write if it does not exist in an atomic operation. Therefore, NFS client >kernels first have to check with the server to see if the file exists, and >send a request to create it if it doesn't. So here's what could happen: > > [ description of resulting race condition omitted ] > >-- >Jonathan Kamens USnail: Thank you Jonathan! I guess the real solution would be for someone to write a "lock" command that makes the NFS lock manager accessible at the user level. Don't think I'll hold my breath though ... ;^) -r
daniel@island.COM (Daniel Smith - Fun Now, Worry Later) (04/19/91)
> In article <1991Apr15.205654.26253@unhd.unh.edu> al@unhd.unh.edu (Anthony Lapadula) writes: > >I'll soon be writing a script (most likely in csh) that has at least > >one critical section ... > > > >I had thought that creating a lock file -- and checking for its presence > >when starting up -- would be a reasonable way to solve the problem. > > > >-- Anthony (uunet!unhd!al, al@cs.unh.edu) Lapadula [someone posted a good solution] And another use of a "lock file" is to flag when to stop doing something. This section is used to check how a tape extraction is going, which happens in the background...the foreground gets the task of reporting how much has been accomplished. It would be easy to check the existence of the lock file and avoind this whole section in the first place. [$CR is set to: `echo X | tr 'X' '\015'`] touch eps_extract_lock (tar xvf $tape_dev > tapelist) >& tape_errs ; rm eps_extract_lock & cat << +++ `clear` Extracting tape... +++ while ( -e eps_extract_lock ) @ so_far=`du -s | awk ' { print $1 }'` set percent_done=`bc` << +++ scale=1 ($so_far * 100) / $disk_space_needed +++ echo -n ' 'percent done: $percent_done + |\ tr '+' "$CR" sleep 1 if ( ! -z tape_errs) then cat << +++ `clear` It looks like there is a problem with the tape: `cat tape_errs` Perhaps one of these things occurred: tape got pulled out of the drive? cable loose or fell out? power turned off to tape drive? full moon? You may wish to try again with a different tape. Exiting this script. +++ goto bail_out endif end -- daniel@island.com Daniel Smith, Island Graphics, (415) 491 0765 x 250(w) daniel@world.std.com 4000 CivicCenterDrive SanRafael MarinCounty CA 94903 dansmith@well.sf.ca.us Fax: 491 0402 Disclaimer: Hey, I wrote it, not IG! falling/yes I'm falling/and she keeps calling/me back again - IJSaF, Beatles