jcbst3@cisunx.UUCP (James C. Benz) (01/07/89)
I am sorry if this has been covered before, or if the subject is boring, but I don't have access to an archive that I know of, and I thought someone might be kind enough to answer a question or two. I am running a database application with Informix on 3B2 computers (ATT SVR2) and I recently read somewhere that setting the sticky bit on a program file will speed up execution time. (Actually, load time) I am considering using this for a couple of C programs that are heavily used on our system. The programs are about 100,000 bytes each, one is interactive and allows shell excape, the other is a stand alone, non-interactive program using stdin and stdout. I would like to know if : 1) this will really help speed things up 2) if there are any security problems or potential problems 3) is there a limit to how much can be stored in the swap space (of course!) and is there a way of increasing it if necessary 4) any other dangers like system crashes, lock up, etc you can think of I follow this group pretty regularly, so posting here is OK, email too. Thanks Jim Benz University of Pittsburgh
jcm@mtunb.ATT.COM (was-John McMillan) (01/10/89)
In article <14750@cisunx.UUCP> jcbst3@unix.cis.pittsburgh.edu (James C. Benz) writes: ... >I would like to know if : >1) this will really help speed things up Setting Sticky Bit [SB] ONLY locks programs' SHARED-TEXT images on the SWAP disk. a) It cannot hasten loading of DATA space. b) It is UNNECESSARY for programs which typically have another incarnation ALREADY RUNNING. Ex: It is usually worthless for SH because the user typically is already running that program and its TEXT is ALREADY on the SWAP disk. c) It is irrelevant for TINY programs whose first block and data blocks have to be read anyway. >2) if there are any security problems or potential problems A SB-program consumes SWAP SPACE for its TEXT even when it it not running. Therefore, you are that much closer to running out of SWAP SPACE -- which is REPORTED as "ENOMEM" -- outta memory -- when it occurs. >3) is there a limit to how much can be stored in the swap space (of course!) > and is there a way of increasing it if necessary Re-defining your partitions/filesystems is little fun if the disk is large and your backup media are small :+) >4) any other dangers like system crashes, lock up, etc you can think of See 2). Also: each SB-program takes up a TEXT-table entry in the kernel -- so re-tuning (NTEXT?) may be appropriate if you're doing more than 1 or 2. First time around, most folk seem to overdo the Sticky-Bit use. jc mcmillan -- att!mtunb!jcm -- muttering for himzelf, only.
dlm@cuuxb.ATT.COM (Dennis L. Mumaugh) (01/10/89)
In article <14750@cisunx.UUCP> jcbst3@unix.cis.pittsburgh.edu (James C. Benz) writes: I would like to know if: 1) this will really help speed things up Yes, under the right conditions on a machine that does swapping. On a system like ATT System V Release 3.1 which has demand paging with sector aligned a.out files the improvement isn't measurable. 2) if there are any security problems or potential problems Yes, I know of one security attack on older releases of UNIX. Also, the file system having the sticky text cannot generally by unmounted, the file cannot be remove or unlinked. Some sticky text files cannot be debugged. The above comments are version and vendor specific and may not be true on your system. 3) is there a limit to how much can be stored in the swap space (of course!) and is there a way of increasing it if necessary On most ATT paging systems swap can be increased on the fly if spare disk partions are available. Of course, Parkinson's Law applies. 4) any other dangers like system crashes, lock up, etc you can think of Getting too smart and having too many sticky texts will hurt. Increasing memory is better, increasing buffer cache works better with paging. -- =Dennis L. Mumaugh Lisle, IL ...!{att,lll-crg}!cuuxb!dlm OR cuuxb!dlm@arpa.att.com
bill@twwells.uucp (T. William Wells) (01/10/89)
In article <1359@mtunb.ATT.COM> jcm@mtunb.UUCP (was-John McMillan) writes: : In article <14750@cisunx.UUCP> jcbst3@unix.cis.pittsburgh.edu (James C. Benz) writes: : ... : >I would like to know if : : >1) this will really help speed things up : : b) It is UNNECESSARY for programs which typically have another : incarnation ALREADY RUNNING. : Ex: It is usually worthless for SH because the user : typically is already running that program and : its TEXT is ALREADY on the SWAP disk. However, an example of where this might be useful: if you use csh, you might want /bin/sh sticky to speed up running shell scripts. : >2) if there are any security problems or potential problems : : A SB-program consumes SWAP SPACE for its TEXT even when it it : not running. Therefore, you are that much closer to running out : of SWAP SPACE -- which is REPORTED as "ENOMEM" -- outta memory -- : when it occurs. Depends on the system. Mine will move stuff out of swap if it's not in use and the space is needed. Read the chmod(2) manual page to see what yours does. : First time around, most folk seem to overdo the Sticky-Bit use. I just did my editor, compiler, make, and ls. That seems to be sufficient. --- Bill { uunet!proxftl | novavax } !twwells!bill
jm@wlbr.EATON.COM (James Macropol) (01/11/89)
In article <1359@mtunb.ATT.COM> jcm@mtunb.UUCP (was-John McMillan) writes: > Setting Sticky Bit [SB] ONLY locks programs' SHARED-TEXT images >on the SWAP disk. > a) It cannot hasten loading of DATA space. > b) It is UNNECESSARY for programs which typically have another > incarnation ALREADY RUNNING. A long time ago and far, far away....... I once modified a V7 PDP-11 UNIX system to save a prototype of the data out on the swap file, too. It took about twenty lines of changes in text.c, text.h and exec.c (I think. My memory is getting hazy). Unlike the standard sticky texts, this DID help shared-text programs that already had another incarnation running, because it DRASTICALLY cut down on the number of reads necessary to initialize the data segment (to one). On some large programs, it cut down program load time by almost two seconds. Obviously, the down side of this scheme was the extra swap space needed to store the data prototype. On the systems we were running, though, program load time was one of our major bottlenecks. When we finally moved to machines that loaded programs by faulting pages into memory (e.g., the VAX), the scheme was abandoned. When we James Macropol jm@wlbr.eaton.com Contel FSD wlbr!jm
debra@alice.UUCP (Paul De Bra) (01/11/89)
In article <314@twwells.uucp> bill@twwells.UUCP (T. William Wells) writes: >... [arguments about sticky bit deleted] > >I just did my editor, compiler, make, and ls. That seems to be >sufficient. > Sufficient for what? I have run several tests on different systems, both with and without virtual memory, and have never found any improvement by setting the sticky bit for any program. The theoretical argument for introducing the sticky bit is that loading a program from the swap-space is faster than loading it from the file-system. This is probably true because processes tend to be more or less contiguous in the swap-space whereas they in general are not contiguous in the file- system (especially not in older, swapping systems, for which the sticky-bit was developed). Unless a file is *really* scattered all over the disk this win is marginal. However, on most older systems setting the sticky bit implies that when the last process executing the program exits the program text has to be "swapped out", which takes time. One needs many slightly faster invokations to compensate for this extra (one-time only) swap-out time. Now, some newer systems don't actually swap out the text but keep it in memory if possible. In that case there may be a win. Considering the annoying side-effects of sticky-bit programs (you cannot overwrite them easily, you cannot unmount file systems, etc) I would not recommend using the sticky bit anymore. Paul. -- ------------------------------------------------------ |debra@research.att.com | uunet!research!debra | ------------------------------------------------------
jcm@mtunb.ATT.COM (was-John McMillan) (01/11/89)
In article <25591@wlbr.EATON.COM> jm@wlbr.eaton.com (James Macropol) writes: ... >I once modified a V7 PDP-11 UNIX system to save a prototype of the data >out on the swap file, too. It took about twenty lines of changes in >text.c, text.h and exec.c (I think. My memory is getting hazy). > >Unlike the standard sticky texts, this DID help shared-text programs >that already had another incarnation running, because it DRASTICALLY cut >down on the number of reads necessary to initialize the data segment (to >one). On some large programs, it cut down program load time by almost >two seconds. Yup -- 'did it three years ago on the 3B1/7300 and it was a FLASH start-up... but o' that swap space! Fortunately, the faster and faster disks have made these tricks rather unnecessary when combined with Paging. PS: 1) I agree with much of what DLM said: the SB does introduce DIS-MOUNT anomalies. One strategy is to only Stick it to ROOTDEV programs. The gain it provides on paging systems can, indeed, be very small -- particularly, when the disk was well-ordered at the time the programs were loaded, when the disk-access rates are fast, when shared libraries are used, and particularly, I would presume, if sequentially allocated disk sectors were allocated to executable files -- a trick I've heard of, but never seen. However, on the 3B1/7300, even with its paging, the times are quite measurable, Dennis! 2) In article <314@twwells.uucp> Bill writes: ... >Depends on the system. Mine will move stuff out of swap if it's not >in use and the space is needed. Read the chmod(2) manual page to see >what yours does. Nice! How common is this? What is your system? Now I've got to scurry through SVR3.2 sources, I suppose ;-{ 'Don't recall the 3B1 being that considerate, however. >I just did my editor, compiler, make, and ls. All nice packages to be sure: But with shared libraries, LS is only 2 to 4 [4K] pages of TEXT. All EDITORS are great targets: you're poised over the keyboard and their TEXT runs towards 30 pages. Shells run 10 to 20 pages, so they're a WIN. Compiler & Make? Lucky you, getting to write software! %-} jc mcmillan -- att!mtunb!jcm -- juzz muttering for hizzelf
jfh@rpp386.Dallas.TX.US (John F. Haugh II) (01/11/89)
In article <8724@alice.UUCP> debra@alice.UUCP () writes: >The theoretical argument for introducing the sticky bit is that loading a >program from the swap-space is faster than loading it from the file-system. >This is probably true because processes tend to be more or less contiguous >in the swap-space whereas they in general are not contiguous in the file- >system (especially not in older, swapping systems, for which the sticky-bit >was developed). Unless a file is *really* scattered all over the disk this >win is marginal. This is quite definitely true. The interface into the swap partition is more or less `raw'. A single read request is issued to bring the pure text into memory [ This is modified by drive hardware, such as request size limitations, and whether or not a read request may span tracks or cylinders. PDP-11 drives seemed to get this right. ] This is vastly more efficient than going through the file system. Unfortunately it is a big loser if there is a large .data segment. The shell got around this by putting the strings into the .text segment, thereby reducing the size of the data segment. And of course .bss takes little or no appreciable file system reads ;-) Proofs of this may be had by measuring file system versus raw partition performance for reading. And no cheating - BSD systems don't count ;-) The real win is to save a complete prototype on the swapper. Or chunk the old space heater and get a paging machine ;-) I forget who brought up the complete-prototype idea - but that's where the big win lies. -- John F. Haugh II +-Quote of the Week:------------------- VoiceNet: (214) 250-3311 Data: -6272 |"Now with 12 percent less fat than InterNet: jfh@rpp386.Dallas.TX.US | last years model ..." UucpNet : <backbone>!killer!rpp386!jfh +--------------------------------------
bill@twwells.uucp (T. William Wells) (01/12/89)
In article <1362@mtunb.ATT.COM> jcm@mtunb.UUCP (was-John McMillan) writes: : 2) In article <314@twwells.uucp> Bill writes: : >Depends on the system. Mine will move stuff out of swap if it's not : >in use and the space is needed. Read the chmod(2) manual page to see : >what yours does. : : Nice! How common is this? What is your system? Now : I've got to scurry through SVR3.2 sources, I suppose ;-{ : 'Don't recall the 3B1 being that considerate, however. I don't know how common it is. SV3.0 from Microport. I should say that I have inferred this by looking at various stats and by noticing differences in startup times after my system has started and then stopped swapping. I haven't gone in with crash to see whether things have really been moved out. : >I just did my editor, compiler, make, and ls. : : All nice packages to be sure: But with shared libraries, : LS is only 2 to 4 [4K] pages of TEXT. Still, if you have a system like mine, which slows horribly when there is any load on the disk, it makes a noticable difference when doing directory listings. And I do a lot of directory listings. --- Bill { uunet!proxftl | novavax } !twwells!bill
bill@twwells.uucp (T. William Wells) (01/12/89)
In article <8724@alice.UUCP> debra@alice.UUCP () writes: : In article <314@twwells.uucp> bill@twwells.UUCP (T. William Wells) writes: : >... [arguments about sticky bit deleted] : > : >I just did my editor, compiler, make, and ls. That seems to be : >sufficient. : : Sufficient for what? I have run several tests on different systems, both : with and without virtual memory, and have never found any improvement by : setting the sticky bit for any program. Program development on my system. It is a 16MHz 386, 4M RAM, 80M 28ms seek time disk. I run Microport's SV/386 3.0e (the latest). Load time on my editor, for example, is cut from about three seconds on initial startup, to under a second. Compile times are cut similarly. (Most of the compile time seems to be related to loading the compiler passes!) : Considering the annoying side-effects of sticky-bit programs (you cannot : overwrite them easily, you cannot unmount file systems, etc) I would not : recommend using the sticky bit anymore. I have no problem with unmounting my /usr file system, even though I have a sticky-bit program (my editor) on it. Overwriting, on the other hand, still requires deleting the file first. --- Bill { uunet!proxftl | novavax } !twwells!bill
dwc@homxc.ATT.COM (Malaclypse the Elder) (01/13/89)
for swap based systems, behrokh samadi has written an expert system performance tuner for unix system v called tunex. one of the things it does is go through accounting files and determine which programs would have benefited from sticky bitting (this is a straightfoward calculation so no "expert" is really needed). i believe that this program has found programs that would benefit from the sticky bit that most system administrators ignore. on demand paging versions of unix system v, it is true that sticky bits no longer do the same thing. but there are benefits that are not related to i/o. we have found that sticky bits seem to reduce the paging overhead. this is because either the shared regions are kept around or the mapping list that maps pages to file blocks (remember on the 3b2, pages are 2k and file blocks are typically 1k) are kept around. i haven't gotten around to figuring out which is the actual reason. also, for non page aligned programs (magic number 410), the sticky bit will result in the text being written out to swap space. although no longer contiguous, it still has the benefit of 1 I/O to read in a page instead of 2 I/Os. the only "definition" of sticky bit that i know of is that it is a bit to identify non-overlapped but commonly used programs. the operating system can use this infomation in any way to improve performance. any problems are simply implementation. one note: i would prefer to see such information kept in the a.out header instead of a file bit since there is really much more information that would be useful and the file system could probably use that bit for other things. danny chen att!homxc!dwc
debra@alice.UUCP (Paul De Bra) (01/13/89)
In article <325@twwells.uucp> bill@twwells.UUCP (T. William Wells) writes: }In article <8724@alice.UUCP> debra@alice.UUCP () writes: }: In article <314@twwells.uucp> bill@twwells.UUCP (T. William Wells) writes: }: >... [arguments about sticky bit deleted] }: > }: >I just did my editor, compiler, make, and ls. That seems to be }: >sufficient. }: }: Sufficient for what? I have run several tests on different systems, both }: with and without virtual memory, and have never found any improvement by }: setting the sticky bit for any program. } }Program development on my system. It is a 16MHz 386, 4M RAM, 80M 28ms }seek time disk. I run Microport's SV/386 3.0e (the latest). Load }time on my editor, for example, is cut from about three seconds on }initial startup, to under a second. Compile times are cut similarly. }(Most of the compile time seems to be related to loading the compiler }passes!) I believe I recently read a message saying that Microport actually keeps sticky-bit programs IN MEMORY, instead of swapping them out to disk. This could easily explain your performance gain. Try running a huge application, and edit and compile after that. (the huge application will absorbe the memory so the sticky-bit applications have to be moved to the swap-space again. I would be surprised if you would still have this 300% improvement. }I have no problem with unmounting my /usr file system, even though I }have a sticky-bit program (my editor) on it. Overwriting, on the }other hand, still requires deleting the file first. There is no reason why a system should be unable to figure out what to do when you unmount a file system. Apparently SV/386 has figured it out. Paul. -- ------------------------------------------------------ |debra@research.att.com | uunet!research!debra | ------------------------------------------------------
w-colinp@microsoft.UUCP (Colin Plumb) (01/14/89)
dwc@homxc.ATT.COM (Malaclypse the Elder) wrote: > one note: i would prefer to see such information kept in > the a.out header instead of a file bit since there is really > much more information that would be useful and the file system > could probably use that bit for other things. But setting the sticky bit on Unix is restricted to super-users for a reason, and there's no way to prevent a user from setting a bit *inside* his file. If the file system needs more bits, it's welcome to them. After all, it has sole and exclusive control over the format of an inode. -- -Colin (uunet!microsof!w-colinp)
bill@twwells.uucp (T. William Wells) (01/15/89)
In article <8739@alice.UUCP> debra@alice.UUCP () writes: : In article <325@twwells.uucp> bill@twwells.UUCP (T. William Wells) writes: : }Program development on my system. It is a 16MHz 386, 4M RAM, 80M 28ms : }seek time disk. I run Microport's SV/386 3.0e (the latest). Load : }time on my editor, for example, is cut from about three seconds on : }initial startup, to under a second. Compile times are cut similarly. : }(Most of the compile time seems to be related to loading the compiler : }passes!) : : I believe I recently read a message saying that Microport actually keeps : sticky-bit programs IN MEMORY, instead of swapping them out to disk. You did. I said it. It comes from the man page for chmod(2). It says 413 executables stay in memory after all processes have stopped using them. 413's are paged straight from the a.out file. : This could easily explain your performance gain. Try running a huge : application, and edit and compile after that. (the huge application will : absorbe the memory so the sticky-bit applications have to be moved to the : swap-space again. I would be surprised if you would still have this 300% : improvement. You are right. : }I have no problem with unmounting my /usr file system, even though I : }have a sticky-bit program (my editor) on it. Overwriting, on the : }other hand, still requires deleting the file first. : : There is no reason why a system should be unable to figure out what to : do when you unmount a file system. Apparently SV/386 has figured it out. At least for removing the stuff when unmounting. It ought to be able to let you overwrite the program as well. Oh well, maybe next version. --- Bill { uunet!proxftl | novavax } !twwells!bill
hsu@kampi.hut.fi (Heikki Suonsivu) (01/15/89)
In article <8724@alice.UUCP> debra@alice.UUCP () writes: >with and without virtual memory, and have never found any improvement by >setting the sticky bit for any program. Depends heavily on hardware. 386 clone running microport V.3 with 8M of memory took 40 seconds to load up emacs, sticky bit saved 10-20 secs. With single user it may be worth it. Low speed is probably due to stupid non-buffering controller, I'm now testing 1:1 interleave controller and it really speeds things up, emacs loads in 10 secs instead of that 40... with same hard disk. PC:s, you know ;-) >was developed). Unless a file is *really* scattered all over the disk this >win is marginal. System V does very bad job of keeping disks unfragmented. Fsanalyze told me that I had my CT miniframe emacs (~550k) in 400 fragments :-)
dwc@homxc.ATT.COM (Malaclypse the Elder) (01/17/89)
In article <259@microsoft.UUCP>, w-colinp@microsoft.UUCP (Colin Plumb) writes: > > But setting the sticky bit on Unix is restricted to super-users for a reason, > and there's no way to prevent a user from setting a bit *inside* his file. > think of it as a hint rather than an absolute requirement (with suitable punishment for providing bad hints). danny chen att!homxc!dwc
zeeff@b-tech.ann-arbor.mi.us (Jon Zeeff) (01/17/89)
In article <18714@santra.UUCP> hsu@kampi.UUCP (Heikki Suonsivu) writes: >System V does very bad job of keeping disks unfragmented. Fsanalyze >told me that I had my CT miniframe emacs (~550k) in 400 fragments :-) The guy who wrote fsanalyze (sdti!mjy) also wrote a program to unfragment disks. It's a big win. I don't know the current status of the program or distribution. Actually, I've never understood why the kernel didn't have some locking support that would allow a daemon to unfragment the disk during idle times. -- Jon Zeeff zeeff@b-tech.ann-arbor.mi.us Ann Arbor, MI umix!b-tech!zeeff My other signature is a good one
w-colinp@microsoft.UUCP (Colin Plumb) (01/18/89)
zeeff@b-tech.ann-arbor.mi.us (Jon Zeeff) wrote: > Actually, I've never understood why the kernel didn't have some > locking support that would allow a daemon to unfragment the disk > during idle times. Actually, you don't ever need to lock the file. Basically, you find the oldest file that hasn't been unfragged (although adding an unsettable last-mod-time helps here), allocate a new inode, copy it to contiguous storage, and then make one magic call, that swaps the two and deletes the old file if it hasn't been changed since the copy started, and deletes the new copy if it has changed. I've been (in the past) involved in writing an FS that provides a background- cleanup daemon, and this is basically how it works. Every now and then, you'll waste effort, but if you start with old files, it's very rare. And not locking the files increases the transparency of the operation. For minumum impact, you need a way to prioritise disk I/O, which I don't think Unix currently has, but can be hacked into an elevator algorithm. And you can live without. -- -Colin (uunet!microsof!w-colinp)