milo@ndmath.UUCP (Greg Corson) (03/06/88)
Is anyone aware of any good programs for unfragmenting unix disks? (collecting
all the files together so they use contiguous blocks). I am looking for
programs for both BSD and SYSV systems.
Also, does anyone know of a way to get UNIX to allocate a file as contiguous
disk space? Preferably, using normal user privleges but a scheme requireing
superuser permission would still be useful.
Please reply by mail as I don't get in to read news very often.
Greg Corson
19141 Summers Drive
South Bend, IN 46637
(219) 277-5306 (weekdays till 6 PM eastern)
{pur-ee,rutgers,uunet}!iuvax!ndmath!milo
zeeff@b-tech.UUCP (Jon Zeeff) (03/07/88)
It is certainly possible to have a program that you would run on an unmounted file system that would do an in place optimization. Something like the Sys V dcopy, but without the need for another drive. Has anyone heard of such a thing? -- Jon Zeeff Branch Technology, uunet!umix!b-tech!zeeff zeeff%b-tech.uucp@umix.cc.umich.edu
knutson@marconi.SW.MCC.COM (Jim Knutson) (03/08/88)
Dump and restore always worked well in the past for defragmenting a disk. The real question is why you would want to do this on a BSD system (assuming it is 4.2 or greater). For AT&T System X, try your favorite method of backup and restore (cpio I suppose). -- Jim Knutson knutson@mcc.com im4u!milano!knutson
hubcap@hubcap.UUCP (Mike Marshall) (03/08/88)
* Dump and restore always worked well in the past for defragmenting a disk. * The real question is why you would want to do this on a BSD system * (assuming it is 4.2 or greater). For the benefit of the poster of the original question: BSD 4.2's fast file system uses a disk management scheme that keeps disk transfer rates near constant over time (not sensitive to fragmentation through use). 4.2 BSD's throughput rates are dependent, instead, on the total amount of free space, which must not be allowed to drop below a certain threshold. -Mike Marshall hubcap@hubcap.clemson.edu ...!hubcap!hubcap
breck@aimt.UUCP (Robert Breckinridge Beatie) (03/08/88)
In article <305@marconi.SW.MCC.COM>, knutson@marconi.SW.MCC.COM (Jim Knutson) writes: > The real question is why you would want to do this on a BSD system Mostly because the BSD FFS doesn't do a perfect job of preventing file system fragmentation. If you remove as well as create files, or if files grow, then you're going to get fragmented files. Maybe the degree of fragmentation is kept small enough that compressing the file system doesn't get you anything back. But that's for him to decide. -- Breck Beatie {uunet,ames!coherent}!aimt!breck "Sloppy as hell Little Father. You've embarassed me no end."
maxwell@ablnc.ATT.COM (Robert Maxwell) (03/08/88)
I also e-mailed this, but for SYSV, the dcopy(1m) is the command specifically intended for disk reorganization. It not only clears fragmentation, but put sub-directories as the first entries in directories to speed up path searches. Cpio (after making a new fs will do a fair amount of cleanup, but dcopy works better. -- ----------------------------------------------------------------------------- R. M. Maxwell AT&T DP&CT | I speak for nobody- Maitland, FL ablnc!delilah!bob | not even myself. -----------------------------------------------------------------------------
paddock@mybest.UUCP (Steve Paddock) (03/09/88)
In article <305@marconi.SW.MCC.COM> knutson@marconi.UUCP (Jim Knutson) writes: >(assuming it is 4.2 or greater). For AT&T System X, try your favorite >method of backup and restore (cpio I suppose). Mightn't fsck -s or mkfs between the backup and restore be helpful on SysV? -- Steve Paddock (uunet!bigtex!mybest!paddock) 4015 Burnet Road, Austin, Texas 78756
Paul_Steven_Mahler@cup.portal.com (03/09/88)
System V provides a dcopy utility for de-fragmenting a disc. You may also use fsck to clean up the inode table. sun!plato!paul
Paul_Steven_Mahler@cup.portal.com (03/09/88)
Aim Technology (415) 856-8649 sells a set of utilities that Gene Dronick wrote which re-organize discs.
daveg@pwcs.StPaul.GOV (Dave Glowacki) (03/11/88)
In article <1097@hubcap.UUCP> hubcap@hubcap.UUCP (Mike Marshall) writes: >For the benefit of the poster of the original question: BSD 4.2's fast file >system uses a disk management scheme that keeps disk transfer rates near >constant over time (not sensitive to fragmentation through use). 4.2 BSD's >throughput rates are dependent, instead, on the total amount of free space, >which must not be allowed to drop below a certain threshold. What is this threshold? Doing a 'df' shows that the system reserves 10% of each partition, since the amounts in the used and available columns only add up to 90% of the total blocks in each partition. My boss maintains that 10% of the AVAILABLE blocks must be kept free, leaving us with only about 81% of the total disk space. I think that the system's already got the space it needs. Could someone PLEASE tell me I'm right, so we can get back all that wasted space? (9% of 3 Fuji Eagles) -- Dave Glowacki daveg@pwcs.StPaul.GOV ...!amdahl!ems!pwcs!daveg Disclaimer: Society's to blame.
mjy@sdti.UUCP (Michael J. Young) (03/11/88)
In article <3755@cup.portal.com> Paul_Steven_Mahler@cup.portal.com writes: >Aim Technology (415) 856-8649 sells a set of utilities that Gene >Dronick wrote which re-organize discs. Has anyone used these? Do they perform in-place reorganization, or do they work like dcopy? The biggest headache with dcopy is the requirement for a second volume. I typically reorganize my disks whenever they become more than 25-30% fragmented. Usually, I just do a mkfs and restore after a normal weekly backup. It's not quite as good as dcopy, but it's close enough, and still results in a big throughput improvement. -- Mike Young - Software Development Technologies, Inc., Sudbury MA 01776 UUCP : {decvax,harvard,linus,mit-eddie}!necntc!necis!mrst!sdti!mjy Internet : mjy%sdti.uucp@harvard.harvard.edu Tel: +1 617 443 5779
bzs@bu-cs.BU.EDU (Barry Shein) (03/12/88)
[note that all of this only applies to 4.2 derived systems] >Doing a 'df' shows that the system reserves 10% of each partition, since >the amounts in the used and available columns only add up to 90% of the >total blocks in each partition. My boss maintains that 10% of the >AVAILABLE blocks must be kept free, leaving us with only about 81% of the >total disk space. I think that the system's already got the space it needs. > >Could someone PLEASE tell me I'm right, so we can get back all that wasted >space? (9% of 3 Fuji Eagles) >-- >Dave Glowacki daveg@pwcs.StPaul.GOV ...!amdahl!ems!pwcs!daveg From "A Fast File System for UNIX*", Marshall Kirk McKusick, William N. Joy, Samaul J. Leffler, Robert S. Fabray. "In order for the layout policies to be effective, a file system cannot be kept completely full. For each file system there is a parameter, termed the free space reserve, that gives the minimum acceptable percentage of file system blocks that should be free. If the number of free blocks drops below this level only the system administrator can continue to allocate blocks." You're right, the boss is wrong, you can have your 10% back, the enforced 10% should cover it. But a few things to note: The 10% figure is a statistical best guess, the algorithm in the file system only knows whether it's succeeding to get a free block by one of three progressively less efficient algorithms (in hashalloc()) each tried in turn. The authors do show in the paper how they arrived at their 10% figure, it's not a randomly chosen number but is derived from measurements. This means you *might* get better performance with even more free space, but chances are also that it's insignificant, 10% is a pretty good margin, if your boss really needs to squeak that much out then tell him/her/it to buy another disk. Much more importantly, most of the hit (but not all) occurs when a block is being allocated, that is, a file is being created and/or extended on a nearly full file system. You won't lose nearly as much if you fill up a file system which then is always (or nearly always) read, such as a partition full of printer fonts or documentation. The caveat was that as cylinder groups fill disk blocks are likely to land all over the disk as the last few percent is allocated so there will likely be some head optimiztion loss on seeks to read blocks. Then again, the buffer caches can negate this effect if the same stuff gets read frequently and/or the caches are large enough, big buffer caches are a big win, forget the extra disk, tell your boss to buy a little more memory and pump up nbuf... This all adds up to a wonderful example of what I was told to do as a student whenever you are asked an operating systems question, always begin with the phrase "Well, it depends..." Cheers, and be sure to read Chris Torek's answer also :-) -Barry Shein, Boston University
steve@polyslo.UUCP (Steve DeJarnett) (03/13/88)
In article <4336@pwcs.StPaul.GOV> daveg@pwcs.StPaul.GOV (Dave Glowacki) writes: >In article <1097@hubcap.UUCP> hubcap@hubcap.UUCP (Mike Marshall) writes: [discussion deleted] >Could someone PLEASE tell me I'm right, so we can get back all that wasted >space? (9% of 3 Fuji Eagles) Yes, you're right. The system does keep 10% extra in reserve. We weren't sure this was true until we filled up our user file system, but sure enough, 'df' showed /usr as being 110% of capacity -- 0 free blocks. I don't know how you can go about changing how 'df' works, short of recoding it, but the extra 10% is definitely usable (although when you get to that point, you'd probably better start shoving stuff on tape and deleting fast -- the last 10% of an Eagle goes pretty fast). ------------------------------------------------------------------------------- | Steve DeJarnett | ...!ihnp4!csun!polyslo!steve | | Computer Systems Lab | ...!{csustan,csun,sdsu}!polyslo!steve | | Cal Poly State Univ. | ...!ucbvax!voder!polyslo!steve | | San Luis Obispo, CA 93407 | | ------------------------------------------------------------------------------- #include <std_disclaimer.h>
chris@trantor.umd.edu (Chris Torek) (03/13/88)
In article <20574@bu-cs.BU.EDU> bzs@bu-cs.BU.EDU (Barry Shein) writes: >[note that all of this only applies to 4.2BSD derived systems] >>... My boss maintains that 10% of the AVAILABLE blocks must be kept free. >>... I think that the system's already got the space it needs. >>Dave Glowacki daveg@pwcs.StPaul.GOV ...!amdahl!ems!pwcs!daveg >You're right, the boss is wrong, you can have your 10% back, the >enforced 10% should cover it. Agreed. >... whenever you are asked an operating systems question, always >begin with the phrase "Well, it depends..." Or, `no and yes' (see _The_Lord_of_the_Rings_, Chapter 3: Three Is Company, very near the end of the chapter). >Cheers, and be sure to read Chris Torek's answer also :-) Seriously, all I can add is that if your boss stubbornly refuses to acknowledge that he might be wrong, you could run tunefs -m 0 /u to change the internally enforced free space to 0% (so that the 10% that you now see will become 19%). Of course you should be ready with an explanation as to where the space came from; `I rolled some old files off to tape' should suffice :-) . -- In-Real-Life: Chris Torek, Univ of MD Computer Science, +1 301 454 7163 Domain: chris@mimsy.umd.edu Path: ...!uunet!mimsy!chris
dkc@hotlr.ATT (Dave Cornutt) (03/15/88)
You're right. In average use, the threshold where the BSD FFS starts to
have trouble finding free blocks is at 90% of the TOTAL file space
available, NOT 90% of the 90%. Incidentally, the reserve is there to
avoid performance problems that occur when the disk gets near full;
it wasn't really intended to be a "system manager reserve", in the
same sense that the last slot in the process table is.
Actually, on file systems consisting mostly of large files, I've been
able to push the reserve down as low as 3% without noticable performance
problems. (You can adjust this using "tunefs". Make sure the file
system is unmounted first.) On something big like an Eagle, this could
mean an extra 30Mb to play with.
--
Dave Cornutt, AT&T Bell Labs (rm 4A406), Holmdel, NJ (Note new address!)
UUCP:{ihnp4,allegra,cbosgd}!hotly!dkc (path stolen from Shelley)
"The opinions expressed herein are not necessarily my employer's, not
necessarily mine, and probably not necessary"rbj@icst-cmr.arpa (Root Boy Jim) (03/16/88)
You're right, the boss is wrong, you can have your 10% back, the enforced 10% should cover it. In fact, as root, you can allocate 111%. This is a good thing to do with system sources, documents, and manual pages. In fact, I loaded the distribution tapes verbatim into a couple of overloaded partitions, and symlinked them to the real /usr/whatever where appropriate. It really cuts down on the size of your operational system. Dumps get smaller too. Of course you should be ready with an explanation as to where the space came from; `I rolled some old files off to tape' should suffice :-) . How about, `Gee, boss, I compressed all your files' :-) (Root Boy) Jim Cottrell <rbj@icst-cmr.arpa> National Bureau of Standards Flamer's Hotline: (301) 975-5688
allbery@ncoast.UUCP (Brandon Allbery) (03/16/88)
As quoted from <305@marconi.SW.MCC.COM> by knutson@marconi.SW.MCC.COM (Jim Knutson): +--------------- | Dump and restore always worked well in the past for defragmenting a disk. | The real question is why you would want to do this on a BSD system | (assuming it is 4.2 or greater). For AT&T System X, try your favorite | method of backup and restore (cpio I suppose). +--------------- Under System V, the way to do it is dcopy: it defragments the disk, spreads free blocks evenly over the disk to slow the effect of further fragmentation, sorts directories to place subdirectories first and thereby speed pathname accesses, etc. -- Brandon S. Allbery, moderator of comp.sources.misc {well!hoptoad,uunet!hnsurg3,cbosgd,sun!mandrill}!ncoast!allbery
jk@Apple.COM (John Kullmann) (03/16/88)
I am not recommending it (yet) because I haven't used it by AIM offers FSTUNE which allows you to selectively compress unix files systems in place (some or all files...). Sounds like a great idea. -- John Kullmann Apple Computer Inc. Voice: 408-973-2939 Fax: 408-973-6489
kurt@hi.unm.edu (Kurt Zeilenga) (03/16/88)
In article <12419@brl-adm.ARPA> rbj@icst-cmr.arpa (Root Boy Jim) writes: > > You're right, the boss is wrong, you can have your 10% back, the > enforced 10% should cover it. > >In fact, as root, you can allocate 111%. Yes, because 10% of 111% is 11% and 111% - 11% is 100% and so the 100% is actually 90% and the 111% is actually 100%. I guess you've never taken new math? :-) -- Kurt (zeilenga@hc.dspo.gov)
mangler@cit-vax.Caltech.Edu (Don Speck) (03/26/88)
In article <4336@pwcs.StPaul.GOV>, daveg@pwcs.StPaul.GOV (Dave Glowacki) writes: > My boss maintains that 10% of the > AVAILABLE blocks must be kept free, leaving us with only about 81% of the > total disk space. I think that the system's already got the space it needs. What counts is the percentage of whole blocks that are free. Minfree is one way of attaining this percentage. If you're able to maintain free space by other means, you can reduce it (our facilities manager enforces enough free space that I set minfree to 5% on our user disks). In article <2452@umd5.umd.edu>, chris@trantor.umd.edu (Chris Torek) suggests: > tunefs -m 0 /u However, this is too far. If you have any small files, some of the free space will be in the form of fragments. If all that's left is fragments, you can't create a file larger than the largest free fragment. When this happens, the disk should be declared full, so that file truncations due to insufficient space will be obvious enough (empty files) that neither humans nor programs (such as sendbatch) will be tempted to use the trashed files. (There oughta be an option...) It's also wise to reserve a few whole blocks for critical operations like updating /etc/passwd (ouch!). Fragments are reassembled into whole blocks only when all the fragments of the block are free at the same time. This sets a lower bound on average free space. If you maintain less, fragments will accumulate and force more unusable free space until adjoining files are deleted. The more file tails that fit in a block, the less likely that all will be free at once. If the median file size is less than 1/3 of a block, you can expect several percent of the space to be lost to fragmentation. The median file size can be increased some by raising the fragment size, but that wastes more space than the fragmentation, so instead you should decrease the blocksize. Usenet articles are usually the worst, causing about 3% fragmentation (assuming a 4K blocksize). Filesystems with mostly static files do a lot better (our /usr/src has 1% minfree). Beyond this bare minimum of free space, you'd like to have enough free that a rotationally optimum block is usually available. The more tracks per cylinder, the more rotationally optimum blocks there are to choose from (thus disks with many heads may be able to run a little more full). If the disk has nonzero head-switch delay (RA81), there is only one rotationally optimum block, i.e. staying on the same track (done with negative rotdelay), so on those you might want to leave more free space. Ditto for contiguous allocation (done with tunefs -a and special drivers). If you're squeezed for space, you should look first at the number of inodes allocated. It's not unusual to be able to reclaim 4% of the total space that way. (I wonder how a free space reserve affects SysV, Counterpoint & Masscomp filesystems)? Don Speck speck@vlsi.caltech.edu {amdahl,ames!elroy}!cit-vax!speck
allbery@ncoast.UUCP (Brandon Allbery) (04/04/88)
As quoted from <5949@cit-vax.Caltech.Edu> by mangler@cit-vax.Caltech.Edu (Don Speck): +--------------- > (long discussion of BSD filesystems deleted) | (I wonder how a free space reserve affects SysV, Counterpoint & Masscomp | filesystems)? +--------------- It doesn't affect System V very much at all: the "old-style" filesystem doesn't have to keep track of fragments and the free list is kept in such a way that either the first block in the free list is available or the file system is full. No degradation on full filesystems at the price of being slightly slower overall -- much *faster* on the systems I use, but then you seem to need CPU to use the Berkeley FFS, and System V is popular on the little *nix boxes. A free space reserve *is* a good idea on the pipedev, though: at least under System III, the system just about stops dead when it can't allocate a block for a pipe. (Even though one would think that pipe blocks would remain in the cache insteda of on disk -- is this a System III bug?) -- Brandon S. Allbery, moderator of comp.sources.misc {well!hoptoad,uunet!hnsurg3,cbosgd,sun!mandrill}!ncoast!allbery
mangler@cit-vax.Caltech.Edu (Don Speck) (04/15/88)
In article <7582@ncoast.UUCP>, allbery@ncoast.UUCP (Brandon Allbery) writes: > you seem to need CPU to use the Berkeley FFS, and System V is popular on the > little *nix boxes. I've heard over and over that the Berkeley Fast Filesystem is considered a CPU hog. So I tried a trivial experiment, a program that does 512 writes of 8K bytes each (enough to exceed the buffer cache) and the times were: 3B20S SysV.2, 1K fs blocksize 32.6 real 0.0 user 11.4 sys vax-750 4.3BSD 8K fs blocksize 8.5 real 0.0 user 8.2 sys BSD beat SysV in system time, even though SysV was running on a CPU twice as fast and optimized for SysV! So why does everybody claim that the BSD filesystem is a CPU hog? Don Speck speck@vlsi.caltech.edu {amdahl,ames!elroy}!cit-vax!speck
jaw@eos.UUCP (James A. Woods) (04/16/88)
From article <6139@cit-vax.Caltech.Edu>, by mangler@cit-vax.Caltech.Edu (Don Speck): > In article <7582@ncoast.UUCP>, allbery@ncoast.UUCP (Brandon Allbery) writes: >> you seem to need CPU to use the Berkeley FFS, and System V is popular on the >> little *nix boxes. > > I've heard over and over that the Berkeley Fast Filesystem is considered a > CPU hog. So I tried a trivial experiment, a program that does 512 writes > of 8K bytes each (enough to exceed the buffer cache) and the times were: > > 3B20S SysV.2, 1K fs blocksize 32.6 real 0.0 user 11.4 sys > vax-750 4.3BSD 8K fs blocksize 8.5 real 0.0 user 8.2 sys > > BSD beat SysV in system time, even though SysV was running > on a CPU twice as fast and optimized for SysV! So why does > everybody claim that the BSD filesystem is a CPU hog? > > Don Speck speck@vlsi.caltech.edu {amdahl,ames!elroy}!cit-vax!speck the rumor probably started at murray hill, where they don't look too kindly on complicated filesystem code. but the payoff with the berkeley filesystem is too large to ignore, as tests against sys5 repeatedly show. it'd be interesting to see stats comparing the edition nine 4k block system with the 4.3 one on the same machines, but i don't know how weinberger handles the horrendous small-file wastage addressed by mckusick's fancy block/fragment code. sure berkeley wastes 10% of the disk (does someone now have a cure for this?), but throwing away 45% by just using large blocks with no fragment scheme would be disaster. some say the advantages theoretically disappear in a multi-user environment where seeks are scheduled randomly, but i've never seen the degradation in real life. ironically, the small boxes need the performance boost just as much -- witness the apple mac2 50k byte/sec fiasco. any machine that can't load a megabyte of ascii in a second or two requires coding contortions for basic word-processing at hand/eye reflex speed. and small machines generally don't run multi-user, so the other random seek myth wouldn't even apply. so again, why do vendors deal with the system5 filesystem at all?
chris@mimsy.UUCP (Chris Torek) (04/16/88)
>In article <6139@cit-vax.Caltech.Edu> mangler@cit-vax.Caltech.Edu >(Don Speck) writes: >>I've heard over and over that the Berkeley Fast Filesystem is considered a >>CPU hog. So I tried a trivial experiment.... BSD beat SysV in system >>time, even though SysV was running on a CPU twice as fast and optimized >>for SysV! So why does everybody claim that the BSD filesystem is a CPU hog? In article <548@eos.UUCP> jaw@eos.UUCP (James A. Woods) answers: >the rumor probably started at murray hill, where they don't look too >kindly on complicated filesystem code. Actually, I remember a Berkeley paper---probably the original fast file system paper---that claimed that the 4.2BSD file system was designed with faster processors in mind; this might lead to such a claim. The 4.3BSD `/usr/doc/smm/14.fastfs' paper (which those who do not understand the 10% free space reserve are encouraged to PLEASE GO READ NOW AND STOP BLATHERING AND ... sorry, I got carried away :-) ) notes that The overhead of allocating blocks in the new system is greater than the overhead of allocating blocks in the old system, however fewer blocks need to be allocated in the new system because they are bigger. The net effect is that the cost per byte allocated is about the same for both systems. >the payoff with the berkeley filesystem is too large to ignore, as >tests against sys5 repeatedly show. I have never done the tests, but given the difference between the 4.2BSD FFS and the old 4.1BSD FS, if the SV FS is much like the 4.1BSD FS (which, I gather, it is), I would have to agree. Writes are at least no more common than reads, and since `the cost per byte allocated is about the same' one has nothing to lose anyway, except perhaps the need for fsdb. >some say the advantages theoretically disappear in a multi-user environment >where seeks are scheduled randomly, but i've never seen the degradation >in real life. I think I have, but so rarely that it matters not. 95%* of the time you win by 85%*, so you are 80% ahead already, or something along those lines. Especially it does not hurt that backups take half the time they consumed under 4.1BSD. [*These numbers were calculated by that time-honoured method, Choosing Numbers that Sound Good Based on Nothing at All. Your mileage may vary, moreso in California and where prohibited by law or customs officials. (The author of this article is suffering from Chocolate Overdose and an overly loud rendition of `Where's the Walrus?' from the _Stereotomy_ CD, if you were wondering.)] -- In-Real-Life: Chris Torek, Univ of MD Comp Sci Dept (+1 301 454 7163) Domain: chris@mimsy.umd.edu Path: uunet!mimsy!chris
guy@gorodish.Sun.COM (Guy Harris) (04/17/88)
> I have never done the tests, but given the difference between the > 4.2BSD FFS and the old 4.1BSD FS, if the SV FS is much like the 4.1BSD > FS (which, I gather, it is), It is. Both are minor variants of the V7 FS; the main difference between the SV and 4.1 versions is that the SV version has a magic number in the super block that indicates whether it's a 512-byte or 1KB file system, whereas the 4.1 FS is always a 1KB file system (at least in the standard 4.1 release).