cfj@isc.intel.com (Charlie Johnson) (11/09/90)
I'm curious if the companies who support Unix on large systems made the necessary file system changes to allow individual files which are larger than 4 gigabytes ?? You'd have to at least stretch the file size in the inode beyond 32 bits and possibly mess around in the super block. Any comments ?? -- Charles Johnson Intel Scientific Computers, MS CO1-01 15201 NW Greenbrier Pkwy Beaverton, OR 97006 phone: (503)629-7605 email: cfj@isc.intel.com
jeff@onion.pdx.com (Jeff Beadles) (11/10/90)
In <1008@intelisc.isc.intel.com> cfj@isc.intel.com (Charlie Johnson) writes: >I'm curious if the companies who support Unix on large systems made the >necessary file system changes to allow individual files which are larger >than 4 gigabytes ?? You'd have to at least stretch the file size in the >inode beyond 32 bits and possibly mess around in the super block. Any >comments ?? Well, that would take one big disk :-) Unix files can not span physical disk partitions, at least on more common version of Unix. (Has anyone changed this?) This pretty well limits the file size more than the kernel internals. Then again, the largest file that I've seen in "real-life" is a 247mb kernel core dump :-) -Jeff -- Jeff Beadles jeff@onion.pdx.com
bernhold@qtp.ufl.edu (David E. Bernholdt) (11/11/90)
In article <1990Nov9.170337.9484@onion.pdx.com> jeff@onion.pdx.com writes: >Well, that would take one big disk :-) Unix files can not span physical disk >partitions, at least on more common version of Unix. (Has anyone changed this?) I think a lot of people (vendors) have worked on this one way or another. One of the most common methods for "minisupers" is disk striping, where several disks are combined into a single logical disk. Take 4 x 1GB disks on 4 controllers and you have a big, fast disk 4GB in size. If you want something really big, make a RAID array of 1GB disks! Also, IBM's AIX on our RS/6000 has a mechanism which allows all kinds of manipulations of the disks into all sorts of logical partitions, including spanning across physical drives. I think Sun has this too. Perhaps others too... Of course it still depends on the kernel if individual files can be > 4GB. There are many users out there who really do need file systems larger than a single drive & even files larger than a single drive. I'm sure its only a matter of time... -- David Bernholdt bernhold@qtp.ufl.edu Quantum Theory Project bernhold@ufpine.bitnet University of Florida Gainesville, FL 32611 904/392 6365
brtmac@maverick.ksu.ksu.edu (Brett McCoy) (11/11/90)
In <1990Nov9.170337.9484@onion.pdx.com> jeff@onion.pdx.com (Jeff Beadles) writes: >In <1008@intelisc.isc.intel.com> cfj@isc.intel.com (Charlie Johnson) writes: >>I'm curious if the companies who support Unix on large systems made the >>necessary file system changes to allow individual files which are larger >>than 4 gigabytes ?? You'd have to at least stretch the file size in the >>inode beyond 32 bits and possibly mess around in the super block. Any >>comments ?? >Well, that would take one big disk :-) Unix files can not span physical disk >partitions, at least on more common version of Unix. (Has anyone changed this?) Unless what I read recently is totally bogus, the latest version of AIX, the version that is on the new RS/6000 machines, has support for spanning filesystems across multiple disks. It does this deep down in the kernal so that to filesystem sees the multiple disks as a single volume. It actually has support for quite a few tricks including disk mirroring. How much of this is actually usable right now I don't know, but supposedly it does work. There is also some company that sells either software or hardware for Sun machines to do disk mirroring and disk spanning. All I know about this is what I saw in the one page ad. I don't even remember the magazine I saw it in anymore. >This pretty well limits the file size more than the kernel internals. No, it is the kernel internals that allows things like disk spanning, so it is the kernel that limits things, not the physical disks themselves. >Then again, the largest file that I've seen in "real-life" is a 247mb kernel >core dump :-) I quite regularly have 300-400M dump files. I have to put them on tape, broken up into multiple pieces, which makes it a pain when looking for a file. Being able to load these onto disk occasionally might make things easier under the right circumstances. -- Every day you scream at me to turn the music low. But if you keep on screaming, you'll make me deaf, ya know. -- Judas Priest Brett McCoy brtmac@maverick.ksu.ksu.edu
buck@siswat.UUCP (A. Lester Buck) (11/12/90)
In article <1990Nov9.170337.9484@onion.pdx.com>, jeff@onion.pdx.com (Jeff Beadles) writes: > In <1008@intelisc.isc.intel.com> cfj@isc.intel.com (Charlie Johnson) writes: > > >I'm curious if the companies who support Unix on large systems made the > >necessary file system changes to allow individual files which are larger > >than 4 gigabytes ?? You'd have to at least stretch the file size in the > >inode beyond 32 bits and possibly mess around in the super block. Any > >comments ?? > > Well, that would take one big disk :-) Unix files can not span physical disk > partitions, at least on more common version of Unix. (Has anyone changed this?) > This pretty well limits the file size more than the kernel internals. > > Then again, the largest file that I've seen in "real-life" is a 247mb kernel > core dump :-) > > -Jeff > -- > Jeff Beadles jeff@onion.pdx.com Anyone who has ever written a disk driver knows that the code to support patching multiple volumes together is very easy. The hard part is adminstering it and making it available through utilities, etc. AIX is doing that now, OSF/1 with the Logical Volume Manager is coming, this is not a tough feature to add. The HARD part is the file size limitation. If you write your own filesystem, you can make the file sizes whatever you want, and can bring all the utilites and user code along with you. But if you want to be able to use the existing Unix utilities and user code that "knows" that stat() returns a long for file size, you are stuck. The only transparent method that supports old code is to have the compiler support long as a 64 bit entity. Cray has done this, I believe. This is a serious efficiency hit on most other machines, where the architecture does not directly support 64 bit arithmetic. -- A. Lester Buck buck@siswat.lonestar.org ...!uhnix1!lobster!siswat!buck
littauer@uts.amdahl.com (Tom Littauer) (11/13/90)
In article <1008@intelisc.isc.intel.com> cfj@isc.intel.com (Charlie Johnson) writes: > >I'm curious if the companies who support Unix on large systems made the >necessary file system changes to allow individual files which are larger >than 4 gigabytes ?? Yes. Up to 6 terabytes is practical, exabytes are theoretically possible. > You'd have to at least stretch the file size in the >inode beyond 32 bits and possibly mess around in the super block. Any >comments ?? File size is now 64 bits, the major/minor is now 32 bits (from 16), fsck is a thing of the past (on 6 terabytes it would've been ugly indeed). Size carries with it other issues, like reliability (we use a RAID-5-like checksum scheme), sequential I/O performance (large blocks, contiguous allocation, and striping), and backup/recovery (special tools). Yes, all of this is still SVID standard. Oops, I'm starting to sound like an ad. Sorry. The point was only that you can't just change the limits... you've got to think through the related issues of *BIG* UNIX systems. -- UUCP: littauer@amdahl.amdahl.com or: {sun,decwrl,hplabs,pyramid,ames,uunet}!amdahl!littauer DDD: (408) 737-5056 USPS: Amdahl Corp. M/S 278, 1250 E. Arques Av, Sunnyvale, CA 94086 I'll tell you when I'm giving you the party line. The rest of the time it's my very own ravings (accept no substitutes).
terry@uts.amdahl.com (Lewis T. Flynn) (11/15/90)
In article <575@siswat.UUCP> buck@siswat.UUCP (A. Lester Buck) writes: [other stuff omitted] >The HARD part is the file size limitation. If you write your own >filesystem, you can make the file sizes whatever you want, and can >bring all the utilites and user code along with you. But if you >want to be able to use the existing Unix utilities and user code >that "knows" that stat() returns a long for file size, you are stuck. >The only transparent method that supports old code is to have the >compiler support long as a 64 bit entity. Cray has done this, I >believe. This is a serious efficiency hit on most other machines, >where the architecture does not directly support 64 bit arithmetic. This is pretty much true, but ignores the problem of binary compatibility. We handled it under the general heading of "expanded fundamental types" in which we relieved several size limits. For file sizes, stat(), lseek(), and everything else which worries about offsets and sizes now use the type "off_t" which in eft mode is a signed 64 bit integer and in non-eft mode is the same old long we all knew and felt luke warm about. Whether a program is in eft mode or not is a compile time option (default non-eft) and libraries are provided for both modes. What all this means is, if you don't need large files (or some other eft feature), don't do a thing. Your binaries and compiles work as before. If you do want to take advantage of this stuff, you'll need to make sure your source uses the correct types on certain system calls and library routines and recompile with the eft option. All commands shipped with UTS 2.1 are eft-ized (I think, I'll be corrected if I'm wrong) and handle large files where feasible. disclaimer: I'm not a spokesman for Amdahl and I believe this is all documented in the manuals someplace.