buck@siswat.UUCP (A. Lester Buck) (05/16/89)
When will common versions of Unix support very large file sizes of >2^32 bytes? A client is moving some massive software packages that deal with file sizes of greater than 4 GB from an MVS hosted environment to a workstation. (I know it takes a long time to just read such a file, but once loaded only small parts are processed in a random access, interactive mode.) I can glue physical drives together into a large filesystem at the driver level, but there is no getting around the 32-bit data type in inode.h . This appears to be the limit in any Unix implementation on 32-bit hardware, from System VR3.2 to AIX. A recent posting mentioned that DG/UX uses 48 bits for block numbers in the filesystem, but I couldn't tell if that included file size. Also, it seems VAX C has already implemented the ANSI C typedef fpos_t, a data type that holds whatever data is necessary to seek anywhere in a file, as a 64 bit entity. [fpos_t is returned by fgetpos() and is input to fsetpos().] Does VMS currently support file sizes of greater than 32 bits? I presume that UNICOS on the Cray and other native 64 bit Unix implementations do currently support filesizes larger than 4 GB. Will System VR4 support such large file sizes? I know we could always implement our own filesystem, but virtually every utility assumes that the file size fits in 32 bits. And the applications could be reworked to split the file into chunks and manage the chunks, but that is not very attractive, either. What does POSIX say about the maximum file size? On a related note, there is a public domain proposal circulating for a data interchange filesystem on WORM disks in the geophysical data processing industry. This file system has a standard generic hierarchical directory structure, lots of room for long filenames (126 bytes), but specifies only 32 bits for file size. And this for 12" WORM media that by the end of this year will have 2 GB/side! This can still be changed, but I hope the ANSI optical filesystem committee forming next month will pick a larger data type for file size. It is interesting to check what people think is the maximum file size in various operating systems. 4 GB is the typical maximum under Unix, not withstanding those who calculate that the triple indirect blocks can keep track of 16+ GB. (Are there any utilities that actually interpret the sign bit of the file size/seek position, which would prevent using the last 2 GB of the 32 bits?) Ask the same question of an MVS expert and there is a significant pause, then some mumbling about 1 primary unit and 255 secondary units, then calculations on how much a 3380(?) tri-density drive (unit) can currently hold (about 1.8 GB/drive, if I remember correctly). So the calculation is much more convoluted, but it does have the interesting property that maximum file size tracks disk technology, so room doesn't run out at a fixed size. (So how, you ask, does MVS seek within such a big file with a 32-bit data type? It seeks by records, of course!) In addition, I was told that for sequential access, MVS disks can be arranged in an unlimited sequence, with a zero count-key indicating physical end of file. And I heard a rumor that IBM hit the wall with AIX/370 under VM, when they reached the VM minidisk maximum size limit, long before the 32 bits ran out. -- A. Lester Buck ...!texbell!moray!siswat!buck