daveh@norge.SGI.COM (David A. Higgen) (01/20/89)
I'm going to be working shortly on ways of extending disk storage over more than one physical disk ("logical volumes", "multi-spindle filesystems" or what-have-you). I'd be glad to have ideas/comments/wishlists from interested parties. Some initial questions. 1) Is your interest in this mainly: a) A need for larger storage, ie more files in a filesystem, or b) A need for really long files? If the latter, what sort of file sizes do you think might arise in practical applications for our systems over the next year or so? 2) Is there any percieved need for large RAW (ie non-filesystem) storage? 3) To address large storage objects, we would have to augment the semantics of 'lseek' in some way. Two possibilities: * extra defines in the "whence" field specifying units of (say) Kbytes instead of bytes. * A new system call taking more than one word to allow a larger range. (Presumably 64 bits, probably typedef'd for convenience). Preferences? Alternatives? Is there really a need to address very large storage objects to byte granularity? 4) How important is extensibility? (ie the ability to add the space of a new disk to an existing filesystem). If this IS considered important, must it be totally dynamic (ie can be done even when the filesystem is mounted and in use) or could we live with needing to unmount the filesystem to extend it? 5) Do you foresee any demand for 'mirroring' ie automatically keeping data on duplicate/triplicate etc storage for reliability? (We are not currently in the database/transaction-processing market where this is common, but who knows)....
tcs@BRL.MIL (Terry Slattery, SECAD) (01/21/89)
> 1) Is your interest in this mainly: > > a) A need for larger storage, ie more files in a filesystem, or > > b) A need for really long files? > > If the latter, what sort of file sizes do you think might arise in practical > applications for our systems over the next year or so? Both. With graphical images, large amounts of storage are needed, especially when making animation sequences. Similarly, these sequences are often stored on some sort of bulk media. Sometimes the bulk media must be used in a way which dictates the use of a single file for efficiency reasons (Exabyte tapes, for example). I would like to take a set of animation files (a large number of 1Mby files) and store them as a single entity and still be able to perform seeks within this entity. This isn't possible if the file is > 2^^31 bytes. > 2) Is there any percieved need for large RAW (ie non-filesystem) storage? Not from me. But what about optical disks? Most WORM drives are best used as raw devices. > 3) To address large storage objects, we would have to augment the semantics > of 'lseek' in some way. Two possibilities: > > * extra defines in the "whence" field specifying units of (say) Kbytes > instead of bytes. > > * A new system call taking more than one word to allow a larger > range. (Presumably 64 bits, probably typedef'd for convenience). > > Preferences? Alternatives? > > Is there really a need to address very large storage objects to byte > granularity? Please make the 64 bit offset your first choice and don't change the semantics of lseek. Other vendors have already started work on "qseek", which takes a quad instead of a long as the offset. Retain the lseek syscall in the kernel for some period of time (e.g., two major releases) and install a library routine named lseek which calls a new qseek syscall, so that existing binaries and sources both continue to work. Old binaries will still work via the retained lseek syscall and any recompilation will be linked to the new qseek syscall via the library. Applications using large files can be fixed to use the new qseek. > 4) How important is extensibility? (ie the ability to add the space of a new > disk to an existing filesystem). If this IS considered important, > must it be totally dynamic (ie can be done even when the filesystem is > mounted and in use) or could we live with needing to unmount the > filesystem to extend it? We generally don't use local disks, preferring instead to keep our user files on NFS servers which are regularly backed up by operators. This feature would be nice on the NFS servers. Since NFS is stateless, it would not be unreasonable to have to unmount the partition in order to extend its size. > 5) Do you foresee any demand for 'mirroring' ie automatically keeping data > on duplicate/triplicate etc storage for reliability? (We are not currently > in the database/transaction-processing market where this is common, but > who knows).... This feature would be nice for critical data. The database folks are not the only people who are concerned about data reliability. You might want to get in touch with Tom Van Baak at Pyramid Technology (pyramid!tvb) about his paper "Virtual Disks: A New Approach to Disk Configuration" which was presented at the Winter 1987 Usenix. The proceedings I have contains a very short two page paper which looks like it may have been truncated; his introduction mentions experimentation results and a more complete description which is missing. He may be able to send you the complete paper. His work allowed 1) single disks, 2) concatenated disks, 3) striped disks, and 4) mirrored disks, all under control of a single configuration file. Thanks for asking. -tcs