karl-d@nisca.ircc.ohio-state.edu (Doug Karl) (06/28/90)
This is a summary of the responses I received from an earlier posting.
With a plea for help in adding an allocate function to future releases of Unix.
REVIEWING EARLIER POSTING:
I am in desperate need to allocate a Unix (BSD 4.3, SUN/OS, System V,
and NeXT/Mach) file prior to using it. By "allocate" I mean that all of
the blocks in the file (10's of megabytes long) should be reserved to that
file instantly). This is possible on all other operating systems except
Unix, because Unix allocates blocks to it's file only when they are first
touched (or written into). This is a well known characteristic of the
Unix file system.
What I am doing is recording several channels of CD/DAT quality sound
simultaniously on generic SCSI disks attached to a computer using a
special SCSI disk controller. The SCSI disk controller sits between the
SCSI bus on the CPU and the disks (it has 2 SCSI ports). It is operating
system and file system independent and therefore works on Unix (assuming
SCSI disks). BUT it expects that a file is fully allocated by the
operating system prior to recording (so it knows where on disk to put
the data) which takes a fraction of a second on VMS, MS-DOS, and MAC/OS
but takes several minutes on Unix. This is because I have to write out
each block of the file for it to be "fully" allocated. After the file
is allocated then I mearly write to the SCSI disk controller the files
logical block mapping and it then can record the sound. Playing or
re-recording is no problem since the file is already allocated.
This is a very significant problem since for the typical example of a
Unix based 16 track recording studio, recording a song 2 minutes long
would take 16 minutes to allocate the files (using the write each block
method).
SUMMARY OF RESPONSES:
- Many people suggested forgetting the Unix file system and writing my own.
--> Ruins the beauty of using existing Unix tools and file system <---
- Others (mostly data base people) said they have the same problem and wrote
their own file system using a raw partition but the code is propriatory.
--> Oh well <--
- Some are writing compressed video to disk and have the same problem.
--> This is another reason to get fast allocate to work on Unix <--
- Some misunderstood the question and told me that Unix was a multi-user system
and suggested I post the request to the MS-DOS news group.
--> ???? <--
- Some misunderstood the problem and suggested using mkfile.
--> This would still take 16 minutes to setup for 2 minutes of recording
since it does a write to each block in order to allocate a file. <--
- Some suggested I record to a raw partition and then copy to a Unix partition.
--> This would take 32 minutes to copy after the 2 minutes of recording <--
- One person pointed out that when I write the blocks they had better not be
full of zeros or the blocks would still not be allocated on some falvors
of Unix.
- One person commented that holey files have been a thorn in the side of Unix
for years and that is just the way it is! (maybe he ment holy files).
The best response was from an engineer at SUN who suggested modifying Unix to
perform the allocate function directly. I presume this would be a kernel mod
that would have to be provided on all of the Unix platforms and be compateble
with NFS. I did find someone who said they have a kernel mod to do such a
thing but I have not received it yet.
PLEA!!:
How do I get the Unix software design engineers and standards community to
consider modifying future releases of Unix of all flavors to allocate files?
With computer based audio recording studios and compressed video to disk
hitting the streets, isn't it a good idea for Unix to be there too? I see this
as a critical issue and easy to fix short comming of the Unix file system.
Is there anyone out there who grasps the importance of such a modification
to data base and real time users of Unix and who sees that such a modification
would not be a hinderance to Unix but an enhancement???
Please respond to me directly with ANY ideas and I will summarize:
THANKS...
Doug Karl,
Senior Computer Specialist
Instruction and Research Computer Center
The Ohio State University
karl-d@osu-20.ircc.ohio-state.edu
(614) 292-4843 (please feel free to call)buck@siswat.UUCP (A. Lester Buck) (06/29/90)
In article <2392@nisca.ircc.ohio-state.edu>, karl-d@nisca.ircc.ohio-state.edu (Doug Karl) writes: > This is a summary of the responses I received from an earlier posting. > With a plea for help in adding an allocate function to future releases of Unix. [ needs to allocate all blocks of a file "instantly" ] > What I am doing is recording several channels of CD/DAT quality sound > simultaniously on generic SCSI disks attached to a computer using a > special SCSI disk controller. The SCSI disk controller sits between the > SCSI bus on the CPU and the disks (it has 2 SCSI ports). It is operating > system and file system independent and therefore works on Unix (assuming > SCSI disks). BUT it expects that a file is fully allocated by the > operating system prior to recording (so it knows where on disk to put > the data) which takes a fraction of a second on VMS, MS-DOS, and MAC/OS > but takes several minutes on Unix. This is because I have to write out > each block of the file for it to be "fully" allocated. After the file > is allocated then I mearly write to the SCSI disk controller the files > logical block mapping and it then can record the sound. Playing or Uh, you have a serious contradiction here. "Operating and file system independent" software doesn't require writing out filesystem logical block mappings to the disk controller. > This is a very significant problem since for the typical example of a > Unix based 16 track recording studio, recording a song 2 minutes long > would take 16 minutes to allocate the files (using the write each block > method). > SUMMARY OF RESPONSES: > - Many people suggested forgetting the Unix file system and writing my own. > --> Ruins the beauty of using existing Unix tools and file system <--- I suggested you use the raw device in email the first time around. A simple flat filesystem (you are talking very large files here) is pretty easy to do. [some suggestions deleted] I will assume your SCSI controller stripes to its disks, something like RAID. Since you have to download your logical mapping of allocated space, I assume it does its own remapping to make the data contiguous for performance, then fakes out the OS. Of course, this is hardly OS independent. Here is the solution to your problem. I presume you are using a well documented filesystem, such as System V or BSD Fast File System. (This was also posted to the AIX group, but all bets are off if you are using TCF and the Locus distributed filesystem!) Unmount the target filesystem and _allocate the space yourself_. This is not nearly as hard as it might sound, if you have read and understood the relevant section in Bach and played with fsdb on a floppy filesystem. Or rag on your vendor to supply the same program. This would be reasonably portable among the System V versions - not sure sure about BSD variants. The kernel is hardly doing anything magic - just grungy. BTW, I can't find an MS-DOS system call to return the file allocation. Why is is okay to directly read out the FAT on MS-DOS, but you expect not to have to directly access the inodes, etc., on Unix? > How do I get the Unix software design engineers and standards community to > consider modifying future releases of Unix of all flavors to allocate files? > With computer based audio recording studios and compressed video to disk > hitting the streets, isn't it a good idea for Unix to be there too? I see this > as a critical issue and easy to fix short comming of the Unix file system. > Is there anyone out there who grasps the importance of such a modification > to data base and real time users of Unix and who sees that such a modification > would not be a hinderance to Unix but an enhancement??? The POSIX.4 Realtime Extensions draft includes a chapter on Realtime Files. These lay out a complete set of functions for specifying various properties of files, including contiguous, pre-allocation, extension, direct I/O, cache usage, aligned transfers, and transfer granularity. Specifically, pre-allocation makes the distinction between reservation and allocation. Reservation deals with guaranteed access to resources, while allocation removes the filesystem overhead during I/O. This is a complicated chapter, but a quick scan does not reveal any way of getting the actual logical blocks allocated to a file on a filesystem. After all, what would this mean for a POSIX implementation on, say, MVS? In general, there is no portable, operating system independent way to get the logical layout of a file on a medium, and certainly not in Unix. So even if some future version of Unix supplies a pre-allocation function, I doubt it will ever support feeding the low level details of logical block allocation back to the user, and I don't think it should. You want to do extremely system dependent, non-portable things, so you get to do extra work. Just look at all those different architectures that VMS, MS-DOS, and Mac OS run on. (Uhhh..., forget I said that. I don't want to start _that_ thread in c.u.aix all over again. :-) ) -- A. Lester Buck buck@siswat.lonestar.org ...!texbell!moray!siswat!buck
mercer@npdiss1.StPaul.NCR.COMDan Mercer) (07/04/90)
Did you attempt to use a holey file for this?
fp = fopen ("cdstuff", "w");
fseek (fp, 0x100000,0);
fputc ('\0',fp);
fclose(fp);
--
Dan Mercer
Reply-To: mercer@npdiss1.StPaul.NCR.COM (Dan Mercer)
"MAN - the only one word oxymoron in the English Language"