markd@silogic.UUCP (Mark DiVecchio) (07/10/90)
An associate has an application where he wants to reserve 4Mb of disk space. The unix file system does not support this since opening the file, seeking to 4Mb and writing one byte does not do the trick. Besides writing out 4Mb of filler, is there any other way to do this? -- Mark DiVecchio, Silogic Systems, 619-549-9841 K3FWT ----- 9888 Carroll Center Road, Suite 113, San Diego, CA 92126 ----- markd@silogic BBS 619-549-3927 ...!ucsd!celerity!celit!silogic!markd celerity!silogic!markd@ucsd.edu
istvan@hhb.UUCP (Istvan Mohos) (07/15/90)
markd@silogic.UUCP (Mark DiVecchio @ Silogic Systems) writes: >An associate has an application where he wants to reserve 4Mb of disk >space. The unix file system does not support this since opening the >file, seeking to 4Mb and writing one byte does not do the trick. >Besides writing out 4Mb of filler, is there any other way to do this? Because disk I/O optimizations are kernel tasks, there is very little you can do unless you're willing to rewrite the buffer cache routines, forcing the system to pop a few free blocks and reassign them to your inode. (Consult Chap. 3 of "The Design Of The UNIX Operating System" by Maurice J. Bach for details.) Personally, I can't blame a philosophy of having to actually write to the disk if one wants to take up space on it; in spite that if (write (fd, malloc(FOURMEG), FOURMEG) != FOURMEG) perror("malloc or write error"), exit (1); does take a few seconds to execute, "delayed-write" or not. Maybe you should just fork off this task and go on to something else. -- Istvan Mohos ...uunet!pyrdc!pyrnj!hhb!istvan RACAL-REDAC/HHB 1000 Wyckoff Ave. Mahwah NJ 07430 201-848-8000 ======================================================================
cpcahil@virtech.uucp (Conor P. Cahill) (07/16/90)
In article <563@hhb.UUCP> istvan@hhb.UUCP (Istvan Mohos) writes: > if (write (fd, malloc(FOURMEG), FOURMEG) != FOURMEG) > perror("malloc or write error"), exit (1); While I know this is just an example & is not intended to be a suggestion of what to do, I just have to comment on it to ensure that some impressionable engineer doesn't think this is a good thing to do. Anyway, my comment is never (not even in an example (unless it is an example of what not to do)) ever use the return of any function that may return NULL before you check it to see if it returned NULL. Your perror() seems to indicated that if the malloc had failed, the write would return -1 and set errno accordingly. In fact, the malloc failure would return a NULL pointer and then write would try to write out 4MB from that location. On most machines this would result in a core dump, either because NULL was not a valid address, or because there was not 4MB in the user's address space following that pointer. I'm not harping on you. I just feel that I must emphasize this point. Too often I see code that does something like the above, although getenv() is more often the culprit. -- Conor P. Cahill (703)430-9247 Virtual Technologies, Inc., uunet!virtech!cpcahil 46030 Manekin Plaza, Suite 160 Sterling, VA 22170
moss@cs.umass.edu (Eliot Moss) (07/16/90)
I don't think tromping through 4 megabytes of newly allocated address space will be much of a winner myself. I'd do something more along these lines (error checking omitted): char buf[ONE_K]; int i; for (i = 0; i < FOUR_K; ++i) write (fd, buf, ONE_K); You might want to tune the size of buf up larger than what I have, but still somewhat "reasonable". Something between the size of a track and a cylinder on the disk might work reasonably well, say 64K or 128K. Enjoy! Eliot -- J. Eliot B. Moss, Assistant Professor Department of Computer and Information Science Lederle Graduate Research Center University of Massachusetts Amherst, MA 01003 (413) 545-4206; Moss@cs.umass.edu
jik@athena.mit.edu (Jonathan I. Kamens) (07/16/90)
In article <MOSS.90Jul15190502@ibis.cs.umass.edu>, moss@cs.umass.edu (Eliot Moss) writes: |> char buf[ONE_K]; |> int i; |> |> for (i = 0; i < FOUR_K; ++i) |> write (fd, buf, ONE_K); I wrote the short program below (which I call "fillspace") when I was testing some bug fixes for bugs that would exhibit themselves only when a disk partition was almost full. I needed to be able to fill a specific amount of disk space in order to get into that situation easily. I've found it useful in other situations as well. How to use it is fairly obvious from the usage message in the code. Jonathan Kamens USnail: MIT Project Athena 11 Ashford Terrace jik@Athena.MIT.EDU Allston, MA 02134 Office: 617-253-8495 Home: 617-782-0710 ------------------------------------------------------------------ #include <stdio.h> extern char *malloc(); main(argc, argv) int argc; char *argv[]; { int k; char *garbage; int blocksize; int block; int i; if (argc < 2 || argc > 3) { fprintf(stderr, "Usage: %s blocks [blocksize]\n", argv[0]); exit(1); } if (argc == 3) blocksize = atoi(argv[2]); else blocksize = BUFSIZ; k = atoi(argv[1]) * blocksize; if (k <= 0 || blocksize <= 0) { fprintf(stderr, "%s: blocks and blocksize must be positive\n", argv[0]); exit(1); } block = 4 * blocksize; garbage = malloc((unsigned) block); if (! garbage) { perror("malloc"); exit(1); } for (i = 0; i < block; i++) garbage[i] = 'a'; while (k) { while (k >= block) { fwrite(garbage, 1, block, stdout); k -= block; } block /= 2; } exit(0); }
cjc@ulysses.att.com (Chris Calabrese[mav]) (07/16/90)
In article <MOSS.90Jul15190502@ibis.cs.umass.edu>, moss@cs.umass.edu (Eliot Moss) writes: > [ ... ] > > char buf[ONE_K]; > int i; > > for (i = 0; i < FOUR_K; ++i) > write (fd, buf, ONE_K); > [ ... ] BTW, while reading the various ways to write some large space on the disk which have been passing by, I thought I'd pass a related tidbit along. On many machine architectures, byte copying from user space to kernel space is quite a bit faster if the buffer is word aligned. For this reason, it is better to use malloc to get your buffer than to allocate it off of the stack (as malloc always returns maximally aligned memory). Of course, this assumes that you'll use the buffer more than once, as the time to malloc() is around the same order as the time to deal with the non-aligned bytes at the beginning and end of the buffer. Name: Christopher J. Calabrese Brain loaned to: AT&T Bell Laboratories, Murray Hill, NJ att!ulysses!cjc cjc@ulysses.att.com Obligatory Quote: ``pher - gr. vb. to schlep. phospher - to schlep light.philosopher - to schlep thoughts.''
martin@mwtech.UUCP (Martin Weitzel) (07/17/90)
In article <13422@ulysses.att.com> cjc@ulysses.att.com (Chris Calabrese[mav]) writes: :In article <MOSS.90Jul15190502@ibis.cs.umass.edu>, moss@cs.umass.edu (Eliot Moss) writes: :> [ ... ] :> :> char buf[ONE_K]; :> int i; :> :> for (i = 0; i < FOUR_K; ++i) :> write (fd, buf, ONE_K); :> [ ... ] : :BTW, while reading the various ways to write some large space on the :disk which have been passing by, I thought I'd pass a related tidbit :along. On many machine architectures, byte copying from user space to :kernel space is quite a bit faster if the buffer is word aligned. : :For this reason, it is better to use malloc to get your buffer than to :allocate it off of the stack (as malloc always returns maximally :aligned memory). Of course, this assumes that you'll use the buffer :more than once, as the time to malloc() is around the same order as :the time to deal with the non-aligned bytes at the beginning and end :of the buffer. What is wrong with the following approach (at least on non-BSD-ish file systems)? while file has not desired size lseek(2) from current position forward disk-block-size bytes minus 1 and write(2) one byte IMHO this should fill the disk and avoids much copying from user-space. -- Martin Weitzel, email: martin@mwtech.UUCP, voice: 49-(0)6151-6 56 83
fnf@riscokid.UUCP (Fred Fish) (07/17/90)
In article <MOSS.90Jul15190502@ibis.cs.umass.edu> moss@cs.umass.edu writes: >char buf[ONE_K]; >int i; > >for (i = 0; i < FOUR_K; ++i) > write (fd, buf, ONE_K); > >You might want to tune the size of buf up larger than what I have, but still >somewhat "reasonable". Something between the size of a track and a cylinder on >the disk might work reasonably well, say 64K or 128K. Enjoy! Eliot This reminds me, I've always wondered why there wasn't a /dev/full as a standard part of unix. It's the obvious counterpart of /dev/null, and simply returns as many null bytes as you ask for. Then all you would have to do is: dd if=/dev/full of=myfile bs=1k count=4k -Fred
ag@cbmvax.commodore.com (Keith Gabryelski) (07/17/90)
In article <13212@mcdphx.phx.mcd.mot.com> fnf@riscokid.UUCP (Fred Fish) writes: >[...], I've always wondered why there wasn't a /dev/full as a >standard part of unix. It's the obvious counterpart of /dev/null, and >simply returns as many null bytes as you ask for. Then all you would >have to do is: > > dd if=/dev/full of=myfile bs=1k count=4k On some of the latest forms of Unix (SVR4 which inherited it from SunOS) there is a /dev/zero which does this. It is used for mmap()ing in the bss segment and ignoring null pointer problems :-). Pax, Keith
stripes@eng.umd.edu (Joshua Osborne) (07/17/90)
In article <13212@mcdphx.phx.mcd.mot.com> fnf@riscokid.UUCP (Fred Fish) writes: >This reminds me, I've always wondered why there wasn't a /dev/full as a >standard part of unix. It's the obvious counterpart of /dev/null, and >simply returns as many null bytes as you ask for. Then all you would >have to do is: > > dd if=/dev/full of=myfile bs=1k count=4k SunOS has a /dev/zero, it does exactly what you want. They use it to get zero mapped pages (ld.so mmap()s shared lib's data space from the /dev/zero file). -- stripes@eng.umd.edu "Security for Unix is like Josh_Osborne@Real_World,The Mutitasking for MS-DOS" "The dyslexic porgramer" - Kevin Lockwood "Don't try to change C into some nice, safe, portable programming language with all sharp edges removed, pick another language." - John Limpert
stripes@eng.umd.edu (Joshua Osborne) (07/17/90)
In article <836@mwtech.UUCP> martin@mwtech.UUCP (Martin Weitzel) writes: >In article <13422@ulysses.att.com> cjc@ulysses.att.com (Chris Calabrese[mav]) writes: >:For this reason, it is better to use malloc to get your buffer than to >:allocate it off of the stack (as malloc always returns maximally >:aligned memory). Of course, this assumes that you'll use the buffer >:more than once, as the time to malloc() is around the same order as >:the time to deal with the non-aligned bytes at the beginning and end >:of the buffer. >What is wrong with the following approach (at least on non-BSD-ish >file systems)? > > while file has not desired size > lseek(2) from current position forward > disk-block-size bytes minus 1 and write(2) > one byte > >IMHO this should fill the disk and avoids much copying from user-space. That may cause alot less copying from user-space, but it causes more context switches from user to kernel & back. Mabie someone wants to try their hand at writev()? I think you can get writev to seek before each write. Alternetly how about mmap? You can mmap a half a meg at a time, that ought to reduce the number of syscalls, only write once every 8K (or whatever the disk block size is) reducing the memmory bandwidth, but will this result in fewer context switches then write? Then writev? -- stripes@eng.umd.edu "Security for Unix is like Josh_Osborne@Real_World,The Mutitasking for MS-DOS" "The dyslexic porgramer" - Kevin Lockwood "Don't try to change C into some nice, safe, portable programming language with all sharp edges removed, pick another language." - John Limpert
lerman@stpstn.UUCP (Ken Lerman) (07/17/90)
In article <836@mwtech.UUCP> martin@mwtech.UUCP (Martin Weitzel) writes: >In article <13422@ulysses.att.com> cjc@ulysses.att.com (Chris Calabrese[mav]) writes: [... much deleted ...] >What is wrong with the following approach (at least on non-BSD-ish >file systems)? > > while file has not desired size > lseek(2) from current position forward > disk-block-size bytes minus 1 and write(2) > one byte > >IMHO this should fill the disk and avoids much copying from user-space. >-- >Martin Weitzel, email: martin@mwtech.UUCP, voice: 49-(0)6151-6 56 83 But if you wrote two bytes at a time, you could write the last byte of one block and the first byte of the next in one write. That would seem to be an advantage. But only the wizards know for sure, and I probably don't qualify in this forum. Ken
moss@cs.umass.edu (Eliot Moss) (07/17/90)
The real problem with writing a few bytes here and there is that it probably leaves holes in the file -- i.e., the space is not really allocated. Copying from /dev/zero might give the same problem (blocks of all zeroes not being stored). The person needed actual disk blocks allocated so that writes could proceed very quickly and not need to go through any block allocation. -- J. Eliot B. Moss, Assistant Professor Department of Computer and Information Science Lederle Graduate Research Center University of Massachusetts Amherst, MA 01003 (413) 545-4206; Moss@cs.umass.edu
istvan@hhb.UUCP (Istvan Mohos) (07/18/90)
cpcahil@virtech.uucp (Conor P. Cahill) writes: :istvan@hhb.UUCP (Istvan Mohos) writes: :> if (write (fd, malloc(FOURMEG), FOURMEG) != FOURMEG) :> perror("malloc or write error"), exit (1); : :While I know this is just an example & is not intended to be a :suggestion of what to do, I just have to comment on it to ensure :that some impressionable engineer doesn't think this is a :good thing to do. The original question from markd@silogic.UUCP (Mark DiVecchio) produced no net responses for a week; just thought I'd stick a twig of controversy into the flames and watch the sparks fly :-) -- Istvan Mohos ...uunet!pyrdc!pyrnj!hhb!istvan RACAL-REDAC/HHB 1000 Wyckoff Ave. Mahwah NJ 07430 201-848-8000 ======================================================================
istvan@hhb.UUCP (Istvan Mohos) (07/18/90)
martin@mwtech.UUCP (Martin Weitzel) writes: >What is wrong with the following approach (at least on non-BSD-ish >file systems)? > > while file has not desired size > lseek(2) from current position forward > disk-block-size bytes minus 1 and write(2) > one byte > >IMHO this should fill the disk and avoids much copying from user-space. This, and the majority of the responses, focus on the merits of byte copying. The real issue is the lack of speed in physically accessing the disk. The program segments posted by others, or your algorithm converted to C (with impending corrections from Conor P. Cahill :-) #include <stdio.h> #include <sys/file.h> #define FOURMEG 4194304 #define DBLK 512 #define ITER (FOURMEG/DBLK) main() { long lseek(); char c = 0; register int fd, ri; if ((fd = open("foo", O_WRONLY | O_CREAT | O_TRUNC, 0644)) == -1) perror("can't write to"), exit (1); for (ri = ITER; --ri >= 0; write (fd, &c, 1)) if (lseek (fd, (long)DBLK-1, 1) == -1) perror("seek error"), exit (2); exit (0); } are all slower then writing 4 Meg in one disk access. -- Istvan Mohos ...uunet!pyrdc!pyrnj!hhb!istvan RACAL-REDAC/HHB 1000 Wyckoff Ave. Mahwah NJ 07430 201-848-8000 ======================================================================
I.G.Batten@fulcrum.bt.co.uk (Ian G Batten) (07/19/90)
Most articles on this subject seem to neglect the fact that many Unix file-systems do not allocate blocks of all zeros. Therefore lseeking ``a lot'' and writing one byte probably only allocates the block for the byte you wrote. ian
pd@ixi.uucp (Paul Davey) (07/23/90)
If you using SunOs, (and other systems supporting NFS booting) would
not mkfile(8) be suitable?
mkfile creates one or more files that are suitable for use
as NFS-mounted swap areas, or as local swap areas. The
sticky bit is set, and the file is padded with zeroes by
default. The default size is in bytes, but it can be
flagged as kilobytes, blocks, or megabytes, with the k, b,
or m suffixes, respectively.
OK so it's meant for NFS swap, but its esentially a (contiguous?)
totally allocated file. I don't see that you couldn't use it for any
other purpose.
--
Regards, pd@x.co.uk IXI Limited
Paul Davey pd@ixi.uucp 62-74 Burleigh St.
...!uunet!ixi!pd Cambridge U.K.
"These are interesting times" +44 223 462 131 CB1 1OJ eliot@chutney.rtp.dg.com (Topher Eliot) (07/30/90)
|> Alternetly how about mmap? You can mmap a half a meg at a time, that ought |> to reduce the number of syscalls, only write once every 8K (or whatever |> the disk block size is) reducing the memmory bandwidth, but will this result |> in fewer context switches then write? Then writev? Well, it would avoid explicit context switches, but would undoubtedly involve some non-trivial number of page faults. Can anyone out there educate us on how many page faults could be expected? Topher Eliot Data General Corporation eliot@dg-rtp.dg.com 62 T. W. Alexander Drive {backbone}!mcnc!rti!dg-rtp!eliot Research Triangle Park, NC 27709 (919) 248-6371 Obviously, I speak for myself, not for DG.