dascher@stanley.cis.brown.edu (David Ascher) (06/21/91)
Hi all. I'm about to start a very large newsfeed and use a good chunk of an rz57 as storage medium... As usual, the problem that comes up is blocksize, since small blocksizes allow for an efficient storage of thousands of very small articles. Problem is, it isn't "legal" to format an rz57 with a blocksize of less than 4k... Which is a real problem. Some people at DEC have told me that the only way around this is, and I quote: "The solution to this problem is to lie about the geometry of the disk and tell it that there are more, but smaller, cylinders". This is where I go "Uh....". Have any of you ever done this? Any experts out there who can either tell me if my fears are unfounded, or where problems may crop up? Thanks for any info. -david ascher Brown University Computing and Information Services David_Ascher@Brown.EDU
alan@shodha.enet.dec.com ( Alan's Home for Wayward Notes File.) (06/21/91)
But I'm not sure how it got here. In article <DASCHER.91Jun20154819@stanley.cis.brown.edu>, dascher@stanley.cis.brown.edu (David Ascher) writes: > Hi all. > > I'm about to start a very large newsfeed and use a good chunk of an > rz57 as storage medium... As usual, the problem that comes up is > blocksize, since small blocksizes allow for an efficient storage of > thousands of very small articles. Problem is, it isn't "legal" to > format an rz57 with a blocksize of less than 4k... Which is a real > problem. When the question was asked originally (internally) I wondered at the specific problem. The first, and one I think the one you percieve is that it would nice to have block size of 2 KB. Sorry. 4 KB and 8 KB are the only ones supported. The other problem, the answer to which you quote, is that on some disks many small files means you'll run out of inodes. There is a newfs (mkfs) parameter that lets you control the number of inodes created. The default value is 2048 bytes of disk/inode. I've had exceptional success making it larger and getting fewer inodes, but not as much getting more. That was a few versions ago and it may well have been broken. The work-around to this problem is: > > Some people at DEC have told me that the only way around this is, and > I quote: > > "The solution to this problem is to lie about the geometry of the disk > and tell it that there are more, but smaller, cylinders". More cylinders mean more groups of inodes and therefore more inodes. If you have a habit of running out of inodes it's a reasonable solution. The real solution is the invention of the "Fat" file system. > > Have any of you ever done this? Any experts out there who can either > tell me if my fears are unfounded, or where problems may crop up? Interesting question. Are you're fears indeed unfounded? I looked my news disk (currently an RA82) and discovered 161,581 files. The average filesize is 2,978 bytes. The distribution goes something like: Number <= KB 31 0 (.04%) 38613 1 (24%) 74794 2 (46%) 25050 3 (16%) 9537 4 (6%) 4766 5 (and so on...) 2551 6 1210 7 798 8 2095 16 499 24 305 32 221 40 309 48 304 56 426 64 71 > 64 KB (largest 1,204,224) So, a lot of the files will be less than 2 KB. But a lot will be larger. Now consider the fragment allocation scheme. In one of the ULTRIX V4 releases we added (or at least documented) a flag to mkfs(8), to set the file system optimization flag. The choices are to optimize for space or time. In the case of space, the file system prefers to allocate a file fragment with existing fragments, rather than allocate a new fragment. So, two 3 KB files will only use one 8 block on disk with 2 KB leftover. In the time case it allocates a new fragment. This way if the file expands, it doesn't have to be copied to a new fragment with enough space. For news, I think the choice would be to optimize for space, which is also the default. So, if you could choose a file system block size of 2 KB you constrain the file system to doing 2 KB I/O for any file, big or small. Hundreds of small files will still packed as tightly together as possible. If you use a larger block size, then reading back the larger files get the benefit of the larger block size. I/O for the small files should only a block size that is "enough". I think your fears are unfounded. If you can get enough inodes. > > Thanks for any info. > > -david ascher > Brown University Computing and Information Services > David_Ascher@Brown.EDU -- Alan Rollow alan@nabeth.cxn.dec.com