[comp.sys.apple] Problems with large files & large sub-directories

IRPGMR7@OUACCVMB.BITNET (05/01/89)

Found a little bug in BinSCII.  Just for fun the other day, I shrunk
(ShrinkIt) all the files in Moria GS and tried to BinSCII the file.
Lo and Behold.....Crash!!!!  Some weird error code from ProDOS.  I was
just wondering:  Has anybody else had a problem with BinSCII and rather
large files?? (upwards of 500K in size??)...

Also found a little bug in ShrinkIt.  It crashed as I was trying to
list the files in the Apple II GS technotes file from Apple2-L.  Any
others with the same problem??

One last general little thing:  What are the acceptable limits in number
of files stored in a directory??  (I've got every file from Apple2-L
stored in a single subdirectory in text file format, and many many programs
I get seem to crash when they try to read the directory and display it.  I
know, I should try store that many files in a subdirectory, but I'm working
on getting them converted to regular program files for a local Apple Users
Group.)

By the way, I find that BinSCII and ShrinkIt are wonderful programs (I'm
not trying to discredit their respective authors with these little bugs)
, although I wouldn't mind seeing a mouse based version of ShrinkIt 2.01.


David William Wrage  a.k.a.  The Frenchman

'I don't have a nifty saying yet, but I'll think of one.....'



Address:  IRPGMR7@OUACCVMB.BITNET

aragorn@blake.acs.washington.edu (Michael Owen) (05/01/89)

In article <8904301302.aa27400@SMOKE.BRL.MIL> IRPGMR7@OUACCVMB.BITNET writes:
>One last general little thing:  What are the acceptable limits in number
>of files stored in a directory??  (I've got every file from Apple2-L
>stored in a single subdirectory in text file format, and many many programs
>I get seem to crash when they try to read the directory and display it.  I
>know, I should try store that many files in a subdirectory, but I'm working
>on getting them converted to regular program files for a local Apple Users
>Group.)

I recently made some rather unscientific tests on how many files a
subdirectory can realistically hold.  In writing my own BBS, I've yet to
decide on a method of saving a user's e-mail.  One option I came up with was
to use a Unix-like method and create a subdirectory for every user in the
e-mail directory, and store a user's mail there.  I've got a maximum userlog
size of 500 users, so I created 500 subdirectories, named user.0001-user.0500.

Then I copied three files to directories 1, 50, 100, 250, 300, 400, 450, and
500, and read them into an array using ModemWorks' & FILES command.  The drive
I used was an Everex 20D SCSI, with a rated access time of 65 ms.  The times
to perform the operations were as follows:

Full pathname: /ea/broken.blade/email/user.nnnn

Filename	Time (approximate)
----------------------------------
user.0001	< 1 sec
user.0050	< 1 sec
user.0100	~ 1 sec
user.0250	~ 1 sec
user.0300	~ 2 sec
user.0400	~ 2 sec
user.0450	~ 3 sec
user.0500	~ 3 sec

The directory file itself (/ea/broken.blade/email) was 39 blocks big (500
files/13 files per directory block).  When I tried a catalog of the drive with
Copy II Plus v8.3, it only read in 191 files; using Cat Doctor, it simply
returned "This directory is too large."  Even when I pared down the directories
to just 50, Cat Doctor still complained "This directory is too large."; Copy
II Plus read them in just fine.

Of course, ProDOS puts no limits on the number of files a subdirectory can
hold, besides that of having enough disk space to store the directory file.
However, from the tests above, and my experiences with various utility
programs, I think there is a practical limit to just how big you should make
your subdirectories.  ProDOS allows 57 files (I think that's the number) to be
stored in the volume directory (assuming a 'standard' version of ProDOS; i.e.,
one that hasn't been modified to use additional volume directory blocks).
This should be enough for any reasonable user to organize his or her files in
an orderly manner.  But, I digress; I certainly don't want this message to
degenerate into a discussion on the right way to organize one's hard drive.

In short: use your directories wisely!

>David William Wrage  a.k.a.  The Frenchman

______________________________________________________________________________
       />    The Broken Blade           Aragorn III (Michael Owen)
      /< ________ ______________        aragorn@blake.acs.washington.edu
C=====[*>_______/|______________>       Starfleet HQ: (206) 783-5589
      \<                                3/12/24 8N1 24 hrs - A ModemWorks BBS
_______\>_____"Ai na vedui!"__________________________________________________

matthew@sunpix.UUCP ( Sun NCAA) (05/01/89)

In article <8904301302.aa27400@SMOKE.BRL.MIL>, IRPGMR7@OUACCVMB.BITNET writes:
| Found a little bug in BinSCII.  Just for fun the other day, I shrunk
| (ShrinkIt) all the files in Moria GS and tried to BinSCII the file.
| Lo and Behold.....Crash!!!!  Some weird error code from ProDOS.  I was
| just wondering:  Has anybody else had a problem with BinSCII and rather
| large files?? (upwards of 500K in size??)...

I've never tried files that large, but I've never had any problem with BinSCII.
What your posting didn't mention is:

1) What the ProDOS error code was.
2) What kind of system you have.
3) Your disk drive type and capacities.  (Assuming that you are trying to BinSCII
   500K in files, you must atleast had two 800K drives, or a hard disk.


| Also found a little bug in ShrinkIt.  It crashed as I was trying to
| list the files in the Apple II GS technotes file from Apple2-L.  Any
| others with the same problem??
| 
| One last general little thing:  What are the acceptable limits in number
| of files stored in a directory??  (I've got every file from Apple2-L
| stored in a single subdirectory in text file format, and many many programs
| I get seem to crash when they try to read the directory and display it.  I
| know, I should try store that many files in a subdirectory, but I'm working
| on getting them converted to regular program files for a local Apple Users
| Group.)

The size of ProDOS subdirectories are dynamically expandable. As you need more 
directory entries, ProDOS will ask for more disk space for a subdirectory.  The 
only limitation on subdirectory size, is available disk space.  What is an
acceptable limit, is a matter of personal taste.  It depends on how many file
entries you are willing to wade through to find the one you want.

| 
| By the way, I find that BinSCII and ShrinkIt are wonderful programs (I'm
| not trying to discredit their respective authors with these little bugs)
| , although I wouldn't mind seeing a mouse based version of ShrinkIt 2.01.
| 
| 
| David William Wrage  a.k.a.  The Frenchman
| 
| 'I don't have a nifty saying yet, but I'll think of one.....'
| 
| 
-- 
Matthew Lee Stier                         |
Sun Microsystems ---  RTP, NC  27709-3447 |        "Wisconsin   Escapee"
uucp: { sun, mcnc!rti }!sunpix!matthew    |
phone: (919) 469-8300 fax: (919) 460-8355 |

lwv@n8emr.UUCP (Larry W. Virden) (05/02/89)

Close - but if I am not mistaken, the 'root' directory has a limit of
50 directory entries in it.  Anyone know if there is perhaps a maximum
number of files which can appear in the directory bitmap (or whatever that
map of names to blocks is called)?

P.S.  I have enountered a few .bny/.bqy files which have intermingled
files and directories.  For instance,

file1
file2
file3
directory/
directory/file4
direcotry/file5
file6
file7
directory2/file8
directory2/file9
file10

But when I try to create these with blu 2.27, I dont seem to be able to.
Anyone have any ideas?  I dont really want to redownload one of the ecp8
bny's from pro-carolina just to get a problem file... sigh. 

The reason that I need it is that I have had problems in the past with
shrinkit 2.01 (and earlier) extracting these types of .bny files.  But
till I can create one, I wont be able to send Andy a sample.


-- 
Larry W. Virden	 674 Falls Place, Reynoldsburg, OH 43068 (614) 864-8817
75046,606 (CIS) ; LVirden (ALPE) ; osu-cis!n8emr!lwv (UUCP) 
osu-cis!n8emr!lwv@TUT.CIS.OHIO-STATE.EDU (INTERNET)
The world's not inherited from our parents, but borrowed from our children.

matthew@sunpix.UUCP ( Sun Visualization Products) (05/05/89)

In article <1055@n8emr.UUCP>, lwv@n8emr.UUCP (Larry W. Virden) writes:
| 
| Close - but if I am not mistaken, the 'root' directory has a limit of
| 50 directory entries in it.  Anyone know if there is perhaps a maximum

The correct count is 53 for a 'root' directory (((13 entries per block) * 
(4 blocks)) - (1 entry for the volume name))

| number of files which can appear in the directory bitmap (or whatever that
| map of names to blocks is called)?

There is only one volume bitmap for the entire 'volume (disk)'.  It will be
as large as necessary so that there is atleast 1 bit for every block on the 
volume.  Under ProDOS8 there is a maximum volume size of 32 meg (2^16 blocks),
and a maximum file size of 16 Meg (2^24 bytes).  The number of blocks required
to hold the volume bitmap is ((number_of_blocks_in_volume / 8) / 512).

| P.S.  I have enountered a few .bny/.bqy files which have intermingled
| files and directories.  For instance,
| 
| file1
| file2
| file3
| directory/
| directory/file4
| direcotry/file5
| file6
| file7
  directory2/
| directory2/file8
| directory2/file9
| file10
| 
| But when I try to create these with blu 2.27, I dont seem to be able to.
| Anyone have any ideas?  I dont really want to redownload one of the ecp8
| bny's from pro-carolina just to get a problem file... sigh. 

You missed an entry for 'directory2'.  The directories are in the list,
because the Binary II unpacking software (in your case: blu 2.27) needs to
know the names of the directories it needs to create as it unpacks the file.

I use BLU v2.28 (which only a minor bug fix above v2.27), and the way to 
create these types of Binary II files is simple.  In your example, just go 
to the directory that holds files:

file1
file2
file3
directory/
file6
file7
directory2/
file10

Mark all these files for packing (even the directories). BLU will then show you
the files in 'directory/'. Mark 'file4' and 'file5' for packing. It will then
show you 'directory2/'.  Mark 'file8' and 'file9' for packing.  Now start the
packing process, and BLU will take care of the rest.


| The reason that I need it is that I have had problems in the past with
| shrinkit 2.01 (and earlier) extracting these types of .bny files.  But
| till I can create one, I wont be able to send Andy a sample.

If your talking about the 'stops process after ~20 files' bug, he already
knows about it.  

| -- 
| Larry W. Virden	 674 Falls Place, Reynoldsburg, OH 43068 (614) 864-8817
| 75046,606 (CIS) ; LVirden (ALPE) ; osu-cis!n8emr!lwv (UUCP) 
| osu-cis!n8emr!lwv@TUT.CIS.OHIO-STATE.EDU (INTERNET)
| The world's not inherited from our parents, but borrowed from our children.


-- 
Matthew Lee Stier                         |
Sun Microsystems ---  RTP, NC  27709-3447 |        "Wisconsin   Escapee"
uucp: { sun, mcnc!rti }!sunpix!matthew    |
phone: (919) 469-8300 fax: (919) 460-8355 |

shawn@pnet51.cts.com (Shawn Stanley) (05/05/89)

matthew@sunpix.UUCP ( Sun Visualization Products) writes:
>The correct count is 53 for a 'root' directory (((13 entries per block) * 
>(4 blocks)) - (1 entry for the volume name))

Oops... 13 * 4 - 1 = 51, not 53.  You may have 51 files in a root directory of
4 blocks with 13 entries per block, after the volume name.

(Not nitpicking, just clarifying.  :-)

UUCP: {uunet!rosevax, amdahl!bungia, chinet, killer}!orbit!pnet51!shawn
INET: shawn@pnet51.cts.com