koster@cory.Berkeley.EDU (David Ashley) (09/10/88)
I was just going through an old issue of Amazing Computing, and I re-read an article called "Fun with the Amiga disk controller." It is a fine article, (it has a mistake: the minterm values are wrong) and it suggests a way of getting 18 percent more than the 880K on a normal 3.5 inch disk. It would mean storing data on the disk in a completely incompatable fashion. The existing handlers for DFn: would have to somehow be replaced with new handlers, which would do their own disk read/writes rather than going through trackdisk. Maybe they can be turned off with an action_inhibit. Anyway, 18 percent is a worthy increase, and at the same time, improvements could be incorporated. I have seen IBM PC's, with their measly 8088, do very fast disk directories. The directories must be stored in a fixed place, so that all the children can be accessed at once. The Amiga, on the other hand, stores file headers as linked lists so that to get a directory will require reads over the whole disk. Also better error checking could be used. The Amiga will throw away an entire track, 11 independent sectors, if any of the checksums in any of the sectors is bad. Don't let anyone fool you; this was a short-cut. The people that made the Amiga just didn't want to spend time trying to recover from disk errors, and they were in a hurry, so they ended up with a piece of shit method: too bad if you get an error..... What is FFS? Does it do this kind of thing? Store files in a different manner? Would it be profitable to work on this kind of project? Is 18 percent increase in storage, faster directories, more reliable storage, and incompatability with the existing system worth enough? I know I won't get any answers to this, but I felt like posting anyway. David Ashley koster@cory.berkeley.edu (Soon to be turned off)
cmcmanis%pepper@Sun.COM (Chuck McManis) (09/12/88)
[Mongo editing, please understand...] In article <5565@pasteur.Berkeley.EDU> (David Ashley) writes: > ... Amazing Computing, ... it suggests a way >of getting 18 percent more than the 880K on a normal 3.5 inch disk. (different encoding maybe ?) > It would mean storing data on the disk in a completely incompatable > fashion. The existing handlers for DFn: would have to somehow be > replaced with new handlers, which would do their own disk read/writes > rather than going through trackdisk. In the 1.3 Mount command (coming soon to a store near you) there is an option FileSystem=<handler> which lets you specify a handler when mounting the device. You can also "unlink" a device from the device list so that the system doesn't know about it. Thus you "unlink" DF1: and then Mount it with a new handler. Voila, your running with the new handler. > Maybe they can be turned off with an action_inhibit. Anyway, 18 percent > is a worthy increase, and at the same time, improvements could be > incorporated. I have seen IBM PC's, with their measly 8088, do very > fast disk directories. The directories must be stored in a fixed > place, so that all the children can be accessed at once. The Amiga, on > the other hand, stores file headers as linked lists so that to get a > directory will require reads over the whole disk. Dave, get a clue. Take that same "measly 8088" and start opening files in a Hard disk directory with 200 - 300 files in it. Yes, the hashed directory methods can be slower on wildcard searches, Yes they are almost always faster on looking up a named file, Yes the layout of the disk could be improved. It also allows for much larger disk sizes (I know about the 52Meg limit in 1.2 but that was a bug and not a designed in limit). Caching the directory does the same thing for AmigaDOS as it does for UNIX (ie makes it fast) then you get *both* fast directories and fast lookups. Most of the floppy cache programs will let you do this easily (FACC being an excellent example here) > Also better error checking could be used. The Amiga will throw away an > entire track, 11 independent sectors, if any of the checksums in any of > the sectors is bad. Don't let anyone fool you; this was a short-cut. > The people that made the Amiga just didn't want to spend time trying to > recover from disk errors, and they were in a hurry, so they ended up > with a piece of shit method: too bad if you get an error..... What are you talking about? A) Entire tracks are read which actually increases performance if you really think about it, and B) What sort of "recovery" is there from a screwed sector? Are you bitching about the disk corrupt requesters? There are ways to avoid those (like careful use of disks) And many useful programs that help recover your data. > What is FFS? Does it do this kind of thing? Store files in a different > manner? Would it be profitable to work on this kind of project? Is 18 > percent increase in storage, faster directories, more reliable storage, and > incompatability with the existing system worth enough? FFS is a new handler that changes the way files are layed out on the disk and to some extent the low level format of those files on the disk. It can improve the disk speed by more intelligent use of the sectors. Writing your own file system handler can be very instructive and if you want to learn about the tradeoffs made, by all means write one and try it out. Also one has to realize the conditions under which the system software was written and is now being maintained (eg many ideas/projects not enough folks). --Chuck McManis uucp: {anywhere}!sun!cmcmanis BIX: cmcmanis ARPAnet: cmcmanis@sun.com These opinions are my own and no one elses, but you knew that didn't you.
daveh@cbmvax.UUCP (Dave Haynie) (09/13/88)
in article <5565@pasteur.Berkeley.EDU>, koster@cory.Berkeley.EDU (David Ashley) says: > > Maybe they can be turned off with an action_inhibit. Anyway, 18 percent is a > worthy increase, and at the same time, improvements could be incorporated. I > have seen IBM PC's, with their measly 8088, do very fast disk directories. You must be using that "Dir" command built into the IBM's COMMAND.COM. A real Dir command, loaded from disk like an AmigaDOS Dir command, can be painfully slow on an IBM (specifically an 8088 based machine like the Bridge Card). Not only that, but you pay for the IBM's disk layout in other ways. For instance, try searching 12 different rather full directories in your search path on an IBM. You barely notice delay on an Amiga. Try large multiple-file copies. The Amiga's file system is inherently better than the IBM's at some tasks, the IBM's is no doubt better than the Amiga's at other tasks. > The Amiga, on the other hand, stores file headers as linked lists so that to > get a directory will require reads over the whole disk. [...[ > Also better error checking could be used. The Amiga will throw away an > entire track, 11 independent sectors, if any of the checksums in any > of the sectors is bad. You're confusing two different pieces of Amiga software here. Sure you're not using that IBM more that you should be? The directory format is a function of the File System, which is a program used directly by AmigaDOS. It determines the layout and allocation of data blocks on a disk, and may include a level of checking (Standard FileSystem runs a simple 32 bit checksum on each block, FFS doesn't). The problem you're seeing with entire tracks getting clobbered is a function of the trackdisk.device, the Amiga EXEC device driver that all programs, file systems included, will use to read, write, or format blocks of data. Similar programs are called by file systems to support hard disks or ram disks. The action a device driver takes is completely up to that driver. > Don't let anyone fool you; this was a short-cut. The people that made the > Amiga just didn't want to spend time trying to recover from disk errors... The trackdisk driver keeps two checksums on each 512 byte block, but they're not very good check mechanisms. They could have been better. The recovery could have been better too, but don't kid yourself; there's alot of code involved in reading the Amiga floppies. The trackdisk device must completely decode the RAW MFM format the Paula chip returns from the floppies. This format has been chosen to be easily decoded in a single blit operation, but it's still not trivial. And unlike IBM machines, there aren't really 11 independent sectors (though an IBM can only give you 9 here anyway); the track is always read and written as a single track. That's the only way it can be accessed. The 11 sectors you get are just a convention of the formatting software. There are more possibilities on a bad track than just a single bad "sector". For example, if the floppy drive died in mid-write, you might end up with two readable copies of 4 or 5 sectors, no valid versions of all the remaining sectors. If you need better recovery of disks gone bad, DiskSalv 1.2 will handle some bad track stuff like this, it's free, and it'll be out in a week or two. This level of recover really doesn't belong in the trackdisk.device. > What is FFS? A filing system that's part of Amiga OS V1.3. It's intended mainly for hard disk drives, and, in concert with a good DMA driven hard disk controller, is one of the fastest filing systems you'll find on any microcomputer. Does that mean I should delete all the stuff I wrote? > David Ashley > koster@cory.berkeley.edu (Soon to be turned off) -- Dave Haynie "The 32 Bit Guy" Commodore-Amiga "The Crew That Never Rests" {ihnp4|uunet|rutgers}!cbmvax!daveh PLINK: D-DAVE H BIX: hazy "I can't relax, 'cause I'm a Boinger!"
daveh@cbmvax.UUCP (Dave Haynie) (09/13/88)
in article <4705@cbmvax.UUCP>, daveh@cbmvax.UUCP (Dave Haynie) says: > The trackdisk driver keeps two checksums on each 512 byte block, but they're not > very good check mechanisms. Really, there's one checksum kept on the actual data area, and one on the private trackdisk space. -- Dave Haynie "The 32 Bit Guy" Commodore-Amiga "The Crew That Never Rests" {ihnp4|uunet|rutgers}!cbmvax!daveh PLINK: D-DAVE H BIX: hazy "I can't relax, 'cause I'm a Boinger!"
peter@sugar.uu.net (Peter da Silva) (09/13/88)
In article <67830@sun.uucp>, cmcmanis%pepper@Sun.COM (Chuck McManis) writes: > increases performance if you really think about it, and B) What sort > of "recovery" is there from a screwed sector? At least other sectors on the track should be readable. Ideally other sectors on the same track should be copied to an unallocated track. -- Peter da Silva `-_-' peter@sugar.uu.net Have you hugged U your wolf today?
peter@sugar.uu.net (Peter da Silva) (09/13/88)
In article <4705@cbmvax.UUCP>, daveh@cbmvax.UUCP (Dave Haynie) writes: > two. This level of recover really doesn't belong in the trackdisk.device. Weren't you just telling me that the device should present an error-free disk to the file system? That was the rationale behind not putting error correcting code there, remember. Where should this code be, then? The drive? Will the Amiga 3000 have SCSI floppies? -- Peter da Silva `-_-' peter@sugar.uu.net Have you hugged U your wolf today?
ericb@athertn.Atherton.COM (Eric Black) (09/14/88)
In article <2618@sugar.uu.net> peter@sugar.uu.net (Peter da Silva) writes: >In article <67830@sun.uucp>, cmcmanis%pepper@Sun.COM (Chuck McManis) writes: >> increases performance if you really think about it, and B) What sort >> of "recovery" is there from a screwed sector? > >At least other sectors on the track should be readable. > But as I understand it, the trackdisk.device crams 880K onto the floppy by not having inter-record gaps separating the (logical) sectors. In other words, there *are* no physical sectors. What's more, to reduce latency in writing to the disk, the same logical sector of data isn't always laid down in the same spot relative to the index mark (i.e., relative to the rotation of the disk). In case of a data error on the track, the trackdisk device could give you the entire track's worth of data with the warning that some or all of it is bogus. An appropriately smart scavenger program can go through all those bits and try to identify the start of logical sectors, but it's not easy. Often the best (sometimes the only) way to do this is to eyeball it and let the user's brain try to recognize data and go backwards to find what looks like a data block header. Not so hot for arbitrary binary data. Here's a useless but maybe helpful analogy (-: I have a byte-addressed memory, 8 bits per byte. The hardware is such that the bits in the byte are only accessed together as a single chunk. Now it happens that these 8 bits are actually stored in a circular shift register of, say, 17 bits. This shift register is constantly circulating, the bits are just spinning around and around and around ... The hardware stores 8 data bits into this byte by injecting the data bits at any arbitrary point, without having to wait for any special point to pass before it begins, along with a special bit pattern in the other bits which tells it where the 8 "real" data bits begin. The only way to tell which of the 17 bits are the real 8 data bits is by grabbing all 17 and looking for that special pattern it put into the 9 non-data bits (call it a "marker"). Now, if alpha particles (or magnets?) blow away a few of the bits in there, the hardware can't find its marker, and can't tell where the 8 data bits are. Now, that overhead of 9 bits for every 8 "real" bits is kind of steep. We can change that by changing the size of the chunk that the hardware handles at one time. For a given chunk size, there is a needed number of overhead bits. If we chunk in multiples of 8 "real" bits, and we know where the first set of 8 starts because of the marker, then we save that marker overhead for each of the remaining sets of 8 bits in the chunk -- each starts right after the one preceding it. What we have here is another example of a trade-off and an engineering decision which weighed many factors in choosing this point along that curve. We have more data on the disk than in formatting schemes which physically separate sectors. Many sector-encoding methods include the track/sector address in a header for each sector which is not visible to anybody but the controller. This takes space away from user data but gives the ability to recover undamaged sectors from a damaged track. I assume that the decision made at Amiga for the trackdisk was that the media was reliable enough that the additional space on each disk was worth the risk. They could have put extra ECC bits to correct errors, but that would take even more space. You might flame that an inappropriate choice of trade-off balance was made, but I haven't heard people saying that, only flaming. Now, if 1.3 allows easy use of different devices (drivers) on different floppy units (as I think Andy/Dave/Somebody said a few days ago), then some enterprising programmer can come up with an alternative for the trackdisk.device which does physically separate the sectors, does put track/sector info on each, maybe even puts extra ECC bits out there for recovery. Performance will be different -- and that's performance as measured in average throughput, burst throughput, sustained throughput, latency, recoverability, vulnerability, storage capacity, and so on. Lots of things to trade off. My personal opinion is that for a home machine, for a game machine, or even a casual use business machine, a reasonable design decision was made. Growing availability of hard disks, with a whole new bag of tradeoffs and design decisions, and overall higher reliability, will make much of this moot for non-casual Amiga users (by that I mean business users and serious home users, even serious game-only users). I have had floppies bite the dust on me, probably a total of 25-30 in the past 3 years of Amiga use, but I Cover My Ass and don't lose much if it happens. And I hate floppy flipping, so I appreciate the extra space on each disk that the current encoding method gives me. Even with four floppy drives on my A1000, and two with a hard disk on my A2000. It all boils down to redundancy. Inter-record gaps between sectors are redundant if you know where the sectors begin. Track/sector id's are redundant if you know where you are on the disk. Data block header id's are redundant if you know what file you are in. Or what directory you are in. Checksums and CRCs and ECC bits are redundant if you know what your bits REALLY are. It all costs you. You gotta decide how much you are willing to pay. And that decision is based on a whole lotta factors. -- Eric Black "Garbage in, Gospel out" Atherton Technology, 1333 Bordeaux Dr., Sunnyvale, CA, 94089 UUCP: {sun,decwrl,hpda,pyramid}!athertn!ericb Domainist: ericb@Atherton.COM
daveh@cbmvax.UUCP (Dave Haynie) (09/15/88)
in article <2619@sugar.uu.net>, peter@sugar.uu.net (Peter da Silva) says: > In article <4705@cbmvax.UUCP>, daveh@cbmvax.UUCP (Dave Haynie) writes: >> two. This level of recover really doesn't belong in the trackdisk.device. > Weren't you just telling me that the device should present an error-free > disk to the file system? That was the rationale behind not putting error > correcting code there, remember. Where should this code be, then? The drive? I like the idea of the device handling the error checking/correcting as much as possible. But it's not always possible in the context of a device driver. For example, you yank out a floppy in the midst of a write and reset your machine. Now plug the floppy back in. On that track that got blasted, there are probably at least two sectors that are totally trashed; chopped in half probably. There's a really good chance that there are also some sectors that aren't there at all, and some that are there in duplicate. Remember, trackdisk as used isn't synced at all. So how does trackdisk give me back those sectors that are still readable? As far as it knows, both versions of sector 5 are equally valid. Without knowledge of the filesystem you happen to be using, how is the device to choose between the two. Similarly, the File System doesn't know about the low level aspects of each individual device driver, so it's not qualified to solve the problem. I can think of two appropriate solutions. First, the device driver could have the capability to pass up several versions of a particular sector if it finds multiple sectors, and let the file system try to determine which is the right one. The other solution is to tell the user he's got a bad disk, and let him go and get a utility specifically designed to recover bad disks. That's what we do now. I think's a decent way to solve the problem at hand. > Peter da Silva `-_-' peter@sugar.uu.net > Have you hugged U your wolf today? -- Dave Haynie "The 32 Bit Guy" Commodore-Amiga "The Crew That Never Rests" {ihnp4|uunet|rutgers}!cbmvax!daveh PLINK: D-DAVE H BIX: hazy "I can't relax, 'cause I'm a Boinger!"
peter@sugar.uu.net (Peter da Silva) (09/15/88)
In article <4733@cbmvax.UUCP>, daveh@cbmvax.UUCP (Dave Haynie) writes: > in article <2619@sugar.uu.net>, peter@sugar.uu.net (Peter da Silva) says: > > In article <4705@cbmvax.UUCP>, daveh@cbmvax.UUCP (Dave Haynie) writes: > >> two. This level of recover really doesn't belong in the trackdisk.device. > > Weren't you just telling me that the device should present an error-free > > disk to the file system? That was the rationale behind not putting error > > correcting code there, remember. Where should this code be, then? The drive? > I like the idea of the device handling the error checking/correcting as much > as possible. But it's not always possible in the context of a device driver. [ example of total catastrophic damage to a track ] That's why I have been arguing for bad-block handling, at least to some degree, in the file system. However, there are less catastrophic failure modes that can be dealt with... for example, a write to a bad spot on the disk. On the verification pass of the write, say, you get an error condition. What you have to do here is write the track into a spare track and set some pointer that indicates this track has moved. Of course, the file system can do this better, since it's already got a bunch of spare blocks floating around to put this stuff in... the device driver would have to have allocated bad block tracks (as I understand SCSI disks do). If the disk is full, of course, there's nothing you can do about it... but at least some of the cases can be handled. Anyway, after you do this you should still put up a requestor so the poor dude can copy his data to a safer medium... > Similarly, the File System doesn't know about the > low level aspects of each individual device driver, so it's not qualified > to solve the problem. True, but if the original data is still around the FS can blast all the sectors back in new spots and update its pointers. One thing the td.device can do is ask the FS for a safe place to put the data, and tell it where the data was stuffed, but this is a bit extreme... > The other solution is to tell the user he's got a bad disk, and let him > go and get a utility specifically designed to recover bad disks. That's what > we do now. I think's a decent way to solve the problem at hand. Except for the occasions where you just guru and let the guy figure out his disk is bad because it crashes whenever he puts it in... -- Peter da Silva `-_-' peter@sugar.uu.net Have you hugged U your wolf today?