[comp.sys.amiga.hardware] Obsessive Multitasking and Amiga File I/O Scheduling

xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) (09/08/90)

jesup@cbmvax (Randell Jesup) writes:
> p554mve@mpirbn.UUCP (Michael van Elst) writes:
[...]
>>On the other side, the Amiga (the filesystem) is very slow when dealing
>>with concurrent tasks. The filesystem deals with requests in
>>first-come-first-served order and concurrent accesses lead to heavy
>>thrashing of the disk. This is because there is no buffer cache, no
>>read-ahead and no "elevator"-algorithm. But it is not necessary most
>>of the time since the Amiga is a single user system and only few users
>>will put stress on the disk by loading several programs or files at once.
[...]
>	Elevator only helps in the case of 3 or more tasks making
>accesses at the same time.  Read-ahead does help in multiple-access 
>situations, but more and more SCSI drives have built-in read-ahead.  Your
>points on single-user are on-target.  Remember, though, that a lot of
>Unix machines are effectively single-user and rarely have multiple-accesses
>occurring (though more often than AmigaDos).

I didn't think my use patterns were that unusual, but maybe so.

Yesterday, in a fairly typical setup, I had half a dozen shells open on
the (A2000,1.3) workbench, doing various combinations of zoo -add,
zoo -extract, lharc a, lharc x, interactive dir-s and info-s and list-s
to make sure I wasn't clobbering Rad:, and on another screen, vt100
running to do downloads.

Slow as molasses, on a 68000, but productive as heck, since I could read
news and mail between downloads, and just flip back to the workbench
every once in a while to get a new process going in the next window.

I sorted through ten megabytes of email and refiled it on floppies this
way, while catching up on yet more correspondence.

I bought my first Amiga, and this latest one, to *multitask*, and I spend
lots of days like the above.  Of course, poverty and indolence keep me
running out of Rad:, but I'm really upset to find out that a machine
whose internal OS is multitasking all the way has such uneven/unbalanced
support for it overall that I'd be really unhappy trying to do the same
thing from a hard disk.

Is there a better disk I/O scheduling algorithm in 2.0, or at least one
in work?  As Randall notes, the solutions are well known, and should be
"in there" by now.  It's been over five years, after all.

Along with the deadly repeated "key error, backup, reformat and reload"
problems reported here so often, this is another market killer for the
business marketplace (which surely won't tolerate the unreliable hard
disk problem like a crowd of hackers/students ready to do the "repairs"
themselves will), and both need fixes Really Soon.

Buying a machine that advertises multitasking, and then hearing your
disk drive attempt to self destruct when you try to do so is almost
as unimpressive as having your _whole_ file system go gaga and need a
long, frustrating recovery to be done, every time a program dies in
mid-write.  Fsck is overdue for AmigaDOS.

So tell me, am I the only one multitasking this sucker within an inch of
its life as a regular way of doing work?  Or is this the general problem
I perceive it to be?

Kent, the man from xanth.
<xanthian@Zorch.SF-Bay.ORG> <xanthian@well.sf.ca.us>

mrush@csuchico.edu (Matt "C P." Rush) (09/09/90)

In article <1990Sep8.050725.14384@zorch.SF-Bay.ORG> xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) writes:
>jesup@cbmvax (Randell Jesup) writes:
>
>I didn't think my use patterns were that unusual, but maybe so.
>
>Yesterday, in a fairly typical setup, I had half a dozen shells open on
>the (A2000,1.3) workbench, doing various combinations of zoo -add,
>zoo -extract, lharc a, lharc x, interactive dir-s and info-s and list-s
>to make sure I wasn't clobbering Rad:, and on another screen, vt100
>running to do downloads.
>
>Slow as molasses, on a 68000, but productive as heck, since I could read
>news and mail between downloads, and just flip back to the workbench
>every once in a while to get a new process going in the next window.

	Sounds like the sort of thing I do, though I'm running a 68010.

>I bought my first Amiga, and this latest one, to *multitask*, and I spend
>lots of days like the above.  Of course, poverty and indolence keep me
>running out of Rad:, but I'm really upset to find out that a machine
>whose internal OS is multitasking all the way has such uneven/unbalanced
>support for it overall that I'd be really unhappy trying to do the same
>thing from a hard disk.

	Multitasking was EXACTLY the reason I bought my first Amiga, and I
haven't had that much trouble thrashing disks.  Sure, if you have multiple
tasks that are accessing different cylinders on the same floppy there's going
to be some head movement.  But that's why Amiga floppies have that extra little
circuit board that PC floppies don't have -- so the drives can be individually
controlled!
	If you don't wanna bang your heads around, get a second (or third, or
fourth) floppy.  Or as you're doing, use a ram disk.

	As far as this sort of thing goes on a Hard Disk, it's just fine.  I
do all my multitasking from my ST157-N running on a Pacific Peripherals DMA
'OverDrive' controller with no problems.
	On the occasions where I know I'm going to be doing a lot of disk
thrashing I use the ol' AddBuffers command.  That's what it's there for!

>Is there a better disk I/O scheduling algorithm in 2.0, or at least one
>in work?  As Randall notes, the solutions are well known, and should be
>"in there" by now.  It's been over five years, after all.
>
>Along with the deadly repeated "key error, backup, reformat and reload"
>problems reported here so often, this is another market killer for the
>business marketplace (which surely won't tolerate the unreliable hard
>disk problem like a crowd of hackers/students ready to do the "repairs"
>themselves will), and both need fixes Really Soon.
>
>Buying a machine that advertises multitasking, and then hearing your
>disk drive attempt to self destruct when you try to do so is almost
>as unimpressive as having your _whole_ file system go gaga and need a
>long, frustrating recovery to be done, every time a program dies in
>mid-write.  Fsck is overdue for AmigaDOS.

	I don't think CBM can be held responsible for Hard Disks that get
trashed because of programs failing during Writing.  This is more the
responsibility of the companies that manufacture the Hard Drive controllers
and associated drivers.

	I used to trash (error validating, key not found) my old Xebec
9720H on my A1000 almost everytime I did an icon snapshot.  This wasn't
any fault with the OS, it was that Xebec didn't know how to program for
a multitasking environment and used a lot of busy waits for timing.
	My current hard drive on my A2000 has never gotten trashed in over
three years -- even if I try!

>So tell me, am I the only one multitasking this sucker within an inch of
>its life as a regular way of doing work?  Or is this the general problem
>I perceive it to be?

	No, on both accounts.  It is a problem of PC developers using PC
techniques on the Amiga, in total disregard of everything that CBM has
warned against since the 1.0 RKM's.

	Of course my Operating Systems Programming instructor would say that
this is not true multitasking because the Amiga doesn't have an MMU.  But
that instructor was a dweeb.

	-- Matt

    *~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*
    %    "Progress is an up-hill battle       %  mrush@csuchico.edu      %
    %    against backwards compatibility."    %  mrush@cscihp.UUCP       %
    %              -- me                                                 %
    %                              Now:  mrush@cscihp.ecst.csuchico.edu  %
    *~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~*
     This is a SCHOOL!  Do you think they even CARE about MY opinions?!

david@twg.com (David S. Herron) (09/10/90)

In article <1990Sep08.210559.19597@ecst.csuchico.edu> mrush@cscihp.UUCP writes:
>In article <1990Sep8.050725.14384@zorch.SF-Bay.ORG> xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) writes:
...
>>I didn't think my use patterns were that unusual, but maybe so.
>>
>>Yesterday, in a fairly typical setup, I had half a dozen shells open on
>>the (A2000,1.3) workbench, doing various combinations of zoo -add,
>>zoo -extract, lharc a, lharc x, interactive dir-s and info-s and list-s
>>to make sure I wasn't clobbering Rad:, and on another screen, vt100
>>running to do downloads.
>>
>>Slow as molasses, on a 68000, but productive as heck, since I could read
...
>	Multitasking was EXACTLY the reason I bought my first Amiga, and I
>haven't had that much trouble thrashing disks.  Sure, if you have multiple
>tasks that are accessing different cylinders on the same floppy there's going
>to be some head movement.  

Disk heads thrashing around the disk as different processes tell it to
go different places is one cause.  Picture in your mind .. remember that
Star Trek episode where you had a bunch of psychics who were all powerful
on this planet and the shorty who was run around this way and that a
complete non-psychic.  Remember how he'd run this way and that back and
forth as different of the bastards got ahold to him.. that's approximately
what's happening with your floppies, Kent.

The other part of what's going on is that you're spending a *LOT* of
time in the process scheduler ... *any* multi-tasking system will, if
you give it too many processes to run, become scheduler-bound.  Don't 
believe me?  Check out OS theory books..

Finally -- there's only so many CPU cycles available in any one machine.
And besides the number of cycles you have to take into account the quality
of the cycles -- how much work can be accomplished in one clock cycle?


> But that's why Amiga floppies have that extra little
>circuit board that PC floppies don't have -- so the drives can be individually
>controlled!

Eh?  Are you sure?  The only extra PC board I knew of was a simple
ID circuit which would respond with some code indicating what kind
of drive it is.

My impression of the hardware interface is that it doesn't prevent
access to multiple floppies at the same time.  It's just that most of
the machines which have multiple floppies (the PC class of machines)
also don't run multi-tasking OS's.  Which makes it harder to create
a condition where you're *trying* to access multiple drives at once.

>	If you don't wanna bang your heads around, get a second (or third, or
>fourth) floppy.  Or as you're doing, use a ram disk.

That's a good idea too..  One of the classic ways of speeding up
Unix systems is to get another drive and split your busy disk
partitions evenly among the two drives.  That way you get some
parallelism working for you in having multiple disk arms moving at
the same time.

...
>>Along with the deadly repeated "key error, backup, reformat and reload"
>>problems reported here so often, this is another market killer for the
>>business marketplace (which surely won't tolerate the unreliable hard
>>disk problem like a crowd of hackers/students ready to do the "repairs"
>>themselves will), and both need fixes Really Soon.
...
>	I don't think CBM can be held responsible for Hard Disks that get
>trashed because of programs failing during Writing.  This is more the
>responsibility of the companies that manufacture the Hard Drive controllers
>and associated drivers.

Regardless of whether or not application programs are poorly written
the file system should be more resistant to being trashed.  Yes,
in theory, there isn't a lot that can be done if a process goes
insane since there isn't any protection in the machine.  My impression
is that the file system is one of the WEAKEST links in this system.
But then I had a long episode last year where my disk got trashed
for some odd reason, so I repaired it and it got trashed again
shortly afterward, and I repaired it again, and it trashed again and..

Like Kent, I would be very happy if an `fsck' were available.

What `fsck' means is:  Fixes the disk in place.  Finds files and
directories which are `inconsistent' and does a best attempt at
recovering the file and putting it *somewhere* in the file system.
If it can't find its original place(es) then put it into a standard
place (`lost+found') with some rationally constructed name.

Dave's program (DiskSalv) is pretty good but it *DOES* *NOT* do it
"in place" meaning I gotta have another disk to recover files to.  But
I only have one hard disk in my system sooo..

It helps a lot if the file system is resilient.  For instance if
there is some critical information somewhere you can try repeating
it at regular spots on the disk.  The Berkeley Fast File System
does this, it repeats the "superblock" at the head of each cylinder
group (cyl-group == 8 or 16 cylinders).  

When viewed from the correct level of abstraction a file system,
or any other part of a computer system, is just a data structure.
Properly constructing the data structure for loss of information
can help a lot.  The example which I know is the Unix file system
-- the `data structure' is a normal everyday bi-directionally
connected tree in which the leaf nodes (files) can be connected to
many interior nodes (directories).  If you lose one of the directories
you simply lose direct reference to the nodes below that node.
Those nodes still reference themselves correctly.


And -- I'm not saying that CBM must do it the Unix way.  The Unix 
way isn't the one-true-and-only way that a lot of people think it is.
There's a lot of problems with Unix.  There's also a *heck* of a lot
of *GOOD* ideas there..
-- 
<- David Herron, an MMDF & WIN/MHS guy, <david@twg.com>
<- Formerly: David Herron -- NonResident E-Mail Hack <david@ms.uky.edu>
<-
<- Sign me up for one "I survived Jaka's Story" T-shirt!

jms@tardis.Tymnet.COM (Joe Smith) (09/20/90)

In article <1990Sep8.050725.14384@zorch.SF-Bay.ORG> xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) writes:
>Along with the deadly repeated "key error, backup, reformat and reload"
>problems reported here so often, this is another market killer ...

They say that this particular problem has been fixed in AmigaDos-2.0.
The data written to the disk is the same as if the file had been completely
closed and reopened after each write.
-- 
Joe Smith (408)922-6220 | SMTP: jms@tardis.tymnet.com or jms@gemini.tymnet.com
BT Tymnet Tech Services | UUCP: ...!{ames,pyramid}!oliveb!tymix!tardis!jms
PO Box 49019, MS-C41    | BIX: smithjoe | 12 PDP-10s still running! "POPJ P,"
San Jose, CA 95161-9019 | humorous dislaimer: "My Amiga speaks for me."

xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) (09/24/90)

jms@tardis.Tymnet.COM (Joe Smith) writes:
> xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) writes:
>> Along with the deadly repeated "key error, backup, reformat and reload"
>> problems reported here so often, this is another market killer ...
>
> They say that this particular problem has been fixed in AmigaDos-2.0.
> The data written to the disk is the same as if the file had been completely
> closed and reopened after each write.

That sounds _so_ nice!  Can anyone confirm:

1) that this change has been made; and
2) that it actually cures the problems that have caused most of the
   hard disk "key error" problems (like over 99% of them?)

Kent, the man from xanth.
<xanthian@Zorch.SF-Bay.ORG> <xanthian@well.sf.ca.us>

xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) (09/28/90)

xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) writes:
>jms@tardis.Tymnet.COM (Joe Smith) writes:
>> xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) writes:
>>> Along with the deadly repeated "key error, backup, reformat and reload"
>>> problems reported here so often, this is another market killer ...
>>
>> They say that this particular problem has been fixed in AmigaDos-2.0.
>> The data written to the disk is the same as if the file had been completely
>> closed and reopened after each write.
>
>That sounds _so_ nice!  Can anyone confirm:
>
>1) that this change has been made; and
>2) that it actually cures the problems that have caused most of the
>   hard disk "key error" problems (like over 99% of them?)

I've gotten two email confirmations of this.  Well done, C-A!

Kent, the man from xanth.
<xanthian@Zorch.SF-Bay.ORG> <xanthian@well.sf.ca.us>

rwm@atronx.UUCP (Russell McOrmond) (09/29/90)

KPD>> xanthian@zorch.SF-Bay.ORG (Kent Paul Dolan) writes:
KPD>>> Along with the deadly repeated "key error, backup, reformat and reload"
KPD>>> problems reported here so often, this is another market killer ...

KPD>jms@tardis.Tymnet.COM (Joe Smith) writes:
KPD>> They say that this particular problem has been fixed in AmigaDos-2.0.

KPD>That sounds _so_ nice!  Can anyone confirm:

I myself HAVE had this problem using the new 2.0 NFFS (New Fast File system? ;-)

But, it's usually myself doing some really NASTY crashing (I'm a developer -
Not neccesary a clean one, and I have wild pointers all too often..)


---
  Opinions expressed in this message are my Own.  My Employer does not even
know what these networks ARE.

  Russell McOrmond   rwm@atronx.UUCP   {fts1,alzabo}!atronx!rwm 
  FidoNet 1:163/109  Net Support: (613) 230-2282
  Amiga-Fidonet Support  1:1/109