U3364521@ucsvc.ucs.unimelb.edu.au (Lou Cavallo) (10/07/90)
G'day,
{Please forgive the length of this article. I hope you find it worthwhile.}
The results of this discussion thread have been as great as I was hoping they
would be. Thanks everyone. Hereafter follows a summary and a new idea I think.
Summary:
The consensus (if there is one) seems to be that a swapping library should be
put together that allows new applications to use routines to implement memory
paging for data files. {Dave Haynie's do the work once principle.}
{I support this idea also. Software may need a set of tools rather than a one
size fits all s/w VM solution...analogously to the different alloc routines?}
Mike Meyer and Matt Dillon have considered the case of an editor and the type
of criteria suitable for VM for file accessing. {Others have suggested their
own ideas here too but I haven't kept track of it all, sorry.}
Some have mentioned Virtual for Mac's (Dave Haynie pointed to a reference for
I believe a way for Mac OS to allocate VM for applications. Si Dave?) and the
VM techniques used on MS-DOS systems. {Virtual is non MacOS VM s/w for Macs.}
A swapping library technique could not offer memory paging for programs that
were not compiled to use it. {Several suggestors here.}
And my original suggestion. Can we find a way to allow (in the interim period
until OS support for VM arrives) applications access files larger than alloc-
atable memory? {Others are suggesting/supporting this limitation also and I'm
again sorry I haven't tracked this as well.}
{There were other suggestions e.g. using more physical memory but I think the
above covers most of them. I offer my apologies if I left your idea out.}
My idea:
A virtual memory fixed disk.
This is not like the VD0: (by ASDG no?) which is a RAM disk that grows as any
data is put in it but only up to the available amount of RAM.
A virtual memory fixed disk (VFD:) would be a file structured RAM disk device
like RAD: that would set up on it a set of memory swapping buffers for files.
The VFD could be mounted to (say) a 100 Mbyte virtual allocation but only use
the user selected amount of RAM for the in memory buffers.
Naturally real disk storage media sufficient to store the whole of a virtual
file would be required. A 100 Meg virtual file is not going to fit onto your
floppy eh? :-)
Generalisations to recoverable and / or dynamic virtual disk devices are left
as exercises for the reader. :-)
{Note that I'm addressing the last of the summary points here. I'll leave the
question of s/w VM to others but I will note there has already been a (doozy)
of discussion on it in this newsgroup.}
This suggestion is specifically to deal with files larger than (allocatable)
memory. It is not a way to give VM for programs for that need allocations for
RAM memory objects larger than that available (I think).
Discussion:
Pros:
All programs and not just those linked with a swapping.library or vm.library
could access files larger than allocatable RAM.
If a user's or programmer's problem requiring VM can be solved using a file
then VFD can provide a crude VM facility.
No changes to the OS should be necessary for this to be implemented assuming
a swapping.library set of routines that a VFD: device handler could be built
with.
This idea (I feel IMHO) solves the problem in a style suiting the Amiga. The
device handler could possibly be written for use with any type of filesystem
or trackdisk level interface. {I defer here, I don't know Amiga internals.}
Cons:
I don't know how to do it. :-) {Discussions as to the actual structure of a
VFD are sought here.}
Would the overhead of maintaining the virtual files in VFD be too much for a
good/acceptable level of performance?
Files put onto a VFD would be in a virtual memory file system but should act
as normal files do. Would there be any problems with features of the Amiga's
file system (say links, or deleting files that are not unlocked) with respe-
ct to the fact that its really implemented in RAM? (I don't think so but...)
VFD: files could be corrupted by errant tasks/processes in memory via trash-
ing of the swapping buffers by errant memory writes.
Large excutables (larger than available RAM) put on a VFD device should still
fail to load into memory as the OS will want to load all of an executable. {I
simplify here, program overlays could be used but assume the overlays are all
too large also.} Actually I'm not sure I'm right here. Perhaps it would work?
yours truly,
Lou Cavallo.
PS: If I've made any obvious points of expository inconsistency it's probably
because I tried to keep each point to a brief minimum.
So what do you think ? Is this a good level to deal with the solution to the
problem or is it all wet and would s/w VM be a simpler/better/more_useful way
than dealing with the one problem of files that are too large?U3364521@ucsvc.ucs.unimelb.edu.au (Lou Cavallo) (10/07/90)
G'day,
LC> In article <1124@ucsvc.ucs.unimelb.edu.au>,
LC> U3364521@ucsvc.ucs.unimelb.edu.au (Lou Cavallo) writes:
LC> {Please forgive the length of this article. I hope you find it worthwhile.}
[...deleted the gory details...]
LC> A virtual memory fixed disk.
[...deleted the gory details...]
LC> So what do you think ? Is this a good level to deal with the solution to
LC> the problem or is it all wet and would s/w VM be a simpler/better/more_
LC> useful way than dealing with the one problem of files that are too large?
Yes I know. It's all wet. :-(
A program that cannot do VM allocation for a file (even virtual!) larger than
memory is not going to be able to deal with a file from the virtual disk that
is also too large.
Hands up all of you reading this that have posted.. err..um "incorrect ideas"
at 3am in the morning to a newsgroup before.
I suppose a virtual disk is a sort of useful thingie but to me at this moment
it looks like an unnecessarily complicated addbuffers system... :-(
yours truly,
Lou Cavallo.jeh@sisd.kodak.com (Ed Hanway) (10/09/90)
In article <1124@ucsvc.ucs.unimelb.edu.au> U3364521@ucsvc.ucs.unimelb.edu.au (Lou Cavallo) writes: >My idea: > >A virtual memory fixed disk (VFD:) would be a file structured RAM disk device >like RAD: that would set up on it a set of memory swapping buffers for files. >The VFD could be mounted to (say) a 100 Mbyte virtual allocation but only use >the user selected amount of RAM for the in memory buffers. You can do this today. Just mount a hard disk partition and use AddBuffers (or equivalent) to select the size of the buffers. No special handler required. This idea's not completely dead, though. There are a few things that could be done to optimize a partition for temporary file use. First, vary the size of the buffers used based on current usage, free RAM, etc. Second, don't write changes through to the disk until pages are flushed from RAM. With these features, either as options to the FastFileSystem or as a separate file system altogether, you could reserve 10-20M of disk space for a fast T: partition optimized for temporary storage. Comments? Suggestions? Volunteers? Ed Hanway uunet!sisd!jeh
new@ee.udel.edu (Darren New) (10/09/90)
In article <1124@ucsvc.ucs.unimelb.edu.au> U3364521@ucsvc.ucs.unimelb.edu.au (Lou Cavallo) writes: >My idea: >A virtual memory fixed disk. Survey says: BUZZZ!!! If your application can use files on a disk, all you need to do is up the number of buffers enough (via FACC, say, or whatever) and this will work. Unfortunately, most applications don't do this because the Unix-style filesystem is painful to use in increments. That is, it is not easily possible to write code to get lines 15, 23, and 75 from a file. (It's not even especially easy to get just one line reliably.) Therefore, for speed, most editors read the entire file into memory that has been allocated (at some level) with AllocMem. Once loaded, the disk is never accessed again. Maybe building a virtual memory fixed disk on top of the swapping library would be worthwhile. But adding the kind of virtual disk you are talking about is not going to give "files bigger than memory" capability to any program that cannot already handle files bigger than memory. -- Darren -- --- Darren New --- Grad Student --- CIS --- Univ. of Delaware --- ----- Network Protocols, Graphics, Programming Languages, Formal Description Techniques (esp. Estelle), Coffee -----
manes@vger.nsu.edu (10/10/90)
One thing has occurred to me about virtual memory for the amiga. Considering the fact that the Amiga operating system does not handle file corruption very well, what would happen if during a swap a power fail occurred? Or the machine was turned off? It appears to me that the swap device MUST use RAM, and if RAM is used would the requirement to have RAM to do Virtual Memory be a bit self defeating? I guess what I am asking is -- By using a swap device to a hard disk, wont we aggrevate the the problems with corrupted files? Doesn't this also bring up the question about a 'shutdown' command again? -mark= manes@vger.nsu.edu
dillon@overload.Berkeley.CA.US (Matthew Dillon) (10/12/90)
>In article <42.2712f4e3@vger.nsu.edu> manes@vger.nsu.edu writes: > >One thing has occurred to me about virtual memory for the amiga. > >Considering the fact that the Amiga operating system does not handle >file corruption very well, what would happen if during a swap a power >fail occurred? Or the machine was turned off? > >It appears to me that the swap device MUST use RAM, and if RAM is used >would the requirement to have RAM to do Virtual Memory be a bit self >defeating? > >I guess what I am asking is -- By using a swap device to a hard disk, >wont we aggrevate the the problems with corrupted files? > >Doesn't this also bring up the question about a 'shutdown' command >again? > >-mark= >manes@vger.nsu.edu It is relatively easy to use a PARTITION on your hard drive for your SWAP space. This is how UNIX does it. You format it yourself and clear the 'swap' space on boot. No files to get corrupted, and since the partition information is never modified the partition itself will never be corrupted, only data inside it which isn't a problem because the data gets cleared when you start up the VM system. In anycase, this is one of the arguments against using a FILE to hold your VM space. Using a partition is less likely to cause problems if you reboot unexpectedly. I've discovered with FMS: (my floppy disk simulator that uses a hard disk file to 'hold' the floppy) that even with a large number of buffers the amiga FFS is not as efficient as one would like Seek()ing in a 1MB file, which leads me to believe that seeking in a 20MB file would be even less efficient. -Matt -- Matthew Dillon dillon@Overload.Berkeley.CA.US 891 Regal Rd. uunet.uu.net!overload!dillon Berkeley, Ca. 94708 USA
jms@tardis.Tymnet.COM (Joe Smith) (10/13/90)
In article <42.2712f4e3@vger.nsu.edu> manes@vger.nsu.edu writes: >I guess what I am asking is -- By using a swap device to a hard disk, >wont we aggrevate the the problems with corrupted files? No. The problems with the file system due to power failure or reboot occur when writing new blocks to a file. It does not occur when rewriting an already-in-use block. Writing to swap space is OK. -- Joe Smith (408)922-6220 | SMTP: jms@tardis.tymnet.com or jms@gemini.tymnet.com BT Tymnet Tech Services | UUCP: ...!{ames,pyramid}!oliveb!tymix!tardis!jms PO Box 49019, MS-C41 | BIX: smithjoe | 12 PDP-10s still running! "POPJ P," San Jose, CA 95161-9019 | humorous dislaimer: "My Amiga 3000 speaks for me."
chanson@isis.cs.du.edu (Chris Hanson) (10/15/90)
>onsidering the fact that the Amiga operating system does not handle file corruption very well, what would happen if during a swap a power fail occurred? Or the machine was turned off? < It is not the OS's fault, it is localized to the FileSystem. OFS and FFS don't deal well with powerdowns/crashes while they are trying to clean up. Solution: FIX the FileSystem. RDB now allows us to put WHATEVER FS we want on an autobooting drive, right? Heck, we could give ourselves (shudder) an autobooting MSDOS-formatted drive! Whaddaconcept. Then we'd have to worry about losing our FAT. (Like Roseanne!) >Doesn't this also bring up the question about a 'shutdown' command again? < Sure. What's wrong with a shutdown command? I think it is very rude to your OS to 'pull-the-rug-out-from-under-it' and expect it to cope with things correctly. UNI* boxes have a shutdown, and I think the MacII series took a step in the right direction by having the ShutDown menu item. Of course, this leads to (on the Mac or Amiga at least) the concept of a virus that turns your computer off again as soon as you turn it on! ;) Chris - Xenon -- #define chanson Christopher_Eric_Hanson || Lord_Xenon || Kelson_Haldane I work, but you don't know who I work for. And they don't know I'm here. "...Other fish to fly." -E. Miles
hclausen@adspdk.UUCP (Henrik Clausen) (10/18/90)
In article <42.2712f4e3@vger.nsu.edu>, manes@vger.nsu.edu writes: > One thing has occurred to me about virtual memory for the amiga. > > Considering the fact that the Amiga operating system does not handle > file corruption very well, what would happen if during a swap a power > fail occurred? Or the machine was turned off? 1) Under 2.0 I've yet to meet problems of HD not validating. Methinks it's not as much of a problem as it used to. 2) If you make a Swap: partition on the HD, it could actually be formatted (Quick) at every boot! Even if the disk structure was completely wrecked, who wouldd need to recover even a single byte off this partition? > I guess what I am asking is -- By using a swap device to a hard disk, > wont we aggrevate the the problems with corrupted files? Using a Swap: partition solves this problem from the ground up, and also avoids the potential problem of disk fragmentation on very loaded systems. So, VM is good for you. -Henrik | Henrik Clausen, Graffiti Data (Fido: 2:230/22.33) | | ...{pyramid|rutgers}!cbmvax!cbmehq!adspdk!hclausen | \__"Do not accept the heart that is the slave to reason" - Qawwali trad__/