sns@acp.OZ (Stuart Nixon) (11/28/88)
I am developing an application that demands large amounts of RAM for
processing. By large I mean the program will not run with less than 2Mb of
FAST ram, and it needs 4Mb to really go. The program is related to Satellite
Image Processing work, and loads parts of 250Mb data sets from WORM drives.
Anyway, the problem I have is this :
The program allocates a number of cache buffers (typically less than 8), which
are set up to take all of available FAST ram. The naive solution is to simply
do :
available_ram = AvailRam(MEMF_FAST) - overhead_ram_needed;
per_buffer_ram = available_ram / number_of_buffers;
/* now allocate each buffer */
[no, I don't normally use variable names that long...]
There are two problems with this :
1. Memory Fragmentation. As FAST ram may be fragmented, the allocation may
fail, even though there is actually ram present. I have considered a number
of ways to solve this, none of which I like. One way is to go and examine
the memory fragment list. This is unfriendly to multi-tasking, and very
internal system dependant [if C=A changes things, it breaks].
Currently, if the allocation fails then I deallocate the caches, and try
again, this time with the known memory pool that I can allocate (e.g. that
consists of large fragements). This method works, but I am unhappy with it.
2. FAST ram vs CHIP ram. As you can see, I only use FAST ram for the cache
buffers. All of CHIP ram is taken up by other operations, such as high-res/
high bitplane screens with superbitmaps. The problem is that the program
is assuming that CHIP ram is always distinct from FAST ram. Is this a
valid assumption? In other words, does [or will :-)] the mythical Amiga
3000 combine CHIP and FAST ram? If so, then I would like to somehow
consider this fact during cache allocation, as the current method will
break on such a system.
Thanks in advance for you comments or suggestions,
sns (Stuart Nixon)
sns Stuart Nixon Software
Phone : +61 9 322 6497
Uucp : ...{uunet,mcvax,ukc}!munnari!acp.oz!sns
ACSnet: sns@acp.ozdaveh@cbmvax.UUCP (Dave Haynie) (11/30/88)
in article <528@acp.OZ>, sns@acp.OZ (Stuart Nixon) says: > Keywords: memory fragmentation, CHIP vs FAST ram on a3000? > There are two problems with this : > 1. Memory Fragmentation. As FAST ram may be fragmented, the allocation may > fail, even though there is actually ram present. Who's fragging the memory. Certainly if there are other programs running who intermix small allocations with your big allocations, this can happen. However, if YOU are the one mixing in small allocations with the big ones, it's a real easy fix. Write a friendly allocation function to handle all of the details. This function would allocate system memory in medium sized chunks, unless otherwise instructed. When you ask for a small chunk of memory, your allocator would take that from a moderate memory chunk. When you ask for a large chunk, your allocator would get that from the system instead. If your moderate chunk size is chosen correctly, none of your small allocations will cause any frags. The OS provides the Allocate() function for allocations from a memory chunk that you own. > 2. FAST ram vs CHIP ram. As you can see, I only use FAST ram for the cache > buffers. All of CHIP ram is taken up by other operations, such as high-res/ > high bitplane screens with superbitmaps. The problem is that the program > is assuming that CHIP ram is always distinct from FAST ram. Is this a > valid assumption? In other words, does [or will :-)] the mythical Amiga > 3000 combine CHIP and FAST ram? If they were combined (seems pretty unlikely), you shouldn't have any problems. The MEMF_CHIP and MEMF_FAST bits in the attribute word aren't mutually exclusive, so at least in theory, you could have memory that's both CHIP and FAST using the current OS definitions. Don't wait up for it... > If so, then I would like to somehow consider this fact during cache > allocation, as the current method will break on such a system. Then you must be doing something wrong. With the simple allocator that's available in the OS now, you have no choice but to ask for MEMF_CHIP when you need CHIP memory for graphics, and MEMF_FAST (and perhaps 0 if that fails) for that buffering. You have to assume that the system memory lists have been set up to give you the best MEMF_FAST for your particular application, but that's about all you really can do. > sns (Stuart Nixon) -- Dave Haynie "The 32 Bit Guy" Commodore-Amiga "The Crew That Never Rests" {uunet|pyramid|rutgers}!cbmvax!daveh PLINK: D-DAVE H BIX: hazy Amiga -- It's not just a job, it's an obsession
sns@acp.OZ (Stuart Nixon) (12/02/88)
In article <5351@cbmvax.UUCP>, daveh@cbmvax.UUCP (Dave Haynie) writes: > in article <528@acp.OZ>, sns@acp.OZ (Stuart Nixon) says: > > Keywords: memory fragmentation, CHIP vs FAST ram on a3000? > > > 1. Memory Fragmentation. As FAST ram may be fragmented, the allocation may > > fail, even though there is actually ram present. > > Who's fragging the memory. Certainly if there are other programs running > who intermix small allocations with your big allocations, this can happen. > However, if YOU are the one mixing in small allocations with the big ones, > it's a real easy fix. Write a friendly allocation function to handle all > [useful comments about a dedicated memory allocation routine deleted] The memory frag problem mainly occurs on the development system, as it not surprisingly frags memory a bit during normal development cycles. My program tries to allocate large chunks (e.g. 200K to 1Mb) first, then does the small allocations. Not withstanding, I've changed to a dedicated memory allocation routine within my program, which grabs *all* fast ram, and then hands that memory down to lower levels. This works just fine. Thanx. > > 2. FAST ram vs CHIP ram. As you can see, I only use FAST ram for the cache > > buffers. All of CHIP ram is taken up by other operations, such as high-res/ > Then you must be doing something wrong. With the simple allocator that's Perhaps I should have explained further. Given an application that requires about 350K of CHIP on a dynamic basic during normal operation, that also wants to grab all of FAST ram, then combining FAST and CHIP ram on the bus causes problems, as by consuming all of FAST ram you have also consumed all of CHIP ram (as they are the same in my hypothetical example of an Amiga 3000 bus). In other words, currently a program can grab all of FAST ram, and then know that there is still about 400K of CHIP ram free (more using ECS chips with 1Mb CHIP ram). I wanted to be sure that I can assume CHIP and FAST will be separate areas of RAM. Anyway, the program now manages CHIP ram internally, so it is a moot point. > available in the OS now, you have no choice but to ask for MEMF_CHIP when > you need CHIP memory for graphics, and MEMF_FAST (and perhaps 0 if that > fails) for that buffering. You have to assume that the system memory lists In my application, I avoid CHIP ram for buffers as the display is typically running OverScanned, with maximum bitmaps in any given resolution (because this is an Image Processing application). Because of this, it is best to avoid CHIP ram to prevent bogging the processor down during the extensive calculations that can occur on the cache data. For the same reason, the program tries to avoid the Slow-FAST ram on Amiga 2000's and 500's for cache buffers. Some of these calculations require tens of 1,000,000 of calculations, so you don't want the processor running at 50% speed or whatever while processing the cache buffers.... > > sns (Stuart Nixon) > -- > Dave Haynie "The 32 Bit Guy" Commodore-Amiga "The Crew That Never Rests" ^^^^^^^^^^^^^^^^^^^^^^ By the way, I would like to express my thanks in general for the time that you and the other CATS/DOGS people spend helping out people on the net. In Australia, we have minimal [read no] support from Australia Commodore-Amiga, and I rely heavily on Usenet as the main line of technical support available. For example, C-A Australia decided not to distribute the Developers Conference notes to Australian developers. This attitude is a pity, as Australia has a very active developers community. sns (Stuart Nixon) sns Stuart Nixon Software, via Australian Computer Products Phone : +61 9 322 6497 Uucp : ...{uunet,mcvax,ukc}!munnari!acp.oz!sns ACSnet: sns@acp.oz