cim2@pyuxv.UUCP (Robert L. Fair) (05/14/87)
Does anyone out there know how DOS handles massive amounts of dynamic
allocation and freeing, in particular memory fragmentation.
In particular some applications which do this run when the machine is
first turned on, but fail later when the machine has been used for
a while.
Specifically:
1. Microsoft C-compiler 4.0 when compiling a VERY large program
(output of a 2000 line yacc grammer with several hundred symbols)
2. Our own product, FORESEE*, when doing large amounts of virtual
aggregation (munching around 10 MB of doubles using a virtual
array via dynamic disk swapping) [FORESEE is written in MS-C.]
Each time 2. is run it gets malloc failures earlier and earlier
into the program, implying that memory is getting eaten up
somehow !
*FORESEE is a trademark of Bell Communications Research.
Systems: IBM PC/AT, IBM PC/XT, PC-DOS 3.1, 640K RAM all true blue.
Any suggestions on whether its an MSC problem, DOS problem or a bug (unlikely)
would be much appreciated - reply by mail is fine.
This query is posed as a private individual and has no connection with
Bellcore/CHC or any other person living or dead.
Bob Fair
Bellcore/CHC
{ihnp4|gamma|ulysses}!pyuxww!pyuxv!cim2dhesi@bsu-cs.UUCP (05/16/87)
In article <287@pyuxv.UUCP> cim2@pyuxv.UUCP (Robert L. Fair) writes: >Does anyone out there know how DOS handles massive amounts of dynamic >allocation and freeing, in particular memory fragmentation. [Examples of problems omitted] I have had problems with using malloc() when compiling with Microsoft C version 3.0. I found that repeated allocations and freeing would eventually result in a hung system. If that had been a bug in malloc() in version 3.0 of Microsoft C, it would likely have been fixed in version 4.0. Since that doesn't seem to have happened, it may be a bug in MS-DOS itself. On the other hand, I have seen the problem occur under both versions 2.1 and 3.1 of PC-DOS, and one would expect the bug to have been fixed going from 2.1 to 3.1. So it's not clear where the bug lies. But it's definitely a bug either in Microsoft C or in MS-DOS or in both. Since the Microsoft C compiler is apparently compiled with itself, it will exhibit the malloc bug like any other C program compiled with Microsoft C. I wrote my own simply memory allocator that calls Microsoft's malloc() function with requests for large blocks and then breaks them up into smaller pieces as needed. This worked fine. -- Rahul Dhesi UUCP: {ihnp4,seismo}!{iuvax,pur-ee}!bsu-cs!dhesi
chips@usfvax2.UUCP (05/20/87)
In article <287@pyuxv.UUCP>, cim2@pyuxv.UUCP (Robert L. Fair) writes: > Does anyone out there know how DOS handles massive amounts of dynamic > allocation and freeing, in particular memory fragmentation. > > In particular some applications which do this run when the machine is > first turned on, but fail later when the machine has been used for > a while. > 1. Microsoft C-compiler 4.0 when compiling a VERY large program > 2. Our own product, FORESEE*, when doing large amounts of virtual > aggregation (munching around 10 MB of doubles using a virtual > array via dynamic disk swapping) [FORESEE is written in MS-C.] > > Each time 2. is run it gets malloc failures earlier and earlier [...] > Systems: IBM PC/AT, IBM PC/XT, PC-DOS 3.1, 640K RAM all true blue. ^^^^^^^^^^^^^^^^^^^^^^^ Bellcore must have $$$. > > Any suggestions on whether its an MSC problem, DOS problem or a bug [...] > > Bob Fair > Bellcore/CHC I ass_u_me that FORESEE is a large-model program, and is thus doing allocation of "far" pointers. (Calling _fmalloc() qualifies, regardless of overall memory model.) Large model memory allocation under MS-C 4.0 asks DOS for little pieces of memory, (slowly I turn, step by step, block by block...) which are returned to DOS only when free() is called. (Or _ffree() if not large model.) DOS does not reclaim these little blocks automatically. Only the block that contains the program's Program Segment Prefix (PSP) is automatically freed on exit. (Good thing, too, since to free your own code could be hazardous.) Conclusion -- you are probably allocating some memory and not freeing it. -- Chip Salzenberg Address: "{gatech,cbatt,akgua}!usfvax2!ateng!chip" AT Engineering, Tampa, FL Redress: "chips@usfvax2.UUCP" "Use the Source, Luke!" My opinions do not necessarily agree with anything.
roy@gitpyr.gatech.EDU (Roy Mongiovi) (05/21/87)
In article <764@usfvax2.UUCP>, chips@usfvax2.UUCP (Chip Salzenberg) writes: > DOS does not reclaim these little blocks automatically. Only the block that > contains the program's Program Segment Prefix (PSP) is automatically freed on > exit. (Good thing, too, since to free your own code could be hazardous.) This isn't at all true. Each block of memory allocated via the DOS functions has a 16 byte (1 paragraph) header which contains, among other things, the PSP of its owner. When a program terminates by any function except one of the terminate and stay resident functions, all memory blocks owned by its PSP are freed. I don't understand the last quoted sentence. How exactly could freeing the code segment for the program that just terminated be hazardous? That seems to me to be exactly what ought to happen.... I sincerely hope that MSC does not use the DOS functions directly to allocate memory, since 16 bytes is a tremendous overhead for small blocks. I don't own Microsoft C, so someone else will have to look into it. -- Roy J. Mongiovi Systems Analyst Office of Computing Services Georgia Institute of Technology Atlanta GA 30332. (404) 894-4660 ...!{akgua, allegra, amd, hplabs, ihnp4, masscomp, ut-ngp}!gatech!gitpyr!roy
geoffs@gssc.UUCP (Geoff Shapiro) (05/22/87)
In article <3617@gitpyr.gatech.EDU> roy@gitpyr.gatech.EDU (Roy Mongiovi) writes: >I sincerely hope that MSC does not use the DOS functions directly to allocate >memory, since 16 bytes is a tremendous overhead for small blocks. I don't >own Microsoft C, so someone else will have to look into it. >-- MSC's memory management routines do indeed call Dos to procure chunks of memory, BUT these chunks of memory are always allocated from Dos in minimum block sizes (you may change the minimum size, described in MSC manual) and they are then submanaged by the runtime library's memory management routines. I am sure that the MSC memory management routines' header overhead for submanaged memory blocks is a bit more concise. Geoff Shapiro Graphic Software Systems (503) 641-2200
chips@usfvax2.UUCP (Chip Salzenberg) (05/22/87)
In article <3617@gitpyr.gatech.EDU>, roy@gitpyr.gatech.EDU (Roy Mongiovi) writes: >In article <764@usfvax2.UUCP>, chips@usfvax2.UUCP (Chip Salzenberg) writes: >> DOS does not reclaim these little blocks automatically. Only the block that >> contains the program's Program Segment Prefix (PSP) is automatically freed on >> exit. (Good thing, too, since to free your own code could be hazardous.) > > This isn't at all true. Each block of memory allocated via the DOS functions > has a 16 byte (1 paragraph) header which contains, among other things, the PSP > of its owner. When a program terminates by any function except one of the > terminate and stay resident functions, all memory blocks owned by its PSP > are freed. I believe you're right. The Terminate and Stay Resident function does not free any memory allocated by the calling process, but the normal Terminate function does. That's one misteak this week. :-) > I don't understand the last quoted sentence. How exactly could freeing the > code segment for the program that just terminated be hazardous? That seems > to me to be exactly what ought to happen.... It would be hazardous for a process to free its own code segment, esp. in a multitasking environment such as Windows. DOS's freeing a process's code segment when that process Terminates is, as you suggest, normal. :-) > I sincerely hope that MSC does not use the DOS functions directly to allocate > memory, since 16 bytes is a tremendous overhead for small blocks. I don't > own Microsoft C, so someone else will have to look into it. I am quite sure that MS-C does call DOS to allocate "far" memory. This `feature' of MS-C makes accurate determination of available "far" memory quite difficult, if not impossible. I don't know whether or not the MS-C library is smart enough to ask for large chunks when a C program asks for many small chunks. Perhaps someone from MS can answer this... (1/2 :-) -- Chip Salzenberg Address: "{gatech,cbatt,akgua}!usfvax2!ateng!chip" AT Engineering, Tampa, FL Redress: "chips@usfvax2.UUCP" "Use the Source, Luke!" My opinions do not necessarily agree with anything.
rap@oliveb.UUCP (Robert A. Pease) (05/22/87)
In article <3617@gitpyr.gatech.EDU> roy@gitpyr.gatech.EDU (Roy Mongiovi) writes: >I sincerely hope that MSC does not use the DOS functions directly to allocate >memory, since 16 bytes is a tremendous overhead for small blocks. I don't >own Microsoft C, so someone else will have to look into it. >-- >Roy J. Mongiovi If you look in your MSC manual, it tells you that the memory allocator asks MS-DOS for a CHUNK of memory and then passes small parts of that chunk to each request from the program. If all the memory in the CHUNK is used up, the memory allocator asks MS-DOS for another CHUNK. -- Robert A. Pease {hplabs|fortune|idi|ihnp4|tolerant|allegra|glacier|olhqma}!oliveb!olivej!rap