LINDAHL@brandeis.BITNET (01/23/87)
Date: Fri, 23 Jan 87 00:50 EST From: <LINDAHL@BRANDEIS.BITNET> Subject: RE: Malloc() Flame To: info-atari16@score.stanford.edu X-Original-To: info-atari16@score.stanford.edu, LINDAHL In digest 87-35, imagen!atari!apratt@ucbvax.Berkeley.EDU (Allan Pratt) writes: > PLEASE don't go around publishing internal addresses like this requests, > since such addresses are GUARANTEED to change and BREAK programs which > count on them. Some parts of an operating system are hidden from > view ON PURPOSE, so the OS can be sure that its data structures will > not be fooled with by user programs. > > The correct procedure in this kind of situation is to say, "Gee, Atari, > I lose memory over time.. I wonder if there's a problem in the memory > allocator? Here is an example program which demonstrates the problem..." Calm down, Allan. I am not planning on running out and distributing programs which depend upon internal addresses, and then flame you when my programs break! Rather, I intend to examine this problem very thoroughly. If I told you, "well, I think that sort of, you know, I lose some memory sometimes after running one of the myriad versions of micro-emacs and maybe at other times with some combinations of a bunch of programs," then you would ignore me. You would ignore me for good reason, because I haven't provided enough information to find the bug. My intent is to explore the problem, and see if I can find a good example. If I can, you'll be the first to know. If I can come up with a fix, and you guys can't/won't fix it, then I _might_ consider distributing a fix with breakable code. But it _would_ be documented as such. Finally, please do not assume that anyone is a moron who needs to have these weighty "operating systems issues" explained to him. Undoubtedly 99.9% of the programmers listening to this conversation realize the folly of using internal addresses in publicly-distributed programs. Greg Lindahl brandeis radio astronomy group (BRAG) bitnet: lindahl@brandeis us snail: box 2522 brandeis university ci$: [76515,1122] waltham mass (usa) 02254-9110 (617) 899-5884 [NSA CIA Cray hackey-sack tie-dye divestment radical civil disobedience Thoreau Locke Hobbes flower-power frisbee physics star wars Teller long hair vegetarians 'and to think that MY tax dollars pay for this!']
apratt@atari.UUCP (01/27/87)
> Calm down, Allan. I am not planning on running out and distributing programs > which depend upon internal addresses, and then flame you when my programs > break! Rather, I intend to examine this problem very thoroughly. If I told > you, "well, I think that sort of, you know, I lose some memory sometimes > after running one of the myriad versions of micro-emacs and maybe at other > times with some combinations of a bunch of programs," then you would ignore > me. You would ignore me for good reason, because I haven't provided enough > information to find the bug. > > My intent is to explore the problem, and see if I can find a good example. > If I can, you'll be the first to know. If I can come up with a fix, and you > guys can't/won't fix it, then I _might_ consider distributing a fix with > breakable code. But it _would_ be documented as such. When seeking a solution to a problem in an open forum, it is often better to attack it directly rather than ask an obscure and infammatory question: asking, "Does Atari know the specifics of the Malloc() problems, and can/will they be fixed?" would have given me more information about what problem you were trying to solve. Had you asked this, I would have told you that we DO know what the problem is, and nothing short of a complete rewrite will fix it, because the memory allocation runs through the OS like a single thread through a whole sweater: you start pulling and the whole thing disintegrates. My advice to you is to forget trying to disassemble ROM or whatever you hoped to do to fix the problem. It permeates the whole OS, and no patch will do. I'd have told you that, and more civilly, had you asked outright. Please see the next posting from me for more on this and other subjects. /----------------------------------------------\ | Opinions expressed above do not necessarily | -- Allan Pratt, Atari Corp. | reflect those of Atari Corp. or anyone else. | ...lll-lcc!atari!apratt \----------------------------------------------/
roeder@sbsvax.UUCP (Edgar Roeder) (04/26/88)
I have some problems with the installation of a system-call via the _shell_p - Interface. Whenever i call system() inside some program P and then call malloc() inside the system()-routine, i get a crash after returning from P. I think this is because the allocated space was owned by the program P and not by the shell. My shell however needs this allocated space for history-substitution, variable-values etc. Now here is my question: Is there some way to tell TOS, that the owner of this Malloc()-ated space is another program ? Or equivalent: can i substitute the current process P by the shell during the Malloc()-call ? (only setting the system-variable _current_proc at 0x602C seems not to work) -- Edgar
hyc@math.lsa.umich.edu (Howard Chu) (08/19/88)
I get the feeling that there really is no solution at this point. How can you
have two different versions of Malloc/Free in the same system? Consider how
this would work. Assume a new ROM with TOS and GEMDOS that use the new, fixed
Malloc. Consider starting an old application from this new GEM desktop. Both
memory allocators will have to be rewritten to know exactly what each other
is doing, or you get programs tromping all over each other. And since you really
can't change the old version of Malloc, what you're really going to have to do
is write the new Malloc such that it allocates memory in such a way that the
old Malloc will honor its allocations. Anything using the old Malloc will still
eat up memory, will still need to be freed in the correct order or else you
lose. It'd be pretty gross. Having two incompatible memory allocators active
in the same system just can't be a good idea... It's not just a question of
a single program only sticking to one version - it's a question of many
programs being able to be loaded up arbitrarily, bearing in mind that the OS
itself is probably using the new allocator.
The only real solution would have been to put in the fixed allocator and tell
anyone with misbehaving code to rewrite it. Unfortunately, as Roy Good has
mentioned, a lot of programs' original writers are no longer in business, so
there's no way to catch everything with an appropriate update. About the only
route then, if Atari put out a fixed Malloc, is for some Good Samaritan to
pull out a debugger and come up with some binary patches. Not exactly a nice
situation.
--
/
/_ , ,_. Howard Chu
/ /(_/(__ University of Michigan
/ Computing Center College of LS&A
' Unix Project Information Systemsgaz@apollo.COM (Gary Zaidenweber) (08/20/88)
This is not really as big a problem as it seems. The old malloc()
can become a wrapper to the new malloc() which knows how to use
the behavior of the old one when it is called by the wrapper
routine. Only the trap code for the new malloc() is published in
header files, though binaries which call the old trap code get
the former behavior. In some future release of the os, the old
malloc() behavior may be deleted, though it doesn't have to if
its not in the way.
This is really a case of discipline. It starts with the vendor
when they publish the do's and don't's of using system calls.
Please, Atari, begin to enforce this discipline on us (in a
reasonable way, of course) by providing this kind of fix. We also
have a strict rule on backward compatibility here at Apollo and
we often use the above mechanism to get there. It increases your
credibility and better code will result from developers.
(note infinite feedback loop here.)
I really hope you guys at Atari (Roy and Allan are the two names
I remember) are feeding this back into your market research
group. Good luck to you and us.
--
UUCP: ...{umix,mit-eddie,uw-beaver}!apollo!gaz
ARPA: gaz@apollo.COM
AT&T: (508)256-6600 x6081
Its never too late to have a happy childhood!apratt@atari.UUCP (Allan Pratt) (08/20/88)
I appreciate the recent wave of comments to the effect that "it doesn't make sense to have both allocators in the same system at the same time." What I believe is missing here is a general understanding of what the new Malloc (the one in the upcoming TOS 1.4 release) really does. The old Malloc (11/20 and Mega ROMs) used OSPool for memory descriptors, but used it VERY BADLY, and there was interference with the 40-folder bug (because folder descriptors also came out of OSPool). That's why there were limits on the number of blocks you could allocate, and the way you freed them. The new Malloc still uses OSPool because it can't use TPA: people mess with memory they don't own, and they expect successive Malloc's to return contiguous memory. But the OSPool has been GREATLY IMPROVED. Now it is managed correctly, with reasonable allocation and freeing and compaction and so on, and furthermore it can be extended (for the use of both folders and memory descriptors) by the same old FOLDR100.PRG. The likelihood of running into trouble with too many folders or Malloc is now MUCH REDUCED, and the fix is easy: use FOLDR100.PRG. Importantly, you no longer get mysterious behavior when you really do run out of pool: Malloc now returns zero, and folder allocations simply fail (you get a message on your screen and the system dies, but I consider that better than mysterious and data-losing behavior). Before you clamour about "The 40-folder bug isn't fixed" let me remind you about the new definition of that limit: 1. The OSPool is statically allocated at boot time, but can be extended with FOLDR100.PRG (as many times as you like). 2. The OSPool is used for open-file handles, directory (folder) descriptors, and some other stuff (in "big" blocks) and Malloc descriptors (in "little" blocks, 1/4 of a big block). 3. The pool is used efficiently: small blocks are compacted when necessary to optimize big-block availablility. 4. A folder only takes up space in the pool when it is "active." A folder is "active" when it is: a) Some drive's root directory, b) Somebody's current directory, c) The parent of some open files, d) Being traversed during the current OS call, or e) The parent of an "active" folder (recursively). The upshot of this is that if there is room for, say, 40 folders in the pool, you can have up to 40 ACTIVE folders at a time. Just getting a directory listing of a folder only makes it active briefly (during the Fsfirst call, but not during Fsnext's). Then it is not active any more, and the space can be re-used if needed. FOLDR100.PRG when used with the new TOS expands the pool for ALL its clients, not just directory descriptors. I hope this clears up some of the confusion, and I hope you understand why we went this way rather than a "two Malloc's" approach or using a "perfect" malloc (with no restrictions) at the expense of breaking some existing programs. ============================================ Opinions expressed above do not necessarily -- Allan Pratt, Atari Corp. reflect those of Atari Corp. or anyone else. ...ames!atari!apratt
Thomas_E_Zerucha@cup.portal.com (08/20/88)
Perhaps it would help if Atari would post a list of things that broke because
of fixing Malloc. I don't know if this would violate any agreements, but
at least it might help us know who to avoid. And if they are out of business
a new opportunity exists to rewrite the products to be compatible.
I generally don't care about games, but would be interested if any
would have problems because I own a few. We don't even know the percentage
of things that broke, nor if any major packages are among these. I find it
hard to believe that very many things have this (Malloc) problem, and would
at least like to know the statistics. And Arkanoid broke on the Mega ROMS,
but I found a patch to let it. Same with Star Trek: the rebel universe.
Did that stop you from coming out with the Mega ROMs? NO!
I think that if you "fixed" the malloc bug, either the dated software
(some of which may already be obsolete) would fade into oblivion, or be
fixed, patched, or some other company would replace it/them.koreth@ssyx.ucsc.edu (Steven Grimm) (08/21/88)
In article <1125@atari.UUCP> apratt@atari.UUCP (Allan Pratt) writes: >I appreciate the recent wave of comments to the effect that "it doesn't >make sense to have both allocators in the same system at the same time." > >What I believe is missing here is a general understanding of what the >new Malloc (the one in the upcoming TOS 1.4 release) really does. [Description of OSPool handling deleted] Thanks! That will clear matters up for a lot of people. Will all that be documented anywhere, for the benefit of developers who don't have access to the net? Someone at Atari should collect all the worthwhile articles like that and add them to the developers' docs, as there has been quite a wealth of information here. --- These are my opinions, and in no way reflect those of UCSC, which are wrong. Steven Grimm Moderator, comp.{sources,binaries}.atari.st koreth@ssyx.ucsc.edu uunet!ucbvax!ucscc!ssyx!koreth
wes@obie.UUCP (Barnacle Wes) (08/21/88)
In article <402@clio.math.lsa.umich.edu>, hyc@math.lsa.umich.edu (Howard Chu) writes: > I get the feeling that there really is no solution at this point. How can you > have two different versions of Malloc/Free in the same system? Actually, you don't need to make another Malloc, just make the current one smarter. The problem with the current Malloc is that it uses a static data area, rather than a linked list, to keep track of allocated memory blocks. This is essentially similar to the idea of file handles. There was a version of MS-DOS once upon a time (I think it was 2.xx) that had an interesting way of expanding the number of files a process could use. The file handle pool was actually allocated in the Program Segment Prefix (PSP), which GEMDOS calls a basepage. There was a pointer to the file handles, which pointed to a 20-byte data area in the PSP, and 20 bytes for file handles. As it turned out, a program could copy the 20 bytes to a (hopefully larger :-) data area, zero the remainder of the data area, and point the pointer to this new data area. There must have also been a datum that described how many handles were available, which also had to be updated. From that point on, the process could use that many file handles, as long as the sytem FILES parameter, set by CONFIG.SYS at boot time, would allow that many files open. If the new GEMDOS used a "Malloc handle" pool that were relocatable in this manner, new applications that new of this feature would be able to Malloc more than the current 20 (or so) blocks, while retaining the essentially static data structures required for compatibility with the current Malloc. cat flames > /dev/null mailx applause money fame wes@obie -- {hpda, uwmcsd1}!sp7040!obie!wes "Happiness lies in being priviledged to work hard for long hours in doing whatever you think is worth doing." -- Robert A. Heinlein --
Thomas_E_Zerucha@cup.portal.com (08/24/88)
Actually, FOLDR100.prg does better than what MSDOS 2.xx may have done - it uses a dynamically linked list which is expandable, and expands upon the current OS Pool. And According to A. Pratt, the NEW half-fixed Malloc will use this *much* more efficiently. The current ROMs can't change the size of the allocated part of the OS pool, or recombine it once returned. Idea: Why not have the Malloc, or the low level OS Pool allocator detect that it has run out of handle space, then allocate a small part of the bottom of a free space (and perhaps return it if not the actual OS Pool). That way, if it failed, you would *really* be out of memory, but the OS Pool wouldn't be a restriction since it could then be extended as part of the normal Malloc process (I know it would take wierd code to do this, but it would fix both problems, since the current crop of Malloc violators don't break, they shouldn't invoke this function ever, and the new fixed applications would then work without worrying about handles).
chasm@killer.DALLAS.TX.US (Charles Marslett) (08/24/88)
In article <402@clio.math.lsa.umich.edu>, hyc@math.lsa.umich.edu (Howard Chu) writes: > I get the feeling that there really is no solution at this point. How can you > have two different versions of Malloc/Free in the same system? Consider how > this would work. Assume a new ROM with TOS and GEMDOS that use the new, fixed > Malloc. Consider starting an old application from this new GEM desktop. Both > memory allocators will have to be rewritten to know exactly what each other > is doing, or you get programs tromping all over each other. And since you really > can't change the old version of Malloc, what you're really going to have to do > is write the new Malloc such that it allocates memory in such a way that the > old Malloc will honor its allocations. Anything using the old Malloc will still > eat up memory, will still need to be freed in the correct order or else you > lose. It'd be pretty gross. Having two incompatible memory allocators active > in the same system just can't be a good idea... It's not just a question of > a single program only sticking to one version - it's a question of many > programs being able to be loaded up arbitrarily, bearing in mind that the OS > itself is probably using the new allocator. > Howard Chu I fail to see the problem -- I am almost always running with three malloc's active at a time now, so four is not such an unreasonable leap of faith. The obvious way is to have a Malloc_new that allocates two kinds of blocks: real ones and Malloc_old blocks that are really double blocks (one that is contiguous with the previous Malloc_old and the other that is used to keep track of it). The only reason for two schemes is that the garbage collection and overhead of the remote header may get expensive. Then if a program uses the old Malloc function, his memory may get fragmented and he may have the same old problems (but we want to be compatible, right (:^) ... and if the program uses only the new functions memory does not fade away. An alternative is to use Malloc_old as the master Malloc, and have some kind of initialization call that would tell Malloc_new to grab all available memory -- one big Malloc, so we can't return it in the wrong order (:^) -- and parcel it out using a reasonable algorithm. This is in fact what my programs do now -- I let MWC grab what it need to support the run time library, then I gram all that is left (less several hundred bytes, just in case) and from then on run my own memory allocator. Not a crash in a million mallocs! So what am I missing? I admit there are probably problems with running a shell that uses one Malloc underneath a shell using the other (spawning subprocesses is a really nasty problem with this scheme, but it never really worked before, so I see that as not much of a real problem), is there another? Enquiring minds want to know! Charles Marslett chasm@killer.dallas.tx.us
poole@forty2.UUCP (Simon Poole) (08/25/88)
One thing has got slightly lost in this discussion, that we are not
talking about just fixing bugs in Malloc, but the whole function of
the Malloc call would change. Until now Malloc has been useful (modulo
the bugs) as a large scale allocator for a limited amount of blocks,
changing Malloc to be a general purpose memory allocator, won't help
anybody who actually has applications out there (or is writing a new
one), because these programs will break on old versions of TOS. With
other words, your application won't run on the computers of the
largest part of your potential customers.
You can naturally get around this by distributing the new TOS on disk
(perhaps together with your program?), but how many program run on a
520ST - 200kB ? Matter of fact you would be supprised how many
programs don't run on a 1040ST - 200kB (I've been booting the Beta-TOS
from disk for the last two months, since the EPROM's never made it
here).
PS: anyway as Moshe Braner pointed out, Malloc is far too slow for any-
thing serious.......
--
----------------------------------------------------------------------------
UUCP: ...mcvax!cernvax!forty2!poole Simon Poole
BITNET: K538915@CZHRZU1A
----------------------------------------------------------------------------glk01126@uxa.cso.uiuc.edu (10/28/89)
Need help with the following code, I've been sweating it out
for days, and can't understand why it works the first time it
is called, but the second through nth times are iffy. Sometimes
free is set to 0, sometimes to a valid count of free mem div
4. Malloc(-1L) after the first time it is called, returns
some value less than 4096, but the memory is available to be
allocated by other parts of the program...
HELP:
*free is set to zero before call.
/*****************************
Compute total free
memory
*****************************/
freemem(free)
int *free;
{
long kilo;
if (Malloc(-1L)>4096L)
{
kilo = Malloc(4096L);
if (kilo)
{
*free += 4;
freemem(free);
Mfree((char *)kilo);
}
}
}
/*****************************
Show free memory in
disk dialog
*****************************/
dial_free(list)
dualfile *list;
{
int free;
free = 0;
freemem(&free);
sprintf(rs_object[rs_trloc[DISK]+MEMOFREE].ob_spec+2,"%4dK, B:",free);
sprintf(rs_object[rs_trloc[DISK]+MEMOLARG].ob_spec,"%ld\0",Malloc(-1L));
sprintf(rs_object[rs_trloc[DISK]+MEMOFILES].ob_spec,"%d\0",
dualcnt(dualfirst(list)) );
dial_draw2(DISK,MEMOFREE,MEMOLARG);
dial_draw(DISK,MEMOFILES);
}
( dial_draw simply redraws the object in the dialog box,
dial_draw2 redraws the rectangle encompassing item1 and item2 )
Thanks a lot if you have any ideas... I really am on the edge of smashing
something. Using Laser C V2.0.
-Spieu!steve@thelake.mn.org (Steve Yelvington) (01/17/90)
[In article <1534@cs.rit.edu>,
ajy2208%ritcv@cs.rit.edu writes ... ]
> I have Mark Williams C, and am curious: What is the difference between
> malloc() and Malloc() ????
Malloc() is a macro that expands to direct invocation of GEMdos function
0x48. GEMdos Malloc() is broken in several ways that *probably* won't bite
you for what you're doing. (Some of these "pecularities" are documented in
the MWC manual.) However, it's not as safe as malloc().
malloc() is a library call. Like Malloc(), it returns a pointer to a block
of memory, or NULL. However, it does not directly call Malloc(). Instead,
it is part of a complicated package of memory management code that Mark
Williams Co. has written to work around the known bugs in the GEMdos
Malloc() function. Each C vendor has had to implement such a scheme.
> I am writing a slideshow/picture viewing
> program, and when I used malloc() to get a 32k block of memory for a
> picture, the picture always had some stray data at the top (looked
> suspiciously like the other variables I was using).. However, when I
> used Malloc() this disappeared. Boy, I almost pulled my hair out
> trying to figure out what was wrong!!!
Remember that malloc() is not required to clear garbage from the memory
block to which it points. If you need it cleared, use calloc() or clear
the block manually.
If MWC's malloc() is indeed pointing you to memory that is in current use
for some other purposes, it's broken -- but I doubt that actually is
happening. MicroEMACS would die horribly. It's the malloc() torture test
from hell.
Also, remember that you must point physbase to a 256-byte boundary (on the
ST, apparently not on the STE), or strange things may happen:
char *ptr, *pbase;
if (((*ptr = malloc(1024*32)) == NULL))
panic(); /* allocation error */
pbase = ptr + (ptr % 256);
... then call Setscreen(). (Note that I'm mallocking 32 kilobytes, while
the screen occupies only 32,000, so the pointer arithmetic is legal.)
--
Steve Yelvington at the (thin ice today*) lake in Minnesota
UUCP path: ... umn-cs.cs.umn.edu!thelake!steve
*16 cars through the ice so far this year! Yes, you, too, can
have that sinking feeling....andyc@hplsla.HP.COM (Andy Cassino) (01/18/90)
alex@athertn.Atherton.COM (Alex Leavens) writes: > (The Malloc() call limit can be raised by using > the FoldrXXX.prg program from Atari.) This raises the question - is Malloc() fixed in TOS1.4 ? %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% % Andy Cassino % % uucp: hplabs!hplsla!andyc domain: andyc%hplsla@hplabs.hp.com % % Hewlett-Packard Lake Stevens Instrument Division % % 8600 Soper Hill Road Everett, WA 98205-1298 % % (206) 335-2211 % %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
jjung@alcor.usc.edu (John Jung) (03/17/90)
There's been a bit of discussion in this newsgroup recently about how the infamous "40 folder bug" can be induced if too many memory blocks are allocated with Malloc(). I'm currently developing an application for the ST in C that uses dynamic storage of linked lists. As a consequence, I'm Malloc()ing blocks of memory left and right. Since I don't want to encounter this bug, I'd like to know if anyone has a solution for me to use? Someone posted in this thread that a solution is to Malloc() one large block at the start of the application, then divide it into smaller blocks with malloc() calls. Does anyone have a sample program or actual code to demonstrate this principle? I'm developing with Laser C; the documentation does not specify any problems with using Malloc(), and the entry for malloc() gives me the impression that it is just a UN*X-compatable alias. Any information will be truly apprciated, either by mail or by a posted reply. Thanks in advance. John
gl8f@astsun9.astro.Virginia.EDU (Greg Lindahl) (03/19/90)
In article <8672@chaph.usc.edu> jjung@alcor.usc.edu (John Jung) writes: >I'm currently developing an application for the ST in C that uses dynamic >storage of linked lists. As a consequence, I'm Malloc()ing blocks of memory >left and right. Since I don't want to encounter this bug, I'd like to know >if anyone has a solution for me to use? I believe Laser C's malloc() does the "right thing"; you can easily test it by simply malloc()ing several thousand little blocks and see what happens. If you want to see source for malloc() that does the right thing, ftp the source code for dLibs from terminator.cc.umich.edu. dLibs is a complete library, and also contains nice things such as the new ANSI str* functions. Greg Lindahl gl8f@virginia.edu Astrophysicists for Choice.
klute@trillian.informatik.uni-dortmund.de (Rainer Klute) (03/19/90)
In article <8672@chaph.usc.edu>, jjung@alcor.usc.edu (John Jung) writes: > I'm currently developing an application for the ST in C that uses dynamic > storage of linked lists. As a consequence, I'm Malloc()ing blocks of memory > left and right. Since I don't want to encounter this bug, I'd like to know > if anyone has a solution for me to use? > I'm developing with Laser C. You don't have to worry at all. Simply avoid using Malloc and use malloc all the time. Laser C will allocate a large memory chunk with Malloc whenever neccessary and pass it on to you in (smaller) chunks you request with malloc. Dipl.-Inform. Rainer Klute klute@heike.informatik.uni-dortmund.de Univ. Dortmund, IRB klute@unido.uucp, klute@unido.bitnet Postfach 500500 |)|/ ...uunet!mcvax!unido!klute D-4600 Dortmund 50 |\|\ Tel.: +49 231 755-4663