[comp.sys.mac.programmer] Heap Scrambling in Debugger

jackiw@cs.swarthmore.edu (Nick Jackiw) (05/09/90)

A week or two ago, a small battery of postings crossed the net about
the importance of spending some time in the debugger before whining
about problems to the net.  Right on.  In my best Davy Crockett manner,
I've been assaulting my wild code from the pits of Macsbug and TMON,
hoping to track down those random bombs which suggest memory mismanagement.
The results have been baffling, however, and I was wondering if any
more seasoned low-level analysts could comment on their experiences in
similar situations.

Both Macsbug and TMON have a "heap-scramble" option, which--whenever a
heap compaction *could* occur---ensures that a heap scramble *will* occur.
If, after a scramble, there's an invalid heap, your application will break
into the debugger.

When I run with heap-scrambling on, I've located two conditions in which my
program behaves aberrantly. Depending on the debugger, I get either a 
"Heap Error" break-to-monitor [TMON], or one of the following errors in
MacsBug [6.1]
	* "Invalid heap"
	* "Divide by Zero exception"
	* Multifinder "Application quit out of memory" error (no break to debug)

In the case of the divide by zero exception, the pc is *not* at--or even near--
a divide instruction.  Likewise, with the third error, there is very little
chance that the application is out of memory--its running in a partition
far too large to be filled by the mem-requests it makes before halting.

Worse yet, the invalid heap is detected when the PC is located in system
code. (In specific, it's either in Pack3-SFGetFile, or in NewControl.) 
In that neither of these take arguments which are dereferenced handles,
and in that I assume my application passed the heap-check the last time
it made an allocation-request, I can't see what possible error of mine
could result in the debugger-break.

And even more maddening, the errors themselves are not consistent.  While
I'll always break at the same place in one example (SFGetFile, called by
Procedure X in segment Y), if I break to the debugger on LoadSeg#Y, and
then--without touching anything--continue, the heap-error doesn't occur
in SFGetFile, but someplace later (again, in system code).

Could some noble soul provide a few words on heap scrambling, beacons to
illuminate my path out of this quagmire of self-doubt?  I'm faced with
self-contradictory evidence, and don't see how to proceed. Is heap
scrambling reliable?  Are standard routines like SFGetFile and NewControl
reliable?  What conditions do these debuggers recognize as constituting
an "invalid heap?"

Thanks for any thoughts you may have on the matter.






--
---
jackiw@cs.swarthmore.edu  |  Smoggo: Can you prevent the detonation of
jackiw@swarthmr.bitnet    |          the thermonuclear device?
Applelink: D3717          |   Jimbo: No ... I cannot.