[alt.hackers] PTRACE_ATTACH

eirik@wonton.TN.Cornell.EDU (Eirik Fuller) (02/26/91)

I just ran out of memory in a GNU emacs window.  Nearly anything I
tried to do complained about tired memory ("Memory exhausted").
Unfortunately,, this included C-x k (kill-buffer).

Since I was somewhat reluctant to close that window and open another,
I decided to try to get it back into a working condition.  I started
another emacs window, for M-x gdb emacs, then used the gdb attach
command on the process id of the tired emacs window.  The error
message came from the fact that new values from malloc were not valid
lisp pointers, since they were outside the 24 bit range (yes, I know I
should rebuild with two more bits; I've already done that on our "big"
machines ...).

The strings it was trying to allocate for any key bindings with
prompts did not fit into the current_string_block.  I noticed,
however, that the contents of that variable included a piece of the
first buffer I wanted to kill.  So I contrived a useful lie, in which
I claimed that the string currently in use was somewhat shorter than
it really was.  That left room for the new prompt strings, instead of
calling malloc to make room for more.  The truncated data that
resulted from the lie was gone after I killed the buffer.

So that emacs window is cheerfully humming along, with no more
complaints about memory shortage.  Of course, the very next thing I
did was C-x C-b, followed by a bunch of d's and x's.


This was not the first time I found gdb's attach command useful.  Back
when we were still running X11R3, my screen would occasionally freeze
in the middle of drawing a line.  One day I got sick of killing the
server from elsewhere, so I instead attached to it with gdb.  I found
that it was stuck in some line-drawing function, apparently in some
infinite loop; I used the debugger to return a bogus value from that
function.  I doubt the line got drawn, but my mouse started tracking
again ...

eirik@wonton.TN.Cornell.EDU (Eirik Fuller) (02/26/91)

In my last posting, I described my use of gdb's attach command to
salvage a tired emacs window.  Kids, don't try this at home!

Seriously, though, the emacs process dumped core soon thereafter.  It
was a useful hack in that I was able to save some buffers before it
died, but a complete hack would have cleaned up the mess I made in
current_string_block before the garbage collector tripped over it.
Gee, I can't wait ... time to start visiting core files until I run
out of memory again ... :-)

ObHack: I fixed the debugger as a result of the emacs core dump.  It
seems it dumped core trying to read the core dump.  Then it dumped
core trying to read its own core dump.  That was simple enough to fix;
a function argument needed an &.  So the emacs core dump turned out to
be useful after all; I'm not sure when I would have noticed that the
debugger couldn't read core files otherwise.


What?  That's too tame for an ObHack?  How about the use of
conditional debugger breakpoints with commands as a means of patching
code, as an alternative to recompilation?  Must be smalltalk that got
me into that bad habit ... :-)