lynn@rave.phx.mcd.mot.com (Lynn D. Newton) (10/25/89)
I am running GNU Emacs 18.54/18.82 variously on a Motorola 68030 based system and a Motorola 88100 RISC based system. (Both Unix System V.) I'm sure this question has asked before, but I don't know the answer, and it has suddenly become a major issue where I am working. I'm planning on cross-posting to 3 groups, so please excuse the duplication. Apparently it is known and possibly even according to design that GNU Emacs continues to grow and never releases memory when buffers are closed or at any other appropriate times, so that it eventually grows into a monster. It is not uncommon for me to see processes that grow to 700-800 4K pages of memory, and I have seen as high as 1200. I am under the impression that such is the case with all strains of GNU Emacs. Naturally I would like to see it otherwise. If it is in fact by design, then what is the, and if it is a bug that has been around for a long time and which everyone experiences, then why can't it be fixed, and if it is a bug which only we have experienced and have lived with all this time thinking that it was supposed to be this way, what can be done about it? -- ================================================================= Lynn D. Newton | System Test Motorola MCD, Tempe, AZ | (Department of Heuristic Neology) (602) 437-3739 | "The bug stops here!" lynn@rave.phx.mcd.mot.com |
rlk@think.com (Robert Krawitz) (10/26/89)
In article <LYNN.89Oct24164609@rave.rave.phx.mcd.mot.com>, lynn@rave (Lynn D. Newton) writes:
]Apparently it is known and possibly even according to design that
]GNU Emacs continues to grow and never releases memory when
]buffers are closed or at any other appropriate times, so that it
]eventually grows into a monster. It is not uncommon for me to see
]processes that grow to 700-800 4K pages of memory, and I have
]seen as high as 1200.
This is a problem with all Unix programs that I know, or more generally
with all non-GC'ing memory management systems (emacs does have a GC for
lisp, but the underlying malloc doesn't garbage collect, so it never
releases any memory). In principle, it would be possible to free memory
down to the highest point at which memory is allocated (the top of the
heap), but in practice I haven't seen a version of malloc that does
this. Malloc can't do compaction (i. e. lowering the top of the heap by
rearranging memory) because it has no way of tracking down all pointers
to a given area of memory. This is to some extent a limitation of the C
language, in that pointers are not distinguished in any way in the
run-time code. In Lisp and other garbage collecting languages, where a
run-time system manages all memory, it is possible to use tag bits to
distinguish pointers, and to otherwise keep track of all outstanding
pointers. However, the Lisp GC combined with free(1) to free memory
allocated with malloc will give you a savings: emacs will preferentially
allocate itself memory that it already owns rather than ask the system
for more.
BTW, a 5 Mbyte emacs image isn't overly big. My emacs is now 10 Mbytes,
and it's only been around for 2 days. It's been known to grow to 20M.
Granted, this is only practical on big machines and personal
workstations, but it's a normal part of emacs use.
--
ames >>>>>>>>> | Robert Krawitz <rlk@think.com> 245 First St.
bloom-beacon > |think!rlk (postmaster) Cambridge, MA 02142
harvard >>>>>> . Thinking Machines Corp. (617)876-1111tale@pawl.rpi.edu (David C Lawrence) (10/26/89)
In <LYNN.89Oct24164609@rave.rave.phx.mcd.mot.com> lynn@rave.phx.mcd.mot.com
(Lynn D. Newton) writes:
Lynn> I'm planning on cross-posting to 3 groups, so please excuse the
Lynn> duplication.
[Aside: duplication is more easily overlooked if you did all the
cross-posting in one article, by specifying a comma-seperated list for
the Newsgroups: line.]
Lynn> Apparently it is known and possibly even according to design that
Lynn> GNU Emacs continues to grow and never releases memory when
Lynn> buffers are closed or at any other appropriate times, so that it
Lynn> eventually grows into a monster. It is not uncommon for me to see
Lynn> processes that grow to 700-800 4K pages of memory, and I have
Lynn> seen as high as 1200.
In <31052@news.Think.COM> rlk@think.com (Robert Krawitz) explains
why this happens. I'd like to expand on what he's said, to hopefully
further clarify it.
Robert> This is a problem with all Unix programs that I know, or more
Robert> generally with all non-GC'ing memory management systems (emacs
Robert> does have a GC for lisp, but the underlying malloc doesn't
Robert> garbage collect, so it never releases any memory).
Quite true. C programmes (by far the most common type on Unix
systems) have the *alloc(), free(), brk() and sbrk() functions
available for memory management. The basics of it all is that most
programmes are written using simply malloc() and free() for memory
management. malloc() will apportion memory as is available on the
heap; free() returns previously allocated memory back to the heap.
The non-decreasing behaviour shows up because malloc() will sbrk()
memory from the system when it needs more, but free() will not brk()
memory back to the system. (This all assumes a "standard" malloc
library.)
Robert> In principle, it would be possible to free memory down to the
Robert> highest point at which memory is allocated (the top of the
Robert> heap), but in practice I haven't seen a version of malloc that
Robert> does this. Malloc can't do compaction (i. e. lowering the top
Robert> of the heap by rearranging memory) because it has no way of
Robert> tracking down all pointers to a given area of memory.
There is now a new malloc library in alpha/beta test. It uses brk()
when appropriate, which has been observed to actually shrink the size
of a running Emacs. (*Gasp*) If all goes well through the testing
stages it is likely that it will be shipped with v19.
Robert> BTW, a 5 Mbyte emacs image isn't overly big. My emacs is now
Robert> 10 Mbytes, and it's only been around for 2 days. It's been
Robert> known to grow to 20M.
Sure it's overly big, and 20M is obscene. I am, as the programmer of
the new malloc package describes, "an Emacs weenie." This means that
I'll keep a single Emacs process around for days, even weeks, on end.
I do just about everything through Emacs from mail to news reading to
compiling to interactive conferencing. Oh yeah, and editing.
Generally all that should grow an Emacs image to that size is editing
of several _huge_ files. In "normal" usage, it shouldn't even get as
big as five meg. Features loaded in this Emacs:
(sort tags irc gnuspost rnews gnus-user-tale gnus-etc nntp gnus connect sup-misc supercite baw-alist mail-utils sendmail gin-mode mh-e x-mouse ehelp electric electric-minibuffer uniquify scroll backquote header crypt)
Process information (ps v):
PID TT STAT TIME SL RE PAGEIN SIZE RSS LIM %CPU %MEM COMMAND
27143 p3 S 7:19 0 99 399 3784 1616 xx 0.0 10.4 emacs
It was below two meg until I read in a 1.9 meg binary image to
purposefully grow the image for the sake of this posting.
Dave
--
(setq mail '("tale@pawl.rpi.edu" "tale@itsgw.rpi.edu" "tale@rpitsmts.bitnet"))kjones@talos.uucp (Kyle Jones) (10/26/89)
In order for GNU Emacs to give memory back to the operating system, it will be necessary to perform compaction. This is because under most, if not all UNIXes, you can only give memory back by adjusting the break, that is, the end of a process' data segment. You cannot give memory back in disjoint bits and pieces. It has to be contigous and adjacent to the break. If Emacs has to manage the break, this also means that Emacs has to be intimitely intertwined with malloc and any C (and X!) library routines that call malloc. "Gnarly." kyle jones <kjones@talos.uu.net> ...!uunet!talos!kjones
jkh@meepmeep.pcs.com (Jordan K. Hubbard) (10/27/89)
>If Emacs has to manage the break, this also means that Emacs has to be >intimitely intertwined with malloc and any C (and X!) library routines >that call malloc. Why? It seems that all one would need is a more intelligent malloc library (Emacs already uses its own malloc() anyway) that did storage compaction and break management on free() calls. I didn't say that this was easy, of course, since one cannot just blithely compact just any part of the pool (and expect active pointers into it to remain valid), but you can do a pretty reasonable job if the malloc() requests are mostly for large contiguous blocks (which they are for buffers, at least). I assume that Emacs free's buffers when they're destroyed, no? :-) I think the real answer is that yes, this can be done with a good malloc() package and little or no modification to the application, no it's not easy to do it right (and stay portable) so it has not been done (at least to my knowledge). Jordan -- PCS Computer Systeme GmbH, Munich, West Germany UUCP: pyramid!pcsbst!jkh jkh@meepmeep.pcs.com EUNET: unido!pcsbst!jkh ARPA: jkh@violet.berkeley.edu or hubbard@decwrl.dec.com
pcg@rupert.cs.aber.ac.uk (Piercarlo Grandi) (10/28/89)
In article <1989Oct26.043743.15022@rpi.edu> tale@pawl.rpi.edu (David C Lawrence) writes:
Robert> In principle, it would be possible to free memory down to the
Robert> highest point at which memory is allocated (the top of the
Robert> heap), but in practice I haven't seen a version of malloc that
Robert> does this. Malloc can't do compaction (i. e. lowering the top
Robert> of the heap by rearranging memory) because it has no way of
Robert> tracking down all pointers to a given area of memory.
There is now a new malloc library in alpha/beta test. It uses brk()
when appropriate, which has been observed to actually shrink the size
of a running Emacs. (*Gasp*) If all goes well through the testing
stages it is likely that it will be shipped with v19.
It so happens that I did post a shrinking, fully tested malloc in
alt.sources that does release memory; it takes great pains to do
this in every possible situation where this may be done (it does
refrain from this if there is evidence that the user bypassed
malloc and called sbrk itself). Well, GNU Emacs *does not* shrink.
Micrognu does, spectacularly.
The problem, I suspect, is that GNU Emacs does not allocate
memory in a stack like fashion. What I mean is: after fetching a file into
a buffer, deleting that buffer should release all the memory that was
allocated to it, and allow the break to be shrunk appropriately.
Unfortunately, I think that GNU Emacs does an allocation before
freeing the buffer and its contents, so that there is allocated
storage *above* the newly free large chunk of memory, and the
break cannot be taken back. Too bad... Also, GNU Emacs has, and
this is far worse, *extremely* poor locality of reference. Now,
if RMS were to rewrite the garbage collector so that it used
breadth first compaction...
--
Piercarlo "Peter" Grandi | ARPA: pcg%cs.aber.ac.uk@nsfnet-relay.ac.uk
Dept of CS, UCW Aberystwyth | UUCP: ...!mcvax!ukc!aber-cs!pcg
Penglais, Aberystwyth SY23 3BZ, UK | INET: pcg@cs.aber.ac.ukallbery@NCoast.ORG (Brandon S. Allbery) (10/31/89)
As quoted from <JKH.89Oct27134436@meepmeep.pcs.com> by jkh@meepmeep.pcs.com (Jordan K. Hubbard): +--------------- | I think the real answer is that yes, this can be done with a good | malloc() package and little or no modification to the application, no it's | not easy to do it right (and stay portable) so it has not been done | (at least to my knowledge). +--------------- It can be done, sort of, by tagging memory allocated by Lisp somehow and compacting that memory. The problem is that you must compact memory before handing any out to a malloc() *not* for Lisp (e.g. for the X library or etc.), or you risk ending up unable to compact beyond a chunk of memory which cannot be moved without making emacs dump core: +--------------------+ | initialized data | +--------------------+ | uninitialized data | +--------------------+ | Xlib memory | +--------------------+ | | | Lisp pool | | | +--------------------+ | something in Xlib | +--------------------+ | compactible memory | +--------------------+ You can't relocate the memory unless you can go in and change pointers to it, which is only possible in areas where you have complete control of the memory (e.g. the Lisp memory allocator). Worse, even if you do compact before servicing a malloc() for something other than the Lisp memory allocator, you can end up with sizeable gaps if the user kills a large buffer after the malloc(). The only way to do it right is to write your own versions of libraries that use malloc() and make them use something that can easily be adjusted, like the "handles" used by the Mac (flames to /dev/null) and Mis-Windows. In other words, it can be done up to a point, but it won't do all that much good. ++Brandon -- Brandon S. Allbery: allbery@NCoast.ORG, BALLBERY (MCI Mail), ALLBERY (Delphi) uunet!hal.cwru.edu!ncoast!allbery ncoast!allbery@hal.cwru.edu bsa@telotech.uucp *(comp.sources.misc mail to comp-sources-misc[-request]@backbone.site, please)* *Third party vote-collection service: send mail to allbery@uunet.uu.net (ONLY)* >>> Shall we try for comp.protocols.tcp-ip.eniac next, Richard? <<<