jis@mtgzx.att.com (J Mukerji) (03/17/90)
I was playing around with help to see how much memory it uses, and was
amazed to see how quickly it eats up memory. As an example here is a
list of actions that were taken in help and the result of running "pstat
-s" on a Sun 4/60 with 12M memory 16M swap space: (pstat -s gives the
disposition of the swap space)
Before starting help
6980k allocated + 2920k reserved = 9900k used, 6480k available
Bring up help (tour.help 5300 bytes)
7588k allocated + 3736k reserved = 11324k used, 5056k available Delta: 1424k
Tag console in help (console.help ~19k)
7716k allocated + 3736k reserved = 11452k used, 4928k available Delta: 128k
See help for sh involving troff-ing the manual page (sh.1 ~41.5k)
8512k allocated + 3756k reserved = 12268k used, 4112k available Delta: 816k
Tag table help (table.help 56.4k)
8640k allocated + 3760k reserved = 12400k used, 3980k available Delta: 132k
Tag ness help (ness.help 11k)
8696k allocated + 3744k reserved = 12440k used, 3940k available Delta: 80k
See help for ls involving troff-ing the manual page (ls.1v 7.7k)
8864k allocated + 3736k reserved = 12600k used, 3780k available Delta: 160k
Ooops clicked on ls again (ls.1v 7.7k)
9008k allocated + 3732k reserved = 12740k used, 3640k available Delta: 140k
back to console (console.help ~19k)
9052k allocated + 3736k reserved = 12788k used, 3592k available Delta: 48k
clicked on console again (console.help ~19k)
9076k allocated + 3740k reserved = 12816k used, 3564k available Delta: 28k
Exit help
7004k allocated + 2916k reserved = 9920k used, 6460k available Delta: -2896k
I have included the size of the raw help files that are being read in at
each stage.
I have also shown the rough difference in available memory after each
step in the field marked "Delta: xxxk". What boggles my mind is that
enormous amounts of memory seem to be getting allocated and it seems
very little of it is getting freed for reuse. In short there seems to be
a very large memory leak somewhere in Help.
Is this a phenomenon peculiar to SPARCs or is it a general memory leak
bug in Help?
Either way has anyone got any ideas about how I or someone else ought to
go about plugging this leak?
Thanks.
Jishnu Mukerji,
jis@mtgzx.att.com,
+1 201 957 5986,
AT&T Bell Laboratories,
MT 3K-423, 200 Laurel Ave.,
Middletown NJ 07748tom@ICASE.EDU (Tom Crockett) (03/20/90)
Excerpts from info-andrew: 16-Mar-90 Help and its memory usage b.. J Mukerji@mtgzx.att.com (2317+0) > Is this a phenomenon peculiar to SPARCs or is it a general memory leak > bug in Help? > Either way has anyone got any ideas about how I or someone else ought to > go about plugging this leak? I think there is another memory leak in Andrew as well. Try the following scenario: Sun 4/60, SunOS 4.0.3c X11R4 + fixes 1-5 compiled w/ "cc -O" Andrew + patches 001-003 compiled w/ "cc" (1) Start Xsun (via xinit or whatever -- I use xdmshell) (2) Start console with lots of dynamic stuff in it (load graphs, etc.) (3) Use "pstat -T" to look at memory usage for the Xsun server (4) Start xlock and let it run for several hours (5) Look at "pstat -T" again. The server has grown considerably. I originally reported this to the MIT X folks, thinking it was a memory leak in the Xsun server, and their response was along the lines of "You must be running some crufty old client that's eating server memory". This only seems to be a problem while xlock is active. Without xlock, server memory usage remains constant. Overnight, the server will easily expand by 5 or 6 MB. In a few days, if I just let things go, I run out of swap space and lots of nasty things start to happen. Tom Crockett ICASE Institute for Computer Applications in Science and Engineering M.S. 132C e-mail: tom@icase.edu NASA Langley Research Center phone: (804) 864-2182 Hampton, VA 23665-5225
Craig_Everhart@TRANSARC.COM (03/20/90)
OK, I'm confused. What does running ``xlock'' have to do with ``another memory leak in Andrew''? Craig
jis@mtgzx.att.com (J Mukerji) (03/20/90)
In my original message on this subject I had made the observation that "enormous amounts of memory seem to be getting allocated and it seems very little of it is getting freed for reuse. In short there seems to be a very large memory leak somewhere in Help". I had based this on a set of experiments that showed that after all the dynamic objects needed for displaying a particular type of help document had been loaded, an inexplicably large amount of memory is getting allocated for each document that is displayed. In that example presented in that message the following documents were displayed in the order in which they appear in this list: [An Andrew ToolKit view (a spreadsheet) was included here, but could not be displayed.]So in the process of displaying these few documents help grew by 1.5M from its startup size of ~1.4M. This was on a Sun 4/60 (Sparcstation 1) running Sun OS 4.0.3c, Andrew and X11 compiled using Sun's C compiler. The workstation has 12M SIMMS and 16M swap space, and is diskless. The server is a Sun 4/330. Charles Hayden looked a little further into it and discovered the following (excerpt from his message to me): Excerpts from mail: 16-Mar-90 Re: Help and its memory usa.. C C Hayden@mtgzx.att.com (1411+0) > I monitored the mallocs with gdb. I found that it rereads all the > directories in the help search path every time, and does not properly > deallocate them. This is actually two bugs: reinitializing every time, > which chews up time, and not properly freeint the space. In my case, > there were 11 directories of 8K each, accounting for 88K on each query. > In addition, if you select an item from the "programs" menu or type it > in, it rereads it. If you select it from the history, I don't think it > does. It is reading into buffers, which it does not free until it > quits. So each time you select the same things from "programs", you get > another copy. The space will consist of at least the file size, plus > overhead for the gap, the style sheets and formatting information, etc. > Some of these data structures are allocated in fairly large chunks. > Your measurements seem to indicate that there are some significant data > structures allocated by the roff processing, that are not being > deallocated. I did not measure this. I would like to bring this to the attention of ITC and seek their help in fixing this problem in Help. Help is a very popular program here, but unfortunately if the users do not quit out of it often they keep running out of swap space. I would like to know if this problem is peculiar to the Sparc platform or is this a general problem. In either case what would be the quickest way to start getting it fixed? Thank you Jishnu Mukerji, jis@mtgzx.att.com, +1 201 957 5986, AT&T Bell Laboratories, MT 3K-423, 200 Laurel Ave., Middletown NJ 07748
tinglett+@rchland.ibm.com (Todd Inglett) (03/21/90)
Excerpts from ext.cmu.info-andrew: 19-Mar-90 Re: Help andits memory usa.. Tom Crockett@RCHGATE.rch (1550+0) > I think there is another memory leak in Andrew as well. Try > thefollowing scenario: > Sun 4/60, SunOS 4.0.3c > X11R4 + fixes 1-5 compiled w/ "cc -O" > Andrew + patches 001-003 compiled w/ "cc" > (1) Start Xsun (via xinit or whatever -- I usexdmshell) > (2) Start console with lots of dynamic stuff in it (loadgraphs, etc.) > (3) Use "pstat -T" to look at memory usage for the Xsunserver > (4) Start xlock and let it run for several hours > (5) Look at "pstat -T" again. The server has grown considerably. This is interesting, because I have seen it as well on an RT running AIX andBSD. Ordinary screen blanking also causes the leak. I have spent many hoursof my spare time to track it down and I am getting close, but I don't know theserver that well. Perhaps someone at MIT would care to help? The problem seems to be in the backing store code (mibstore.c). If you runyour server with -bs (no backing store) you will find that it does not consumememory. I have tracked the leak down to line 3199 of mibstore.c: if(pWindowPriv->pBackingPixmap->drawable.serialNumber != pBackingGC->serialNumber) { ValidateGC((DrawablePtr)pWindowPriv->pBackingPixmap, pBackingGC); } The ValidateGC winds up calling miBSCheapValidateGC (seems odd) which inturn calls the device specific validate (in my case apa16ValidateGC). Myapa16ValidateGC calls miRegionCreate to create the clipping region for the GC. Each time this validate happens (on line 3199), this clipping region getsreallocated and nobody seems to be getting rid of the old region. I put in code to track all regions created by the server and found the sameclipping region (described above) allocated thousands of times!! By the way,the code in apa16ValidateGC is virtually identical to mfbValidateGC, so Ithink there are many (perhaps all) servers with this problem. Anyone out there ever work on the backing store code? -todd inglett
nsb@thumper.bellcore.com (Nathaniel Borenstein) (03/21/90)
Clearly the business with the directory search path is simply a bug, which should be fixed. The matter of the buffers is probably more disputable, so I wanted to add my opinion: Buffers make sense in ez, where users often know about buffer-oriented operations. They make very little sense in help. I'm not convinced there's any reason to use buffers at all in help, and I'm highly skeptical that the buffers should be preserved once they're no longer visible. (Conceivably you might want a heuristic, so that the longer troff-derived help files, such as "csh", would be preserved in a buffer so that you didn't have to run rofftext on them twice. But mostly, it is so quick to read in a new help file that it is silly to waste the memory by keeping it around.) Finally, I want to draw to the attention of everyone concerned that there is a wonderful tool included in Andrew for digging out core leaks. This is the "plumber" function, included with Andrew's malloc (and therefore available only if you've compiled with ANDREW_MALLOC_ENV defined). If you've got this defined, the text object will add some bindings (^X^@t and ^X^@m) that will, respectively, write to a file or print on the printer a lot of information about what has been malloced so far, including the address of the routine that made the malloc calls. You can scan through this list for the big offenders (especially for the leaks by comparing consecutive malloc dumps) and then use a debugger to find out what procedure corresponds to the offending hex address (in gdb, type "br *0xhex-addr" and it will tell you the name of the routine if the appropriate symbol table has been loaded). In the past, I've found this an invaluable tool for digging out core leaks, so I thought I should mention it on this list.
wjh+@ANDREW.CMU.EDU (Fred Hansen) (03/21/90)
Excerpts from mail: 20-Mar-90 Re: Help and its memory usa.. Nathaniel Borenstein@thu (1792) > This is the "plumber" function, included with Andrew's malloc (and > therefore available only if you've compiled with ANDREW_MALLOC_ENV > defined). ANDREW_MALLOC_ENV is defined in the distributed ATK system in the file .../src/andrew/config/allsys.h. To get the plumber your .../src/andrew/config/site.h file must include #define DEBUG_MALLOC_ENV 1 To get the new malloc in most applications, the only directories that would need to be recompiled would be .../andrew/overhead/malloc and .../andrew/atk/apps. (I think.) FredH
jis@mtgzx.att.com (J Mukerji) (03/22/90)
Excerpts from info-andrew: 20-Mar-90 Re: Help and its memory usa.. Nathaniel Borenstein@thu (1793) > I'm not convinced there's any reason to use buffers at all in help, and I'm highly skeptical that the buffers should be preserved once they're > no longer visible. (Conceivably you might want a heuristic, so that the > longer troff-derived help files, such as "csh", would be preserved in a > buffer so that you didn't have to run rofftext on them twice. But > mostly, it is so quick to read in a new help file that it is silly to > waste the memory by keeping it around.) Buffer or no buffer, Help at present proceeds to run troff on the same file over and over anyway. I think even if it is decided to continue to use buffers for some documents like 'csh' for caching as suggested by Nathaniel, a way must be provided to get rid of that document to release memory without getting out of Help altogether. When one has been in Help for a while there is much state information that is stored in the panels that one would rather not lose, simply to get rid of some memory hog document that one happened to have stumbled upon at some point. Jishnu Mukerji, jis@mtgzx.att.com, +1 201 957 5986, AT&T Bell Laboratories, MT 3K-423, 200 Laurel Ave., Middletown NJ 07748
gk5g+@ANDREW.CMU.EDU (Gary Keim) (03/22/90)
Excerpts from misc: 21-Mar-90 Re: Help and its memory usa.. J Mukerji@mtgzx.att.com (1265+0) > I think even if it is decided to continue to use buffers for some > documents like 'csh' for caching as suggested by Nathaniel, a way must > be provided to get rid of that document to release memory without > getting out of Help altogether. Andy Palay has found many substantial core-leaks wrt help and is in the process of patching them. Stay tuned for a much leaner Help application. Gary Keim ATK Group PS - help doesn't use buffers and the path lookups aren't unreasonable.