john@acorn.co.uk (John Bowler) (01/04/89)
Some time ago (11 Dec) David Rosenthal posted a message prompted by a comment from Jon Greenblatt about a server running out of memory when running with a program which used a large amount of backing store:- > I have pointed out for a long time: > > - that current X11 clients fail abruptly as soon as they get an Alloc > failure from the server, And what about the server - the sample server is not proof against malloc returning NULL - look at the BSD implementation of Xalloc. > - that making them survive Alloc errors is hard, because of the > asynchronous nature of error notification, > - that current X11 clients make little use of the memory-consuming > parts of the X11 protocol, because these are the parts that weren't > there in X10, > - and that this is skewing our ideas of how much memory an X11 server > in normal use will require. > > Unless the X community works out how to write clients that don't simply > collapse at the first hint of resource exhaustion, the suppliers of X > terminals are going to have a lot of VERY unhappy customers. Remember, > my experiments over a year ago showed that even a 0.5% alloc failure > rate rendered the system totally unusable. But writing a client which deals with resource exhaustion may well mean implementing the same thing twice - for example once using backing store (no redraw required on expose events) and once using explicit redraw if the backing store isn't available. I cannot believe that such an application would achieve acceptable performance in both cases (else why bother with the backing store implementation) so effectively the application doesn't work unless sufficient backing store is available. So much for portability. The best (worst?) example of this kind of thing is in xman - take a release 2 colour Sun server (say on a 3/60) and run xman. Open the directory listing for section 3 (or any section with several hundred entries) - now examine the size of the server. The xman allocates a pixmap in the server to hold the listing (effectively do-it-yourself backing store) then tiles (yes tiles!) the window background with this, resulting (in release 2) in a copy being made of the pixmap. On a large machine running a monochrome server this is not too much of a problem - it consumes 1/4 to 1/2 MByte of memory. On an 8 bit per pixel colour server or on a system with only 1 or 2Mbyte of memory it is a disaster. Resource exhaustion doesn't necessarily become apparent to the client on a UNIX system - the system just grinds to a halt. The problem is not particularly that clients ``simply collapse at the first hint of resource exhaustion'' - it is that some of the programming paradigms which are encouraged by X11 (and not discouraged by accompanying documentation) simply do not work in some machine environments. A few other examples:- - xmille/xfig/mazewar/xtrek are unuseable on 864x480 (``vga'') displays because the window which they produce does not fit on the screen. - applications which use colours without allocating a private colour map (ico -faces, twm with lots of colours, kaleidoscope) behave poorly on colour systems with (say) only four bits per pixel. [Ok I accept that the absence of the ICCCM+window managers which handle colour maps sensibly make private colour maps very difficult to use.] - The font caching strategy used within the server (caching whole fonts, not individual characters - contrast with PostScript) means that applications which use a lot of fonts consume large quantities of memory in the server. And again the message from K. Richard Magill (``memory vs toolkit'', 17 Dec):- > Is anyone out there creating and deleting lots of widgets dynamically? > > Seems there are some major holes in memory reclamation although I > suppose it could be malloc. > > You might think it doesn't really matter in a vm environment but > consider a 3/50 who now has one piece of in-use memory on each of a > zillion pages. Can you say "thrashing"? This highlights another problem. I have seen no real evidence of memory loss under these circumstances, but I know, from experience with NeWS, that heavy use of malloc/free causes both the Sun storage allocator and (even more so) the BSD4.3 allocator to consume large amounts of vm due to fragmentation. I see no easy solution to these problems. Issues of storage fragmentation within the server (from allocation of space for fonts or large numbers of windows) need to be addressed within the server. The current Acorn server (Release 2) responds to the SIGHUP from init by exiting - this at least guarantees that heap fragmentation in the server is corrected when the user who caused it (:-) logs out. It also uses a special purpose version of malloc designed to minimise memory fragmentation - I can see no justification for using a general purpose storage allocator in program such as the server whose storage use is so great and characteristic. Use of alloca can also help - but alloca is impossible to port to some procedure call conventions (notice that the ``portable'' release 3 alloca leaks store between alloca(0) calls). But the overall issues of writing applications to work with a variety of X server implementations (with different architectural limitations) must be addressed at the toolkit level, if not higher. Is it not the case that this is a very real limitation of the true portability of an X application? In particular I would have thought that large X applications (eg anything in the CAD field) will still need to be coded for the particular server environment under which they will be run. So the issue, as I see it, is not writing X clients to deal with resource exhaustion correctly, but providing facilities and guidelines to allow a greater set of X applications to be written portably. John Bowler, Acorn Computers Ltd jbowler@acorn.co.uk
dc@graphon.UUCP (Darren Croke) (01/07/89)
In reply to John Bowlers article on servers running out of memory. (posted Tue, Jan 3). I was disturbed to read the following comment. > But writing a client which deals with resource exhaustion may well mean > implementing the same thing twice - for example once using backing store > (no redraw required on expose events) and once using explicit redraw if > the backing store isn't available. I cannot believe that such an application > would achieve acceptable performance in both cases (else why bother with > the backing store implementation) so effectively the application doesn't > work unless sufficient backing store is available. So much for portability. The X11R3 protocol spec clearly states on page 33 that selecting backing-store for a window informs the server that maintaining contents would be beneficial. ---------- The spec goes on to say that "the server may stop maintaining contents at any time", which, correct me if I'm wrong, means that your client will receive expose events. Hopefully writers of clients will cater for expose events even when backing-store is advertised. Failure to do this will result in clients that won't run on most of the low end X terminals which don't have virtual memory and obviously have a finite limit as to how much backing-store they can maintain. Darren Croke.
dshr@SUN.COM (David Rosenthal) (01/08/89)
Just to make it quite clear - setting backing store on a window is a HINT to the server, it does not absolve the client from the responsibility of responding appropriately to Expose events. There is no sensible way to write an X client without dealing with Expose events: - There is no way to tell when to start drawing in your window(s) without waiting for an Expose event. Stupid X client #1 is the one who says "I created a window, mapped it and drew in it but nothing was visible". - Even if you set the backing-store hint to "Always", the server may lazy-evaluate the hint and not actually devote the memory until you map the window the first time. So even with backing-store Always you can't create the window and paint in it without waiting for an Expose event. - The server may stop maintaining backing store for a window at any time, and start generating Expose events on it. The bottom line is - if you think having the server pay attention to the backing-store hint on any of your windows is essential to the functioning or performance of your client, re-think the way you're writing the client. Never depend on anyone taking the hint. If you really mean that you want to be sure that you only draw the image once, what you want to do is: - Create a Pixmap the size you want the image to be. - If you did not actually get a Pixmap (remember, the error is asynchronous, and you will need to install a custom error handler to avoid the default Xlib bail-out. See Section 8.12 of the Xlib manual): - Check the error code. If it is an Alloc: - Explain to the user that the server is out of resources and ask that other clients be terminated to free up resources. - Wait for confirmation that this has been done, and try again. - If it isn't an Alloc, you wrote a bug. - Draw the image into the Pixmap. - Create a top-level Window that you are going to use to display all or part of the image in. Set up a mapping between the Pixmap coordinates and the Window coordinates, that can be adjusted to scroll the Window around over the Pixamp (remember, you may not get the size of top-level Window you ask for). Select for Expose and ConfigureNotify on the Window. (Really, this should be a Widget with appropriate scrollbars to control the mapping). - When you get an Expose, use CopyArea to copy the appropriate part of the Pixmap to the Window. - When you get a ConfigureNotify, change the coordinate mapping to correspond to the new window size. Note that even this isn't entirely free of resource exhaustion problems. But at least it anticipates the thing that is likely to fail (creating a large Pixmap) and does something sensible - instead of just failing with an Xlib error. Of course, this doesn't guarantee success. The server may never be able to create the Pixmap you want even with no other clients active. X does not guarantee portability, and this is only one of the ways in which the guarantee is not made. Sorry, but we never promised you a rose garden. X makes portability much more likely, but there are no guarantees. Even PostScript, which makes much stronger efforts than X to attain portability, does not guarantee it - see Appendix B of the PostScript Red Book. Another area in which X does not guarantee portability (Visuals) is the subject of the paper Dave Lemke & I will be giving at Usenix. There is a good paper waiting to be written on techniques for writing BadAlloc-proof X clients (I don't want to write this one, so I don't mind telling you about it). There are equally good papers waiting to be written on how to deal with other areas in which X doesn't guarantee portability (I do want to write some of these, so I won't tell you what they are just yet). David.