harkcom@spinach.pa.Yokogawa.CO.JP (02/09/91)
I've been pondering the allocation of resource ID's. I understand how a window allocates an ID and sends a request to the server for creating new resources. I also understand that the server checks to insure that the id is a valid one. But what I don't understand is what happens if a client continues creating and destroying resources to the point of using up all of the valid ID's. Does X have some mechanism for resetting the resource_id member when it overflows into the SERVER_BIT? If not, clients which manage a lot of resources for a long time may run out of valid ID's. If such a mechanism doesn't exist, has there been any talk about implementing a feature which allows the re-use of 'used' resource ID's? thanks Al P.S. One must be either extremely bored or extremely ex[c]ited by such a topic to actually spend a friday night digging through source code. Well, nobody's asked me go out for a beer so I think I'll keep reading source. It's cheaper than paperbacks in this 'greedy' country...
harkcom@spinach.pa.Yokogawa.CO.JP (02/14/91)
In article <HARKCOM.91Feb8215355@spinach.pa.Yokogawa.CO.JP> harkcom@spinach.pa.Yokogawa.CO.JP (that's me) writes: =} But what I don't understand is what happens if a client continues =} creating and destroying resources to the point of using up all of the =} valid ID's. Does X have some mechanism for resetting the resource_id =} member when it overflows into the SERVER_BIT? If not, clients which =} manage a lot of resources for a long time may run out of valid ID's. =} =} If such a mechanism doesn't exist, has there been any talk about =} implementing a feature which allows the re-use of 'used' resource ID's? Since there were no replies, I figure nobody knows. I checked Xlib and could find nothing that does such a thing. I also discovered that there is no mechanism for setting your own ID when making a request to create a resource. The client just keeps sucking up valid ID's until there are none left and them is completely helpless... If I am wrong about this, please, PLEASE, somebody tell me before my next free Friday night, or I might just create a support library to handle this. BTW: How does one go about suggesting to the powers that be that this might be a good addition to Xlib. [for ex. XReCreateWindow(Display *, XID)]
mouse@lightning.mcrcim.mcgill.EDU (02/14/91)
> I've been pondering the allocation of resource ID's. > I understand how a window allocates an ID and sends a request to the > server for creating new resources. I also understand that the server > checks to insure that the id is a valid one. > But what I don't understand is what happens if a client continues > creating and destroying resources to the point of using up all of the > valid ID's. It can't, in one sense. Once a resource has been destroyed the ID can be reused. The protocol document says: The resource-id-mask contains a single contiguous set of bits (at least 18). The client allocates resource IDs for types WINDOW, PIXMAP, CURSOR, FONT, GCONTEXT, and COLORMAP by choosing a value with only some subset of these bits set and ORing it with resource-id-base. Only values constructed in this way can be used to name newly created resources over this connection. [...] The client is not restricted to linear or contiguous allocation of resource IDs. Once an ID has been freed, it can be reused, but this should not be necessary. "should not" because creating 262144 resource objects over the life of a program seems a bit excessive. But if you must, you *can* reuse XIDs. (If you have a client that is perverse enough to want to have more than 262144 resource objects in existence simultaneously, you will most likely have trouble. Such clients cannot be fully portable, though they may work on servers that give out resource-id-masks with more than 18 bits set.) > Does X have some mechanism for resetting the resource_id member when > it overflows into the SERVER_BIT? This is an Xlib issue. I don't know, but I suspect not. It seems unlikely, simply because nobody would ever dream of its being necessary. > If not, clients which manage a lot of resources for a long time may > run out of valid ID's. This seems like a silly thing to worry about. However, suppose someone has some program running for weeks, creating and destroying windows, GCs, whatever, and then as it gets old, it crashes when its Xlib creates an invalid XID...unless Xlib does get this right, of course! (I haven't checked the source.) Hmmm, 262144 XIDs, that would mean 8738+ a day in order to run out in a month, which, in turn, means one every 9.9- seconds (8738 XIDs per day / 60*60*24=86400 seconds per day = 1 XID per 9.8877- seconds). That's on the average, over the whole month. Hmmm, perhaps it's not something we have to worry about after all. der Mouse old: mcgill-vision!mouse new: mouse@larry.mcrcim.mcgill.edu
dbrooks@osf.org (David Brooks) (02/14/91)
In article <9102140823.AA17984@lightning.McRCIM.McGill.EDU>, mouse@lightning.mcrcim.mcgill.EDU writes: |> This seems like a silly thing to worry about. However, suppose someone |> has some program running for weeks, creating and destroying windows, |> GCs, whatever, and then as it gets old, it crashes when its Xlib |> creates an invalid XID...unless Xlib does get this right, of course! |> (I haven't checked the source.) Actually, this happened to a customer of ours, but the other way round. They had a long-running client that did a lot of repetitive mapping/unmapping windows, which eventually translated into a sequence of grab/ungrab (Ihope I'm getting the details right). The server allcoated a new Server-private XID for each, and eventually overflowed its private bit into the client space. On the next free, it freed some random XID, much to the client's surprise, and all hell broke lose (well, it didn't actually; usually it froze). That took some debugging, especially as it takes hours to make it happen each time... The customer redesigned the application, I think. -- David Brooks dbrooks@osf.org Systems Engineering, OSF uunet!osf.org!dbrooks "Home is the bright cave under the hat." -- Lance Morrow
ekberg@asl.dl.nec.COM (Tom Ekberg) (02/15/91)
> Since there were no replies, I figure nobody knows. I checked Xlib and > could find nothing that does such a thing. I also discovered that there > is no mechanism for setting your own ID when making a request to create > a resource. The client just keeps sucking up valid ID's until there are > none left and them is completely helpless... If I am wrong about this, > please, PLEASE, somebody tell me before my next free Friday night, or > I might just create a support library to handle this. > > BTW: How does one go about suggesting to the powers that be that this > might be a good addition to Xlib. [for ex. XReCreateWindow(Display *, XID)] Sorry I wasn't on xpert when you sent your original message. As a former server implementer, I know that the number of bits available to a particular client are server-dependant. During connection setup, the server tells the client how many bits are available (see page 101 of the X11R4 protocol specification) as well as other server specific information. On the MIT sample server running on a Sun, I see 22 bits for each client. If the client allocates 1 resource every second, it will run out of resource bits in 48.6 days. I suppose you could create a specific server to give you more bits, to work around the problem. The following table, shows that 25 bits should be enough, but you will only be able to have at most 128 clients. Power 2**Power Num clients sec/day Days Years 22 4194304 1024 86400.00 48.55 0.1329 23 8388608 512 97.09 0.2658 24 16777216 256 194.18 0.5316 25 33554432 128 388.36 1.0633 26 67108864 64 776.72 2.1266 27 134217728 32 1553.45 4.2531 28 268435456 16 3106.89 8.5062 29 536870912 8 6213.78 17.0124 30 1073741824 4 12427.57 34.0248 31 2147483648 2 24855.13 68.0497 A better solution would be to have more smarts in the server to allocate resource ids better. A simple approach would be to have a bitmap which indicates the allocated/free state of each resource. With a 22 bit resource id, this would be 512k bytes for each client. The idea is that a long-running client would request this feature so the server would need this table only for certain clients. -- tom, ekberg@aslss02.asl.dl.nec.com (x3503)
gjw@spurr.wr.usgs.GOV (Gregory J. Woodhouse) (02/15/91)
What happens to a window if its parent is destroyed? More precisely, is the window ID freed, or does it turn into something akin to a zombie process under Unix (but one that hangs around forever)? Unless I'm misunderstanding the problem here, it seems that destroying orphaned windows (after the process that created them exits or its rootwindow is destroyed, of course) would solve your problem. Gregory Woodhouse U.S. Geological Survey gjw@ags.wr.usgs.gov (415) 329-4694
harkcom@spinach.pa.yokogawa.co.jp (02/15/91)
In article <HARKCOM.91Feb13172705@spinach.pa.Yokogawa.CO.JP> harkcom@spinach.pa.Yokogawa.CO.JP writes: A hearty "Thank you!" to everyone who mailed me info. The reults of my little sorti were: The allocation of resource ID's is handled by Xlib and there is no mechanism for reusing used ID's. One can easily be constructed though... The idea that a client could use up all of the valid ID's is pretty far-fetched. A client would have to be run continuosly for an awfully long time to use them all up. Pardon the bandwidth... Al
sarima@tdatirv.UUCP (Stanley Friesen) (02/16/91)
In article <9102140823.AA17984@lightning.McRCIM.McGill.EDU> mouse@lightning.mcrcim.mcgill.EDU writes: |... "should not" because creating 262144 resource objects over the life of |a program seems a bit excessive. But if you must, you *can* reuse |XIDs. (If you have a client that is perverse enough to want to have |more than 262144 resource objects in existence simultaneously, you will |most likely have trouble. You will likely have more trouble than just XID availablility! I seriously doubt that there are any existing X servers that can provide that many simultaneous X resources anyhow. [i.e. you will get a BAD_ALLOC error long before reaching 262144 XID's]. -- --------------- uunet!tdatirv!sarima (Stanley Friesen)
bret@codonics.COM (Bret Orsburn) (02/16/91)
>A better solution would be to have more smarts in the server to allocate >resource ids better. A simple approach would be to have a bitmap which >indicates the allocated/free state of each resource. With a 22 bit resource >id, this would be 512k bytes for each client. The idea is that a long-running >client would request this feature so the server would need this table only for >certain clients. Sounds like somebody's been working with virtual memory systems too long. :-) -- ------------------- bret@codonics.com uunet!codonics!bret Bret Orsburn
mouse@lightning.mcrcim.mcgill.EDU (02/16/91)
> What happens to a window if its parent is destroyed? It too is destroyed. > More precisely, is the window ID freed, or does it turn into > something akin to a zombie process under Unix (but one that hangs > around forever)? The protocol document simply says that "[t]he window and all inferiors are [...] destroyed". As far as I can see, nowhere does it explicitly say that destroying a window frees the ID of that window for reuse, though it seems to me that it's rather too obvious to need saying. > destroying orphaned windows (after the process that created them > exits or its rootwindow is destroyed, of course) Root windows can't be destroyed. der Mouse old: mcgill-vision!mouse new: mouse@larry.mcrcim.mcgill.edu
mikew@fx.com (Mike Wexler) (02/21/91)
harkcom@spinach.pa.yokogawa.co.jp writes: >The reults of my little sorti were: > The allocation of resource ID's is handled by Xlib and there > is no mechanism for reusing used ID's. One can easily > be constructed though... > The idea that a client could use up all of the valid ID's > is pretty far-fetched. A client would have to be run > continuosly for an awfully long time to use them all up. I have actually seen this happen on a real application under a very heavy workload. The application basically displays error messages. When a new one comes in it blinks (changes the background color back and forth). Because this was a Motif/Xt based program and at least one of the two colors used for blinking wasn't used by another widget, whenever the application blinked, it would use up a resource id. The application would fail after a few days of receiving messages constantly (fast enough so that the blink was continuous). -- Mike Wexler (mikew@fx.com)
john@acorn.co.uk (John Bowler) (02/25/91)
In article <9102140823.AA17984@lightning.McRCIM.McGill.EDU> mouse@lightning.mcrcim.mcgill.EDU writes: >> If not, clients which manage a lot of resources for a long time may >> run out of valid ID's. > >This seems like a silly thing to worry about. However, suppose someone >has some program running for weeks, creating and destroying windows, >GCs, whatever, and then as it gets old, it crashes when its Xlib >creates an invalid XID...unless Xlib does get this right, of course! >(I haven't checked the source.) Over christmas 1989 I ran Xvstress from the X Test Suite on (against :-)) one of our servers. When I came back (after about two weeks) the XVSpixmap test had failed with a BADIDChoice error. This test creates two pixmaps from bitmap data, sets a window background to one of the pixmaps, clears the window and frees both pixmaps. This was running in parallel with the other 16 or so tests, and the server had the ``normal'' number of client bits (ie 20). OK, this is a stress test, but it still seems entirely believable that some serious, long lived, applications will end up creating and freeing that number of resources over a long time period. I fixed up XVSpixmap.c by remembering display->resource_id before the test and resinstating it after (when the pixmaps had been freed). Although this is a hack which depends on knowledge of how the default display->resource_alloc works (display->resource_id++ :-)) the basic approach could be used in an application which knows that resource id's used in a particular sequence of operations will (all) have been freed at the end of that sequence. Detecting wrap-round is easy, but dealing with it is not. In general a (probably) small number of resource id's will be permanently allocated as an application starts up; it is essential that no attempt is made to reuse these when the resource id counter wraps round! There are obvious heuristics (wrap round to client_base+<small number> rather than client_base), but the application may also allocated permanent id's at arbitrary times while it is running... The general solution is to have a private display->resource_alloc procedure which ``knows'' whether an id is only required for a short time or for a long time. It can then maintain two counts, and use one for ``permanent'' allocations and the other (which will probably wrap round) for ``temporary'' ones. A possible implementation is for the caller of an Xlib routine to set a ``permanent_resource_id_required'' global flag before making a permanent allocation and reset it afterwards. John Bowler (jbowler@acorn.co.uk)
mouse@lightning.mcrcim.mcgill.EDU (03/02/91)
>>> If not, clients which manage a lot of resources for a long time may >>> run out of valid ID's. >> This seems like a silly thing to worry about. However, [...]. > Over christmas 1989 I ran Xvstress from the X Test Suite on (against > :-)) one of our servers. When I came back (after about two weeks) > the XVSpixmap test had failed with a BADIDChoice error. [...] > I fixed up XVSpixmap.c by remembering display->resource_id before the > test and resinstating it after (when the pixmaps had been freed). > [...] > Detecting wrap-round is easy, but dealing with it is not. In general > a (probably) small number of resource id's will be permanently > allocated as an application starts up; it is essential that no > attempt is made to reuse these when the resource id counter wraps > round! There are obvious heuristics (wrap round to > client_base+<small number> rather than client_base), but the > application may also allocated permanent id's at arbitrary times > while it is running... It seems to me that the solution to this is to reuse IDs right away. For example, keep a table of IDs in use - or perhaps just their resource_id values, which amounts to the same thing - and mark them as taken or free as they are used or freed. Then choosing a known-free ID is as simple as looking for a free slot in the table. Of course, it won't necessarily be implemented as a simple array. Something like this, perhaps: typedef struct Xlib_free_id { /* Xlib internal */ struct Xlib_free_id *link; XID xid; } FreeID; typedef struct _XDisplay { .... FreeID *free_ids; .... } Display; FreeID *free_free_ids = 0; XID new_xid(register Display *disp) /* Xlib internal */ { register FreeID *fid; register XID id; fid = disp->free_ids; if (fid) { disp->free_ids = fid->link; id = fid->xid; fid->link = free_free_ids; free_free_ids = fid; } else { id = (disp->resource_id++ << disp->resource_shift) | disp->resource_base; } return(id); } free_xid(register Display *disp, XID xid) /* Xlib internal */ { register FreeID *fid; fid = free_free_ids; if (fid) { free_free_ids = fid->link; } else { fid = malloc(sizeof(FreeID)); } fid->xid = xid; fid->link = disp->free_ids; disp->free_ids = fid; } This should give decent performance with small memory overhead (except for pathological programs that allocate a huge number of resources simultaneously and then free most of them, which I wouldn't worry about). The major damage I can see is that when you use a resource after freeing it, you are much less likely to get a BadWindow/BadGC/BadDrawable/etc error; instead, you will all too likely just get the wrong resource. Perhaps this algorithm should be used only on request (XHeavyResourceUsage(Display *, Boolean) perhaps?). It does require hooks in the FreePixmap, FreeGC, DestroyWindow, etc, routines. While this will not always detect when an XID can be reused (other clients can explicitly destroy resources, and multiple windows with effectively impossible-to-determine XIDs can be destroyed with a single call to XDestroyWindow), it will cover most cases. It will certainly come just about as close as any Xlib-based solution can. (Watching for DestroyNotify events might help the window situation a little bit.) > The general solution is to have a private display->resource_alloc > procedure which ``knows'' whether an id is only required for a short > time or for a long time. Aside from the strict stack discipline imposed on the temporary IDs by this, a serious program (as opposed to stress test) may not always know ahead of time whether a resource will stick around for a long time. (Some will, certainly; this is a reasonable approach for them.) der Mouse old: mcgill-vision!mouse new: mouse@larry.mcrcim.mcgill.edu
sarima@tdatirv.UUCP (Stanley Friesen) (03/05/91)
In article <5366@acorn.co.uk> john@acorn.co.uk (John Bowler) writes: <In article <9102140823.AA17984@lightning.McRCIM.McGill.EDU> mouse@lightning.mcrcim.mcgill.EDU writes: <>> If not, clients which manage a lot of resources for a long time may <>> run out of valid ID's. <>This seems like a silly thing to worry about. ... <Over christmas 1989 I ran Xvstress from the X Test Suite on (against :-)) <one of our servers. When I came back (after about two weeks) the <XVSpixmap test had failed with a BADIDChoice error. ... I have just run into another, more serious example of this problem. Twice now my xlock has bombed, apparently because it ran out of ID's. This is a potential security problem, since my workstation was sitting there, logged in, all weekend. [Sigh, I normally do log out, but I wanted to save an error message to show someone today]. xlock, at least, needs to be fixed to run indefinately. -- --------------- uunet!tdatirv!sarima (Stanley Friesen)