daveb@rtech.ARPA (Dave Brower) (05/19/85)
> In a random discussion with a co-worker the other day, I recalled a > scheme for reducing the size text segments of programs used in an > otherwise excreble and unnamed operating system: Shared runtime > libraries. I've received references to systems that have kernel & hardware support for this sort of thing (the multiplicity of segments noted in the original posting): Multics, VMS, HP-9000, Macintosh, even someone who recognized my comment about Perkin-Elmer's (nee Interdata) OS/32... A summary of the responses follows, but what I'm interested in is how to get shared libraries on the run-of-the mill System V that does NOT have any support beyond shared memory. In particular, how would the average Joe do it with the OS he got off the shelf? Examples are systems that are difficult (or impossible) to obtain kernel source for: Amdahl UTS (but who cares how big you are!) 3b5 3b2 ATT 7300 Pyramid in SV "universe" ========================================================================== **** "Letters, we get your letters, we get your letters every day..." **** ========================================================================== [Assume an implicit :-) in these comments...its not that serious really!] From @mtxinu.UUCP:mirror!prism!zrm Sat May 18 13:02:50 1985 If you would like to have the definitive overview of shared program text systems, read the Multics book by Eliot Organick. I afraid I don't have mine near me and so cannot tell you the publisher. ["You never outgrow your need for Multics..."] Briefly, Multics uses the concept of a linkage section -- a bunch of pointers to indirect through to make a procedure call. The Macintosh, and lots of ad hoc overlay systems do this too. When you try to call through an invalid pointer, the system detects it, and in Multics terminology, resolves a "linkage fault." By now you may have realized that all this can be done at runtime, so that when new libraries are installed, all the applications in the system don't have to be relinked. [I think I'd be willing to live with the need to relink, though jumptables reduce the need...] What remains is the problem of the data area for shared libraries -- where do you put it even if you know you are going to use it? On a VAX, say, I could imagine bashing my data segment's page table on entry and exit from a shared library do that every shared library I'm using is free to assume its data area begins at 0. No longjumps across library boudaries! [OK on the surface, but what about a library routine that is interrupted, and a handler in the user space that longjumps out? Needed on BSD because of the non-interruptable system calls] From @amdahl.UUCP:hplabs!hpfcla!ajs Fri May 17 17:05:31 1985 HPUX (Hewlett-Packard Bell-derived and largely compatible UNIX) supports shared libraries much the way you envision them, with the results you would hope for. [The 9000 architecture descriptions I've seem make it look like an ideal candidate for the next Multics port. Each program can have up to 1000 segments of up to 512k each. Unfortunately, there doesn't appear to be hardware paging support, so if you want a swappable segment, your access time is 4 or 5 times slower. I've heard rumours that going through SV shared memory also takes a performance hit. Does anyone know?] From: @amdahl.UUCP:ihnp4!ptsfa!jmc I believe such a scheme was presented in the January, 1984 Uniforum in Washington. You might take a look at the proceedings to see what they did. Unfortunately, they were not allowed to tell all due to corporate realities, so some wheel reinventing will be necessary. [This looks interesting. Anyone with a copy of those proceedings who'd like to summarize? Maybe this one below is it?] From: @mtxinu.UUCP:mips!mash (John Mashey) This actually works & can be done fairly easily - A guy who used to work for me hacked this sort of thing together for CT MiniFrames in about a week. A tuned version of the method is used for the libraries on an ATT UNIX PC - saved immense amount of space (esp. on 10MB disk!); even more important, program loads were much faster, especially given the baggage that 68K programs carry around with them. [Do these implementations require kernel mods? Can I link to these libraries on my 7300 as delivered? Can you post it to net.sources?] There are interesting tradeoffs among 1) speed 2) flexibility and 3) administration nightmarishness amongst the variants of this method that I've heard of. [Administrative nightmarishness is a MAJOR problem, hence the desirability of hardware support.] From pc@unisoft.UUCP (n) Tue May 14 08:29:27 1985 The way to do solve these problems are very much the same as VMS does it: [Others speak below about vms...] 1. Provide kernel support for mapping in the shared library (which is after all just another sort of shared text, mapped at a different address, why not generalise this?) at exec time. [cheating... I want it on a vanilla, or even UniPlus+ SV] 2. Make all the entry points into the library through a set of jump vectors (this way the entry point is always the same on many different versions of the library). This can be done by the same program that makes a stripped down library for linking. [everybody loves jump tables...] 3. Library data ... ugh ... the hard part, either rewrite the routines so that they don't use it (the easy way out, unfortunatly there are many that already do this ... ie stdio) or provide a separate, copy-on-reference segment, also at a known address to store the data in. [For all our love of uniform address space machines like the VAX or 32032 or 68k, segments keep popping up as a useful concept. But only if they can be buried invisible to the user program, which stills wants to see abig array of bytes. This leads to twiddling with the page tables...] From: @ihnp4.UUCP:uiucdcs!liberte (Daniel LaLiberte) The Macintosh uses two concepts that might aid in implementing shared libraries in Unix... Instead of direct pointers to objects that might move, a "handle" to a master pointer is used instead. The master pointer never moves. Objects that are pointed to are, in fact, code segments. [There are difficulties in doing it mechanically, e.g, Sumacc doesn't yet support it. Mac segmentation is so programmer visible that I've given up and and saving my pennies for a 7300.] From @mtxinu.UUCP:decvax!jmcg Sat May 18 06:11:50 1985 The idea of shared libraries or putting at least some of the routines like printf into the kernel has been kicking around for a long while. Bill Jolitz claimed to have implemented it for the NS 32000. One difficult topic you neglected is where to put the library routines' global variables (errno, for instance) so that the library code can always refer to them in the same place. If you're willing to make a lot of modifications to the C compiler, you can overcome this and other problems. [Or the iob[] array. An important thing is that we DON'T want to stick more in the kernal; We just want to be able for user processes to share code. This would conceivably let us take stuff OUT of the O/S, thus reducing the size of the non-pageable kernel] Eventually, you end up with something as tangled as VMS shared libraries. Not a pleasant prospect. [Doesn't anybody have a kind word for VMS? I mean, at least they've GOT these features, like shared libraries, asynchronous traps, and a real lock manager.] From chuck@dartvax.UUCP (Chuck Simmons) Thu May 16 00:20:50 1985 Here at Dartmouth, we are slowly working on bringing our operating system into the 80's ... When (if) we are through, all the common runtime routines will be kept in a canonical location and will be sharable by all user programs. First off, we have a table that tells where every defined object is within the common runtime library. Each compiler ... will know the offset within this table of each defined item. Secondly, when a user program starts up, it opens up the library routines as a memory segment. The operating system handles the problems of allowing users to share this file in some magic manner. Thirdly, each user program requests another memory segment from the operating system. This other memory segment is used to store impure data and impure code for the common runtime routines. The common runtime routines know that the nth memory segment was created for them. [Wasn't Multics the same general vintage as DTSS :-)] ======================================================= **** Keep those cards and letters coming in folks! **** ======================================================= WARNING: Don't try to reply via the .ARPA in the send path. It won't work. -- {amdahl|dual|sun|zehntel}\ | Overheard in Berkeley: {ucbvax|decvax}!mtxinu >!rtech!daveb | "Have you got an Apple?" ihnp4!phoenix ___________/ | "Oh, yeah, of course."