daveb@rtech.ARPA (Dave Brower) (05/13/85)
In a random discussion with a co-worker the other day, I recalled a
scheme for reducing the size text segments of programs used in an
otherwise excreble and unnamed operating system: Shared runtime
libraries. Though I have no current application, it seemed like an
interesting topic to bounce around.
It works like this: You link the pure code of common routines
(libc for example) into a module, keeping the symbol table around. This
module is loaded into a publicly accessible part of memory either at
boot time (like a daemon) or on demand by client programs.
When the client program is linked, it is resolved to the symbol table
of the previously linked shared library. When it fires up, somehow it
attaches to the runtime library and goes along its merry way, sharing
the code with all other active client programs. None of the clients has
the common code in its text segment. Note that this is NOT the much
discussed dynamic linking-- the symbols are completely resolved when the
client program is created by the loader.
There are some interesting benefits on systems with marginal resources,
e.g., workstations with < 300 mB disk and < 4mB memory:
(1) The size of all the clients AS STORED ON DISK would be reduced by
the amount of common library code used.
(2) The amount of physical AND virtual memory used for each process
could be reduced by the amount of common code used.
Some obvious difficultites are:
(1) Ensuring that the executables are resolved to the currently
loaded version of the library. This can probably be solved with version
and/or time stamps.
(2) OS and/or Hardware support.
On operating systems that support numerous segments, it is simple
enough to stash the library in some reserved and public segment. Things
are less clear on UNIX. My current sketchy thinking involves the use of
System V shared memory and runs along the following lines. It assumes
that a shared memory segment can be made both executable and read-only.
(1) A client program forks and execs a common library process, which
sets up a shared memory segment for the client to use. This involves
many library processes, but all presumably have the same text segment
and a set sticky bit so that it is not too expensive in terms of real
memory or execution time. The major cost is having a separate shared
memory segment for each client process, of which there are probably only
a small number available because of hardware limitations.
(2) A library daemon sets up a well-known shared memory segment that
can be used by multiple client programs. There is likely some limit on
the number of processes that may attach to a shared memory segment.
The major trick is in figuring out how to link the client programs.
Where would one locate the library code? It would need to be at the
same fixed location for all client programs. On an intellegent virtual
system, you could just stuff it a bunch of meg past the end of the space
you're really going to use. You're in trouble if the system creates
disk pages for the holes between your real data and the library. On
dumber systems, like System V's that swap, you'd probably want to start
at some low address, like 0. But you'd lose most of the size advantage
if the load image had to have a big physical hole where the library
segment was to be mapped. And you'd like to be able to have the client
be able to attach to multiple libraries. Then there's the nettlesome
problem of doing something smart with the impure data used by the
library. Well, I didn't say it would be simple!!!
Questions about the sanity of the idea belong in net.arch; ideas about
how to do this for real belong in net.unix-wizards(?). Please send
truly inspired flames by mail-- I'll summarize the best to the net :-)
WARNING: Don't try to reply via the .ARPA in the send path. It won't work.
--
{amdahl|dual|sun|zehntel}\ | Overheard in Berkeley:
{ucbvax|decvax}!mtxinu >!rtech!daveb | "Have you got an Apple?"
ihnp4!phoenix ___________/ | "Oh, yeah, of course."pc@unisoft.UUCP (n) (05/14/85)
<munch> The way to do solve these problems are very much the same as VMS does it 1. Provide kernel support for mapping in the shared library (which is after all just another sort of shared text, mapped at a different address, why not generalise this?) at exec time. 2. Make all the entry points into the library through a set of jump vectors (this way the entry point is always the same on many different versions of the library). This can be done by the same program that makes a stripped down library for linking. 3. Library data ... ugh ... the hard part, either rewrite the routines so that they don't use it (the easy way out, unfortunatly there are many that already do this ... ie stdio) or provide a separate, copy-on-reference segment, also at a known address to store the data in. Paul Campbell ..!ucbvax!unisoft!paul
chuck@dartvax.UUCP (Chuck Simmons) (05/16/85)
> In a random discussion with a co-worker the other day, I recalled a > scheme for reducing the size text segments of programs used in an > otherwise excreble and unnamed operating system: Shared runtime > libraries. Though I have no current application, it seemed like an > interesting topic to bounce around. Here at Dartmouth, we are slowly working on bringing our operating system into the 80's. Recently, two programmers around here rewrote the operating system to take advantage of the virtual mode hardware. Presently a few people are rewriting various compilers, the linker, the loader, and the common runtime routines. When (if) we are through, all the common runtime routines will be kept in a canonical location and will be sharable by all user programs. This is implemented as follows: First off, we have a table that tells where every defined object is within the common runtime library. Each compiler (pl1, pascal, basic, and maybe someday fortran and cobol) will know the offset within this table of each defined item. Each table entry contains a pointer to the actual location of the defined item. This extra level of indirection makes this scheme slightly less gross than it otherwise would be. Secondly, when a user program starts up, it opens up the library routines as a memory segment. The operating system handles the problems of allowing users to share this file in some magic manner. Thirdly, each user program requests another memory segment from the operating system. This other memory segment is used to store impure data and impure code for the common runtime routines. The common runtime routines know that the nth memory segment was created for them. Now I wonder... was anyone interested in that? Hopefully Gelhar will correct any blatant errors in my description... -- Chuck Simmons
robert@gitpyr.UUCP (Robert Viduya) (05/20/85)
> > In a random discussion with a co-worker the other day, I recalled a > scheme for reducing the size text segments of programs used in an > otherwise excreble and unnamed operating system: Shared runtime > libraries. Though I have no current application, it seemed like an > interesting topic to bounce around. > A couple of notes on the subject: A) The IBM PL/I Optimizing compiler running under VM/CMS has what they call a "transient" library. I'm not sure about all the details, but I believe this library can be set up (by the system administrator) to reside in a non-volative section of memory. I think OS/MVS has the same thing. B) I understand the new AT&T 7300 micro supports some type of shared libraries. Again, I'm not sure of all the details, but I have heard that you can't use the debugger with a program using the shared libraries. Anyone want to clarify? robert -- Robert Viduya Georgia Institute of Technology ...!{akgua,allegra,amd,hplabs,ihnp4,masscomp,ut-ngp}!gatech!gitpyr!robert ...!{rlgvax,sb1,uf-cgrl,unmvax,ut-sally}!gatech!gitpyr!robert
mzal@phoenix.UUCP (Mike Zaleski) (05/23/85)
>> I understand the new AT&T 7300 micro supports some type of >> shared libraries. Again, I'm not sure of all the details, >> but I have heard that you can't use the debugger with a >> program using the shared libraries. I'm not sure about the 7300 micro, but on our 3b2's we run a shared runtime library with our XXX CLASSIFIED XXX project. The basic idea is that a set of shareable routines is built with the loader and loaded by a special process at boot time (or manually while the system is running if no one is using the shared runtime library). The routines are accessed by small assembler routines which are in a different library which developers use to link their programs with. Because I'm not doing development on our project, I'm not sure about the debugger statement. What I *think* is the case is that SDB can be used, but is not very helpful on errors in the shared runtime library. We've been using this setup for a while and it works quite nicely. -- "The Model Citizen" Mike^Z Zaleski@Rutgers [ allegra, ihnp4 ] pegasus!mzal
adam@npois.UUCP (Adam V. Reed) (05/24/85)
The AT&T UNIX PC has shared libraries under System V. Check it out. Adam Reed ihnp4!npois!adam
eby@pegasus.UUCP (Robert P. Eby) (05/29/85)
As a followup to Mike's (phoenix!mzal) article, sdb is useable with a process running with (our version of 3B2) shared libraries. However, it will not handle addresses in the shared library in any easy to use fashion (you have to interpret the hex for yourself). The implementation we have was developed originally for the 3B5, and in theory, should be doable on any System V machine configured with shared memory. The things to watch out for are; - shared memory has to be executable. This is true by default on the 3B20 & 3B2, but requires a two line kernel mod on the 3B5 (not sure about the vax). - you have to be sure to attach the shared library at a fixed virtual address, which is usually differant between differant machine types. This is so that the ``interface routines'' can find it later when needed. Bob Eby pegasus!eby