[net.unix-wizards] Shared runtime libraries - SV shared memory?

daveb@rtech.ARPA (Dave Brower) (05/13/85)

  In a random discussion with a co-worker the other day, I recalled a
scheme for reducing the size text segments of programs used in an
otherwise excreble and unnamed operating system:  Shared runtime
libraries.  Though I have no current application, it seemed like an
interesting topic to bounce around.

  It works like this:  You link the pure code of common routines
(libc for example) into a module, keeping the symbol table around.  This
module is loaded into a publicly accessible part of memory either at
boot time (like a daemon) or on demand by client programs.

  When the client program is linked, it is resolved to the symbol table
of the previously linked shared library.  When it fires up, somehow it
attaches to the runtime library and goes along its merry way, sharing
the code with all other active client programs.  None of the clients has
the common code in its text segment.  Note that this is NOT the much
discussed dynamic linking-- the symbols are completely resolved when the
client program is created by the loader.

There are some interesting benefits on systems with marginal resources,
e.g., workstations with < 300 mB disk and < 4mB memory:

  (1)  The size of all the clients AS STORED ON DISK would be reduced by
the amount of common library code used.

  (2) The amount of physical AND virtual memory used for each process
could be reduced by the amount of common code used.

Some obvious difficultites are:

  (1)  Ensuring that the executables are resolved to the currently
loaded version of the library.  This can probably be solved with version
and/or time stamps.

  (2)  OS and/or Hardware support.

  On operating systems that support numerous segments, it is simple
enough to stash the library in some reserved and public segment.  Things
are less clear on UNIX.  My current sketchy thinking involves the use of
System V shared memory and runs along the following lines.  It  assumes
that a shared memory segment can be made both executable and read-only.

  (1) A client program forks and execs a common library process, which
sets up a shared memory segment for the client to use.  This involves
many library processes, but all presumably have the same text segment
and a set sticky bit so that it is not too expensive in terms of real
memory or execution time.  The major cost is having a separate shared
memory segment for each client process, of which there are probably only
a small number available because of hardware limitations.

  (2) A library daemon sets up a well-known shared memory segment that
can be used by multiple client programs.  There is likely some limit on
the number of processes that may attach to a shared memory segment.

The major trick is in figuring out how to link the client programs.
Where would one locate the library code?  It would need to be at the
same fixed location for all client programs.  On an intellegent virtual
system, you could just stuff it a bunch of meg past the end of the space
you're really going to use.  You're in trouble if the system creates
disk pages for the holes between your real data and the library.  On
dumber systems, like System V's that swap, you'd probably want to start
at some low address, like 0.  But you'd lose most of the size advantage
if the load image had to have a big physical hole where the library
segment was to be mapped.  And you'd like to be able to have the client
be  able to attach to multiple libraries.  Then there's the nettlesome
problem of doing something smart with the impure data used by the
library.  Well, I didn't say it would be simple!!!

Questions about the sanity of the idea belong in net.arch; ideas about
how to do this for real belong in net.unix-wizards(?).  Please send
truly inspired flames by mail-- I'll summarize the best to the net :-)

WARNING: Don't try to reply via the .ARPA in the send path.  It won't work.
-- 
{amdahl|dual|sun|zehntel}\			| Overheard in Berkeley:
{ucbvax|decvax}!mtxinu    >!rtech!daveb 	| "Have you got an Apple?"
ihnp4!phoenix ___________/			| "Oh, yeah, of course."

pc@unisoft.UUCP (n) (05/14/85)

<munch>
	The way to do solve these problems are very much the same as VMS does it

	1.	Provide kernel support for mapping in the shared library (which 
		is after all just another sort of shared text, mapped at a
		different address, why not generalise this?) at exec time.

	2.	Make all the entry points into the library through a set of
		jump vectors (this way the entry point is always the same on
		many different versions of the library). This can be done by
		the same program that makes a stripped down library for linking.

	3.	Library data ... ugh ... the hard part, either rewrite the 
		routines so that they don't use it (the easy way out,
		unfortunatly there are many that already do this ... ie stdio)
		or provide a separate, copy-on-reference segment, also at a
		known address to store the data in.



			Paul Campbell		..!ucbvax!unisoft!paul

chuck@dartvax.UUCP (Chuck Simmons) (05/16/85)

>   In a random discussion with a co-worker the other day, I recalled a
> scheme for reducing the size text segments of programs used in an
> otherwise excreble and unnamed operating system:  Shared runtime
> libraries.  Though I have no current application, it seemed like an
> interesting topic to bounce around.

Here at Dartmouth, we are slowly working on bringing our operating system
into the 80's.  Recently, two programmers around here rewrote the
operating system to take advantage of the virtual mode hardware.  Presently
a few people are rewriting various compilers, the linker, the loader, and
the common runtime routines.  When (if) we are through, all the common
runtime routines will be kept in a canonical location and will be sharable
by all user programs.

This is implemented as follows:  First off, we have a table that tells
where every defined object is within the common runtime library.  Each
compiler (pl1, pascal, basic, and maybe someday fortran and cobol) will
know the offset within this table of each defined item.  Each table entry
contains a pointer to the actual location of the defined item.  This extra
level of indirection makes this scheme slightly less gross than it otherwise
would be.  

Secondly, when a
user program starts up, it opens up the library routines as a memory
segment.  The operating system handles the problems of allowing users
to share this file in some magic manner.  Thirdly, each user program requests
another memory segment from the operating system.  This other memory segment
is used to store impure data and impure code for the common runtime routines.
The common runtime routines know that the nth memory segment was created
for them.

Now I wonder...  was anyone interested in that?  Hopefully Gelhar
will correct any blatant errors in my description...

-- Chuck Simmons

     

robert@gitpyr.UUCP (Robert Viduya) (05/20/85)

> 
>   In a random discussion with a co-worker the other day, I recalled a
> scheme for reducing the size text segments of programs used in an
> otherwise excreble and unnamed operating system:  Shared runtime
> libraries.  Though I have no current application, it seemed like an
> interesting topic to bounce around.
> 

A couple of notes on the subject:

	A) The IBM PL/I Optimizing compiler running under VM/CMS has what
	   they call a "transient" library.  I'm not sure about all the
	   details, but I believe this library can be set up (by the
	   system administrator) to reside in a non-volative section of
	   memory.  I think OS/MVS has the same thing.
	
	B) I understand the new AT&T 7300 micro supports some type of
	   shared libraries.  Again, I'm not sure of all the details,
	   but I have heard that you can't use the debugger with a
	   program using the shared libraries.

Anyone want to clarify?

			robert
-- 
Robert Viduya
Georgia Institute of Technology

...!{akgua,allegra,amd,hplabs,ihnp4,masscomp,ut-ngp}!gatech!gitpyr!robert
...!{rlgvax,sb1,uf-cgrl,unmvax,ut-sally}!gatech!gitpyr!robert

mzal@phoenix.UUCP (Mike Zaleski) (05/23/85)

>> I understand the new AT&T 7300 micro supports some type of
>> shared libraries.  Again, I'm not sure of all the details,
>> but I have heard that you can't use the debugger with a
>> program using the shared libraries.

I'm not sure about the 7300 micro, but on our 3b2's we run
a shared runtime library with our XXX CLASSIFIED XXX project.
The basic idea is that a set of shareable routines is built
with the loader and loaded by a special process at boot time
(or manually while the system is running if no one is using the
shared runtime library).  The routines are accessed by small
assembler routines which are in a different library which developers
use to link their programs with.

Because I'm not doing development on our project, I'm not sure
about the debugger statement.  What I *think* is the case is that
SDB can be used, but is not very helpful on errors in the shared
runtime library.

We've been using this setup for a while and it works quite nicely.

-- "The Model Citizen" Mike^Z
   Zaleski@Rutgers  [ allegra, ihnp4 ] pegasus!mzal

adam@npois.UUCP (Adam V. Reed) (05/24/85)

The AT&T UNIX PC has shared libraries under System V. Check it out.
						Adam Reed
						ihnp4!npois!adam

eby@pegasus.UUCP (Robert P. Eby) (05/29/85)

As a followup to Mike's (phoenix!mzal) article, sdb is useable with a
process running with (our version of 3B2) shared libraries.  However, it
will not handle addresses in the shared library in any easy to use
fashion (you have to interpret the hex for yourself).

The implementation we have was developed originally for the 3B5, and
in theory, should be doable on any System V machine configured with
shared memory.  The things to watch out for are;
	- shared memory has to be executable.  This is true by default
	  on the 3B20 & 3B2, but requires a two line kernel mod on the
	  3B5 (not sure about the vax).
	- you have to be sure to attach the shared library at a fixed
	  virtual address, which is usually differant between differant
	  machine types.  This is so that the ``interface routines'' can
	  find it later when needed.

				Bob Eby
				pegasus!eby