liam@cs.qmw.ac.uk (William Roberts) (08/11/90)
I am currently trying to shoehorn a workable A/UX root partition into
10 Megabytes (well, it keeps me off the streets). During this exercise,
I have noticed that some binaries are nice and small, whilst others
are quite a lot larger for not much more functionality.
For example:
A/UX 2.0 SunOS 3.5
/bin/chmod 3604 bytes 19292 bytes (+15688)
/bin/chgrp 32812 bytes 46620 bytes (+13808)
The A/UX 2.0 version is using the shared libc, but so is the chgrp
and I don't understand why it is so much bigger.
When I do a "what" on these files, I get
/bin/chmod:
Copyright (c) 1984-85 AT&T-IS, 1985-87 UniSoft Corporation, All Rights
Reserved. {Apple version 1.3 89/10/10 08:10:18}
compatmode.c 1.2
/bin/chgrp:
Copyright (c) 1984-85 AT&T-IS, 1985-87 UniSoft Corporation, All Rights
Reserved. {Apple version 1.4 90/03/16 13:34:16}
compatmode.c 1.2
get_myaddress.c 1.1 86/09/24 Copyr 1984 Sun Micro
The curious thing is that "get_myaddress.c" is part of libc.a, and this
is supposed to be shared.... isn't it?
Turning to my trusty BSD 4.2 source code, I try compiling chmod.c and
chgrp.c, only to discover that I can't seem to match the size that
the distributed /bin/chmod achieves. Furthermore, there are no clear
examples in the A/UX 2.0 distribution (online manuals only) to say
exactly HOW to compile and link for shared libraries.
Looking at the files with "size", I get
Size of /bin/chgrp: 83988
Section Size Physical Address Virtual Address
.text 26596 328 328
.data 5824 4196652 4196652
.bss 16160 4202476 4202476
.lbt1206 26156 1206910976 1206910976
.lbd1207 9108 1207697408 1207697408
.lbb1207 80 1207706516 1207706516
.lib 64 0 0
Size of /bin/chmod: 38972
Section Size Physical Address Virtual Address
.text 2036 328 328
.data 1176 4196668 4196668
.bss 352 4197844 4197844
.lbt1206 26156 1206910976 1206910976
.lbd1207 9108 1207697408 1207697408
.lbb1207 80 1207706516 1207706516
.lib 64 0 0
So chgrp is bigger by 24000 bytes of text and 4700 bytes of data.
The sizes of the chmod.o and chgrp.o files are
chmod.o text=1284 data=156
chgrp.o text= 852 data=188
so all that extra is coming from unshared text & data extracted from
libc. Running "strings /bin/chgrp | wc" shows that there are 3771 bytes
of printable string (or thereabouts), most of which is the standard
error strings used by perror: chgrp uses perror, chmod doesn't.
Can anyone at Apple shed any light on all this? Have I just found a
way to make the A/UX 2.0 distribution smaller by 20k for most of
the 729 "executable" things listed in /FILES (a modest saving of 14 Meg!)?
PS: gcc makes all string constants part of the text segment (and hence
non-writeable) by default. The same effect could be achieved with
ld (I think) and would be valuable if applied to errlst.c in the
C library - this just defines the standard error strings and would
put that 2492 bytes of data into somewhere that is truly shared rather
that "copy-on-write" shared.
This treatment of string constants would be useful in lots of
places - just type "strings" on any executable over 40K...
--
William Roberts ARPA: liam@cs.qmw.ac.uk
Queen Mary & Westfield College UUCP: liam@qmw-cs.UUCP
Mile End Road AppleLink: UK0087
LONDON, E1 4NS, UK Tel: 071-975 5250 (Fax: 081-980 6533)abyss@Apple.COM (Ananthan Srinivasan) (08/14/90)
The shared library implementation of libc need not (and does not for A/UX 2.0) mean that the entire libc is shared. Only the routines within libc that are used frequently are combined to form the sharable part of libc. The rest of the routines are not shared. Another of the considerations to get the best out of shared library implementation is to keep files with large 'data' spaces out of the shared library - as 'data' is not shared. As a result, both perror and get-myaddress are not included in the sharable portion of libc. (perror uses the standard error strings which are not part of the sharable portion of libc because of its big contribution to the 'data' space - similarly all the routines in libc which use perror/the standard error strings have been left out of the sharable portion.) Sure if our compiler makes all the string constants part of the 'text' space, it could be used to get more savings and I will pass that info. on to our compiler experts. [As an aside: Even if the string constants were made part of the 'text' space, it is not obvious that 'error-related-routines' would be made sharable. In most cases they probably result in smaller executables, but then the 'text' portion of the shared library which is around in main memory has stuff which is used only in case of an error and thus programs/daemons/etc. which are used very frequently (if not all time) may end using main memory inefficiently.] Shared libraries are linked just like any other standard libraries (use '-lfoo' with ld to use the shared library /lib/libfoo assuming that the corresponding target portion of the library is in the /shlib directory). Srinivasan, A.B.
coolidge@casca.cs.uiuc.edu (John Coolidge) (08/14/90)
liam@cs.qmw.ac.uk (William Roberts) writes: >I am currently trying to shoehorn a workable A/UX root partition into >10 Megabytes (well, it keeps me off the streets). During this exercise, >I have noticed that some binaries are nice and small, whilst others >are quite a lot larger for not much more functionality. I can shed some light on this. I've got a version of the SVR3 shared library documentation, which appears to describe a superset of the A/UX (SVR2) shared library system. There's a section describing (but not going into detail) the construction of the shared libc. They mention that, for performance reasons (not binding in dead data and init code to applications) they split off "commonly-used" parts of libc and made them shared while keeping the rest statically linked. If you look at libc with nm, you'll find that about 195 entries have bizarre names (hft*) --- these are the shared parts. The other parts with normal names are normal library entries and will be linked statically. >Turning to my trusty BSD 4.2 source code, I try compiling chmod.c and >chgrp.c, only to discover that I can't seem to match the size that >the distributed /bin/chmod achieves. Furthermore, there are no clear >examples in the A/UX 2.0 distribution (online manuals only) to say >exactly HOW to compile and link for shared libraries. That's an understatement :-). I'd be dead in the water except for the persence of the SVR3 documentation I'm reading (for those interested, the 3B1 or the SGI IRIX documentation is very close). The gcc I've released will link with shared libraries, or you can build a test program and link with -lc_s and use cc -v to find out what's being linked in (the important bits are: use crt1 and crt2 first, crtn later, and shlib.ld LAST!). >[...] Running "strings /bin/chgrp | wc" shows that there are 3771 bytes >of printable string (or thereabouts), most of which is the standard >error strings used by perror: chgrp uses perror, chmod doesn't. >Can anyone at Apple shed any light on all this? Have I just found a >way to make the A/UX 2.0 distribution smaller by 20k for most of >the 729 "executable" things listed in /FILES (a modest saving of 14 Meg!)? It looks like it, indeed. Given the simplicity of perror it shouldn't even be hard to write one and built a shared version of it which incorporates the strings from libc. Perhaps I'll try it tonight... (Note: this is being written away from my A/UX machine; I reserve the right to discover that this is a bad idea :-)). --John -------------------------------------------------------------------------- John L. Coolidge Internet:coolidge@cs.uiuc.edu UUCP:uiucdcs!coolidge Of course I don't speak for the U of I (or anyone else except myself) Copyright 1990 John L. Coolidge. Copying allowed if (and only if) attributed. You may redistribute this article if and only if your recipients may as well.
domo@tsa.co.uk (Dominic Dunlop) (08/14/90)
In article <2631@sequent.cs.qmw.ac.uk> liam@cs.qmw.ac.uk (William Roberts) writes: > I am currently trying to shoehorn a workable A/UX root partition into > 10 Megabytes (well, it keeps me off the streets). During this exercise, > I have noticed that some binaries are nice and small, whilst others > are quite a lot larger for not much more functionality: > > A/UX 2.0 SunOS 3.5 > > /bin/chmod 3604 bytes 19292 bytes (+15688) > /bin/chgrp 32812 bytes 46620 bytes (+13808) > > The A/UX 2.0 version is using the shared libc, but so is the chgrp > and I don't understand why it is so much bigger. > >[And so on, at some length...] > > Can anyone at Apple shed any light on all this? Well, I'm not at Apple, but maybe I can shed some light on it nevertheless. If you get hold of a copy of _The UNIX System V Programmer's Guide_, AT&T/Prentice Hall, 1986 edition (this is important), ISBN 0-13-940438-4, you will find that chapter 8 describes COFF shared libraries, how to build and use them, and why you might wish not to include each and every one of your library functions is a shared library. This information may well be missing from the current edition of the manual: AT&T has, for a variety of reasons, including the inflexibility and other drawbacks of COFF shared libraries, switched to a new format, ELF (Extensible Linkage Format) for System V Release 4. So Apple's only just caught up with something that AT&T has already replaced. The information in the manual finishes with a commentary on the creation of a COFF shared library for the standard C library functions. A major drawback of COFF shared libraries is that, where a library function utilizes static data or provides globals, each and every program which links to the shared library must allocate data space for the function _whether it calls the function or not_. It is therefore necessary to examine every library function and determine a) whether it can be rewritten to cut down on its use of static and global data; and b) if not, whether it is used by a sufficient proportion of the programs linking to the library to merit cluttering up the data space of all the rest. If the answer to both questions is ``no'', then the function is not a suitable candidate for inclusion in the shared library. It should instead be part of a normal library, meaning that applications which use it incorporate a copy, and so increase in size. I suspect that get_myaddress(), identified by William as being in /bin/chgrp, is such a function. This being the case, if get_myaddress() were incorporated in the shared library, then, yes, /bin/chgrp and other programs using it would be smaller, but every other binary (and/or memory image, depending on whether the data is initialized or not) would be larger by the size of its data area. > [Could this be] > way to make the A/UX 2.0 distribution smaller by 20k for most of > the 729 "executable" things listed in /FILES (a modest saving of 14 Meg!)? Assuming Apple and/or their suppliers know what they're doing (a fair bet), probably not. -- Dominic Dunlop
coolidge@casca.cs.uiuc.edu (John Coolidge) (08/15/90)
coolidge@casca.cs.uiuc.edu (John Coolidge) writes: >It looks like it, indeed. Given the simplicity of perror it >shouldn't even be hard to write one and built a shared version >of it which incorporates the strings from libc. Perhaps I'll >try it tonight... (Note: this is being written away from my >A/UX machine; I reserve the right to discover that this is a bad >idea :-)). Well, I discovered that it's not a stunning idea, anyway :-). The size savings is only about 3K using shared perror. It's an interesting idea, though... What should be done is to build several shared libraries with pieces of libc. For instance, the AT&T notes mention a shared libnet which includes all the network code. If all the system programs were linked with such a library, only those who used networking would use the shared library (and hence pay the space penalty) --- the rest would just ignore it. The same goes for shared perror, shared math routines, etc. At any rate, I've build a shared liberr which includes perror, sys_nerr, and sys_errlist. I simply grab the strings from libc to implement sys_errlist. Testing indicates that my liberr is fully compatible with Apple's implementation (unless I missed something during testing :-)). Since it's small, I'm appending it to this posting. If you've got gcc, I advise (strongly!) compiling it with 'make CC=gcc'. If you don't have gcc, I advise getting it :-) Enjoy! --John -------------------------------------------------------------------------- #!/bin/sh # This is a shell archive (shar 3.10) # made 08/14/1990 17:56 UTC by coolidge@cs.uiuc.edu # Source directory /home/johnson/coolidge/liberr # # existing files WILL be overwritten # # This shar contains: # length mode name # ------ ---------- ------------------------------------------ # 854 -rw-r--r-- README # 717 -rw-r--r-- Makefile # 566 -rwxr-xr-x create_errlist # 609 -rw-r--r-- liberr.spec # 1141 -rw-r--r-- perror.c # 342 -rw-r--r-- pointer.c # 369 -rw-r--r-- shared.h # touch 2>&1 | fgrep '[-amc]' > /tmp/s3_touch$$ if [ -s /tmp/s3_touch$$ ] then TOUCH=can else TOUCH=cannot fi rm -f /tmp/s3_touch$$ # ============= README ============== sed 's/^X//' << 'SHAR_EOF' > README && XThis package will build a shared liberr_s which implements Xperror(3) and sys_nerr/sys_errlist. The error table is Xextracted from the one used in /lib/libc.a (I was in a hurry; Xbesides, it's easily kept up to date). X XA good part of the motivation for doing this was to release Xan example of working shared library code. The import Xmechanism for shared libraries used here should be a good Xexample of how to import external functions (here we Ximport errno, strlen, and writev). X XThere's one bit of minor customization needed --- if you Xwant the shared part of the library to be somewhere other Xthan /usr/local/shlib, you need to edit the #target Xdirective to point at the correct location. X XThis whole package is in the public domain. I'd appreciate Xsome mention if you use it in a package you distribute, Xthough... X X--John Coolidge Xcoolidge@cs.uiuc.edu SHAR_EOF chmod 0644 README || echo "restore of README fails" if [ $TOUCH = can ] then touch -am 0814125290 README fi # ============= Makefile ============== sed 's/^X//' << 'SHAR_EOF' > Makefile && X# Makefile for liberr_s, a shared library implementation of perror and X# sys_errlist/sys_nerr. X# X# John L. Coolidge, coolidge@cs.uiuc.edu 8/14/90 X X# Public domain (I'd appreciate a mention if you use it, though) X X XSHELL=/bin/sh X Xall: liberr_s X Xliberr_s: perror.o errlist.o pointer.o X mkshlib -s liberr.spec -h liberr_s.a -t liberr_s X Xerrlist.c: create_errlist /lib/libc_s.a X# Grab sys_errlist out of errlst.o (this makes us compatible with future X# versions --- just remake the library). X ar x /lib/libc_s.a errlst.o X $(SHELL) create_errlist X rm errlst.o X Xclean: X rm -f *.o X rm -f liberr_s.a liberr_s errlist.c X Xperror.c: shared.h Xerrlist.c: shared.h SHAR_EOF chmod 0644 Makefile || echo "restore of Makefile fails" if [ $TOUCH = can ] then touch -am 0814124490 Makefile fi # ============= create_errlist ============== sed 's/^X//' << 'SHAR_EOF' > create_errlist && X#! /bin/sh X X# create_errlist: Munch the strings table of errlst.o and produce a X# file errlst.c which implements the same strings. X# Intended to allow for shared library implementation under A/UX X X# John L. Coolidge, coolidge@cs.uiuc.edu 8/14/90 X X# Public domain (I'd appreciate a mention if you use it, though) X Xstrings errlst.o | awk -F% '\ XBEGIN {$num = 0; printf "char * sys_errlist[]={\n"}\ X{printf "\"%s\",\n",$1; $num = $num+1}\ XEND {printf "\"\"}; \n\nint sys_nerr = %d;\n",$num } X' | sed 's/gError 0/Error 0/' >>errlist.c SHAR_EOF chmod 0755 create_errlist || echo "restore of create_errlist fails" if [ $TOUCH = can ] then touch -am 0814124490 create_errlist fi # ============= liberr.spec ============== sed 's/^X//' << 'SHAR_EOF' > liberr.spec && X## liberr.spec: mkshlib specification file for shared liberr_s X## Intended to allow for shared library implementation under A/UX X X## John L. Coolidge, coolidge@cs.uiuc.edu 8/14/90 X## Public domain (I'd appreciate a mention if you use it, though) X X## Use the lowest addresses in user shared lib space for now. X#address .text 0x48000000 X#address .data 0x48040000 X X#target /usr/local/shlib/liberr_s X X#branch X perror 1 X X#objects X perror.o X errlist.o X pointer.o X X#init perror.o X errno _perror_errno X strlen _perror_strlen X writev _perror_writev SHAR_EOF chmod 0644 liberr.spec || echo "restore of liberr.spec fails" if [ $TOUCH = can ] then touch -am 0814124490 liberr.spec fi # ============= perror.c ============== sed 's/^X//' << 'SHAR_EOF' > perror.c && X/* X * perror.c: implement the perror(3) library function. X * Intended to allow for shared library implementation under A/UX X * X * John L. Coolidge, coolidge@cs.uiuc.edu 8/14/90 X * X * Public domain (I'd appreciate a mention if you use it, though) X */ X X#include <sys/types.h> X#include <sys/uio.h> X X#include "shared.h" X Xextern char* sys_errlist[]; Xextern int sys_nerr; X Xextern int errno; Xextern int strlen(); Xextern void writev(); X Xperror(char* s) X{ X struct iovec errvec[4]; X char* errmsg = sys_errlist[errno]; X if( errno > sys_nerr ) { X errvec[2].iov_base = "Unknown error"; X errvec[2].iov_len = 13; X } X else { X errvec[2].iov_base = (caddr_t)errmsg; X errvec[2].iov_len = strlen(errmsg); X } X errvec[3].iov_base = "\n"; X errvec[3].iov_len = 1; X if( !s ) writev(2, errvec+2, 2); X else { X errvec[0].iov_base = (caddr_t)s; X errvec[0].iov_len = strlen(s); X errvec[1].iov_base = ": "; X errvec[1].iov_len = 2; X writev(2, errvec, 4); X } X} SHAR_EOF chmod 0644 perror.c || echo "restore of perror.c fails" if [ $TOUCH = can ] then touch -am 0814124490 perror.c fi # ============= pointer.c ============== sed 's/^X//' << 'SHAR_EOF' > pointer.c && X/* X * pointer.c: Declaration of external pointers for perror. X * Intended to allow for shared library implementation under A/UX X * X * John L. Coolidge, coolidge@cs.uiuc.edu 8/14/90 X * X * Public domain (I'd appreciate a mention if you use it, though) X */ X X#include "shared.h" X Xvoid errno=0; Xvoid strlen=0; Xvoid writev=0; SHAR_EOF chmod 0644 pointer.c || echo "restore of pointer.c fails" if [ $TOUCH = can ] then touch -am 0814124490 pointer.c fi # ============= shared.h ============== sed 's/^X//' << 'SHAR_EOF' > shared.h && X/* X * shared.h: Redefine external symbols for importing. X * Intended to allow for shared library implementation under A/UX X * X * John L. Coolidge, coolidge@cs.uiuc.edu 8/14/90 X * X * Public domain (I'd appreciate a mention if you use it, though) X */ X X X#define errno (*_perror_errno) X#define strlen (*_perror_stren) X#define writev (*_perror_writev) SHAR_EOF chmod 0644 shared.h || echo "restore of shared.h fails" if [ $TOUCH = can ] then touch -am 0814124490 shared.h fi exit 0 -------------------------------------------------------------------------- John L. Coolidge Internet:coolidge@cs.uiuc.edu UUCP:uiucdcs!coolidge Of course I don't speak for the U of I (or anyone else except myself) Copyright 1990 John L. Coolidge. Copying allowed if (and only if) attributed. You may redistribute this article if and only if your recipients may as well.