tom@opera.chorus.fr (Michel Tombroff) (12/15/90)
Here is a question related to the "semantics" of the brk()/sbrk()
UNIX system calls. If this has already been discussed on the NewsNet,
can somebody just send me pointers. Thanks.
sbrk(incr) : incr more bytes are added to the program's data
and a pointer to the start of the new area is returned.
In all the implementation of sbrk() I have seen, the data area
of a process is a region that grows towards increasing virtual addresses.
This means that successive calls to sbrk() with positive increments
will return increasing addresses.
My questions are the following:
1/ is this feature part of the "semantics" of the sbrk() system
call, or is it a property of most actual UNIX implementations
to have bss segments growing upward?
2/ would it make sense to implement a bss into a region growing towards
lower addresses. Suppose for instance my bss is a region whose
most positive virtual address is fixed, and trying to extend its
size would make it extend towards a smaller address:
before after sbrk(SIZE);
_____________ _____________
| | | | <-- high virtual addr
| | | |
| | | |
| | | |
| BSS | | BSS |
| | | |
| | | |
| | | |
| | | |
------------- | |
| |
| |
| |
SIZE bytes | |
added | |
| |
| |
------------- <-- low virtual addr
Note: we suppose here that there are no problems of overlapping, ...
with the text/stack regions.
3/ in the case this makes sense, what is the amount of UNIX
code ( I mean commands, programs, utilities, X servers ) that would
have to be rewritten. In other words, how much of UNIX programs
make the assumption of "growing upward" memory regions.
Thanks,
-tom
________________________________________________________________________
Michel Tombroff Tel: +33 (1) 30 64 82 77
Chorus Systemes Fax: +33 (1) 30 57 00 66
6 Av. Gustave Eiffel Email: tom@chorus.fr
78182 Saint-Quentin-En-Yvelines
Francebarmar@think.com (Barry Margolin) (03/09/91)
This question came up at work yesterday: when sbrk(2) is used to increase
the size of the data segment, is the newly-allocated memory guaranteed to
be empty? The man page (on SunOS 4.1.1 and Ultrix 3.x) doesn't say
anything about the contents of the new memory. For security reasons it
makes sense that data from an old process should not slip into another
process's address space, but this is Unix so I don't think such assumptions
are always safe :-(
Assuming most implementations provide zero-filled memory, is this generally
done using some kind of optimization of all-zero memory pages, or does it
have to explicitly zero lots of memory and swap space pages (assuming you
give sbrk() a large value)?
--
Barry Margolin, Thinking Machines Corp.
barmar@think.com
{uunet,harvard}!think!barmargwyn@smoke.brl.mil (Doug Gwyn) (03/11/91)
In article <1991Mar8.180132.12025@Think.COM> barmar@think.com (Barry Margolin) writes: >This question came up at work yesterday: when sbrk(2) is used to increase >the size of the data segment, is the newly-allocated memory guaranteed to >be empty? You should not assume that. If you want it to contain 0-valued bytes, use memset() to fill it with 0 bytes.
boyd@necisa.ho.necisa.oz.au (Boyd Roberts) (03/11/91)
In article <1991Mar8.180132.12025@Think.COM> barmar@think.com (Barry Margolin) writes: > >Assuming most implementations provide zero-filled memory, is this generally >done using some kind of optimization of all-zero memory pages, or does it >have to explicitly zero lots of memory and swap space pages (assuming you >give sbrk() a large value)? > On System V paging systems the right thing is done. The process' page tables are expanded and the corresponding DBDs (disk block descriptors) are marked to be demand-fill zero. The memory isn't allocated until a validity fault occurs on the first reference to the page. A page is attached to the pte, then it's zeroed, then the process continues. You'd hope that other paging systems would use the same scheme. It's simple and it makes sense. Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...''
richard@aiai.ed.ac.uk (Richard Tobin) (03/11/91)
In article <1991Mar8.180132.12025@Think.COM> barmar@think.com (Barry Margolin) writes: >This question came up at work yesterday: when sbrk(2) is used to increase >the size of the data segment, is the newly-allocated memory guaranteed to >be empty? There's one case where it amlost certainly won't be zero, which is when memory has been previously alocated and released (eg by calling sbrk() with a negative argument). So it's unwise to rely on it even if the operating system allocates zeroed (or zero-fill-when-referenced) pages. -- Richard -- Richard Tobin, JANET: R.Tobin@uk.ac.ed AI Applications Institute, ARPA: R.Tobin%uk.ac.ed@nsfnet-relay.ac.uk Edinburgh University. UUCP: ...!ukc!ed.ac.uk!R.Tobin
guy@auspex.auspex.com (Guy Harris) (03/12/91)
>You'd hope that other paging systems would use the same scheme. It's >simple and it makes sense. Yup, BSD and the SunOS 4.x/System V Release 4 VM system (which is different from the paging S5R2/S5R3 system described) also give you zero-fill-on-demand pages when you "sbrk()". V6, V7 and non-paging S3 and S5 zeroed the data out explicitly, as I remember.
guy@auspex.auspex.com (Guy Harris) (03/13/91)
>There's one case where it amlost certainly won't be zero, which is when >memory has been previously alocated and released (eg by calling sbrk() >with a negative argument). So it's unwise to rely on it even if the >operating system allocates zeroed (or zero-fill-when-referenced) pages. Umm, why would it not be zero in that situation? If the memory has truly been released, as in "handed back to the kernel", it should be re-zeroed if allocated to a process again. (I.e., calling "sbrk()" with a negative argument, in most if not all versions of UNIX, doesn't just set some user-mode pointer so that the memory stays in the address space of the process.)
richard@aiai.ed.ac.uk (Richard Tobin) (03/13/91)
In article <6582@auspex.auspex.com> guy@auspex.auspex.com (Guy Harris) writes: >Umm, why would it not be zero in that situation? If the memory has >truly been released, as in "handed back to the kernel", it should be >re-zeroed if allocated to a process again. Yes, of course. > (I.e., calling "sbrk()" with >a negative argument, in most if not all versions of UNIX, doesn't just >set some user-mode pointer so that the memory stays in the address space >of the process.) Even if it does this, will it not release memory in page-size units? A quick check on a Sun, Sequent, BSD Vax and Orion all reveal that the bytes before the first page boundary in the (re-)allocated space are unzeroed (and indeed contain the data unchanged from before). So you can't rely on sbrk() space to be completely zeroed, unless you're sure nothing has already allocated and freed memory. And you can't be sure of that (maybe the system did it before calling main(), for example). While experimenting, I was reminded of another hazard of sbrk(). For example, this program produces a segmentation fault on a Sun4 under SunOS 4.1: #include <stdio.h> extern char *sbrk(); main() { sbrk(100000); printf("hello world\n"); sbrk(-100000); printf("hello again\n"); exit(0); } -- Richard -- Richard Tobin, JANET: R.Tobin@uk.ac.ed AI Applications Institute, ARPA: R.Tobin%uk.ac.ed@nsfnet-relay.ac.uk Edinburgh University. UUCP: ...!ukc!ed.ac.uk!R.Tobin
boyd@necisa.ho.necisa.oz.au (Boyd Roberts) (03/15/91)
In article <4312@skye.ed.ac.uk> richard@aiai.UUCP (Richard Tobin) writes: |While experimenting, I was reminded of another hazard of sbrk(). For |example, this program produces a segmentation fault on a Sun4 under |SunOS 4.1: | | #include <stdio.h> | | extern char *sbrk(); | | main() | { | sbrk(100000); | printf("hello world\n"); | sbrk(-100000); | printf("hello again\n"); | exit(0); | } Well, I thought everybody new that sbrk(2) and malloc(3) just don't mix. Odds on printf(3) calls malloc and gets some memory whose address is > sbrk(0) + 100000*. The second printf references this memory after the second sbrk and it's length violation city! malloc(3) and sbrk(2) -- be careful out there. Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...'' * Obviously rounded up to the next page.
stan@Dixie.Com (Stan Brown) (03/15/91)
>While experimenting, I was reminded of another hazard of sbrk(). For >example, this program produces a segmentation fault on a Sun4 under >SunOS 4.1: ** Program Deleted *** OK, I give up. Why ? -- Stan Brown P. c. Design 404-363-2303 Ataant Ga. (emory|gatech|uunet) rsiatl!sdba!stan "vi forever"
mouse@thunder.mcrcim.mcgill.edu (der Mouse) (03/15/91)
In article <6582@auspex.auspex.com>, guy@auspex.auspex.com (Guy Harris) writes: >> There's one case where it amlost certainly won't be zero, which is >> when memory has been previously alocated and released (eg by calling >> sbrk() with a negative argument). > Umm, why would it not be zero in that situation? If the memory has > truly been released, as in "handed back to the kernel", it should be > re-zeroed if allocated to a process again. True. The problem is that releasing of memory, in this sense, happens with page granularity. And unless your page size is one byte, this means it's possible for some garbage to be left at the end of the last valid page, above the sbrk()-set end pointer. der Mouse old: mcgill-vision!mouse new: mouse@larry.mcrcim.mcgill.edu
mouse@thunder.mcrcim.mcgill.edu (der Mouse) (03/15/91)
In article <2045@necisa.ho.necisa.oz.au>, boyd@necisa.ho.necisa.oz.au (Boyd Roberts) writes: > In article <4312@skye.ed.ac.uk> richard@aiai.UUCP (Richard Tobin) writes: >> While experimenting, I was reminded of another hazard of sbrk(). >> For example, this program produces a segmentation fault on a Sun4 >> under SunOS 4.1: [edited -dM] >> sbrk(100000); >> printf("hello world\n"); >> sbrk(-100000); >> printf("hello again\n"); > Well, I thought everybody new that sbrk(2) and malloc(3) just don't > mix. Odds on printf(3) calls malloc and gets some memory whose > address is > sbrk(0) + 100000*. The second printf references this > memory after the second sbrk and it's length violation city! You are correct, of course, but on SunOS 4.1 it's even worse than this. Apparently Sun decided that nobody wants to use sbrk and malloc in the same program. malloc and sbrk are incompatble. Even when all arguments to sbrk are positive multiples of the page size, you *still* get a core dump when you mix sbrk() and malloc()! I don't know what possessed them to introduce this bug; I guess it's just another instance of Sun deciding they didn't care about compatibility with the rest of the world. I don't know why they even bothered to provide sbrk(), because nearly everything winds up using some library routine or other which eventually calls malloc(). (If I sound annoyed, it's because I am. I spent days trying to track down that core dump....) der Mouse old: mcgill-vision!mouse new: mouse@larry.mcrcim.mcgill.edu
richard@aiai.ed.ac.uk (Richard Tobin) (03/15/91)
In article <2045@necisa.ho.necisa.oz.au> boyd@necisa.ho.necisa.oz.au (Boyd Roberts) writes: >| sbrk(100000); >| printf("hello world\n"); >| sbrk(-100000); >Well, I thought everybody new that sbrk(2) and malloc(3) just >don't mix. There is no reason for sbrk() and malloc() to "not mix" provided you're careful not to free memory you didn't allocate. On the other hand, there's not usually a good reason to not just use malloc(). >Odds on printf(3) calls malloc and gets some memory >whose address is > sbrk(0) + 100000*. Yes, this is what happens. Malloc() uses sbrk() to get some memory, and the user's sbrk() frees it. -- Richard -- Richard Tobin, JANET: R.Tobin@uk.ac.ed AI Applications Institute, ARPA: R.Tobin%uk.ac.ed@nsfnet-relay.ac.uk Edinburgh University. UUCP: ...!ukc!ed.ac.uk!R.Tobin
boyd@necisa.ho.necisa.oz.au (Boyd Roberts) (03/19/91)
In article <4319@skye.ed.ac.uk> richard@aiai.UUCP (Richard Tobin) writes: > >There is no reason for sbrk() and malloc() to "not mix" provided you're >careful not to free memory you didn't allocate. On the other hand, >there's not usually a good reason to not just use malloc(). > Well there may be no reason, but in reality most malloc(3) implementations assume that it and no one else has called sbrk(2). I'm sure that pre-System V implementations were known to break if you mixed sbrk(2) and malloc(3). I could be wrong, but I know I've been burnt by this before. Don't call sbrk(2) after malloc(3). Core dumps assured. Boyd Roberts boyd@necisa.ho.necisa.oz.au ``When the going gets wierd, the weird turn pro...''
gwyn@smoke.brl.mil (Doug Gwyn) (03/19/91)
In article <2048@necisa.ho.necisa.oz.au> boyd@necisa.ho.necisa.oz.au (Boyd Roberts) writes: >Well there may be no reason, but in reality most malloc(3) implementations >assume that it and no one else has called sbrk(2). I'm sure that pre-System V >implementations were known to break if you mixed sbrk(2) and malloc(3). The SVR2 version, at least once I got through fixing its bugs, was VERY careful to allow incrementing uses of sbrk() to be interleaved with malloc(). (I was impressed!) Of course, decrementing the break would cause problems. Incidentally, the only malloc/free implementation I have ever seen that didn't have at least one bug is the one in (the draft I reviewed of) K&R 2nd Edition.
clewis@ferret.ocunix.on.ca (Chris Lewis) (03/20/91)
In article <2048@necisa.ho.necisa.oz.au> boyd@necisa.ho.necisa.oz.au (Boyd Roberts) writes: >In article <4319@skye.ed.ac.uk> richard@aiai.UUCP (Richard Tobin) writes: >>There is no reason for sbrk() and malloc() to "not mix" provided you're >>careful not to free memory you didn't allocate. The big trouble is knowing when your sbrk isn't going to free malloc'd memory. >Well there may be no reason, but in reality most malloc(3) implementations >assume that it and no one else has called sbrk(2). I'm sure that pre-System V >implementations were known to break if you mixed sbrk(2) and malloc(3). Shore did. I remember porting V6 stuff to V7, converting to stdio along the way. Many things blew out royally. -- Chris Lewis, clewis@ferret.ocunix.on.ca or ...uunet!mitel!cunews!latour!ecicrl!clewis Psroff support: psroff-request@eci386.uucp, or call 613-832-0541 (Canada) **** somebody's mailer is appending .bitnet to my From: address. If you
kpv@ulysses.att.com (Phong Vo[drew]) (03/21/91)
In article <2048@necisa.ho.necisa.oz.au>, boyd@necisa.ho.necisa.oz.au (Boyd Roberts) writes: > In article <4319@skye.ed.ac.uk> richard@aiai.UUCP (Richard Tobin) writes: > > > Well there may be no reason, but in reality most malloc(3) implementations > assume that it and no one else has called sbrk(2). I'm sure that pre-System V > implementations were known to break if you mixed sbrk(2) and malloc(3). > Reality points the other way. Most malloc implementations that I've seen do try to handle sbrk when this is used to obtain space. Whether they do it right is another issue. In any case, sbrk should never be used directly by an application to reduce space. This is because free blocks are usually kept in data structures that involved links of some type and unpredictable consequences can arise when the space pointed to by these links suddenly disappear. I do know of one malloc implementation from a friend that uses no links. Its free strategy is rather wasteful but it's a great malloc for programs that manipulate space in a stack-like manner.
richard@aiai.ed.ac.uk (Richard Tobin) (03/26/91)
In article <534@bria> uunet!bria!mike [whose articles arrive without a From: line] writes: >The stock mallocs are piggishly slow beasts. Rephrase: >``On the other hand, there's not usually a good reason to not just use >malloc() unless you need reasonable performance.'' malloc() is unlikely to be slower than sbrk() unless *extremely* badly implemented, since sbrk() is a system call. For good performance, it may be best to allocate a large block of space and allocate fragments of it yourself, especially if you need fixed-size allocations. If you do this, then it won't make much difference (to speed) how you allocate the large block. Someone else remarked that it is difficult to be sure that you're freeing the right memory when you use sbrk(). Actually it's easy - just use sbrk(0) to check that the break is at the end of the memory you're freeing. If it isn't, someone else has allocated memory so you can't free it. -- Richard -- Richard Tobin, JANET: R.Tobin@uk.ac.ed AI Applications Institute, ARPA: R.Tobin%uk.ac.ed@nsfnet-relay.ac.uk Edinburgh University. UUCP: ...!ukc!ed.ac.uk!R.Tobin