gordoni@chook.ua.oz.au (Gordon Irlam) (05/08/90)
Now that we have figured out how to put lots of memory in a Sparcstation 1 the next problem is trying to use it. SunOS tends to have problems handling anything more than around 16M on a Sparcstation as described below. We have been trying to run a Sparcstation 1 with 48M of memory, and have been having problems making SunOS 4.03 make full use of the memory. Based on our experiences with SunOS 4.1 on a Sun 4/330 we suspect that the same problem persists under SunOS 4.1 on a Sparcstation. The problem we encounter is that with this much memory a very large number of address translations faults are occurring, up to 3000 per second, and with each fault taking around 300ms to handle this means that around 90% of the total CPU time is spent in system mode handling these faults. Getting into the kernel it appears that SunOS allocates a fixed sized area to keep track of mappings between virtual and physical addresses. On a Sparcstation this area can hold up to 128 page management groups. Each page management group maps a contiguous range of addresses up to 256k in size. One page management group will be needed to map each of the text, data, and stack segments of a process, and one appears to be used by the operating system for the process. Thus a minimum of four page management groups are used by a process, more are used if any of its segments exceed 256k. When a process needs to access a page in a page management group that is not in the kernel table, and no free page management groups exist, it steals a page management group from another process. When the other process next goes to access a page in this page management group it will get a fault and also have to steal a page management group from another process. Having got a page management group back all the page table entries associated with that page management group are marked invalid, and thus the process will receive address translation faults when it goes to access each of the 64 4k pages associated with the page management group. The problem is compounded by SunOS swapping out processes whose resident set size is zero. If all the page management groups belonging to a process get stolen from it the kernel determines that the process's resident set size is zero, and promptly swaps the process out. Fortunately this swapping only involves moving all of the processes pages onto the free list, and not to disk. But the CPU load associated with doing this is significant. We are finding around five processes per second get swapped to the free list. With 128 page management groups it will be possible to map up to 32M of virtual memory, although many processes have page management groups that map less than their full 256k, so that a total virtual address space of around 16M will be more typical. Thus the problem described is probably apparent to some extent on any Sparcstation with a total active virtual address space exceeding 16M, and sufficient physical memory to hold the entire virtual address space without paging. Shared libraries, and text segments mean that the actual physical memory on the machine could be less than 16M, and these problems could still occur. To get a feel for the cost of these problems you can have a look at the hatcnt data structure in the kernel. # nm /vmunix | grep hatcnt f80cbf00 B _hatcnt # od -D /dev/kmem +0xf80cbf00 | head -2 f80cbf00 0002129059 0002034884 0019942909 0003173659 f80cbf10 0002685512 0000000000 0000000000 0000000000 The 4th word is the total number of page management group allocations (3173659), the 5th word is the number of page management group allocations that stole a page management group from another process (2685512). Figuring 300ms per fault and around 32 faults for each page management group that has been stolen from another process a total of 7 hours has been spent handling these faults on a machine that has been running for 2 days, and is idle at night. We have made one unsuccessful attempt to patch /vmunix so as to increase the number of page management groups allocated by the npmgrps variable, and controlled by the NPMGRPS_60, and MNPMGRPS constants. Has anybody managed to increase the number of page management groups under SunOS (on any Sun 4 machine, not necessarily a Sparcstation), either with or without recompiling the sources, and if so what is the secret? Gordon Irlam Adelaide University, Australia (gordoni@cs.ua.oz.au)
mhorstm@swbatl.swbt.com (Mark Horstman 5-3417) (05/09/90)
In article <7463@brazos.Rice.edu> gordoni@chook.ua.oz.au (Gordon Irlam) writes: >SunOS tends to have problems >handling anything more than around 16M on a Sparcstation as described >below. > >The problem we encounter is that with this much memory a very large number >of address translations faults are occurring, up to 3000 per second, and >with each fault taking around 300ms to handle this means that around 90% >of the total CPU time is spent in system mode handling these faults. Is this true? I was planning on purchasing 4 4MB SIMMS to extend the standard memory (8 1MB SIMMS) that came with my SS1. Mark Horstman Southwestern Bell Telephone mhorstm@swbatl.sbc.com uunet!swbatl!mhorstm