skh@vi.ri.cmu.edu (Steve Handerson) (04/06/90)
When I was working with lisp I'd have loved to limit the amount of memory that lisp could hog for paging. Otherwise a few seconds pause would swap my editor out. I thought of writing (this is in MACH) a "pager shell" you would run programs under, and would limit the number of in-core pages everything under it could have. Even better, it might be nice to allocate memory among processes, or have "upper limits" of unshared swapping memory per process. It would be even better if the user could prioritize use of all swapping resources. I don't care if lisp grinds to a halt, if I can edit unhampered at the same time (if my editor's swap requests always take priority). Do any operating systems already do this? -- Steve
loc@uunet.UU.NET (Leigh Clayton) (04/11/90)
In article <2402@darkstar.ucsc.edu> Steve Handerson talks about one process
(especially a 'background' one) hogging too much memory.
I have struggled much with this problem in building a load manager for the
Sharp APL system, and in thinking about relaxing some of the things we now use
to manage this sort of problem. This is a long message, but I thought it might
be worth posting (to someone, anyway).
At present, we actually keep the entire memory area for each user together
and swap it as a whole, so that our problem is considerably altered from that
in a free paging environment (each user has essentially a binary paging state).
I have always believed that this was a both a simplification and an advantage,
but I no longer consider it an advantage (there's no question that it is a
simplification). This is because of the sheer bandwidth required to manage a
thousand regions when each is 1 or 2 meg in size (Obviously I'm using a
mainframe, but I don't think that really alters the basic issues).
It does seem to me, though, that a user's current working set (assuming that
you have a way of determining that with reasonable accuracy) should be managed
this way, and I feel a bit vindicated in this belief by the fact that IBM (MVS)
did finally begin to manage their 'swap' this way. This means that the problem
reduces to two parts - keeping track of the current working set, and deciding
whose working sets to keep swapped in at any time. Demand paging will make up
for any miscalculations and trim down the working sets, but we can ignore that.
Within this framework, I think you can safely apply the rule of thumb that
we've used for years -- that is, divide tasks up into those that use a very
small amount of CPU time occasionally (vi?), and those (like the lisp process
discussed in the referenced message) that are making steadier resource demands.
The occasional users then get allocated a fairly small portion of the memory
but a fairly large portion of the bandwidth to the swap device, while most
of the memory is dedicated to the resource-consumers.
This system will tend to use quite a lot of 'swap' banmdwidth, but will in
fact perform very well, as long as users are promoted from 'occasional' to
'steady' consumers quite promptly.
These guidelines are really only of value, of course, if you are managing
a system which has a large real-memory deficit (that is, if there is a lot
of virtual memory compared to the amount of real memory). Given the cost of
memory lately, I'm beginning to wonder if this sort of configuration is
worth worrying about.
One alternative that might be worth considering is simply fencing each
task's minimum working set (assuming that they have such a thing). That is,
give each task a region of "(total_memory*fence_fraction)/no_users" that can
*not* experience steals for other users. 'fence-fraction' would be set to
the fraction of total memory to be dedicated to fenced task portions -- let's
say 0.5 for now. The rest of memory would be open for ordinary demand paging.
Obviously, if the fence_fraction was set too low, or memory utilisation were
set too high, this wouldn't work, but given the huge memories becoming common
these days, this shouldn't be too much of a problem. As long as a task's
working set fit within it's fence, it would experience essentially no paging
and of course swapping never happens. Tasks whose working sets overflow
their fences will need to go through the sort of workload management that they
do now, with the exception that only the part sticking outside of their
fences would be subject to swapping or stealing while swapped (whichever way
the 'swap' is implemented).
I trust it's obvious that I don't mean to imply the fences are anything but
logical -- there only need be a single frame pool.
I apologise for the length of this, I'd welcome any comments (posted or
Emailed) of course.
.../Leigh
-----------------------------------------------------------
- Men argue, Nature acts /Voltaire -
- -
- ...for we never know what is enough, -
- until we know what is more than enough. /Blake -
-----------------------------------------------------------
Leigh Clayton, loc@tmsoft.UUCPciotti@wilbur.nas.nasa.gov (Robert B. Ciotti) (04/12/90)
In article <2402@darkstar.ucsc.edu> skh@vi.ri.cmu.edu (Steve Handerson) writes: > > >When I was working with lisp I'd have loved to limit the amount >of memory that lisp could hog for paging. ... >swapping resources. I don't care if lisp grinds to a halt, >if I can edit unhampered at the same time (if my editor's swap >requests always take priority). > >Do any operating systems already do this? > >-- Steve The 205/ETA10 (possibly the 203) machines from Control Data running the VSOS/EOS operating systems had a feature to limit/allocate the number of working set pages for a process. It would be interesting to know if this made it into VE (Another CDC proprietary OS). This particularly necessary feature however never made it into UNIX for the ETA machines and was really a problem because a single process (say doing a wide stride on a very large vector), could bring the whole machine to a halt. I was going to go in and modify the page stealer to limit working set sizes by stealing pages from the process causing this type of behavior, but time was short... Bob