rmiller@gmu90x.UUCP (Richard Miller) (07/22/89)
Newsgroups: comp.binaries.ibm.pc Subject: Lightweight Tasks Reply-To: rmiller@gmu90x.UUCP (Richard Miller) Distribution: comp.arch,comp.realtime Organization: George Mason University, Fairfax, Va. In the recent past I know I have been exposed to the term "lightweight task", but I can't seem to find any references that will define the term. Does anyone out there in netland know of a published reference for the for it? No fat jokes please! Please reply via mail. I will post a summary of responses in a week or so. -------------------------------------------------------------- Rick Miller - S.I.T.E. - George Mason University, Fairfax, VA. -----> Any opinions are my own - but that's ALL I've got <---- --> My future's so bright I have to wear a catcher's mask <--- --------------------------------------------------------------
gillies@p.cs.uiuc.edu (07/23/89)
Lightweight is to tasking as RISC is to computer architecture. "Lightweight tasks" became popular about 8 years ago. It is a buzzword that indicates the operating system can do a context switch quickly. Lightweight tasks came about because most UNIXes take several milliseconds to perform a context switch. This is *death* to a well-structured, layered protocol implementation. Some early machines (like the Xerox DLION) could perform a task switch as fast as a procedure call (in about 20 microseconds), using some hardware support. Other operating systems minimize the task context block so that very little state must be changed in order to switch tasks. Sometimes, all the lightweight tasks run together in the same address space (In the DLion, and in MIT's IBM PC/IP network protocol package, and in its unix V7 TCP/IP) There is no precise definition, just as there is no precise definition of RISC. RISC was a knee-jerk reaction to the hideous complexity of the VAX; Lightweight tasks were a reaction to the hideous slowness of UNIX tasking.
hascall@atanasoff.cs.iastate.edu (John Hascall) (07/25/89)
In article <129300004@p.cs.uiuc.edu> gillies@p.cs.uiuc.edu writes: [...defines lightweight tasks...] >There is no precise definition, just as there is no precise definition >of RISC. RISC was a knee-jerk reaction to the hideous complexity of >the VAX; Lightweight tasks were a reaction to the hideous slowness of ^^^ >UNIX tasking. ... which has hardware support for fast context switches hmmm... --John
zs01+@andrew.cmu.edu (Zalman Stern) (07/28/89)
I have to disagree with Jim Gillies' description of lightweight tasks. I have always heard the term used to describe an entity which has a stack and a register set (including the program counter). A lightweight task does not have its own address space or other context (i.e. signal masks, uid, file descriptor table etc...) Light weight tasks are often called threads (e.g. MACH, Concert MultiThread architecture, Modula 3), or LWPs for Light Weight Processes. This definition is consistent with the usage of the terms in MACH, various Xerox projects, work from DEC Systems Research Center including modula-3, Concert MultiThread architecture which originated at DEC Western Research Lab (I think), Synthesis (from Columbia), and various products from companies like Apollo and Sun. My understanding is that lightweight tasking predates UNIX' popularity by quite a bit. Especially if you consider coroutine schemes to be LWP systems. (The internals of coroutines and non-premptive LWPs are just about identical.) In any event, it is not a valid comparison to RISC vs. VAX in that lightweight processes solve different problems than UNIX processes. Most systems that support LWPs also have heavy weight processes or "address spaces" to allow one to partition subsystems so that they cannot access each other's memory. (This last comment does not apply to systems which don't have virtual memory.) Finally, there are implementations of LWP done using UNIX processes and shared memory to emulate multiple threads of control within a single address space (i.e. Xerox PCR). With a decent UNIX implementation (some of which do exist), this solution has acceptable performance. Sincerely, Zalman Stern Internet: zs01+@andrew.cmu.edu Usenet: I'm soooo confused... Information Technology Center, Carnegie Mellon, Pittsburgh, PA 15213-3890
tvf@cci632.UUCP (Tom Frauenhofer) (07/28/89)
In article <129300004@p.cs.uiuc.edu> gillies@p.cs.uiuc.edu writes: > >Lightweight is to tasking as RISC is to computer architecture. Uh-oh. I smell trouble afoot. ( :-) ) >RISC was a knee-jerk reaction to the hideous complexity of >the VAX; Lightweight tasks were a reaction to the hideous slowness of >UNIX tasking. A couple of points: 1) Yes, RISC was a knee-jerk reaction to the VAX. Fortunately, we needed this "knee-jerk" to make us take a hard look at the instruction sets and processor complexity that the hardware architects were coming up with (unfortunately, the RISC proponents exaggerated the advantages of their solutions, just as the CISC-ers go overboard in ragging on RISC, but that's for another group). 2) Lightweight threads are more than just a reaction to the slowness of UNIX tasking; they are, instead, a re-evaluation of what defines a "context". This, to me, is the real value of the Lightweight/Heavyweight dialogue. Thomas V. Frauenhofer ...!rutgers!rochester!cci632!ccird7!tvf *or* ...!rochester!cci632!ccird7!frau!tvf *or* ...!rochester!rit!anna!ma!tvf1477 FRAU BBS: (716) 227-8094 2400/1200/300 baud - log in as "new" to register "The Earth? I'm going to blow it up. It obstructs my view of Venus" - Martin
peter@ficc.uu.net (Peter da Silva) (07/28/89)
In article <EYnoUGm00Vs8M9wZ0z@andrew.cmu.edu>, zs01+@andrew.cmu.edu (Zalman Stern) writes: > A lightweight task does not > have its own address space or other context (i.e. signal masks, uid, > file descriptor table etc...) Is this an observation or part of the definition. After all there need be no processing overhead for these other resources (i.e., a file descriptor does not require explicit action on a context switch), so there seems to be no advantage from leaving this much context out of a thread. -- Peter da Silva, Xenix Support, Ferranti International Controls Corporation. Business: peter@ficc.uu.net, +1 713 274 5180. | "...helping make the world Personal: peter@sugar.hackercorp.com. `-_-' | a quote-free zone..." Quote: Have you hugged your wolf today? 'U` | -- hjm@cernvax.cern.ch
hascall@atanasoff.cs.iastate.edu (John Hascall) (07/29/89)
In article <5323@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: >In article <EYnoUGm00Vs8M9wZ0z@andrew.cmu.edu>, zs01+@andrew.cmu.edu (Zalman Stern) writes: >> A lightweight task does not >> have its own address space or other context (i.e. signal masks, uid, >> file descriptor table etc...) >Is this an observation or part of the definition. After all there need be >no processing overhead for these other resources (i.e., a file descriptor >does not require explicit action on a context switch), so there seems to >be no advantage from leaving this much context out of a thread. But they do involve overhead during thread creation/deletion. Processes under VMS (for example) context switch quickly, but creation is quite lengthy. John Hascall ISU Comp Center
gillies@m.cs.uiuc.edu (07/30/89)
Zalman Stern, Let me see if I can summarize your definition.
Lightweight tasks --
(1) have stack & register set.
(2) have no address space, signal masks, uid, file descriptor tables, etc.
(3) are also called threads, LWPs.
(4) predate UNIX [Ed -- then why the new name?]
(5) solve different problems than UNIX tasks.
(6) Most LWP systems have Heavy Weight Processes too
[Not true in Xerox DLion/Pilot/Cedar, with VM + lightweight tasks]
(7) Some implementations of LWPs exist within UNIX processes.
I have no qualms with (1), (2), (3). About (4) -- I think it is
historically correct to say that network protocols were a major
impetus for developing lightweight tasks (in the three systems I
worked with -- PDP11 MIT V7 TCP/IP/Telnet & C-gateway, IBM PC/IP, and
Xerox Pilot OS, this was precisely the reason for them). Certainly
lightweight tasks allow faster response time to realtime events (such
as network packet arrivals and ack timeouts).
As for (6) and (7) -- it is possible to layer heavy weighted-ness upon
lightweight tasks, by adding status blocks to the lightweight task.
This is what is done in the Xerox DLion with the Pilot kernel.
The least successful lightweight tasking system I know of was built
within heavyweight UNIX tasks. If you must schedule a heavyweight
UNIX task before you can get to a lightweight task, then I call this
a heavyweight tasking system.
We can quibble about whether lightweight tasks were a reaction to the
slowness of UNIX tasking; It is certainly true that UNIX tasks are
inappropriate for a well-structured layered protocol implementation,
and hence most of the protocol implementation in 4.2BSD ended up in an
awful place -- in the kernel. Lightweight tasks were used in other
systems to get this code out of the UNIX (and other) kernels.
Don Gillies, Dept. of Computer Science, University of Illinois
1304 W. Springfield, Urbana, Ill 61801
ARPA: gillies@cs.uiuc.edu UUCP: {uunet,harvard}!uiucdcs!gilliesperry@apollo.HP.COM (Jim Perry) (08/01/89)
Here's my two cents. First off, I think "lightweight tasks" is a corrupted and misleading form. The term "task" by itself embodies the concept in question. I don't know the origins of the term, I think it came from IBM-land (certainly tasks occur, with this meaning, in IBM PL/I). It (the term and the concept) was used extensively in the Dartmouth Timesharing System (DTSS), mid-'60's. Multitasking predates time-sharing (sometimes called multiprocessing, but that gets confusing when you let multiple CPUs into the picture), let alone UNIX. The historical origins do come from I/O optimization, but not particularly network protocols. Mainframes (another line of discussion) were very concerned with "throughput" -- getting as much data as possible processed as fast as possible, and tended to have fancy hardware to handle I/O in parallel with CPU processing. Instead of waiting around for e.g. a read operation to complete, optimal programs would continue processing, e.g. sorting the previous buffer. When the read completed the program would be notified and logically continue that alternate thread of control (temporarily suspending the sorting thread). This was termed multitasking, and the abstraction of "task" applied to the individual threads of control. With timesharing came the idea of running multiple programs, each with a "virtual machine" -- address space, registers, etc. apparently for its own use. In UNIX, via Multics, this became a "process". These systems optimize cycles by running another process when one is blocked on I/O. Other systems (e.g. DTSS) maintain the potential asynchrony of I/O and thus potential multitasking within a process. For instance, the DTSS command line interpreter (shell equivalent) SIMON is a single program/process. As each new user logs on, a communication file (similar to a pipe) connected to that user's terminal connection is passed to SIMON, which creates a task to handle it (and thus may be handling several hundred tasks at once). Interactive commands (even editing, to an extent), are handled by SIMON and a new process doesn't get created until a user explicitly runs a program. This scheme allows considerably more simultaneous users for the same horsepower CPU. Anyway, multitasking is an old and useful concept. The same results can be achieved with multiple communicating processes, especially in the presence of shared memory and IPC, but in UNIX, for instance, this can be expensive. "Lightweight process" seems to have been coined to discuss tasks by analogy to UNIX (nee Multics) processes. It is possible through various nefarious schemes to create a semblance of multitasking within a UNIX process, and various such schemes exist; hopefully future iterations will make it even easier. -- Jim Perry perry@apollo.com HP/Apollo, Chelmsford MA This particularly rapid unintelligible patter isn't generally heard and if it is it doesn't matter.
scotth@tekcrl.LABS.TEK.COM (Scott R. Herzinger) (08/01/89)
What about additional shades of gray between "heavy" and "light"?
At the USENIX conference in San Diego last January, someone from Data
General presented a paper about their changes to fork() semantics and the
organization of process context in order to provide something they called
variable weight processes. I don't have the proceedings at hand, so I
can't give details and must disclaim that errors are mine, not the paper's
author's.
The idea is that process context is partitioned into about several
(about seven, I think) parts, each comprising things that roughly
"belonged together" or had interdependencies such that they couldn't be
separated. A variant of fork() is provided that permitted the caller
to create a new process for which any of the parts could be copied or
shared. A child that copies only the stack and otherwise shares
*everything* is a light[est]weight task. Normal fork() is provided to create
children that copy everything (on-write, of course :-)). There are
lots of combinations in-between. I suppose that some of them are
very useful and some probably aren't interesting or useable at all.
Instead of adding a new layer of context partitioning (e.g. lightweight
tasks) to UNIX processes, graduated partitioning is provided. Context
management depends on the amount of non-shared context. Lightweight
tasks are available, but within a fairly conventional UNIX framework.
Hope this is interesting,
Scott
--
Scott Herzinger scotth%crl.labs.tek.com@relay.cs.net
Computer Research Lab, Tektronix, Inc.
PO Box 500 MS 50-662, Beaverton, OR 97077reeder@stanley.hf.intel.com.ogc.edu (08/01/89)
In article <70900004@m.cs.uiuc.edu> gillies@m.cs.uiuc.edu writes: > >Zalman Stern, Let me see if I can summarize your definition. >Lightweight tasks -- > (1) have stack & register set. > (2) have no address space, signal masks, uid, file descriptor tables, etc. > (3) are also called threads, LWPs. > (4) predate UNIX [Ed -- then why the new name?] > (5) solve different problems than UNIX tasks. > (6) Most LWP systems have Heavy Weight Processes too > [Not true in Xerox DLion/Pilot/Cedar, with VM + lightweight tasks] > (7) Some implementations of LWPs exist within UNIX processes. > >I have no qualms with (1), (2), (3). About (4) -- I think it is >historically correct to say that network protocols were a major >impetus for developing lightweight tasks (in the three systems I >worked with -- PDP11 MIT V7 TCP/IP/Telnet & C-gateway, IBM PC/IP, and >Xerox Pilot OS, this was precisely the reason for them). Certainly Another "lightweight/threads" impetus was simply that it was the only sensible implementation available. In 1973 I was involved with the production of a commercial multi-user DBMS on IBM mainframes running the predominant OSs of the time (MFT, MVT, VS/1, SVS, etc. etc). To efficiently control multi-user access to a single resource (the database) you created a "monitor" (i.e., simple OS) that "multi-threaded" user's database requests within the single address space (process) controlled by the monitor. The OSs of the time were batch oriented and developers had to fabricate all of the necessary environment (such as compilers that supported stacks) to have a decent implementation. This was a common practice in the database community and many of the "multi-threaded" DBMSs would now be said to support "lightweight processes." > ... etc. ... Ed Reeder Intel Corp Hillsboro, OR
peter@ficc.uu.net (Peter da Silva) (08/01/89)
In article <1271@atanasoff.cs.iastate.edu>, hascall@atanasoff.cs.iastate.edu (John Hascall) writes: > In article <5323@ficc.uu.net> peter@ficc.uu.net (Peter da Silva) writes: > >In article <EYnoUGm00Vs8M9wZ0z@andrew.cmu.edu>, zs01+@andrew.cmu.edu (Zalman Stern) writes: > >> A lightweight task does not > >> have its own address space or other context (i.e. signal masks, uid, > >> file descriptor table etc...) > >Is this an observation or part of the definition. After all there need be > >no processing overhead for these other resources [...], so there seems to > >be no advantage from leaving this much context out of a thread. > But they do involve overhead during thread creation/deletion. Maybe, maybe not. If you set the thread control data structures up right the overhead could be limited to that of the actual resources used by the process. If you don't open any files, for example, no resources need be allocated to the process for files. > Processes under VMS (for example) context switch quickly, > but creation is quite lengthy. True, but a bit of a slippery-slope argument. Processes under VMS have a near-psychotic amount of context. -- Peter da Silva, Xenix Support, Ferranti International Controls Corporation. Business: peter@ficc.uu.net, +1 713 274 5180. | "The sentence I am now Personal: peter@sugar.hackercorp.com. `-_-' | writing is the sentence Quote: Have you hugged your wolf today? 'U` | you are now reading"
tneff@bfmny0.UUCP (Tom Neff) (08/01/89)
Intel's iRMX series of realtime multitasking OS's also embody the lightweight task idea. Tasks own little more than a priority and a stack plus a few flags for the dispatcher to look at (like whether to restore the numeric coprocessor's context). The goal is to make creating and using tasks very cheap, so embedded systems can run reliably and fast. iRMX uses the "job" to contain all the extra bushwah associated with UNIX "processes." Jobs own resources, user IDs and so forth. It works quite well. (If anyone but Intel had come up with it, it'd probably be the standard by now. :-) ) As I see it UNIX jams the two disparate "job" and "task" concepts together into one unhappy identity (the process) to the detriment of performance. That's why special "real time" UNIXen are necessary to begin with. -- "We walked on the moon -- (( Tom Neff you be polite" )) tneff@bfmny0.UU.NET
aral@calliope.Encore.COM (Ziya Aral) (08/02/89)
In article <4405@tekcrl.LABS.TEK.COM> scotth@tekcrl.LABS.TEK.COM (Scott R. Herzinger) writes: > >What about additional shades of gray between "heavy" and "light"? > >At the USENIX conference in San Diego last January, someone from Data >General presented a paper about their changes to fork() semantics and the >organization of process context in order to provide something they called >variable weight processes. I don't have the proceedings at hand, so I >can't give details and must disclaim that errors are mine, not the paper's >author's. "Data General"?!!!??! Sheesh... The paper was by myself, Ilya Gertner, Greg Schaffer and Alan Langerman from Encore and Tom Doepner and Jim Bloom from Brown University.... "Data General"! That I can never forgive :-) > >The idea is that process context is partitioned into about several >(about seven, I think) parts, each comprising things that roughly >"belonged together" or had interdependencies such that they couldn't be >separated. A variant of fork() is provided that permitted the caller >to create a new process for which any of the parts could be copied or >shared. A child that copies only the stack and otherwise shares >*everything* is a light[est]weight task. Normal fork() is provided to create >children that copy everything (on-write, of course :-)). There are >lots of combinations in-between. I suppose that some of them are >very useful and some probably aren't interesting or useable at all. > >Instead of adding a new layer of context partitioning (e.g. lightweight >tasks) to UNIX processes, graduated partitioning is provided. Context >management depends on the amount of non-shared context. Lightweight >tasks are available, but within a fairly conventional UNIX framework. > >Hope this is interesting, >Scott >-- >Scott Herzinger scotth%crl.labs.tek.com@relay.cs.net > Computer Research Lab, Tektronix, Inc. > PO Box 500 MS 50-662, Beaverton, OR 97077 This is quite accurate except for "Data General" (sheesh again).... Basically the idea was that multiprocessors change the machine model which the operating system presents. In place of a process implementing a virtual machine roughly corresponding to single user on a time sharing system and with hard partitions (seperate addresses spaces, system resources, etc.) implemented at kernel-level, multi's typically allocate many tightly communicating virtual machines to a single user. Here the issue is resource sharing semantics and the system "weight" of that many processes as reflected by process creation/deletion times, context switch times, and by demands on the kernel non-paging memory pool. MACH tasks/threads represent one approach to this problem by essentially allowing many "threads" to exist within one task. A task maps logically to a traditional process and defines an address space etc., while threads are in effect simple virtual cpus.... I suppose it could be called a virtual shared-memory multiprocessor. One task with one thread is the equivilant of a UNIX process. The nUNIX kernel implemented at Encore took a slightly different tack. Instead of instantiating multiple contexts within a process, we broke up the traditional process control block by removing individual resources (address space, file descriptors, signal handlers, process statistics, etc.) and giving them an independent existence as Resource Descriptors. fork() semantics were then changed to allow either the copying or the sharing of individual resource descriptors at fork() time through the addition of a new resctl() call. Essentially resctl() set the bits in a resource mask which specified the behavior of fork()'s for that particular process. If a process specified that its file descriptors were private but its address space shared, then at fork() time a new fd resource descriptor (a copy of the original) would be created, but a new address space (i.e. new page tables) would not. Instead, the pcb of the child process would simply point to the address resource descriptor (i.e. page tables) of the parent. In the final implementation, only the resctl() call was added and all existing process control calls (fork, exit, wait, etc. ) were retained. Since the default behavior specified all private resources, complete backward compatibility with existing UNIX fork()'s was maintained. Also the need for two different paradigms was avoided. Resource descriptors were managed by assosciating a reference count with each one. Each time a resource descriptor was referenced by a process, the count was incremented and each time such a process died, the count was decremented. When the count hit zero, the descriptor was destroyed. The results with this mechanism have been very encouraging. Process creation times dropped from 35 - 128 ms. (System V.3 - Encore APC machine) for data sets ranging from 0 to 16 megs., to a flat 2.5 ms. for all shared processes. This compares with a creation time of around 10 ms. for MACH threads on the same hardware. Context switch times also have improved significantly while kernel memory requirements drop by over two thirds per process. Best of all, it took only a few hundred lines of code to implement given that we already had a multi-threaded kernel with sharable resources (I admit, it is a big "given"). As a final aside, "variable weight" is probably a misnomer as almost all the weight of UNIX processes (startup, switch, and memory requirements) comes from the need to create, initialize, switch, and maintain page tables. All the others count as "noise" from the standpoint of effeciency although they do provide the possibility of highly flexible combinations of shared and private resources. Hope this helps.... Ziya Aral Encore Computer
scotth@tekcrl.LABS.TEK.COM (Scott R. Herzinger) (08/04/89)
In article <11843@xenna.Encore.COM>, aral@calliope.Encore.COM (Ziya Aral) writes: > >At the USENIX conference in San Diego last January, someone from Data > >General presented a paper > > "Data General"?!!!??! Sheesh... The paper was by myself, Ilya Gertner, Greg > Schaffer and Alan Langerman from Encore and Tom Doepner and Jim Bloom from > Brown University.... "Data General"! That I can never forgive :-) I apologize to the authors for this slip up. I recalled two interesting papers from the conference; the DG paper and the one referred to above. I should have said "I can't remember the attribution" of the paper, or asked someone, instead of misattributing, as I did. Again, apologies. At least someone noticed my mistake. Thanks Compliments to the authors. I found the presentation one of the most interesting at the conference. For what it's worth, my copy of the proceedings is pretty inaccessible at the moment, as it is on some ship, in transit between Le Havre, France, and San Francisco, which is why I didn't refer to it before finishing my posting. One of the perils of long-distance relocations. -- Scott Herzinger scotth%crl.labs.tek.com@relay.cs.net Computer Research Lab, Tektronix, Inc. PO Box 500 MS 50-662, Beaverton, OR 97077