std-unix@ut-sally.UUCP (06/09/87)
From: boba@iscuva.ISCS.COM (Bob Alexander) Modern, memory managed operating systems (like UNIX) have addressed quite nicely certain special requirements of executable files. In particular (1) the file (text and data) need not be loaded into memory in its entirety to begin executing, and (2) the pages can be shared among processes that are executing them (both on disk and in memory). As far as I know, those capabilities are not made available to interpreters for their pseudo-code and data, even though they would be equally as applicable as they are to "real" programs. If 15 users are running a program written in an interpretive language, the interpreter code is shared, but the p-code exists separately for each user. This results in a major disadvantage in the use of interpretive languages to produce production programs. Interpretive systems are in quite wide use today (e.g. shells, SQLs, (((Lisp))), Icon, etc., etc., [even BASIC]), and as processor speeds increase, use of interpreters will likely continue to grow. There are a few ways of working this problem with existing UNIX facilities, but the ones I've come up with so far are kluges. My reason for posting to this newsgroup is to get your reaction to a possible new UNIX facility for this purpose. I'll express my suggestion in SVID format, sort of: ------------------------------ NAME vread -- read from a file into memory [but not really, maybe]. SYNOPSIS int vread(fildes, bufptr, nbyte) int fildes; char **bufptr; unsigned nbyte; DESCRIPTION The function "vread" attempts to read "nbyte" bytes from the file associated with "fildes" into an allocated buffer whose address is returned in "bufptr". This function is similar to read(ba_os) [read(ba_os) is SVIDese for read(2)] except for its implications concerning virtual memory and that it allocates a buffer rather than being given one. In a memory managed system, the contents of the file are not transferred into the program's memory space. Instead, the file is "mapped" into an area of the caller's data space (involving no actual data transfer) and demand-paged into real memory, directly from its disk file, as accessed by the program. As long as any such page remains pure, it never needs to be swapped out to disk, and can always be swapped in from its original location on disk. If a page becomes dirty, it will have separate swap space allocated for it on disk and the page will be re-mapped to that space. [This technique is often used for the initialized data portion of executing programs]. Therefore, "vread" produces the appearance of reading from a file into memory, but no data actually transferred (in a memory managed system), and the system is afforded the opportunity to optimize by sharing the data among all processes accessing the file. From the program's point of view, this operation is indistinguishable from an actual data transfer. In non-memory-managed versions of UNIX, "vread" is implemented as a true data transfer. Therefore, "vread" calls are portable between memory-managed and non-memory-managed systems. Since the system decides the address at which the space will be allocated, specific memory management requirements (such as page size and alignment) are hidden from the caller and are therefore of no concern to a program using this facility. In a memory managed system, use of "vread" can provide a significant optimization when large portions of files must be available in their entirety, but are sparsely and/or randomly accessed (such as the pseudo-code for an interpreter), and when it is desirable to share large, read-only files. RETURN VALUE Same as read(ba_os). ERRORS Same as read(ba_os). ------------------------------------- For interpreters to take full advantage of this facility, they would have to interpret their p-code "as is" as it sits on disk. If they modify the code, much of the advantage would be lost. I'd be interested in hearing your comments and suggestions regarding this idea; alternative ideas to solve this problem, ways other OSs have dealt with it, implementation problems, or gross oversights. What would you think of a "read only" option for this function (a fourth argument?), where the data would be mapped as read only (i.e. protected). -- Bob Alexander ISC Systems Corp. Spokane, WA (509)927-5445 UUCP: ihnp4!tektronix!reed!iscuva!boba Volume-Number: Volume 11, Number 55
johnl@ima.ISC.COM (John R. Levine) (06/16/87)
From: johnl@ima.ISC.COM (John R. Levine) In article <8250@ut-sally.UUCP> boba@iscuva.ISCS.COM (Bob Alexander) writes: >[interpretive languages would work better if they could page their data >memory like compiled languages do, so how about a vread() call that maps >instead of reading?] When we were designing AIX, the Sys V port for the RT PC, the same question of file mapping came up, this time in the context of doing data base work. The original proposal was for something like the vread() call, but we eventually decided on, essentially: pointer = filemap(fd); where fd is an open file descriptor. It maps the entire file into the address space, somewhere, and gives you a pointer to it. It fails if there isn't enough address space. This had the advantage over the vread() approach that it generalizes better for file writing -- if the file is mapped read/write, anything you write to the segment goes into the file, adding new blocks into the file if need be. If you don't want the file's size to be a multiple of a page, use ftruncate() to set its length. Notice that vread() can easily be simulated by this scheme, but not vice versa. I agree that this scheme fails dismally if you don't actually have paged memory, but I'd rather have this as the underlying call and vread() as a library routine that maps on systems where it can and reads on systems where it can't. John Levine, ima!johnl or Levine@YALE.somethingorother --- John R. Levine, Javelin Software Corp., Cambridge MA +1 617 494 1400 { ihnp4 | decvax | cbosgd | harvard | yale }!ima!johnl, Levine@YALE.something U.S. out of New Mexico! Volume-Number: Volume 11, Number 68
gwyn@brl-smoke.arpa (Doug Gwyn ) (06/19/87)
From: gwyn@brl-smoke.arpa (Doug Gwyn ) In article <8281@ut-sally.UUCP> ima!johnl@harvard.harvard.edu (John R. Levine) writes: > pointer = filemap(fd); Hey, I really like this. I can even see how to implement it on a mickey-mouse (non-paged) architecture, and how to fake the facility (always return NULL). This is a whole lot better than vread() and kin. Volume-Number: Volume 11, Number 84
cc1@cs.ucla.edu (Michael Gersten) (06/19/87)
From: cc1@cs.ucla.edu (Michael Gersten) vread() is a nice idea, but fairly (very) limited. In particular, it requires that the p-code being interpreted be stored on the disk in semi-compiled form, and used as pure. May I remind you that INTERACTIVE interpreters (which is the whole point of interpreters) do not do this; most BASIC's tokenize the text as they read it in, so it is not pure; any decent interactive system will have dificulty utilizing this because they do not p-compile first. Michael Gersten Volume-Number: Volume 11, Number 72