torek@elf.ee.lbl.gov (Chris Torek) (02/15/91)
>In <1991Jan15.204849@IASTATE.EDU> spam@IASTATE.EDU (Begley Michael L) writes: >>What I'd like to do is execute a stream. [eg] >> uncompress -c microemacs|execute In article <118868@uunet.UU.NET> rbj@uunet.UU.NET (Root Boy Jim) writes: >I once suggested to Chris Torek that the kernel should execute >compressed programs. He groaned. Here is why: A compressed file (that is, one which has had redundancy removed by the compress(1) program) is a variable-length bit stream in which the meaning of any given set of bits depends on an arbitrarily large number of preceding bits (well, up to the compression table sizes). In other words, there is no way to tell that bit 71342 is the start of a 9 bit long field whose meaning is `the sequence "hello"' without starting at bit 0 and uncompressing until bit 71342 is reached. There are forms of compression that can be uncompressed on a per-block basis; these are somewhat more suitable. There remains one stumbling block: Unix systems like to run executables out of their original files (at least in part), but the mapping between `original file blocks' and `memory pages' is rather `lumpy'. With regular files there is a direct mapping in both directions. This too could be changed, at some expense; alternatively, the kernel could run compressed binaries by uncompressing them into a swap image, and running the swap image (this would be easier in pure swapping systems) or by uncompressing them into ordinary unlinked files and running those. The implementation of the latter is trivial even in user code. If one were to do this, however, the best approach would probably be the one outlined by some other poster: run a nightly script to compress executables that have not been read recently, and leave executables uncompressed once they have been run. -- In-Real-Life: Chris Torek, Lawrence Berkeley Lab EE div (+1 415 486 5427) Berkeley, CA Domain: torek@ee.lbl.gov
roy@phri.nyu.edu (Roy Smith) (02/15/91)
torek@elf.ee.lbl.gov (Chris Torek) writes: > the kernel could run compressed binaries by uncompressing them into a > swap image, and running the swap image (this would be easier in pure > swapping systems) or by uncompressing them into ordinary unlinked files > and running those. The implementation of the latter is trivial even in > user code. Gaak! Compression does not belong in the kernel (he says with great authority and religious overtones), unless maybe you're talking about a driver for a DES chip. All you really need to implement compressed executables (as Chris hinted) is to start the executable files with "#!/bin/uncompressexec" which would uncompress the rest of the file, put it back in place of itself (if you want to cache uncompressed executables, which is probably a good idea) and then overlay itself with the new executable. A nightly cron job could recompress stuff that hasn't been run in a while. -- Roy Smith, Public Health Research Institute 455 First Avenue, New York, NY 10016 roy@alanine.phri.nyu.edu -OR- {att,cmcl2,rutgers,hombre}!phri!roy "Arcane? Did you say arcane? It wouldn't be Unix if it wasn't arcane!"
rbj@uunet.UU.NET (Root Boy Jim) (02/16/91)
In article <9950@dog.ee.lbl.gov> torek@elf.ee.lbl.gov (Chris Torek) writes: >>In <1991Jan15.204849@IASTATE.EDU> spam@IASTATE.EDU (Begley Michael L) writes: >>>What I'd like to do is execute a stream. [eg] >>> uncompress -c microemacs|execute > >In article <118868@uunet.UU.NET> rbj@uunet.UU.NET (Root Boy Jim) writes: >>I once suggested to Chris Torek that the kernel should execute >>compressed programs. He groaned. [Good explanation deleted] Well, I was half kidding. On the other hand, consider files as objects. Right now, the only "message" you can send them is "execute yourself with these arguments". Right now, we embed the messages in programs which know about the format of their data. Strip removes the symbol table from executables, for example. What would it mean to strip a perl, awk, or shell script, or perhaps an include file? Most likely, decommenting, squishing, and unifdefing. Consider the message "delete yourself". Smart files could figure out where to archive themselves, or stubbornly refuse to die. Objects and executables would refuse to "print". Isn't it time for oonix? I became intrigued with what kinds of interpreters one could slap on the beginning of files besides #! /bin/sh. There is "more +2" to make a file self page, as well as awk and perl. Around here, we have some huge log files which people tail -f. So I prepended #! tail -f to them and made them executable! It would be nice to be able to prepend #! uncompress+exec to a compressed binary and have it do what Chris suggested, leave the file uncompressed then execute it. In fact, that's what I did with my autoload, load, and unload scripts which I posted to c.u.w. when this whole discussion started. Clearly this can all be done in user code. And it should be. -- [rbj@uunet 1] stty sane unknown mode: sane
dhesi%cirrusl@oliveb.ATC.olivetti.com (Rahul Dhesi) (02/16/91)
In <1991Feb15.150413.14443@phri.nyu.edu> roy@phri.nyu.edu (Roy Smith) writes: >Gaak! Compression does not belong in the kernel (he says with >great authority and religious overtones).. Actually, even the kernel doesn't belong in the kernel. As Mach 3.0 gets nearer, this becomes more and more true. -- Rahul Dhesi <dhesi%cirrusl@oliveb.ATC.olivetti.com> UUCP: oliveb!cirrusl!dhesi
bhoughto@pima.intel.com (Blair P. Houghton) (02/16/91)
In article <9950@dog.ee.lbl.gov> torek@elf.ee.lbl.gov (Chris Torek) writes: >In article <118868@uunet.UU.NET> rbj@uunet.UU.NET (Root Boy Jim) writes: >>I once suggested to Chris Torek that the kernel should execute >>compressed programs. He groaned. > >Here is why: A compressed file (that is, one which has had redundancy >removed by the compress(1) program) is a variable-length bit stream in >which the meaning of any given set of bits depends on an arbitrarily >large number of preceding bits (well, up to the compression table >sizes). In other words, there is no way to tell that bit 71342 is the >start of a 9 bit long field whose meaning is `the sequence "hello"' >without starting at bit 0 and uncompressing until bit 71342 is reached. One wonders if you guys saw the newgroup of alt.comp.compression, the first article of which mentioned an "unbalanced," "fractal" compression algorithm. It further said one needs a supercomputer to do the compressing, but apparently an abacus and a microsecond for uncompressing. If so, we may be on to something, here... --Blair "Or not; as the dichotomy may eventuate."