vvawh@convx1.lerc.nasa.gov (Tony Hackenberg) (10/04/90)
Does anyone know of any studies on the effect of allowing vi usage by many users on supercomputer performance? Does anyone know if there are versions of "vi", "emacs", or other full screen editors that are written to minimize performance hits on supercomputers? -- Tony Hackenberg NASA/Lewis Research Center M.S.142-2, 21000 Brookpark Rd., Cleveland, OH 44135 Internet: vvawh@convx1.lerc.nasa.gov Phone: (216) 433-5201 (work) Opinions are mine & do NOT represent any official position by my employer.
booker@network.ucsd.edu (Booker bense) (10/05/90)
In article <1990Oct4.050509.11405@eagle.lerc.nasa.gov> vvawh@convx1.lerc.nasa.gov (Tony Hackenberg) writes: >Does anyone know of any studies on the effect of allowing >vi usage by many users on supercomputer performance? > >Does anyone know if there are versions of "vi", "emacs", or other >full screen editors that are written to minimize performance >hits on supercomputers? > > > >-- Several people in the systems group here have studied such things, they have even developed an rvi which allows remote editing of files on the YMP. My "limited" understanding of it is that is uses tcp/ip sockets to grab the file down to your workstation. The actual editing takes place there and when you save the file , the changes go back to the YMP. People to contact about this would be schroeder@sdsc.edu or zhengc@sdsc.edu. In my experience the biggest effect of many small interactive jobs is that is makes getting any performance benefits out of auto-tasking very difficult. The more processes there are the more difficult it is to get all of your processes into the cpu. I hear rumors that Unicos 6.0 will include the idea of a "family" of processes and schedule all of them into the cpu at the same time. Also editing( depending on how the scheduler works and how big your ssd is ) can be a very frustrating task. Being swapped out for 40 secs is not fun. One of the local modifications to the kernel we have made here is to include a second-swapper for interactive jobs. Another change made was include a penalty for excessive system calls in the swapper. As a general rule , if the system overhead gets much above 7-8% , everyone feels the effects. - Booker C. Bense /* benseb@grumpy.sdsc.edu These are my own ramblings, no offical opinion , endorsement or actual fact is implied. */
alan@uf.msc.umn.edu (Alan Klietz) (10/06/90)
Credit please. Rvi was written at the University of Minnesota in 1987 and posted to comp.sources.unix volume 4. The most recent version is available via anonymous ftp to uc.msc.edu in the directory /staff/rvi. Secondly, it does not copy the file but rather it generates ``ed'' commands for execution on a remote machine. So anyway, let's talk philosophy. Simply put, you have to weigh to machine cost of keystrokes against the benefit of better user interaction. The average # of keystrokes that can reach a supercomputer remain somewhat constant while the number of cycles per unit of time that a super can offer has historically increased exponentially with each new model. So the fraction of supercomputer processing power dedicated to keystroke processing eventually crosses a critical threshold where it becomes acceptible. It reflects not only raw CPU speed but also multithreaded kernels, multiheaded machines, better I/O devices (e.g. DX Hyperchannels), better Internet latency, and so on. Historical note: Rvi was written before we did a study that showed that a single processor Cray-2 w/slow memory had less than 7% overhead in dealing with 32 simulated users in vi typing one keystroke per second. So we junked rvi and we ported vi and X10 to our machine. This was back when the debate was still raging at Cray Research. Today they offer X11 as a standard product. Granted, autotasking complicates things somewhat because it makes a context switch much more painful where stragglers can slow down striped loops. I don't know how UNICOS 6 does things, but as you suggested it ought to dedicate processors to autotasking or greatly penalize switches of active autotasked jobs. Swapping can be avoided by setting your sched parameters appropriately. There is nothing intrinsically wrong with running a screen editor on a supercomputer. -- Alan E. Klietz Minnesota Supercomputer Center, Inc. 1200 Washington Avenue South Minneapolis, MN 55415 Ph: +1 612 626 1737 Internet: alan@msc.edu
booker@network.ucsd.edu (Booker bense) (10/09/90)
In article <2720@uc.msc.umn.edu> alan@uh.msc.umn.edu (Alan Klietz) writes: >Credit please. Rvi was written at the University of Minnesota >in 1987 and posted to comp.sources.unix volume 4. The most recent >version is available via anonymous ftp to uc.msc.edu in the directory >/staff/rvi. Secondly, it does not copy the file but rather it generates >``ed'' commands for execution on a remote machine. > - This is a coincidence of names, the rvi developed here effectively incorpates ftp within vi. It operates on a client server model in which the client requests the server to either read in or write out a file. All the editing takes place on the workstation. I asked the author for an explaination of how it worked. I think the point of this work is to make more effective use of both the workstation and Ymp cpu cycles, not necessarily to avoid having screen editors on the YMP. i.e making it easier for the user to use the machine effiecently and cheaply. Our version is still undergoing testing, and is not generally available yet. I also believe a version of remacs is in development. ( I'm not working on this project, so don't take anything I say as a more than water cooler gossip. ) [stuff about philosphy deleted...] > >There is nothing intrinsically wrong with running a screen editor on a >supercomputer. > - I would second this statement. Any problems we've had with system overhead here have not been due to screen editors. - Booker C. Bense /* benseb@grumpy.sdsc.edu All disclaimers apply, i.e. if caught the Secretary will deny all knowledge of this article's existance. This disclaimer will self-destruct in 10 seconds */
boylanr@silver.ucs.indiana.edu (ross boylan) (10/09/90)
vvawh@convx1.lerc.nasa.gov (Tony Hackenberg) writes: >Does anyone know of any studies on the effect of allowing >vi usage by many users on supercomputer performance? >Does anyone know if there are versions of "vi", "emacs", or other >full screen editors that are written to minimize performance >hits on supercomputers? I recall that some documentation from NCSA (Crays under UNICOS) said that it was OK to run emacs; it didn't degrade system performance.
khb@chiba.Eng.Sun.COM (Keith Bierman - SPD Advanced Languages) (10/09/90)
In article <3831@network.ucsd.edu> booker@network.ucsd.edu (Booker bense) writes:
....
I also believe a version of remacs is in development.
I've know many gnuemacs users to use hacks to the open (spelt find)
and save (spelt write) commands. These are included in all the
versions of gnuemacs I've seen for a couple of years as
ftp-find-file
Function: FTP to HOST to get FILE, logging in as USER with password PASSWORD.
ftp-write-file
Function: FTP to HOST to write FILE, logging in as USER with password PASSWORD
Perhaps this may obviate the need for a new emacs.
--
----------------------------------------------------------------
Keith H. Bierman kbierman@Eng.Sun.COM | khb@chiba.Eng.Sun.COM
SMI 2550 Garcia 12-33 | (415 336 2648)
Mountain View, CA 94043de5@de5.ctd.ornl.gov (Sill D E) (10/09/90)
In article <3814@network.ucsd.edu>, booker@network.ucsd.edu (Booker bense) writes: >In article <1990Oct4.050509.11405@eagle.lerc.nasa.gov> vvawh@convx1.lerc.nasa.gov (Tony Hackenberg) writes: >> >>Does anyone know if there are versions of "vi", "emacs", or other >>full screen editors that are written to minimize performance >>hits on supercomputers? > >Several people in the systems group here have studied such things, they >have even developed an rvi which allows remote editing of files on the >YMP. And the there's GNU Emacs, which, right out of the box, can edit files remotely using ftp. Add Andy Norman's "ange-ftp" package, and it's almost transparent. It even does filename completion. -- Dave Sill (de5@ornl.gov) Martin Marietta Energy Systems Workstation Support
seanf@sco.COM (Sean Fagan) (10/11/90)
In article <boylanr.655424875@silver> boylanr@silver.ucs.indiana.edu (ross boylan) writes: >I recall that some documentation from NCSA (Crays under UNICOS) said >that it was OK to run emacs; it didn't degrade system performance. Ah, yes, Crays: machines where a 1-2Mb EMACS process is considered small. It isn't that the emacs didn't degrade performance, it was that it was found that using emacs (or other editors, for that matter) locally on the cray was more efficient in *programmer time* than ftp'ing the file from the cray, editing locally on your workstation, and ftp'ing it back. And, yes, programmer time is still very expensive. -- -----------------+ Sean Eric Fagan | "Never knock on Death's door: ring the bell and seanf@sco.COM | run away! Death really hates that!" uunet!sco!seanf | -- Dr. Mike Stratford (Matt Frewer, "Doctor, Doctor") (408) 458-1422 | Any opinions expressed are my own, not my employers'.
cedman@lynx.ps.uci.edu (Carl Edman) (10/12/90)
In article <8144@scolex.sco.COM> seanf@sco.COM (Sean Fagan) writes: In article <boylanr.655424875@silver> boylanr@silver.ucs.indiana.edu (ross boylan) writes: >I recall that some documentation from NCSA (Crays under UNICOS) said >that it was OK to run emacs; it didn't degrade system performance. Ah, yes, Crays: machines where a 1-2Mb EMACS process is considered small. It isn't that the emacs didn't degrade performance, it was that it was found that using emacs (or other editors, for that matter) locally on the cray was more efficient in *programmer time* than ftp'ing the file from the cray, editing locally on your workstation, and ftp'ing it back. And, yes, programmer time is still very expensive. It still seems like a terrible squandering of resources to use valueable cray preformance to do such a simple job with low (by cray standards) preformance requirements as editing files and at the same time use your local workstation as dumb terminal. The solution ? Use "ange-ftp" , a free emacs lisp file. It allows you to read and write files on any filesystem reachable via ftp completely transparently. f.e. if I want to edit a file "/tmp/foo" on the cray "y1.sdsc.edu" and use emacs here to do it I simply open a file (by the standard method) and give "/y1.sdsc.edu:/tmp/foo" as a file name. Reading, writing, dir-edit-ing, and even file name completion works just as if the entire cray filesystem was local. Isn't emacs great ? (No, please no editor flamewar. vi, is great,too) Carl Edman Theorectial Physicist,N.:A physicist whose | Send mail existence is postulated, to make the numbers | to balance but who is never actually observed | cedman@golem.ps.uci.edu in the laboratory. | edmanc@uciph0.ps.uci.edu
mike@BRL.MIL (Mike Muuss) (10/13/90)
Carl - The problem with your argument is that you implicitly assume that it is cheaper to transfer an entire file to another machine, than it is to do a little editing on the file on the Cray. In my measurements (Cray XMP48 running COS 116, Cray XMP48 running UNICOS 3, 4, and 5, Cray-2 running UNICOS 4 and 5), for "medium" sized source files, it actually used *less* Cray time to suck the file into a screen editor (JOVE and VI were tested), modify a "few" lines, and write it back out, than it was to move the file to a "front end" and then back to the Cray. (obviously, no screen editor on COS, I just measured file xfer times, and found it more expensive of Cray CPU time than a UNICOS FTP or RCP). Now, I would not suggest developing new code from scratch on the Cray, nor would I suggest using the Cray for document processing, but there is no reason in the world that Cray users should not be allowed to modify their Cray files using the Cray! Keep the bloody intermediate machines (err, "front ends") out of my way, please. Obviously, new code and document processing should be done on your workstation. Best, -Mike Muuss U. S. Army Ballistic Research Lab
booker@network.ucsd.edu (Booker bense) (10/15/90)
In article <9010130128.aa04704@WOLF.BRL.MIL> mike@BRL.MIL (Mike Muuss) writes: >Carl - > >The problem with your argument is that you implicitly assume >that it is cheaper to transfer an entire file to another machine, >than it is to do a little editing on the file on the Cray. > >In my measurements (Cray XMP48 running COS 116, Cray XMP48 running >UNICOS 3, 4, and 5, Cray-2 running UNICOS 4 and 5), for "medium" sized >source files, it actually used *less* Cray time to suck the file into >a screen editor (JOVE and VI were tested), modify a "few" lines, >and write it back out, than it was to move the file to a "front end" >and then back to the Cray. (obviously, no screen editor on COS, >I just measured file xfer times, and found it more expensive of >Cray CPU time than a UNICOS FTP or RCP). > >Now, I would not suggest developing new code from scratch on the Cray, >nor would I suggest using the Cray for document processing, but >there is no reason in the world that Cray users should not be allowed >to modify their Cray files using the Cray! Keep the bloody intermediate >machines (err, "front ends") out of my way, please. > >Obviously, new code and document processing should be done on your >workstation. > Best, > -Mike Muuss > U. S. Army Ballistic Research Lab - While I would be be the last person to say that emacs and vi should not be on a CRAY. I believe the efforts toward distributed editing to be extremely worthwhile. The consideration here is performance of the 'overall' system. An editing process requires an entry that remains in the process table for a significant amount of time using a significant portion of that time in system calls. Also your test of editing 'just a few lines' is IMHO not a very typical example of how screen editors are used. A more reasonable scenerio is the following. I ftp my code to the cray. It doesn't compile ( I tested it on a vax $%#@%$) I begin editing the offending file. I scroll up and down , Moving from declarations to the actual line of the error. Figure out that the error is a vax non-standard type declaration. Edit a few lines. quit the editor. Recompile , it runs and crashes. etc.... So while I have only made a few changes, I have been in the editor for many minutes. All this time the editor is patiently waiting for keypresses, unless of course my process got swapped out. Note: in this example I have not used any more cpu time than in the test examples above, however I have caused the scheduler that much more grief. While I do not pretend to know the inner workings of the UNICOS scheduler, I know trying to edit files at around 3pm here is an exercise in patience. As several earlier posters recommended I got the 'ange-ftp' package and I highly recommend it. The difference in response time in editing on my workstation is tremendous. I had not realized how `slow' editing on the YMP is. Also, with a windowing system, I now have the best of both worlds. In one window I'm logged onto the YMP using compile commands , in another I'm editing in the emacs running on the workstation. In summary, a screen editor should be part of any supercomputer that supports an interactive environment, however every effort should be made to move editing off that machine. This effort should be as transparent to the user as possible. -Booker C. Bense /*benseb@grumpy.sdsc.edu*/ These are my own opinions, who else would be this foolish? %-) No coffee &-) 2-3 cups 8-) Ready to read news.
ddickey@aspen04.cray.com (Dan A. Dickey) (10/15/90)
Excuse my ignorance here, but doesn't anyone out there use NFS? It seems like a fairly ideal method of editing files on your workstation, and then compiling & testing on the Cray. Just have the workstation NFS mount the Cray file system. Or am I missing something? -Dan --------- Dan A. Dickey ddickey@aspen.cray.com
mike@brl.mil (Mike Muuss) (10/18/90)
Dan - The problem with using NFS is that, in the scenario under discussion, the Cray would be required to NFS transfer the *entire* file to the editor running on the workstation, and then receive the *entire* file back to the Cray again using NFS. For changing a few lines in a medium-sized source file, this is STILL more expensive of Cray time than running the editor locally on the Cray. For highly modularized source code, where the source files are only a few dozen Kbytes, using NFS in this way is certainly a performance "win". But running a GREP through your sources via NFS is likely to be a big loose. Face it, folks: using the Cray interactively in the edit-compile-test cycle is a good, efficient use of time. Both Cray time and user time. Best, -Mike