ted@evi.com (Ted Stefanik) (03/01/91)
So there I was, waiting for another 15 minute G++ compilation caused by
a very simple change to a single class header file.
GEEEEEEEZZZ! There are six other DECStations here all twiddling their thumbs.
WHY CAN'T THIS SYSTEM USE THOSE IDLE WORKSTATIONS TO MAKE THIS MAKE GO FASTER?
WOW! GNU Make has a -j switch to kick off multiple simultaneous independent
tasks from the Makefile.
BUT... This isn't a parallel processor, its a TCP/IP and NFS-connected set of
workstations. That -j switch just makes each job go N times slower on this
system, and garbles up the command output as well.
FOO!
********************
Does this story sound familiar to you? Then you'll like the happy ending:
********************
I KNOW! I'll take the gsh perl script found in the perl-3.0@44/eg/g directory,
and totally revamp it to handle distributing a command to the least loaded
system. And I'll add an output disentangler. That -j GNU Make flag will work
just fine!
Now my 15 minute recompile only takes 5 minutes!
*******************************************************************************
#! /bin/sh
# This is a shell archive. Remove anything before this line, then unpack
# it by saving it into a file and typing "sh file". To overwrite existing
# files, type "sh file -c". You can also feed this as standard input via
# unshar, or by typing "sh <file", e.g.. If this archive is complete, you
# will see the following message at the end:
# "End of shell archive."
# Contents: mysh.1 mysh
# Wrapped by ted@blt on Thu Feb 28 16:30:49 1991
PATH=/bin:/usr/bin:/usr/ucb ; export PATH
if test -f 'mysh.1' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'mysh.1'\"
else
echo shar: Extracting \"'mysh.1'\" \(3732 characters\)
sed "s/^X//" >'mysh.1' <<'END_OF_FILE'
X.TH MYSH 1
X.SH NAME
Xmysh \- execute a command on the least loaded system
X.SH SYNOPSIS
X.B mysh
X.RB [ \-d ]
X.RB [ \-s ]
X.RB [ \-c ]
X.RI command
X.SH DESCRIPTION
X.I Mysh
Xis a wrapper for
X.B ruptime, rsh,
Xand
X.B sh
Xthat executes a command on the least
Xloaded system. The systems should have some sort of file system sharing
Xlike NFS.
X.PP
X.I Mysh
Xfirst reads a
X.I ghosts
Xconfiguration file to determine the list of systems that
Xyou will allow the command to be executed on. It then does a
X.B ruptime
Xto determine the least loaded system. Next, it wraps the command up and
Xsends it via
X.B rsh
Xto the least loaded system to execute. (The wrapper includes a
X.B cd
Xto the current directory; hence the need for a shared file system.)
XLast, it collects the results from the command and prints them all at once.
X.PP
X.B Mysh
Xuses a "busy" file to avoid overloading any particular system, and a lock file
Xto arbitrate use of the busy file plus single thread command result outputs
Xfrom multiple invocations.
X.PP
XThe available command line options are:
X.TP 5
X.B \-l
XTurns on debugging output.
X.TP
X.B \-s
XUses
X.B rsh
Xand
X.B pstat
Xto find system with most swap space instead of using
X.B
Xruptime
Xto find the system with the least load.
X.TP
X.B \-c
XIgnored if present, so
X.B
Xmysh
Xcan eat a
X.I mysh -c
Xcommand originally meant for the Bourne shell.
X.TP
X.I command
XThe command to be executed.
X.PP
XThe command line options must be used in the order specified above.
X.PP
X.I Mysh
Xalso looks for two environment variables,
X.B MYSHLOCAL
Xand
X.B MYSHQUIET:
X.TP
X.B MYSHLOCAL
Xcontains a list of commands that
Xmust be executed on a particular system. For example:
X.sp
X.in +.5i
Xsetenv MYSHLOCAL "echo make g++:ralph"
X.in
X.sp
Xmeans to execute the echo and make commands only on the local system,
Xand execute the g++ command only on the system named ralph.
X.TP
X.B MYSHQUIET
Xtells
X.B mysh
Xto execute quietly. If it contains a numeric value, then commands are
Xtruncated to that number of tokens. If it is blanks or contains only
Xnon-numerics, then commands are never echoed. However, MYSHQUIET will not
Xquash output from the remotely executed command.
X.sp
X.nf
X.in +.5i
Xsetenv MYSHQUIET "" -- Don't display commands or systems
Xsetenv MYSHQUIET 5 -- Display system and truncated commands
X.in
X.fi
X.sp
X.SH USING MYSH WITH GNU MAKE
XYou can specify MYSHLOCAL and MYSHQUIET in a GNU Make Makefile; GNU Make
Xexports all definitions into the environment. For example:
X.sp
X.nf
X.in +.5i
XMYSHLOCAL=sccs date echo make
XMYSHQUIET=5
X.in
X.fi
X.PP
X.B Mysh
Xwas designed to be used with GNU Make's
X.RB -j
Xoption, allowing your makes to be executed in parallel on multiple processors
Xconnected by only the rsh command and a shared file system. For example, if
Xyou have at least four mostly idle processors,
X.sp
X.in +.5i
Xmake -j 4 SHELL=mysh
X.in
X.sp
Xshould cause your make to execute 2 to 3 times faster.
X.SH FILES
X.RB . \^ghosts
X.br
X.RB $HOME/ . \^ghosts
X.br
X.RB /usr/local/lib/ghosts
X.br
X.RB $HOME/ . \^mysh.busy
X.br
X.RB $HOME/ . \^mysh.lock
X.SH RESTRICTIONS
X.B Mysh
Xassumes that the
X.I command
Xis destined for execution under the Bourne shell.
X.PP
X.B Mysh's
Xmethod of preserving quoting may break under some shells and quoting
Xmechanisms.
X.PP
X.B Mysh
Xassumes that the file system can recognize the first element in the path
Xas a system name. For example,
X.I blt
Xis a system name in:
X.sp
X.in +.5in
X/blt/usr/users/ted/tests
X.in
X.sp
XApollos use something like this (the first element of
Xthe path is preceeded by two slashes), but you'll have to modify
X.B mysh
Xto have it work under Domain OS. For NFS, you must either sanely choose your
Xmount points, or manually set up symbolic links.
X.SH AUTHOR
XTed Stefanik, Expert Views, Inc.
X.sp
XSend bugs and remarks to <ted@evi.com> .
END_OF_FILE
if test 3732 -ne `wc -c <'mysh.1'`; then
echo shar: \"'mysh.1'\" unpacked with wrong size!
fi
# end of 'mysh.1'
fi
if test -f 'mysh' -a "${1}" != "-c" ; then
echo shar: Will not clobber existing file \"'mysh'\"
else
echo shar: Extracting \"'mysh'\" \(9995 characters\)
sed "s/^X//" >'mysh' <<'END_OF_FILE'
X#!/usr/bin/perl
X
X#
X# mysh - A wrapper for sh and rsh that executes a command on the least loaded
X# system of a NFS-connected set of systems.
X#
X# Created by: Ted Stefanik <ted@evi.com>
X# February, 1991
X#
X# Copyright 1991 by Ted Stefanik and Expert Views, Inc. All Rights Reserved.
X#
X# Permission to use, copy, modify, and distribute this software and its
X# documentation for any purpose and without fee is hereby granted, provided
X# that the above copyright notice appear in all copies.
X#
X# This code is distributed in the hope that it will be useful, but WITHOUT ANY
X# WARRANTY. Ted Stefanik and Expert Views, Inc. disclaim all warranties with
X# regard to this software, including all implied warranties of merchantability
X# and fitness, and in no event shall be liable for any special, indirect or
X# consequential damages or any damages whatsoever resulting from loss of use,
X# data or profits arising out of or in connection with the use or performance
X# of this software.
X#
X
X&InitConstants();
X&ParseArgs();
X&GetBusySystems();
X&ReadHosts();
X&ReadRups();
X&ReadSwps();
X&PickLucky();
X&SetBusySystems();
X&DoCommand();
X&ResetBusySystems();
X
Xexit($Status);
X
X
X#
X# Initialize the Manifest Constants
X#
X# Note: You may want to change $LIB.
X#
Xsub InitConstants
X{
X $TRUE = 1;
X $FALSE = 0;
X
X $LOCK_SH = 1;
X $LOCK_EX = 2;
X $LOCK_NB = 4;
X $LOCK_UN = 8;
X
X $L_SET = 0;
X $L_INCR = 1;
X $L_XTND = 2;
X
X $LIB = '/usr/local/glib';
X $HOME = $ENV{'HOME'} || $ENV{'LOGDIR'} || (getpwuid($<))[7] || '';
X $LOCKFILE = "$HOME/.mysh.lock";
X $BUSYFILE = "$HOME/.mysh.busy";
X
X select (STDOUT); $| = $TRUE;
X
X chop($hostname = `hostname`);
X chop($cwd = `pwd`);
X local(@cmd) = split(m|/|, $cwd);
X local($testhost) = gethostbyname($cmd[1]);
X if (defined($testhost))
X {
X $nfsd = $cwd;
X }
X else
X {
X $nfsd = "/$hostname$cwd";
X }
X
X if (defined($ENV{'MYSHLOCAL'}))
X {
X @special = split(/\s+/, $ENV{'MYSHLOCAL'});
X foreach $spec (@special)
X {
X ($cmd, $host) = split(/:/, $spec);
X $host = $hostname
X if ($host eq '');
X $require{$cmd} = $host;
X }
X }
X
X return (undef);
X}
X
X
X#
X# Parse the argument list.
X#
X# Note: No syntax errors are possible.
X#
Xsub ParseArgs
X{
X if (@ARGV[0] eq '-d')
X {
X $debug = $TRUE;
X shift(@ARGV);
X }
X
X $swap = 1;
X if (@ARGV[0] eq '-s')
X {
X $swap = -1;
X shift(@ARGV);
X }
X
X shift(@ARGV)
X if (@ARGV[0] eq '-c');
X
X @ToDo = @ARGV;
X @ToDoPrn = split(/\s+/, "@ToDo");
X $basecmd = $ToDoPrn[0];
X
X if (defined($ENV{'MYSHQUIET'}))
X {
X if ($ENV{'MYSHQUIET'} =~ /(\d+)/)
X {
X @ToDoPrn = split(/\s+/, "@ToDo");
X splice(@ToDoPrn, $1, $#ToDoPrn - $1 + 1,
X ($#ToDoPrn >= $1) ? '...' : '');
X }
X else
X {
X undef(@ToDoPrn);
X }
X }
X
X return (undef);
X}
X
X
X#
X# Get the list of interesting hosts from the ghosts file.
X#
Xsub ReadHosts
X{
X @ARGV = ();
X
X push(@ARGV, '.ghosts')
X if (-f '.ghosts');
X push(@ARGV, "$HOME/.ghosts")
X if ($#ARGV == -1 && $HOME ne '' && -f "$HOME/.ghosts");
X push(@ARGV, "$LIB/ghosts")
X if ($#ARGV == -1 && -f "$LIB/ghosts");
X
X die("\nCouldn't find any hosts!\n")
X if ($#ARGV < 0);
X
X line: while (<>)
X {
X next line if /^#/;
X next line if /^$/;
X next line if /=/;
X ($host) = split;
X $Wanted{$host} = $TRUE;
X push(@Hosts, $host);
X $found = TRUE;
X }
X
X die("\nCouldn't find any hosts!\n")
X if (!$found);
X
X return (undef);
X}
X
X
X#
X# Get the systems' load averages from the ruptime command
X#
X# Note: Each system that is running a mysh command from this user is
X# penalized an extra amount. See ReadBusyFile() for more details.
X#
Xsub ReadRups
X{
X open(RH, 'ruptime|') || die "\nCan't run ruptime: $!\n";
X
X while (<RH>)
X {
X ($host,$upness,$foo,$users,$foo,$foo,$load) = split;
X chop($load);
X $load += $penalty{$host};
X if ($Wanted{$host} && $upness eq 'up')
X {
X $load{$host} = $load;
X $host{$load} .= "$host ";
X }
X }
X
X close(RH);
X
X return (undef);
X}
X
X
X#
X# If requested, get the systems' free swap amounts from "rsh <sys> pstat -s".
X#
Xsub ReadSwps
X{
X return(undef)
X if ($swap == 1);
X
X @ups = keys(%load);
X undef(%load);
X undef(%host);
X
X foreach $host (@ups)
X {
X print STDERR "Checking free swap on $host... ";
X open(RH, "rsh $host -n pstat -s|") || die "\nCan't run ruptime: $!\n";
X
X while (<RH>)
X {
X $load = $1/1000
X if (/^\s+(\d+)k free/);
X $load{$host} = $load;
X $host{$load} .= "$host ";
X }
X
X close(RH);
X print STDERR "done\n";
X }
X
X return (undef);
X}
X
X
X#
X# Pick the lucky system that has the most resources available (CPU or swap).
X#
X# Note: Any system whose resource level (load average or megabytes of free swap
X# space) is within .5 of the system with the best resource level is
X# a candidate. The candidate that is nearest the top of the ghosts file
X# is one selected.
X#
X# Note: The MYSHLOCAL environment variable can override the selection.
X#
X#
Xsub PickLucky
X{
X sub loadorder { $a > $b ? (1 * $swap) : $a < $b ? (-1 * $swap) : 0; }
X @loadlist = sort(loadorder values(%load));
X $best = $loadlist[0];
X
X if ($debug)
X {
X print STDERR "Loadlist=|@loadlist|\n";
X foreach $host (keys(%load))
X {
X printf STDERR "%-10s%5.2f\n", $host, $load{$host};
X }
X print STDERR "Best=$best\n";
X }
X
X sub abs { return ((@_[0] < 0) ? -@_[0] : @_[0]); }
X getbest: foreach $sys (@Hosts)
X {
X if (defined($load{$sys}) && &abs($load{$sys} - $best) <= .5)
X {
X $Lucky = $sys;
X last getbest;
X }
X }
X
X $Lucky = $require{$basecmd}
X if (defined($require{$basecmd}));
X
X print "Using $Lucky for \"@ToDoPrn\"\n"
X if ($#Hosts != 0 && defined(@ToDoPrn));
X
X return (undef);
X}
X
X
X#
X# Execute the requested command on the lucky system.
X#
X# Note: Uses a command like "rsh $lucky -n exec sh -c $command" to execute.
X#
X# Note: Hacks and wacks to preserve the quoting through the local system()
X# command and the target system's csh invoked by the rsh.
X#
X# Note: Hacks and wacks to preserve the command's exit status which is lost
X# by rsh.
X#
Xsub DoCommand
X{
X $ToDo = "@ToDo";
X $ToDo =~ s/'/\033/g;
X $ToDo =~ s/\033/'"'"'/g;
X $ToDo1 = "sh -c 'cd $nfsd ; $ToDo 2>&1 ; echo @@@Status@@@ = \$?'";
X $ToDo2 = $ToDo1;
X $ToDo2 =~ s/'/\033/g;
X $ToDo2 =~ s/\033/'"'"'/g;
X $cmd = ($Lucky eq $hostname) ? $ToDo1 : "rsh.inf $Lucky -n 'exec $ToDo2'";
X
X open(SH, $cmd . "|") ||
X die("\nCan't execute remote command on $Lucky\n");
X
X while (<SH>)
X {
X if (/@@@Status@@@ = (\d+)/)
X {
X $Status = $1;
X }
X else
X {
X push(@Results, $_);
X }
X }
X
X close(SH);
X
X $Status |= $? | ($? >> 8);
X
X return (undef);
X}
X
X
X#
X# Get the list of systems that this user is currently running mysh commands on.
X#
Xsub GetBusySystems
X{
X &LockBusyFile();
X &ReadBusyFile();
X
X return (undef);
X}
X
X
X#
X# Rewrite the list of systems on which this user is currently running mysh
X# commands, and include this mysh command as one of them.
X#
Xsub SetBusySystems
X{
X $pids{$$} = $Lucky;
X &WriteBusyFile();
X &UnLockBusyFile();
X
X return (undef);
X}
X
X
X#
X# Rewrite the list of systems on which this user is currently running mysh
X# commands, but delete this mysh command from the list.
X#
X# Note: The results of this mysh command are written out during the time the
X# lock file is locked so that the results from multiple mysh commands
X# are kept separate.
X#
Xsub ResetBusySystems
X{
X &LockBusyFile();
X if ($#Results != -1)
X {
X print "Results from $Lucky for \"@ToDoPrn\":\n"
X if ($#Hosts != 0 && defined(@ToDoPrn));
X print @Results;
X }
X &ReadBusyFile();
X delete $pids{$$};
X &WriteBusyFile();
X &UnLockBusyFile();
X
X return (undef);
X}
X
X
X#
X# Lock the file that contains the list of systems on which this user is
X# currently running mysh commands.
X#
Xsub LockBusyFile
X{
X if (! -f $LOCKFILE || ! -f $BUSYFILE)
X {
X system("touch $LOCKFILE $BUSYFILE");
X sleep(2);
X }
X
X open(LH, "+>>$LOCKFILE") ||
X die("\nCan't open lock file \"$LOCKFILE\": $!\n");
X flock(LH, $LOCK_EX);
X
X return (undef);
X}
X
X
X#
X# Unlock the file that contains the list of systems on which this user is
X# currently running mysh commands.
X#
Xsub UnLockBusyFile
X{
X flock(LH,$LOCK_UN);
X close(LH);
X
X return (undef);
X}
X
X
X#
X# Read the file that contains the list of systems on which this user is
X# currently running mysh commands.
X#
X# Note: Each system that is running a mysh command from this user is
X# has its system load penalized an extra .75; this prevents four
X# simultaneous myshs from starting up a job on the same system.
X# It also makes sense from the point of view of the three minute
X# ruptime delay. And last, it tends to evenly scatter a long string
X# of simultaneous myshs over all available hosts.
X#
Xsub ReadBusyFile
X{
X undef %penalty;
X undef %pids;
X
X open(BH, "+>>$BUSYFILE") ||
X die("\nCan't open busy file \"$BUSYFILE\": $!\n");
X seek(BH, 0, $L_SET);
X
X actives: while (<BH>)
X {
X chop;
X ($host, $pid) = split;
X
X last actives if ($host eq "DONE!");
X next actives if (!kill(0, $pid));
X
X $pids{$pid} = $host;
X $penalty{$host} += 0.75;
X }
X
X if ($debug)
X {
X @list = %pids;
X print STDERR "Current pids/hosts: |@list|\n";
X @list = %penalty;
X print STDERR "Current penalties: |@list|\n";
X }
X
X return (undef);
X}
X
X
X#
X# Write the file that contains the list of systems on which this user is
X# currently running mysh commands.
X#
Xsub WriteBusyFile
X{
X seek(BH, 0, $L_SET);
X
X while (($pid, $host) = each %pids)
X {
X print BH "$host $pid\n";
X }
X print BH "DONE!\n";
X
X close(BH);
X
X return (undef);
X}
END_OF_FILE
if test 9995 -ne `wc -c <'mysh'`; then
echo shar: \"'mysh'\" unpacked with wrong size!
fi
chmod +x 'mysh'
# end of 'mysh'
fi
echo shar: End of shell archive.
exit 0