[comp.unix.admin] How many administrators needed per site?

boyko@rastro.uucp (christina boyko) (01/16/91)

A while ago, I saw a posting calling for input about how many administrators
any given site should have.  I believe the poster was going to gather the
info and publish the stats.  I never caught the results.

Well, due to "budget cutbacks", I am in danger of losing my
partner/fellow administrator.  It seems they think I'm good
enough to handle this site all by myself.  I'm flattered, but mostly,
I'm  panicked.  I don't think that this is a 1-person job. (I have
~50 Suns on site and ~15 PC's)

So to justify my partner's existance, I need to know:
 What's the average administrator-to-machine ratio?
   
 1:20, 1:50, 1:100? (This assumes that the users do absolutely
 no administrative tasks.---They don't even have root access.)
(My personal feeling is 1:25 at most, but management thinks 1:60....)
 
 How about the average administrator-to-USER ratio. Is it different
 from admin:machine?
 
 If anyone remembers the posting I'm referring to and can
 recall the stats, please let me know. I'm also willing to
 accept input and compile the info myself. Either
 post, of better still, send me email.  I appreciate any
 help I can get. 
 
 Thanks again,
 
 Christina Boyko
 System Administrator
 rastro!boyko@uunet.UU.NET

verber@pacific.mps.ohio-state.edu (Mark Verber) (01/16/91)

   A while ago, I saw a posting calling for input about how many administrators
   any given site should have.  I believe the poster was going to gather the
   info and publish the stats.  I never caught the results.

I didn't recall seeing the results either.

   Well, due to "budget cutbacks", I am in danger of losing my
   partner/fellow administrator.  It seems they think I'm good
   enough to handle this site all by myself.  I'm flattered, but mostly,
   I'm  panicked.  I don't think that this is a 1-person job. (I have
   ~50 Suns on site and ~15 PC's)

   So to justify my partner's existance, I need to know:
    What's the average administrator-to-machine ratio?

    How about the average administrator-to-USER ratio. Is it different
    from admin:machine?

The administrator to machine ratio that is reasonable depends on what
level of service is expected and what responsibilities the sysadm has.
In addition to these two parameters, there are two other issues that
will temper the equation: (1) Single sysadm sites are not very
desirable (except financially) in most cases.  Is that it is difficult
for a single person to have the breadth of knowledge and experience to
run a really first class operation, no matter how few machine you
have.  There will always be some area that a single person will be
weak on.  It also means that when the sysadm is on vacation, (or gets
run over by a bus) the site is vulnerable.  Carrying a sky-pager on
vacation isn't my idea of fun, and no one can predict when a bus might
strike.  (2) The larger a site is, the less people that are needed
because you can get a good economy of scale.  I have seen at larger
sites at a 1:100 sysadm-to-machines where things ran pretty well,
although more people would have been nice.

I am not sure that there are any hard and fast rules for ratios.  The
thought of administrating 60 Suns that are more or less alike, a
server or three with mainly diskless or dataless clients that are
clones of each other wouldn't scare me.  That is a pretty manageable
task.  You would *need* need to be proactive rather than reactive to
tasks, make use of existing tools or building some for yourself that
permits your work to be multiplied, eg. running something like
rdist,target,sup for automatic updates, setting up backups to run
automatically (how did we ever live without exebytes), etc, etc.

A lot would depend on the level of support your users expect.  The
best way to figure out how many people are needed, it to figure out
what level of service your users expect and whether the environment
helps or hurts the maintainces of software.  Rarely are sysadms doing
only UNIX sysadm tasks.  Here is a list of things that I find myself
doing in addition to the normal "sysadm" tasks like backups, account
installs, OS maintaince, etc.

(1) User Services

How much hand holding is expected?  Some sites have users than are
pretty self-sufficient.  Other sites have users than need their hands
held for everything.  Can yours take care of themselves, or do the
need/want the admin to do the simplest tasks for them.  For example, I
have a friend whose users would demand him to do the most basic
things for them.  Things like moving their files from one directory to
another.  (Anything that wasn't running the text editor and reading
mail was Unix, therefore a job for the adm).  This sort of support
requires something like a 1:2 ratio.

Does the site want you to conduct workshops, prepare extensive local
documentation?  To what extent is the sysadmin expected to consult on
technical issues?  Just using Unix, or other realms.  For example,
lets say your site has heavy users of FrameMaker, TeX, Mathamatica,
Common Lisp, C++, X11, PostScript, and Sybase.  Is the sysadms suppose
to be able to answer detailed questions on all those topics?  One
person can not be an expert on all these things.  Something that
many people don't appreciate is that if someone is to consult on
a topic area, they need some time to play, experiment, and generally
develop in that area.

(2) Diversity of Arch/OS/Setup

Of course, each type of machine (arch/OS) that a site has multiples
the complexity of the support task, especially if your users expect
all machines to behave identically.  Each install program will have
to be installed N times.  OS upgrades will have to be done at least
once by hand for each arch/OS, etc, etc.

Workstations can be arranged in ways that make it easier, or harder to
do distributed administration on.  If you can configure all your
workstations to use the identical configuration, and have identical
tools, etc, it is easy to support a larger number of machines.  If
each your machines is configured differently: swap sizes different,
some diskless, some dataless, some diskful, with different software
installed, etc. you are going to have headaches.  If you can have one
'prototypical' machine that you can clone from, installs, upgrades,
etc can be done pretty easily/automatically.  For example, lets say
you run a dataless configuration with /, /var, and parts of /usr on a
local disk, and everything else accessed via an automounter.
Your could have a diskless partition on a server that would boot
up, install SunOS on a disk, do your local customizations, and reboot
as a newly configured workstation ready for action just by editing
your ethers file and a configuration file.  If you have to do each
machine by hand, you will have to waste a lot of times every time you
install a new machine or have to do an OS upgrade.

(3) Software Support?

How much PD/freeware software do people want installed, and what level
of support are they expecting?  Just compiling and installing software
doesn't take much time, but often times (and rightly so) a sysadm is
expected not only to compile and install new software, but to test the
software before people get at it, to debug any problems, port it if
necessary, and be a general expert.  This takes time which varies with
the quality and complexity of the software.  Keeping a current version
of kermit or perl isn't hard (I wish everyone did as nice a job as
Larry has with perl), keeping ontop of g++ takes a quite a bit more
time.

(4) Custom Software?

Most places not only expect the sysadms to keep the world running, but
create tools for the user population when needed.  This is
understandable, especially in small site where the sysadm might be the
decent programmer.  If there is this expectation, time must be given
for the development process.

(5) Site Planning/Admin Overhead

How much site planning is the sysadm expected to handle.  Are you
going to have to know about AC/heating loads and power?  How much
paperwork is there?

(6) Hardware/Network Maintaince

Who crawls through the ceiling to pull wires?  How finds the flaky
transceiver when the ethernet starts to go crazy?  When a terminal or
workstation dies do you just call your vendor and wait, or are more
creative solutions required.  Do you buy all of your peripherals ready
to install, or do you save money by purchasing components and do the
integration yourself?  All of these things take time.

(7) Leading Technology

Is the sysadm suppose to anticipate new technology and advise the
company as to where they might like to take there environment?
Most places I have been, the syadm were expected to have a good feel
for what was state of the art and what was looking promising.  Not
just products, but research.  Keeping up with what is going isn't
easy.  trade rags can give you a picture of what is being sold, but
they aren't particularly good at helping people to anticipate what
might be in a year or three.  Forward looking is often necessary given
many sites have a 2-5 year planning and/or depreciation schedule.

Cheers,
Mark

steve@archone.tamu.edu (Steve Rikli) (01/17/91)

In article <1991Jan15.230613.8451@rastro.uucp> boyko@rastro.uucp (christina boyko) writes:
>
> What's the average administrator-to-machine ratio?
>   
> 1:20, 1:50, 1:100? (This assumes that the users do absolutely
> no administrative tasks.---They don't even have root access.)
>(My personal feeling is 1:25 at most, but management thinks 1:60....)
> 
> How about the average administrator-to-USER ratio. Is it different
> from admin:machine?
  
Here in the College of Architecture's Visualization Lab, we have

	- a sun4 server
	- ~15 sun3's
	- ~10 sparc's
	- 2 NeXT's
	- 8 SGI's of various flavors
	- 40-odd PC's
	- 30-odd Mac's

There are approximately 100 users with active accounts, but I would
say only about 20-30 are regular users.

Of the Mac's and PC's, only a handful have direct connections to the
server, so they are not directly our concern.  In addition, there
are various printers, scanners, and other peripherals that fall under
our jurisdiction.

The "we" above consists of a full-time system administrator, and a 
half-time assistant system-administrator (who is a grad student).
There is also a full-time staff member to handle most Mac and PC
problems, so there will naturally be some overlap.

          _________________________________________________________
         /  steve@archone.tamu.edu   /   Work:  (409) 845-3465    /
        /  srr2632@sigma.tamu.edu   /   Home:  (409) 696-0910    /
       /___________________________/____________________________/
      /  Steve Rikli, Assistant System Manager                 /
     /  Visualization Lab, College of Architecture            /
    /  Texas A&M University                                  /
   /  College Station, TX 77844                             /
  /________________________________________________________/

anselmo-ed@CS.YALE.EDU (Ed Anselmo) (01/17/91)

Yale Computer Science has a Facility composed of 1 Facility Director,
1 Manager of Development, 2 Sr. Systems Prog. (I'm one of them), 1
Manager of Operations, 1 Operations programmer, 1 Operations staff.

This is down from 4 Sr. Programmers, 2 Operations programmers, and 2
Operations staff 2 years ago.  And they've cut out the weekend backup
operators too.

We manage about 150 Suns + miscellaneous other machines (Connection
Machine, Intel Hypercube, Encore Multimax, Sequent Symmetry, some
IBM-RT's, IBM RS/6000, DEC-5000's).  Maybe 200 machines in all.

The user community is maybe 600 (undergrads, grads, faculty, and
staff).

I was the sole support person in my last job (5 Suns, 40 IBM-PCs, 10
Macs).  They expected me to do development when I could barely keep
all the machines and printers alive from day to day. Yeah, sure.

>> (1) User Services

Users get daily backups, except on weekends.  A staff person is on
call on weekends for extreme emergencies, otherwise we're a 9-to-5
weekdays operation.

We seem to be fair game for questions on anything that falls under the
"supported" category:  TeX, X, mail, news, networking, etc.  Each the
development staff is at least dimmly aware of every major software
package we support, though "The Other Guy" handles TeX, I field news,
mail, and networking questions, and we split up the X support.

>> (2) Diversity of Arch/OS/Setup

Since CS switched to all Sun Sparcstations, maintenance has become
much easier.  2 years ago, the facility supported 4 workstation
architectures: Apollo, HP, IBM, and Sun.  Now it's pretty much just
Sun, though we just got in 10 DECstations, and a similar number of
Macs.

>> (3) Software Support?

The raging issue amongst the Powers That Be is just what software is
to be supported and at what levels.  At our current staffing level,
it's basically impossible to do feature enhancement; fixing serious
bugs is doable; but mostly, we just install/port the software, and
rely on others to provide us with fixes.

>> (4) Custom Software?

We support several locally written pieces of software, like the
magical software that hides all the userids in CS behind the
"lastname-firstname" alias for the purposes of news and mail, the User
Database program (manages accounts/uids/mailing-lists), the "autodump"
program that manages the file backups, an editor, and a mail reader.

>> (5) Site Planning/Admin Overhead

The facility director gets stuck with this job.

>> (6) Hardware/Network Maintaince

We have on-site Sun hardware support.  Plus a relatively new building
with professionally installed thicknet.  I did my time in the Midnight
Wiring Crew at my last job.  Ugh.

We let our hardware technician take care of most things.  (Thankfully,)
he yells at me when I start poking around the multiports in the comm.
closets.  So I let him do the work.

>> (7) Leading Technology

"I'm reading news so that I can keep up with new technologies.  YEAH,
that's it, NEW TECHNOLOGIES, that's the ticket."
-- 
Ed Anselmo   anselmo-ed@cs.yale.edu   {harvard,cmcl2}!yale!anselmo-ed

seeger@thedon.cis.ufl.edu (F. L. Charles Seeger III) (01/17/91)

In article <1991Jan15.230613.8451@rastro.uucp> boyko@rastro.uucp (christina boyko) writes:
| A while ago, I saw a posting calling for input about how many administrators
| any given site should have.  I believe the poster was going to gather the
| info and publish the stats.  I never caught the results.

I, too, would like to see more information on this subject, especially from
big name CS departments.  Frankly, I'm looking for ammunition for improving
the situation here.

| Well, due to "budget cutbacks", I am in danger of losing my
| partner/fellow administrator.  It seems they think I'm good
| enough to handle this site all by myself.  I'm flattered, but mostly,
| I'm  panicked.  I don't think that this is a 1-person job. (I have
| ~50 Suns on site and ~15 PC's)

Depends on more details, but it is very unlikely to be a 1-person job.

| So to justify my partner's existance, I need to know:
|  What's the average administrator-to-machine ratio?

Here is brief description of the situation here.  I consider this to be
a very bad (understaffed) situation.  I don't know if this will help
bolster your position or not.  I don't think your employers will want
their system staff to be as overloaded as we are.  Everyone suffers.

We have one Senior Sys. Programmer (me), one Sys. Programmer, and one
Electronic Technician.  We have about 10 part time student employees
(20 hrs/wk for an assistant for the E-Tech, 20 hrs/wk for backups, about
25 hrs/wk for software and documentation support, and about 105 hrs/wk for
manning our workstation lab).  We have no secretarial or other administrative
support assigned for managing our facilities.  The second Sys. Prog.
position has existed only since the first of last July.

The E-Tech spends much of his time handling administrivia such as shipping
and receiving, inventory control, purchasing supplies, handling support
contracts.  I do all the vendor interaction for purchasing new equipment.
Accounting and other business functions in our department office are a
sick joke.

We have 80+ Suns, 7 HPs, 4 RTs, 1 RS6k, 1 Ardent, 1 Gould PowerNode, various
terminals, several PCs and two Macs.  We have 27 faculty members, a passwd
file with about 1000 entries, and about 6 GB of user disk space with 3 more
on the way.  We also have some non-standard equipment, such as two different
Transputer systems and two different specialized computer vision systems.
Our /local partition consumes much more space than all of SunOS (including
OW2 and the unbundled compilers).

We also run Florida's biggest news feed and uucp machine.  Mail sent to
postmaster@ufl.edu goes into my mailbox.  Thank dmr that someone else
handles assigning IP addresses on this campus.

Enough griping about the situation here.  How do *real* CS departments
stack up?  We've heard from UMd Eng (but not CS), Yale, Ohio State Physics
(but not CS).  What about Stanford, UCB (or UC*), Michigan, Illinois, CMU,
MIT, Harvard, Purdue, PSU, Utah, GaTech, Texas, etc.?

Bombard me with useful ammunition, please.

Regards,
Chuck
--
  Charles Seeger    E301 CSE Building             Office: +1 904 392 1508
  CIS Department    University of Florida         Fax:    +1 904 392 1220
  seeger@ufl.edu    Gainesville, FL 32611-2024

romig@brachiosaur.cis.ohio-state.edu (Steve Romig) (01/18/91)

>   Enough griping about the situation here.  How do *real* CS departments
>   stack up?  We've heard from UMd Eng (but not CS), Yale, Ohio State Physics
>   (but not CS).

We've got ~230 diskless SLCs, served by 21 Sun 3/180 file servers (and
a 4/280 and a 4/330); ~10 diskless HP somethings served by 1 HP file
server; 4 Pyramids; 1 Multimax; 1 Butterfly; 300+ Macs; and some odds
and ends.  There are roughly 9 different hardware/software platforms
that we currently support (sun3 running sunos 4.1, sun4 running sunos
4.1, etc).  This is all connected through 1 main ethernet (our
backbone), 24+ Ethernet subnets and a bunch of Appletalk stuff that I
don't want to know anything about.  

Our users: roughly 1700-1800.  45 faculty, 200 grad students, rest are
undergrads or guest acounts.  These facilities are for instruction and
research in the Computer and Info Sciences Department at OSU.  That
count doesn't include the students using the Macs in the low level
courses, which is probably another 1500 folks or so.

Our staff: We're split into 3 parts: software, hardware and
operations.

    Software staff deals with software development and systems
    installation, maintenance and bug tracking/fixing.  Consists of 8
    full time folks (6 Unix, 1 Mac, 1 Unix/parallel research support)
    and 8 part time folks (grads and undergrads).  We're all (but 1)
    general Unix folks, though we each tend to specialize in different
    areas (X, postscript, printers and text processing stuff,
    networks, mail, news, strange languages, ntp, nameservers, etc).

    Hardware staff deals with hardware install, maintenance, advice on
    upgrades and etc.  We do almost all of our Sun (and I think most
    of our Mac support) in house, at the board component level.  We
    also do most of our own peripheral integration (select, buy and
    install disks, tapes, etc).  The rest is through support contracts
    with the vendors.  Hardware consists of 2 full time folks and
    something like 6 part time folk.  Oh, they take care of the nets
    too.

    Operations staff deals with keeping things running: acount
    installation, maintenance, file system stuff (creating and
    maintaining user and project directories, backups, restores,
    handling common problems and emergencies) annnnd they are
    stationed in the labs when they are open to handle problems,
    answer questions, and keep people from walking away with or
    destroying machines.  1 full time person, 35 part time.

We don't do much in the way of course-ware development, but do do alot
of consulting type things with our various users.  The software staff
(especially) is expected to and is trying to do more development type
work (make new/better sysadmin tools, better user interface type
things, etc), though our main "purpose" is to keep things running and
reasonably up to date.

Lessons we've learned:

    Keep everything as much the same as possible.  All of our Sun
    clients are clones of a master copy, all of the servers are clones
    of a master server, all of the Pyramids look alike, etc.

    Reduce the number of platforms as much as possible.  We used to
    have something like 13 platforms, we're down to 9, and may soon be
    down to 7 if we lose the Pyramids...In my mind, though reducing
    the number of platforms is nice, you have to balance that against
    having a rich environment, which is also nice.

    Localize local changes to /usr/local (or some scheme like that) as
    much as possible, which makes upgrades easier.  Try to refrain
    from hacking on and reinstalling local versions of things in /bin,
    /usr/ucb, and so on.

    Diskless workstations are your friend, as long as you have enough
    memory on them.  It takes me about 2 hours to install a new copy
    of / on all 220+ diskless SLCs, including shutting them down,
    copying the stuff and bringing everything up again.

    You have to strike a balance between keeping things up to date and
    spending too much time keeping things up to date.  I try to settle
    on a SunOS release that seems reasonably stable and stay there for
    a long time, for example.  We were at SunOS 3.5.1 for a very long
    time.  We're at 4.1 now, and I'm still searching for a point of
    stability...:-)  

    Build tools to do things, rather than doing it "by hand" - if you
    do something once, you'll do it again.

    Beg, borrow and steal (only kidding) software from others when you
    can. 

(Mark came up with the last two suggestions...)

--- Steve