[comp.software-eng] more on estimation

cliffhanger@cup.portal.com (Cliff C Heyer) (07/22/90)

[The following by cliffhanger@cup.portal.com] Bill writes...
>The ability to ship a prototype as a finished product is one of the
>curses of our field. Imagine where other engineering disciplines [and society]
>would be if it were possible to clone a breadboard or a prototype car at a
>cost comparable to that of turning out a production engineered version.
>A lot of quality is usually added to a product during production engineering.

This is an excellent alternative way to express my point! Thanks Bill.

> No contractor
>is going to bid on a project unless the design product is demonstrably
>well thought out.  ....
or where he has prior art as a reference. 

>Nobody has a comparable concern with the internal details
>of software.
>
Software "contractors" are routinely asked to "bid" on projects with NO
internal details, and with NO accessible prior art. Then these people
are told they are "bad" because they could not make an estimate. I
intend to give these contractors some ammunition to fight back with
to stop the injustice of this situation.

I hope my message reaches some programmers in the situation
I have been in - that is being wrong in your estimate with MBA type
managers tell you you are not a good programmer. This so angered
me that I went to the work to think why this is wrong and thus my
postings on the net.

When you are educated to the facts, you can say the right things
so that the MBA managers can't push you around. What can they 
say when you explain that programming is NOT like assembly of
components in a pre-determined way? They are forced to 
acknowledge what programming REALLY IS. Without this 
acknowledgement, they will determine what programming is to 
support their own interests - And YOU pay the price. (Of course
they may be so arrogant that they will laugh at you. If this happens,
work on a critical project and then quit just before delivery.)

Scott McGregor writes...
>> I don't think that hardware engineering works this way.  Talking to
>> my HW engineering friends I find that they are faced with deadlines
>> and schedules just like I am. 
>
>I think that John is correct.  In all disciplines management views
>their job as making tradeoffs between time, quality and functionality.
Note none of these have anything to do with "happiness" of the
programmer. 

>In general, management of software does want solid schedules. 
Management's focus on this triangle together with often
promoting the misnomer that programming IS accurately estimatable
causes low programmer morale and fosters programmer obstinacy, 
all the items Scott attributed to "programmer psychology" below>

>I I find
>that  engineers often internalize their own goals and are less 
>flexible in changing functionality and quality than management might be.
If engineers were treated in accordance with the reality of their work,
I think they would be as flexible as any manager could want. I speak from
personal experience.

I have to fight to force those who hire me to understand what I am 
speaking of here, because when they do I can be flexible and produce
12-50 lines of tested and debugged code per hour, 10 times the published
averages described in IEEE Software and Computer articles. Otherwise
I am obstinate and sacrifice my own future along with those I work for
and code at 1-2 lines of tested and debugged code per hour.

I demand those I work for to respect me, rather than allow them to 
treat me as a "younger less experienced person who knows less 
than the manager and therefore should shut up."

>I believe that this is a social aspect.  
I disagree. It is a result of the fact that a company's need to make money
has nothing in common with the need of an engineer being happy in his
job. I don't propose a solution to this dilemma - it is the effect of
the existence and use of money.

>Engineers don't like to shave
>functionality because it is the thing that they want to do--that's why
>they entered the trade. 
But if they are treated correctly than they won't mind shaving
functionality. The engineer attitude you describe results from
bad management of "of people" - eg. management of the product
without concern that "people" are working for you and being
sensitive to their needs. It is very easy to be obstinate when
the person you work for shows insensitivity and arrogance.

>Engineers don't like to shave  quality because
>of self-integrity, they want to feel they did a good job by their own
>internal measures.  
The engineers' focus on "internal measures" is CAUSED my management in
the first place, not a result of the engineer's psychology.

Because management does not provide measures that the engineer can
feel satisfaction by (eg. blame them for bad estimation) they are forced
to develop their own internal measures so they can feel satisfaction. 
After this happens, the engineer is no longer responsive to management.

>Time frames are typically less important to software
>engineers, so they would feel better sliding the schedule and holding the
>other aspects constant. 

This "axiom" is a perfect example of what I am talking about.
Let's just say that software engineers are a certain way, and that's
the way they are. Never mind how they got that way, for example,
because of our own actions as managers.

Since I started my own business to "live" the philosophy of my postings
in this collection, I've seen my own productivity zoom to an excess
of 10 times what I was capable of in the past or in a typical
company setting. The "manager attitude" embodied in Scott's posting is
what I escaped from.

>Since managers often get measured on what ships
>and not how much work was done, 
Yes, the reality of profit being incompatible with employee happiness.
This is a sad fact of life, for which I have no solution. I guess I'm
proposing the searing crossfire of ideologies be moved up to upper
management who get paid more, rather than be subjected to engineers
by measuring them by the accuracy of their estimates.

>their is a social opposition here. I
>believe   these are the reasons why we see tension between software
>developers and their managers on "unreasonable schedules".  
Well it all depends on whether you are running a company to make money
or running it to make employees happy as possible. If the only goal is only
to make money, employees more likely to be treated in a way that will
cause them to be obstinate. If you run it for the employees, the company
may go broke. 

I think there is a balance that will net programmer productivity far 
above where it is now. With a goal of profit companies shoot themselves
in the foot. If the goal was for employee happiness, then employees would
work harder and in the end the company would make more money than if it
forced the issue of profit and created obstinate employees.

[Robert I. Eachus's comments deleted but this writer agrees.]

Scott McGregor continues...
>>[Cliff wrote... ]attempt to create an accurate measure of something that is not
>> measurable in the first place.
>
>Software development time is surely measurable.
Yes, you are right.  Substitute "accurately measurable" for "measurable" - 
I forgot a word.

>But truthfully everyone who builds tools for predictions
>of this sort is usually quite upfront about the fact that they create
>predictions of MEANS with some VARIANCE. 
Well I think I'd be happier to see more management acknowledgement
of VARIANCE. Right up to the stockholders.

>Clearly, software schedules are predicted by people, and so ipso facto are
>predictable, it is merely that the predictions may not be very
>accurate.  
Increased management focus on this will boost programmer moral and
thus productivity.

>
>> Such estimates are bound to be useless because numerous coding difficulty
>> assumptions will be wrong without support of actual time measurements.
>> These errors propagate in an exponential manner throughout layers of code.
>...The [estimates] aren't useless merely because they don't offer certainty 
>for [program completion]. 
"Useless" was too much of a work for me to use. Obviously something is better
than nothing.

Even an estimate 10 times off can still be used to distinguish a 2 month project 
from a 5 year project, and this is useful.

> The alternative,
> NO estimate, is often psychological unacceptable to risk averse persons.
I don't mean to say it is practical to work with NO estimate. I think that
there needs to be MUCH more focus on the fact that the estimate 
is not accurate, and programmers should NOT receive demerits for
a bad estimate when there is no prior art to go by. (prior art meaning
development of exactly the same thing, like a car company making a
new model car, which is most of the time in software.)

[Scott's discussion of HW estimation deleted.]

Scott confirmed a key point - that the complexity of hardware component is
FIXED, while with software components the complexity is not well understood.
(I'm not talking about PLDs, etc.)
This ties in with a previous posters' concept of DOMAINS. Software
is applied to any domain "on the fly" - it is so flexible. There is no chance
to develop components for any one domain, much less share them with
other domains. So the software engineer has the work of the hardware
engineer PLUS the need to develop all the components at the same time.

On top of that, the laying out of circuit boards is pretty much automated.
But with software, connecting components requires use of program
statements - which must be developed.

No wonder there is such an estimation problem!

>I believe that it is the power of constraint relationships to predict
>complexity that accounts for why despite everything lines of code has
>usually the strongest relationship to development time of any typically
>measured predictor variable.

It's nice to see at least ONE relationship you can count on in software.
But unfortunately we can't estimate lines of code any more than development
time, so we are no better off!

Cliff

jtaylor@axion.bt.co.uk (Julie Taylor) (07/23/90)

From article <31989@cup.portal.com>, by cliffhanger@cup.portal.com (Cliff C Heyer):
> so that the MBA managers can't push you around. What can they ...

Surprising to find MBA managers being thought the least competent managers;
one would expect those individuals who had realised that managers
need to develop their skills, not just rely on what they were born with,
to operate more effectively than non-MBA managers.
An MBA program would include Management Theory, 
introducing the idea that productivity improves if the `workers' are 
involved in defining objectives, plans, etc. and treated as people
not simply resources.
I would hope that software engineers would aspire to be `professional'
managers also, if their career path so dictates.

usenet@linus.mitre.org (Usenet News) (07/24/90)

less expensive materials, and plan for manufacturing techniques that 
are more efficient and less costly than those we used during prototyping.
The analogy with software breaks down here because software isnt mass-produced.
From: davis@mwunix.mitre.org (David Davis)
Path: mwunix.mitre.org!davis

Two goals that are similar in both hardware and software prototyping are to 
test our ideas and to market a product to our own management or an outside 
customer.  There is an implication in doing software prototyping that 
one of the  reasons for doing a prototype is that we dont really believe 
that if we use a full-fleged life-cycle approach that we can achieve 
success.  That is, implementing a cohesive design that has acceptable 
perfomrance and is delivered in a timely way.


>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
Dave Davis                  davis@mwunix.mitre.org
MITRE Corporation           me := disclaimer.all
7525 Colshire Dr.

reggie@dinsdale.paradyne.com (George W. Leach) (07/24/90)

In article <1990Jul23.154606.5854@axion.bt.co.uk> jtaylor@zaphod.axion.bt.co.uk writes:

>Surprising to find MBA managers being thought the least competent managers;

     Well that depends.  If we are talking about a recent MBA grad who
goes right into management that is different from someone who is not
a manager or has recently been promoted to that level and has returned
to school to pursue an MBA.

            I don't equate an MBA with industrial experience.  I think
the two skill sets should complement one another.  In practice, I don't
know if this is true.  

>one would expect those individuals who had realised that managers
>need to develop their skills, not just rely on what they were born with,
>to operate more effectively than non-MBA managers.

            Unfortunately, it has been my experience that there is little
middle ground.  Many people I have seen promoted from the ranks of software
development groups over the years have received little or no training in
any way shape or form on many topics ranging from dealing with people to
managing budgets, etc......  It seems like you get all that through on
the job training.

>An MBA program would include Management Theory, 
>introducing the idea that productivity improves if the `workers' are 
>involved in defining objectives, plans, etc. and treated as people
>not simply resources.

       There certainly seems to be a demand for such information.  Witness
all the books that have sold over the past number of years in the category
of the "One Minute Manager" or "In Search of....".

>I would hope that software engineers would aspire to be `professional'
>managers also, if their career path so dictates.


       That depends upon if the "software engineer" is a professional
at what they do in the first place.  That is often debatable.

George

George W. Leach					AT&T Paradyne 
(uunet|att)!pdn!reggie				Mail stop LG-133
Phone: 1-813-530-2376				P.O. Box 2826
FAX: 1-813-530-8224				Largo, FL 34649-2826 USA

ejp@icd.ab.com (Ed Prochak) (07/25/90)

In article <114473@linus.mitre.org>, usenet@linus.mitre.org (Usenet
News) writes:
 [some deleted stuff]
> From: davis@mwunix.mitre.org (David Davis)
> Path: mwunix.mitre.org!davis
> 
> Two goals that are similar in both hardware and software prototyping are to 
> test our ideas and to market a product to our own management or an outside 
> customer.  There is an implication in doing software prototyping that 
> one of the  reasons for doing a prototype is that we dont really believe 
> that if we use a full-fleged life-cycle approach that we can achieve 
> success.  That is, implementing a cohesive design that has acceptable 
> perfomrance and is delivered in a timely way.

That's exactly right. Often we don't have all the information
needed to do that cohesive design. Things missing include:
unstated or changing customer requirements,
unknown performance of software in a particular hardware environment,
unknown costs in the tradeoffs between mantainability and performance,
and I am sure there are others.

The point is: hardware types do prototypes, why should
              we be forced to deliver final product without
              doing prototypes?

and doing prototypes does not mean abandoning an orderly
development process, including timely delivery.
> 
> 
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> Dave Davis                  davis@mwunix.mitre.org
> MITRE Corporation           me := disclaimer.all
> 7525 Colshire Dr.

Edward J. Prochak   Voice: work-(216)646-4663  home-(216)349-1821
               Email: {cwjcc,pyramid,decvax,uunet}!ejp@icd.ab.com
USmail: Allen-Bradley, 747 Alpha Drive, Highland Heights,OH 44143
Wellington: ENGINEERING is "the ability to do for one dollar,
                            what any damn fool can do for two."

mcgregor@hemlock.Atherton.COM (Scott McGregor) (07/27/90)

Warning to readers.  This turned out MUCH longer than I expected.
It is a thoughtful response to Cliff Heyer's extremely provocative postings
on estimation and project management.  If that's your
interest, I suggest you print it and read it at you convenience,
rather than page through it on line.  If not, skip now. --Scott

I believe that I share many of Cliff's  goals.  I would like to see
better software produced, I would like to see the art of software
management (and the level of practice by various managers) grow,
and I would like to see engineers feel rewarded by what they do.
I agree with Cliff that situations are common today where these goals don't
occur.

My biggest difference of opinion with Cliff regards as to what to do about it.
I believed from Cliff's first posting that Cliff was looking for an
evolutionary way of improving things.  But when I read Cliff's latest
posting, I felt less sure of that.  I felt a lot more revolutionary
fervor and anger against managers in general.  I'd like to help
anyone interested in improving s/w management, but I can't share
the "me vs. them" approach I felt when I read the posting.  Perhaps
I'm reading too much emotion in to it (I've been around email
and net  postings long enough to know that sometimes things are
misinterpretred).  In any case, if some of my moderate approach ideas are
interesting, read on. 

Let me share some of what I believe about software management.

Predicting the future (of which schedules is one part) is a basicly 
determined by normal statistical forecasting laws.  Accuracy is based
upon past data, and the similarity of current cases to past cases.
For any given person, past data is their own past experience plus what
they read of others past experiences.  By no means everyone is this
way, but a lot of people who are attracted to software developement
choose to spend little time reading about other projects or writing up their 
own projects.   As a result, people are left to make predictions based upon
their own experience alone, and that can only grow a little bit at a time.
Even if one wanted to, as some do, there is little industry knowledge to
borrow.
Newly promoted project managers are thus at an incredible disadvantage.
Very little can be done to improve their performance except give them time
and see if with their growing experience they get better at estimating.
If they don't you can weed them out and start with someone else, but
then they need to start getting experience too.  The bad problem of course
is that while these new managers are learning, engineers are working for
them, and  presumably suffering for their manager's mistakes.   I 
believe this is one of the most unfortunate problems with our industry,
and will remain so the mean number of years of experience in the industry
starts getting in the high teens and above. This problem is exacerbated
in areas where there are large projects because you have fewer data
points to learn from, and the variance from project to project is such
that you can't leverage as much.

The truth of the matter is that it is very hard to get good estimates
without experience.  In looking over my own past twenty years in this
business, I can see my own naivete in the early years really lead to
bad estimates and unfortunate results.  When I became responsible for
setting schedules for others there was much to be concerned about because
others would suffer along with me for my mistakes.  I tried to get
better fast, but clearly made some mistakes along the way and I am
very sorry for those who worked for me at that time and shared in
the problems my lack of experience caused.  On the other hand, I
am now considerably better at this and so this doesn't happen as
often these days.  That's little consolation for those who suffered
through my learning, but good news for those who have to suffer along with
me these days.

Some of the common failures of inexperience project managers are the
following.  They want to please their superiors who won't see any
direct benefit until the project ships.  So they put on strong pressure
on themselves and others to get "aggressive schedules".  (More mature
managers will say of some efforts, "sorry, I can't do it sooner, if
an earlier date is needed for market reasons we just can't compete
for that particular market.  We'd just blow our money if we tried.
What are some other market opportunities where we really can be competative?")
They often get unrealistic schedules because these schedules pare out
all the possible things that can go wrong.  Unfortunately Murphy's law
often prevails, and if not all things go wrong, at least some do, and
as a result you are still late.   They then hope that at least people
will understand the "excuses" that are slips are "due to unforseen
problems and things outside our control".  (More experienced managers
will often put in enough slack to cover these unforseen things that
always do happen.  How much to add is, well a matter of experience...)

Another reason green managers underestimate schedules is that they often
under-appreciate the amount of indirect uses of a persons time occurs.
There are always staff meetings, company announcement meetings, various
classes, conferences, vacations, illnesses, etc. that chew up much more
time that most people expect.  There is also preparation for meetings,
demos, and waiting for approvals.  This kind of time theft from
main tasks can vary considerably.  I've been in environments where
this took no more than 20% of my time to environments where this took
75% of my time.  I don't doubt that there are even worse extremes.
Failure to include these other time uses can undermine estimates
even where the task is well understood and repeatable.  This can also
be a trap for seasoned managers who move to new environments where
time traps are different than where they gained their experience.

One of the biggest areas where time sinks are unappreciated is where
lots of communications with other people are necessary.  Fred Brooks
pointed out that adding people to a late software project makes it
later.  The reason is increased communications overhead is necessary.
Software designs can be complex and it takes a number of meetings and
discussions to make sure that enough information has been communicated, and 
that what was understood is what was meant.  Too often projects are
scheduled assuming that all implementation decisions will be made
on the spot, but more often they are deferred until the engineer
has a chance to think about them and then discuss them with others.
These implementation decisions may even affect the underlying design
decisions which will cause even more time to be lost.   While what
decisions will have this effect are impossible to predict, the fact
that this is likely to occur regarding SOME decision, and that therefore
time must be allocated to this should be predicted and planned for.

I believe that the above points are very important above the underlying
statistical prediction problems, because industry prediction errors 
don't follow a normal distribution around the mean, they tend to be
skewed toward over-optimism.    Nevertheless, it is important for
decision makers to understand the level of certainty or uncertainty
of estimates being made.  It is not the case that estimates are
either exactly correct or totally useless.  Levels of confidence,
measured in variance do matter.  Being off 10% in an initial 
estimate is not the same as being 80% off, and being 80% off is
not the same as being 800% off.  When you make an investment
decision, you usually don't want to just make it on a 50-50% basis,
so you want to make sure that things will still be profitable even
under reasonable unforseen circumstances. Thus the amount of variance
of your estimates (possibly in the form of worst case schedules)
is valuable in assuring management profitability goals.

Unfortunately, understanding and widespread use of statistics,
especially concepts of variance, confidence intervals and sensitivity
analysis  is not common among managers, even middle managers, in the
software development field.  In part this is due to the large number
of managers who have risen from technical ranks and have not been
"professional" managers, but there are many notable "professionally"
trained managers who seem lacking in this area.  Clearly, statistics
courses are available (sometimes required) for MBA degrees, but this
has not brought the results than might be hoped for.  As a result,
engineers and other managers need to expend tremendous effort trying
to train those around them in this area.  It is a long process and
a thankless one, but once a manager finally integrates these ideas
in their planning the scheduling process can be smoother for both
manager and employee.

>From time to time in any career you will find it necessary to
work with some one that is difficult and unpleasant.  In such circumstances
the best one can do is get things as quickly and painlessly as
possible while trying to extricate yourself from the situation.
If one is working for someone where they have a major communication problem, I
recommend getting it fixed or getting out as quickly as possible. 
Sticking in such a situation out of loyalty to the organization or
to try to "fix" the situation often leaves people feeling sour.  Rightly
or wrongly, how well people communicate with their bosses usually has
a large effect in how valuable they are perceived to be, how they
are rewarded, etc.   It is really useful to keep trying to seek out
a situation where you have that kind of good relationship with a manager;
it usually pays off in many ways.   

So, I don't believe that adversarial relationships between managers
and engineers are all that's possible.  I believe cooperative relationships
can occur and that they are generally more successful.  Less successful
managers and engineers are usually weeded out in any organization with
any sort of darwinian reward system, but the time frames that these
weeding out occurs over is long enough that you don't want to spend
much of your career waiting for it to just occur.  

For cooperative relationships to work it is important to understand
people's differing reward systems and be aware that others may
have very different reward systems and circumstances than oneself.
It is also important to make sure others understand your reward
system.  As I pointed out before, most managers are rewarded based
upon products shipping and profits received.  Many engineers are
evaluated and rewarded based on technical aspects of their work.
This can lead to differences in what people want.  It is easy to
see this leading to adversarial situations such as Cliff pointed
out here:

> I think there is a balance that will net programmer productivity far 
> above where it is now. With a goal of profit companies shoot themselves
> in the foot. If the goal was for employee happiness, then employees would
> work harder and in the end the company would make more money than if it
> forced the issue of profit and created obstinate employees.

I have seen companies lose sight of profit, and focus on employee happiness.
The loyalty of the engineers was astounding.  They built great things
on tight deadlines.  However, in the end, they built the wrong things and
the company couldn't ship what they were building.  Finally the whole effort
was costing so much that it was shut down.  This made many of the displaced
engineers very bitter.   In truth, I think it is important that a
company pay attention to both profit AND employee happiness. In fact,
one can see, as Cliff did, that employee happiness is one step toward
profit.   Employees on the other hand need to see the company's attempt
to be profitable as one step toward long term employment and growth
for the employee.  Both sides need to make allowances for the needs
of the other group. 

Admittedly, there are many managers who do not seem to value their
employee's happiness.  I can only hope that their employees will
keep leaving them and finally they will fail in their organizations
because they won't be able to meet their committments without people,
and in that way perhaps they will learn the value of their people.
It is easy to take people for granted when they stay with you even under
intolerable situations, so I would hope that they would not.
(Obviously, people have their reasons, and I will not second guess
them, but I wish that they could act more freely in this regard).
Unfortunately, I have found little else that makes much of an impression
on managers with this predisposition to ignore their implicit dependence on
their employees.  I have been in management of people classes with
such managers and they ignore all that is taught, until it is too late.
I feel quite sorry for anyone who is their victim. 

But the good news is that all managers are not this way.  Some see their
success as being dependent upon also making their employees feel
successful by their own personal measures.  Seek these people out and
help them be successful in comparison to those who are not so disposed.
With these people one can begin building the kind of environment
that rewards you as well as the organization.  I would wager that
you'll still be asked to make predictions about the future to help
the organization protect its vitality through profitability, but
there should be some understanding about the reasons for poor
estimates and therefore some tolerance for schedules that alot time
for unforseen difficulties.  Absolute accuracy will not be expected,
but people will want to know just how accurate or inaccurate you
might be so they can make investments as wisely as possible given
uncertain information.  Tolerance for unforseen circumstances will
be normal, but committments (on both sides) will be expected to be
met regardless of changing circumstances through use of plans that
are stable in the face of a wide number of unexpected events.
 
Scott McGregor
mcgregor@atherton.com

The remainder of this article is a few specific remarks on some
specific quotes from Cliff's last posting that bear reiteration

> Software "contractors" are routinely asked to "bid" on projects with NO
> internal details, and with NO accessible prior art. Then these people
> are told they are "bad" because they could not make an estimate. I
> intend to give these contractors some ammunition to fight back with
> to stop the injustice of this situation.

In the end, even nonmangers need to learn how to become good businessmen
to protect their own interests.  A good negotiator recognizes that
some offers must be walked away from, even if the result is that
the deal is totally lost.  To the extent that software "contractors"
are free agents "ammunition" will not help them.  They already have
the most powerful ammunition they can use--if every contractor
walks away the bidder is left without their project complete.  They
either live with that, or change the terms to be more palattable.
An unfortunate observation is that perhaps our industry has too many
over-eager people willing to underbid for contracts.  If they could
really deliver this would benefit them, the bidders and the ultimate
consumers through lower costs.  If they can't deliver, it probably
sours opportunities for other people who won't underbid until the
bidders wise up and look for more realism.  Undoubtedly this will
happen over time, but there is a lot of personal miseries that occur
in the meantime.  

Programmers DO need to learn to make an estimate under any circumstances.
It should be okay, and in fact expected that they hedge exactly how
much variance there might be so that people can prepare for a worst
case.  If they don't like that estimate they can try someone else
until they get one they like.  If it is an underestimate the company
will just pay later unintentionally rather than avoiding the expense
knowledgably today.  Sometimes knowledge of uncertainty helps rather
than hinders a decision.

> I hope my message reaches some programmers in the situation
> I have been in - that is being wrong in your estimate with MBA type
> managers tell you you are not a good programmer. This so angered
> me that I went to the work to think why this is wrong and thus my
> postings on the net.

I can't see this as being a bad programmer, but I can see this
as being unreliable in meeting ones committments.  The trick of
course is to learn to only make meetable committments and only do
business with people who will accept reasonable estimates based on
your track record of having met your committments.  Looking
to get satisfaction from people who won't be reasonable is going
to make one bitter in the end. 

> When you are educated to the facts, you can say the right things
> so that the MBA managers can't push you around. What can they 
> say when you explain that programming is NOT like assembly of
> components in a pre-determined way? They are forced to 
> acknowledge what programming REALLY IS. Without this 
> acknowledgement, they will determine what programming is to 
> support their own interests - And YOU pay the price. (Of course
> they may be so arrogant that they will laugh at you. If this happens,
> work on a critical project and then quit just before delivery.)

I think it is very important for both engineers and managers to
understand the problems inherent in software scheduling.  I encourage
people to spend time to understand this better.   However, I do not
think it is a panacea.  Some people will always be persuaded to
go with a more optimistic solution even in the face of argument.
At that point, YOU have to be willing to walk away from them.  While
I am not promoting people intentionally sabotage others, I do believe
that over time unreasonable people either learn from their mistakes
are removed from situations where the make them, IF people stop
cooperating with them.  Unfortunately this is a source of very
stressful situations for everyone involved.

> Note none of these have anything to do with "happiness" of the
> programmer. 

This is very true.  However, it is not likely that managers who
do not currently value their employees happiness will change their
attitude because of engineering obstinancy.  What can help is
trying to find solutions that meet BOTH the manager's and the 
employee's needs.  So figure out how to meet your manager's need
and how by the manager meeting your needs you can help meet theirs.
Illustrate that with your cooperation they will get what they want
and they won't without it.  If they aren't reasonble in negotiating
from this position see above discussion.

> If engineers were treated in accordance with the reality of their work,
> I think they would be as flexible as any manager could want. I speak from
> personal experience.

I find that engineers treated with respect are very flexible, however,
they have strong attachments to certain things like product quality
and functionality that make them try to hold on to these things
tenatiously in any negotiation, more so than many managers.  It
is good for managers to realize these aspects are personlly important
to many engineers and make allowances for this if they are to contribute
to engineer's happines. 

> I have to fight to force those who hire me to understand what I am 
> speaking of here, because when they do I can be flexible and produce
> 12-50 lines of tested and debugged code per hour, 10 times the published
> averages described in IEEE Software and Computer articles. Otherwise
> I am obstinate and sacrifice my own future along with those I work for
> and code at 1-2 lines of tested and debugged code per hour.

On the one hand I am glad that people reward those who treat them
well, and that people do less well for those who do not.  However,
I hope that people will not sacrifice their futures, and will have
the sense to remove themselves from impossible situations before this
occurs.  If one cannot extricate oneself, you have my sympathy 

> I demand those I work for to respect me, rather than allow them to 
> treat me as a "younger less experienced person who knows less 
> than the manager and therefore should shut up."

In part this is the problem of experience and historical data points
in estimating.  A younger less experienced person might naturally
have fewer data points to make your estimates on, and unless they have
learned from other people experience they will naturally have difficulty
making any estimate, and their variance in estimates is likely to be
large (probably making them even more reluctant to estimate, for fear
of being wrong).  Naturally this poses a difficulty for a manager who
must make a decision based on some estimate.  Hesitancy to set an
estimate oneself, and to stick by that estimate, naturally encourages
second guessing from the manager. Experience, and a track record of
being correct is the one most important thing that will help resolve
this but that is hard to wait for.  In the meantime, conservative
judgement, and sticking to ones guns, walking away where compromise
is impossible is about the best one can do.  

> But if they are treated correctly than they won't mind shaving
> functionality. The engineer attitude you describe results from
> bad management of "of people" - eg. management of the product
> without concern that "people" are working for you and being
> sensitive to their needs. It is very easy to be obstinate when
> the person you work for shows insensitivity and arrogance.

I don't speak of obstinancy here, but rather of values.  Engineers
often value the quality and functionality more,  they might negotiate
on these points with sensitive management, but they often value them
more highly than managers.  As negotiators, this is important information
to both sides seeking to understand why agreement is elusive.

> The engineers' focus on "internal measures" is CAUSED my management in
> the first place, not a result of the engineer's psychology.

I don't agree.  I believe that ALL people are internally motivated. 
External motivations can develop through their tie to internal motivations.
A manager might be driven by personal advancement needs.  An engineer
might be driven by the need to MAKE something OF VALUE for the world.
Whether these come into conflict or not depends upon whether the manager
can find someway to advance their career (making a PROFITABLE product)
through helping the engineer achieve their goal (the making of that
product).  They can come into conflict where the manager wants to 
make a tradeoff against value in favor of profitability.  This is
not unresolvable, but it is a starting point for a negotiation between
the two. 

> Because management does not provide measures that the engineer can
> feel satisfaction by (eg. blame them for bad estimation) they are forced
> to develop their own internal measures so they can feel satisfaction. 
> After this happens, the engineer is no longer responsive to management.

I wish more engineers and managers understood this.  If company goals
don't tie into personal goals then it is impossible to motivate people
and they will do what they want to do whether it pleases you or not.
A negotiation between manager and employee can usually get some
(often most) of what a manager wants, whereas imposition of decisions
will often become self-defeating as tuned out employees persue their
own needs.  But note that this needs for satisfaction are already there.
They didn't develop because of something that management did.  Rather
management has the opportunity to tie the two together or not.
Also, the question of management  provided measures is not fairly
treated here.  Clearly people don't feel satisfaction when they
fail, and aren't intended to.  But what would have happended if
they HAD made good estimates.  Would they have been rewarded?
Would they have felt satisfaction from the reward of having made
a realistic estimate, having stuck with it, of it proving accurate
and of having met their committments.  If not, management is failing them.

> Let's just say that software engineers are a certain way, and that's
> the way they are. Never mind how they got that way, for example,
> because of our own actions as managers.

In one sense, it really doesn't matter how people got there.  If managers
are doing something that is causing this dichotomy between what
managers want and what engineers want, then yes they should work on that.
But they also do need to deal with the fact that it is a reality today
that a managers and engineers needs are different.  Any negotiation
aimed at satisfying both parties cannot afford to ignore this today,
even if it goes away tomorrow.  I personally do not believe that
it can ever go away, because I believe that managers and engineers
get rewarded for different things.  When these things can work
together everyone can benefit, but each side only REALLY cares about
what benefits them.  Managers typically get rewarded for longer term
events than do engineers.  (In fact there is a similar incongruity of
goals between functional managers (we built what you asked for on time
and in budget) and general managers (yeah, but when customers didn't
like it we lost money).   Another longer term vs. shorter term 
incongruity sometimes occurs between senior executives and middle
managers.


I remember hearing  a story about a software development experiment
over ten years ago where four individuals, A,B,C, and D were asked
to write a software program, A and B were told that the would be
measured (and $rewarded) on the basis of minimum code size.  C and D
were told that they would be measured on readability and clarity
(including comments).
A and C were put in one room and B and D in another.  Additionally
B and D were told to produce ONE program among the two of them.
A and C tried to write a program together but fought about it and
wound up creating two separate programs, each of which met their goals
but did not at all contribute to the others. B and D DID write one
program that met both goals better than a control group (with no
particular measures) did.  Since I have never been able to track down
a paper discussing this experiment it may be apocraphal, but I find
it believable enough that it might have happened. If so, it is certainly
a reasonable example of how people act to meet their own goals (happiness)
and may or may not compromise in doing so.

> Since I started my own business to "live" the philosophy of my postings
> in this collection, I've seen my own productivity zoom to an excess
> of 10 times what I was capable of in the past or in a typical
> company setting. The "manager attitude" embodied in Scott's posting is
> what I escaped from.

I don't doubt that Cliff's productivity is up.  I think it is great
he got away from the situations that weren't satisfying to him.  One
question is why these promise of great productivity didn't sway
managers to cooperate with him.  Certainly they just might have
been green themselves and might not have learned his value yet.
But additionally, they may have been concerned that where Cliff's
productivity was directed may not have been where they needed it.
This is where it is important for both sides to negotiate for success.

> Yes, the reality of profit being incompatible with employee happiness.
> This is a sad fact of life, for which I have no solution. I guess I'm
> proposing the searing crossfire of ideologies be moved up to upper
> management who get paid more, rather than be subjected to engineers
> by measuring them by the accuracy of their estimates.

This was the most disturbing part of Cliff's posting for me.  I totally
disagree that profit is incompatible with employee happiness.  In fact
I believe that employee happiness cannot occur at all without sufficient
profit, nor can profit be sustained without some employee happiness.
But it is clear that there are things one can do as a manager or
an engineer that only promote one but not the other.  Or which contribute
to one at the expense of the other.  Good communications and negotiation
can find compromises that increase both long term.  Improving
such communications and enriching the knowlege of people on both side
of the negotiation seems like a good idea to me.  But I fear that
turning this into a "searing crossfire of ideologies" will just
polarize the participants who will then walk away with nothing more
than the satisfaction that they  hurt the other.

> Well it all depends on whether you are running a company to make money
> or running it to make employees happy as possible. If the only goal is only
> to make money, employees more likely to be treated in a way that will
> cause them to be obstinate. If you run it for the employees, the company
> may go broke. 
> 
> I think there is a balance that will net programmer productivity far 
> above where it is now. With a goal of profit companies shoot themselves
> in the foot. If the goal was for employee happiness, then employees would
> work harder and in the end the company would make more money than if it
> forced the issue of profit and created obstinate employees.

I agree that if you run a company for employee happiness then you will
can have less obstinate employees than if you run it ONLY for profit.
However, a company that has gone broke is little able to continue to
contribute to employee happiness.  A fair compromise between the
two can be in the best interests of all.

> Well I think I'd be happier to see more management acknowledgement
> of VARIANCE. Right up to the stockholders.

So would I.  I work on teaching this to people every day.  It is not
easy.  It takes years of work.  I only see small improvements.  But
I do see improvements over time, more quickly in my chain of management
where I focus attention, less quickly elsewhere, but still happening.
If there is some way more I can contribute in this way, I'd like to
hear about it.

> >Clearly, software schedules are predicted by people, and so ipso facto are
> >predictable, it is merely that the predictions may not be very
> >accurate.  
> Increased management focus on this will boost programmer moral and
> thus productivity.

I work on this too.  I repeatedly bring up sensitivity analysis and analysis
of variance.  People get sick of it (and me).  Finally, they start
incorporating
it into the way they do things.  It changes them as they begin to really
understand measurements of uncertainty.  It gets the results that I
want in the long run.

> I don't mean to say it is practical to work with NO estimate. I think that
> there needs to be MUCH more focus on the fact that the estimate 
> is not accurate, and programmers should NOT receive demerits for
> a bad estimate when there is no prior art to go by. (prior art meaning

Cliff and I are close to agreement but not exactly.  I think that the
extra focus should be on HOW MUCH the estimate is not accurate by
(i.e. the variance), not just the fact that reality is not a point
estimate.  This is important because it shifts the measure and application
of demerits from a binary (correct estimate vs. incorrect estimate) to
regularly completes project within the estimate plus confidence interval vs.
regularly exceeds estimates plus confidence intervals.   At some point
there is value in accountability for what we forecast.   You can't and
shouldn't be held responsible for results that are still well within the
realms of chance.  But if you repeatedly exceed chance liklihoods then
there is a problem with your estimation techniques.  Commitments should
therefore only be made upon the most conservative estimates so that
you can get credit for being reliable. Being pleasantly agreeable
but unreliable in meeting committments can seriously undermine an
organization, and should be measured and dealt with (not necessarily
with demerits--perhaps with training by experts).

The flip side of this is that people who are considerably better
estimators should be detected through these measures and their
expertise should be capitalized upon to help bring up to speed
the less experienced people.  To the extent that they are meeting their
committments contributes to the success of the company they probably
deserve some extra rewards too.

> No wonder there is such an estimation problem!

Absolutely!

Scott McGregor
mcgregor@atherton.com

joshua@athertn.Atherton.COM (Flame Bait) (07/28/90)

jtaylor@zaphod.axion.bt.co.uk writes:
>Surprising to find MBA managers being thought the least competent managers;
>one would expect those individuals who had realised that managers
>need to develop their skills, not just rely on what they were born with,
>to operate more effectively than non-MBA managers.

I think people are objecting to managers with only MBA experiance; the
person who was running a fruit packing plant, then went to MBA school,
and is now a V.P. of software development.

Joshua Levy  (joshua@atherton.com)