klopfens@bgsu-stu.UUCP (Bruce Klopfenstein) (02/28/90)
I am very interested in the discussion going on about forecasting.
I have studied technological forecasting, and what surprises me is
the seeming lack of research into past forecasts. De Sola Pool
published his Retrospective Technology Assessment of the Telephone
and another looked at the trans-Atlantic cable. What I find strange
is the lack or research effort into evaluating past forecasts. What
better way to learn about forecasting than to re-analyze past forecasts
and look, for example, for systematic errors in the process. The con-
clusion I came to on the subject is that forecasting practitioners
discard yesterday's forecast and go on to the next one. Little is
learned from past mistakes. I'm curious what you forecasting theorists
and practitioners think about that.
--
Dr. Bruce C. Klopfenstein | klopfens@andy.bgsu.edu
Radio-TV-Film Department | klopfenstein@bgsuopie.bitnet
Bowling Green $tate University | klopfens@bgsuvax.UUCP
Bowling Green, OH 43403 | (419) 372-2138; 352-4818
| fax (419) 372-2300duncan@dduck.ctt.bellcore.com (Scott Duncan) (02/28/90)
In article <5473@bgsu-stu.UUCP> klopfens@bgsu-stu.UUCP (Bruce Klopfenstein) writes: >I am very interested in the discussion going on about forecasting. [In comp.society.futures] >I have studied technological forecasting, and what surprises me is >the seeming lack of research into past forecasts. > What >better way to learn about forecasting than to re-analyze past forecasts >and look, for example, for systematic errors in the process. > Little is >learned from past mistakes. I'm curious what you forecasting theorists >and practitioners think about that. >Dr. Bruce C. Klopfenstein | klopfens@andy.bgsu.edu >Radio-TV-Film Department | klopfenstein@bgsuopie.bitnet >Bowling Green $tate University | klopfens@bgsuvax.UUCP >Bowling Green, OH 43403 | (419) 372-2138; 352-4818 > | fax (419) 372-2300 It struck me that this applies equally well to software projects. In discus- sing software quality and productivity efforts with a number of people over the past couple of years, this has been one of the major failings noted. There is very little effort put into retrospective analysis of what happened in a pro- ject. My sense is that some large projects (especially those for the military) may have this done to them and some very small projects, where the individual de- termination of a few people prevails, do this. However, the 100K - 5M range (lines of code) does not seem to attract much of this. I have heard many reasons given for this, but the ones repeated most often are: o by the time a project is done, folks are already committed to the next project (or, at least, the next release of the one they just completed), i.e., there is no time alloted in the original (or following) schedule for such activity; o people keep very poor records of what went on in the project, e.g., history, documentation of decision-making, etc., so there is little data to operate on other than people's memory of what happened, i.e., people won't trust or find worth in the effort with such "unreliable data"; o the impression is that this effort is to find out what went wrong (rather than what worked well and then deciding how to repeat those successes) and folks involved don't want to spend time on an activity that can only make them look bad; o it takes a lot of time and human effoprt, i.e., it costs a lot, and nobody's risked doing it enough (and written about it) to show real benefit in subsequent projects from having done retrospectives; o the original teams usually move on to other projects so, even if you uncovered things of interest, the next project will have a dif- ferent staff and is likely, because of individual differences, to display a different profile of behavior. While there is truth in all of them, I think the negative aspects of that "truth" can be overcome. In particular, if such retrospective activity and project history is a pre-planned part of the project, I think much of the prob- lem goes away. What is a problem is justifying, in the short term, the cost of such effort since the long-term benefit is generally where you'd expect to see the cost recouped. On the other hand, my personal belief is that the very next project would prob- ably benefit from such an effort. I look at it as a parallel to code inspec- tions (also labor intensive but effective): there's an up front cost, but peo- ple begin to see the back-end benefit rather quickly once they actually get in- to doing it. I'd be interrested in people's comments about these points. Speaking only for myself, of course, I am... Scott P. Duncan (duncan@ctt.bellcore.com OR ...!bellcore!ctt!duncan) (Bellcore, 444 Hoes Lane RRC 1H-210, Piscataway, NJ 08854) (201-699-3910 (w) 609-737-2945 (h))
doering@kodak.UUCP (Paul F. Doering) (02/28/90)
In <5473@bgsu-stu.UUCP> Bruce Klopfenstein writes that he finds
little evidence that technology forecasters critique past efforts
to refine the forecasting process. I earn my living as a forecaster
of technological developments, and I would cite two explanations:
I believe that most successful forecasters are unknown to the
public or even to their peers. They live inside organizations that
rely on their services. Possibly they can be detected only by their
employer's knack for good decisions. "Famous" forecasters rise on
their own successes and fall precipitously on their first obvious
misstep. That's a consequence of being "employed" by the public.
Private employers are more likely to accept a satisfactory batting
average instead of perfection. What all this means is that the kind
of post-mortum Bruce is seeking probably goes on quietly in scores
of places, with the results hoarded for competitive advantage.
Additionally, I find that I spend much of my time qualifying the data,
not the technique. Any good algorithm fed bad information will give
unreliable results. Clearly, I don't mean to belittle the need for
good algorithms; but it's disappointing how many sources are in
business merely to publish bad data.
... All of which makes my signature block a little ironic --
--
========================= ===============================
Paul Doering (for self) We shall walk together
doering@kodak.com as seekers toward the Future.
========================= ============= -Carl Sandburg ==matheson@portia.Stanford.EDU (David Matheson) (03/01/90)
There are (at least) two things to be learned by looking at past forecasts: 1. Knowledge about the domain of interest (e.g. computers and so on) 2. Knowledge about the forecasting procedures. In this second domain, there has been quite a bit of research in the area of probabilistic forecasting. Most of this research (that I know of) comes from psychology. A recent book, "Judgement under uncertainty: Heuristics and biases" by Daniel Kahneman, Paul Slovic, and Amos Tversky provides a nice summary of this literature. Let me site two well-know biases as examples. People are overconfident in their judgements, and tend to anchor their assessments on reference points. If one has to make a numerical assessment, for example, it is likely that first a reference point will be identified (such as a linear extrapolation, current level, or whatever), and then adjusted to account for the future uncertainty. This procedure systematically leads to too little adjustment. Furthermore, the assessee will be overconfident about his judgement. These biases are pervasive, effecting the judgements of everyone from laymen through experts. Careful practitioners are careful to use procedures designed to decrease the importance of such biases. In complex environments (such as technology forecasting), biases in hindsight will likely hinder the efforts of those trying to determine how good the forecast was. See chapter 23 in the book cited above. For example, there is a strong bias of creeping determinism, which is the conscious beleif that whatever happened had to have happened. Another effect is fellacious learning, in which people find meaning in random sequences of events. There are environments where probabilistic forecasts are extremely accurate, in some sense. In the probability world, accuracy is usually measured in terms of calibration, which is does the event occur with the same frequency as the level of probability you assigned? Weathermen, for example, are extremely well-calibrated. Their domain is the prime example of domains where researchers find good learning. The feedback is unambigious, fast, and there is lots of it. The more interesting types of forecasts are about events that occur only once, such as "will memory technology reach 64M by 2000?". Here there is only one event, the feedback is ambiguous and takes years to get. The only practical way to judge the a forecast (the probabilty that the event is true) is by examining the procedure whereby the probability was elicited. I would certainly agree with the sentiment of wanting to look at past forecasts and find such efforts laudible. We may learn less than we hope from such activity. David -- ______________________________________________________________ David Matheson matheson@portia.stanford.edu 376 College #5, Palo Alto, CA 94306-1545 (415) 328-3515