hanson@ptolemy.arc.nasa.gov (Robin Dale Hanson) (03/02/90)
In <9625@portia.Stanford.EDU> David Matheson writes: > There are environments where probabilistic forecasts are extremely accurate, > in some sense.> In the probability world, accuracy is usually measured in > terms of calibration, which is does the event occur with the same frequency > as the level of probability you assigned? Weathermen, for example, are > extremely well-calibrated.> Their domain is the prime example of domains > where researchers find good learning.> The feedback is unambigious, fast, > and there is lots of it. > > The more interesting types of forecasts are about events that occur only > once, such as "will memory technology reach 64M by 2000?".> Here there > is only one event, the feedback is ambiguous and takes years to get. > The only practical way to judge the a forecast (the probabilty that > the event is true) is by examining the procedure whereby the probability > was elicited. Is it fundamentally harder to have well calibrated forecasts about future memory technology that about the weather? I doubt it. What I think you are seeing instead are differences in the incentive systems for forecasters in different subjects. The future sales prospects of a new corporation are extremely hard to project, yet the estimates of those prospects embodied in stock prices are rather well calibrated, I claim. Conversely, I'll bet used car dealers are very poorly calibrated in the estimates they give me about the reliability of their cars. In <2378@kodak.UUCP> Paul F. Doering writes > I believe that most successful forecasters are unknown to the > public or even to their peers. They live inside organizations that > rely on their services. Possibly they can be detected only by their > employer's knack for good decisions. "Famous" forecasters rise on > their own successes and fall precipitously on their first obvious > misstep. That's a consequence of being "employed" by the public. > Private employers are more likely to accept a satisfactory batting > average instead of perfection. What all this means is that the kind > of post-mortum Bruce is seeking probably goes on quietly in scores > of places, with the results hoarded for competitive advantage. Paul seems closer to the mark. If an employee of a company (or the National Weather Service) can stay within the company for many cycles of prediction and discovery, then he/she can be rewarded based on performance, and so has the incentives to be a "careful practitioner". The incentives of a "visionary" who sells books to the public can be quite different. It is in their interest to tell you that the prospects of artificial intelligence, hypermedia, or virtual reality are so wonderful that you just HAVE to buy their book or hire them as a consultant. Many of the biases mentioned seem more rational when you realize the incentives. People are more impressed by definite predictions that by someone who seems to waffle a lot. And those complex "scientific" analyses are sure to impress the clients, even if they aren't any more accurate. If I am right about the primacy of incentives, then we should be able to get quality well-calibrated predictions about complex one-time problems (like future memory density or even artificial intelligence) by using better incentive systems. IN FACT, I hereby propose that we create a stock-like market mechanism for betting on future predictions, somewhat like the one described in John Brunner's book Shockwave Rider. Markets can be highly calibrated because if anyone notices a consistent bias, they can make money by trading on that bias, and thereby reduce it. And since the prices would be public, the best forecasts might not be "hoarded for competitive advantage". I have actually thought about this proposal in some detail, and have a paper on the subject, should anyone be interested. Robin Hanson hanson@ptolemy.arc.nasa.gov (or hanson@charon.arc.nasa.gov) 415-604-3361 MS244-17, NASA Ames Research Center, Moffett Field, CA 94035 408-735-9257 110 Locksunart Way #6, Sunnyvale CA 94087 soon to be: 47164 Male Terrace, Fremont, CA 94539 @ 415-651-7483