ark@alice.UucP (Andrew Koenig) (11/06/85)
A while ago I posted a quote from a review in which someone tried 16 times to identify which of two amplifiers was playing and got it right 12 times. The authors considered that to be significant. Was it really? Well, maybe. If my arithmetic is correct, the chance of picking the right amp 12 or more times out of 16 if you just chose randomly is 1 in 26. Looks pretty convincing. Of course it would probably have been considered equally significant if the reviewer had gotten 12 wrong out of 16 -- that would mean he could distinguish which amp was which but was mistaken about the particular characteristics of the one he was reviewing. That increases the odds to 1 in 13. Next we have the fact that the reviewer tried to duplicate his feat and failed. Presumably the results would be considered equally significant if he had failed the first time and succeeded the second. The chance of succeeding in one out of two trials is 1 in 6.5, more or less. Finally, we apply the anthropic principle and ask "How many experiments of this type were tried that we weren't told about?" For this purpose, we do not need to restrict ourselves to experiments done for this particular review: we can look at all the other reviews we might have seen and all the other reviewers. When we do this, we realize that dozens of people have probably tried experiments of this sort, so it is not susprising that one reviewer comes up with a result that is statisticly significant in isolation. It is also not surprising that that reviewer would report the results. Putting the idea into different words: if you know a hundred people, you should not be surprised if something happens to one of them that should only occur once every hundred years or so. In fact, you should expect such things to happen to at least one of your acquaintances about once a year. Furthermore, such things are unusual enough when they happen to you that your acquaintance will probably tell you about it. The same thing is happening here.