adam@mtund.UUCP (Adam V. Reed) (09/19/86)
> > > > AG's commission: > >>>> In an attempt to approximate a ``real world'' situation, Malamuth and > >>>> Check (1981) had male and female subjects view full-length features > >>>> as part of a campus cinema showing... Adam Reed: > > > > I stand by my observation that *appropriate controls*, namely ... > > > > were never done. The "pornographic" materials used by Malamuth (et al) > > > > were in fact *violent* as well as pornographic, ... Moreover, > > > > Malamuth uses a significance level of .05, so that *even with proper > > > > controls* the probability of any single contrast yielding a spuriously > > > > significant result would be .05 . ... [the study] involved at least 5 > > > > contrasts , so the probability of at least one spuriously significant > > > > result is .23, about one in 4. And this is the *best* of the studies > > > > cited by the commission. Not much to go on, is it now? > > I am referring to the five separate dependent measures administered in > > *one* study, the only one by Malamuth et al which included anything (Thanks for correcting my typo, Mark) vvvvv > > resembling a control condition: Malamuth and Check (1981). Only one of > > those five contrasts yielded a result significant at the .05 level. Mark Terribile: > Yes. Assuming that the trends all had no less than 1 chance in 20 of being > random occurrences (in other words, that all of the trends were *just* barely > within the level of significance) there is about 1 chance in four that one of > the five trends observed was a random event, in the absence of further data. > The likelyhood of all five were flukes, however, would be 1/(3 200 000). No. First of all, only *one* of the five contrasts administered in Malamuth and Check (1981) yielded a result significant at the .05 level. Since the study lacked proper controls, even this one result is not usable. However, even had proper controls been done, multiple contrasts would have brought the likelihood of a spurious result *for that one study* to about one in 4, *not* one in 20 as you seem to assume in your 1/32000 calculation, which I shall assume refers to the *cumulative* evidence from five separately published articles. Now given the lack of *even an attempt to achieve external validity* in the other 4 studies, bringing them in seems disingenuous. But suppose we had some otherwise valid and similarly significant (in the statistical sense only) results. What would they mean? Since each study used several contrasts, the likelihood of producing a spurious result is much higher than the nominal significance level of those contrasts that, unlike the other planned and administered contrasts, happened to yield "significant" results. Let me leave you with the following exercise. Suppose a researcher and his students conduct 20 studies, incorporating five separate contrasts at the .05 significance level per study, for a total of 100 contrasts. 15 of those 20 studies yield no significant results, and are not published. The other five studies yield one significant contrast and 4 non-significant contrasts each. Assuming no other problems in the interpretation of these studies (ha), what is the probability that all 5 results are spurious? Adam Reed (ihnp4!mtund!adam)