dsstodol@daimi.aau.dk (David S. Stodolsky) (11/09/90)
Consensus Journals: Invitational journals based upon peer consensus David S. Stodolsky Roskilde University Centre DK-4000 Roskilde david@ruc.dk Abstract Computer networks open new possibilities for scientific communication in terms of quality, efficiency, and rapidity. Consensus journals have the economy of invitational journals and the objectivity of journals based upon the peer review. That is, all articles are published and the reader benefits from article selection based upon impartial refereeing. An additional benefit of consensus journals is that the negotiation process, that typically occurs prior to publication, is automated, thus saving efforts of participants. Readers submit reviews that evaluate articles on agreed dimensions. A statistical procedure is used to identify the most knowledgeable representative of each consensus position and these persons are invited to submit articles that justify the review judgments they have submitted. A major advantage of this approach is the ability to develop reputation without article publication. The approach includes a protection mechanism based upon pseudonyms, that substitutes for the protection of anonymity typical with scientific journals. This reduces the potential for irresponsible behavior and facilitates reputation development. The level of quality enhancement is superior to that achievable with anonymous peer review. Eliminating the editor and the delay associated with conventional refereeing makes message quality enhancement available in message systems for educational and business environments. ____________________________________________________________________ This document has been prepared for electronic publication. Underscore characters indicate the start and end of italicized character sequences. Figures and tables assume a monospace font. Citation: Stodolsky, D. S. (1990). Consensus Journals: Invitational journals based upon peer consensus. _Datalogiske Skrifter_ (Writings on Computer Science). No. 29 / 1990. Roskilde University Centre, Institute of Geography, Socioeconomic Analysis, and Computer Science. (ISSN 0109-9779-29) ____________________________________________________________________ Invitational journals can be distinguished from typical scholarly journals by the sequence of events that results in publication of an article. The sequence of events with a typical journal starts with the writing of an article. The article is then transmitted to an editor and refereed. After a successful review, often contingent upon negotiated revisions, the article is published and read. With invitational journals, however, events are reversed. The tentative decision to publish an author is made first, often based upon the reading of previous work by that author. Then negotiation between the editor and author occurs, or there is informal refereeing of a proposal, which if successful, results in the writing of an article. The great advantage of this second sequence -- read, negotiate, write -- is that almost every article written gets published. The disadvantage is that selection of authors is somewhat arbitrary and there is no way an unknown author can get published. The objective of this article is to outline a method of scientific communication that has the economy of invitational journals and the objectivity of journals based upon the peer review. These self-edited journals will be called _consensus journals_ in order to distinguish them from conventional invitational journals. Any reader of an article in a consensus journal can act as a referee. Assume, for simplicity, that referees send reviews to a mediator. At a deadline, the mediator performs calculations and issues invitations to the referees who have been selected as new authors (Figure 1). These calculations are implicit negotiations, that is, they predict which persons would have been selected to respond to the reviewed article if referees has actually negotiated and reached a consensus. One benefit of consensus journals is that the negotiation process is automated, thus saving participant effort. -------------------------------------------------------------------- ------ R: Review ------- M: Invitation ------- | Read |----------->| Calc. |--------------->| Write |-----------> ------ ------- ------- R: Article | | R: Renege R = Referee Calc. = Calculate consensus | M = Mediator V -------------------------------------------------------------------- Figure 1. Simplified cycle of operation for a consensus journal The simplified cycle of operation for a consensus journal shows actions in boxes and messages as arrows. In this simplified cycle, referees invited to publish (and justify the reviews they have submitted) have a choice of submitting their article by a deadline or reneging on the promise implied by their review. This simplified cycle of operation assumes, additionally, that consensus positions can be calculated and that published articles are retained indefinitely. Eliminating these assumptions requires a more articulated cycle of operation and additional message types. Before considering a more articulated cycle of operation, however, it is necessary to note an important feature of peer review that contributes to impartial judgment. This feature is a protection mechanism, typically anonymity, that shields referees from pressures that might be associated with evaluation of a colleague. Further, names and affiliations of authors are often hidden from referees to ensure that only article content is the basis for evaluation. Protection can be alternatively be provided by a pseudonym system. This has the advantage of reducing opportunities for irresponsible behavior as compared to systems based upon anonymity (Stodolsky, 1990). It has a further advantage of permitting reputation development through the refereeing alone, thereby making it possible to establish a reputation without contributing articles. When there are multiple referees, it is important that their judgments are independent, so referee reports must not be made available until all have been submitted. This last requirement can be met by ensuring that reviews transmitted to the mediator are hidden until the deadline. The dynamics and implementation of protection systems are beyond the scope of this article, so only the necessity for the simultaneous release of information is addressed here. Definition of message types While in the simplest case, messages in the consensus journal environment consist of only articles and reviews (Stodolsky, 1990), considerations of effective negotiation and of storage management suggest defining additional message types. There are five types of messages transmitted in the consensus journal environment. _Article Articles, while shorter than those usually seen in conventional journals, will most often play the same role. However, it is quite possible to have an article in a consensus journal that is only a few lines long, and that can only be understood in connection with the review message it follows and its target article. _ Review Review messages must be distinguished from conventional reviews because they are characterized by a vector of numbers that summarize a reader's reaction to an article. If we think of articles as nodes in a graph or pages in a hypertext network, then review messages are the labels on arcs or links that connect the articles. Reviews can go beyond merely evaluating an article, by offering to provide new information that may be essential to support the target article's position. Review messages also serve as a commitment to deliver a justification of the reader's judgment, if invited. _ Invitation Invitations are public, and therefore, impossible to refuse without some loss of reputation. This makes them somewhat different than invitations from an editor of a journal. In effect, the invitation says, "We offer you storage space for an article." Also, a person may post an invitation for themselves during the negotiation stage of review, if they feel confident they can support the position claimed in their review message. _ Cancellation It is possible for an author to cancel an article, thereby releasing the associated storage space. The article then goes off-line (i.e., "out of print") along with its reviews and the articles that were dependent upon it for their place in storage. This would typically occur during explicit negotiation after the author had seen the article's reviews. It could, however, occur much later, when a new criticism was delivered. _Withdrawal Finally, during explicit negotiations, a review may be withdrawn. This eliminates the referee from those prepared to respond to an invitation. Cycle of operation The sequence of events with a consensus journal is the same as with an invitational journal. The review method, however, involves the entire readership, or at least those who offer a judgment. New authors are then selected based upon the review judgements. While most articles will follow from reviews and be connected to their target articles, independent articles are also permitted. However, articles posted without consensus based invitations are less likely to be read and can not be assumed to have support of other referees. If we assume that a consensus journal is already functioning, we can follow the events through a cycle of operation that starts with reading of an article. While it is not essential for smaller readerships, we assume that participants exchange information electronically. All readers are presented with a target article at the same time. A reader offers a review judgment in order to be considered for future authorship. The review message must be received before a certain deadline, say one week later. The review message consists of scores along several preselected dimensions. For instance, a scientific article is expected to be relevant, correct, and original. A more conversational approach might include the dimensions completeness, clarity, and appropriateness. At the deadline, the mediator runs a statistical procedure to determine if there are consensus positions among the referees. The most central referee from each of these positions is invited to submit a new article. These most central referees are also considered most knowledgeable, within the framework of cultural consensus theory. D'Andrade (1987) discusses the evidence supporting this view. Cultural consensus theory is based on the assumptions of common truth (i. e., there is a fixed answer pattern "applicable" to all referees), of local independence (i.e., the referee-dimension response variables satisfy conditional independence), and of homogeneity of items (i.e., each respondent has a fixed "cultural competence" over all dimensions) (Romney, Weller & Batchelder, 1986). Results can be obtained with as few as three respondents, but four are required if the significance of the results are to be calculated (i. e., a degree of freedom is then available in the statistical model) (Batchelder & Romney, 1988). A recent development in the model is the ability to identify two consensual groupings within the population of respondents (Romney, Weller & Batchelder, 1987) This is extremely helpful since it permits a minority to publicize their viewpoint under the same conditions as a majority. Cultural consensus theory assumes that we have no _a priori_ knowledge about referees, that is, they have no reputations. This is extremely valuable when a new topic comes up or when there are violation of assumptions required for calculations concerning a current article based upon previous information (Stodolsky, 1984b). Given that reputations have developed and assumptions are satisfied, however, the theory requires elaboration to be applied most effectively. Cultural consensus theory provides, in effect, a cross sectional estimation of competence. That is, given a sample of responses at a given moment, relative competence is estimated. On the other hand, given a performance history, Bayesian estimation can be used to assess the relative importance of different persons' judgments. That is, there are reputations that give information about relative competence independent of the current responses. This assumes stationarity, that is, that the same area of competence is required for correct response, and that responses are generated in the same manner (e. g., respondents continue to give honest answers). Both methods are based upon likelihood estimation, therefore, a combined theory should be achievable. The combined sources of information would likely make achieving an implicit consensus more frequent. The mediator issues an invitation report showing submitted judgments, the degree of consensus achieved, the number of consensus positions identified, degree of knowledge of each referee, and so on. If consensus has been reached, invited referees are expected to submit articles. Negotiation must proceed explicitly if no consensus can be identified (Figure 2). In that case, referees may look at the judgments submitted and decide if their positions have sufficient support. If not, they could reconsider their review judgments, and either revise them or withdraw from the review process. The author of an article might, on the basis of these judgments, cancel an article, thus avoiding potential reputation damaging criticism. -------------------------------------------------------------------- ------ R: Review ------- M: Invitation ------- | Read |----------->| Calc. |--------------->| Write |-----------> ------ ------- ------- R: Article ^ | | R: Review | | M: No consensus | R: Renege | | | | V V ----------- R: Withdrawal | Reconsider|--------------> ----------- | | A: Cancelation | V Key A = Author M = Mediator Calc. = Calculate consensus R = Referee -------------------------------------------------------------------- Figure 2. Cycle of operation for a consensus journal (A referee becomes an author only after a submitted article has been published by the mediator [not shown in figure]). Assuming that the article was not cancelled, the combined effects of withdrawal by referees with most deviant judgments and reconsideration by others would likely lead to consensus, particularly if the requirements for consensus were successively relaxed. This assumes that revision of judgments would be in the direction of dominant view points, a common finding. The result would be an invitation issued by the mediator to selected referees. Subsequent submission of an articles by selected referees and their publication by the mediator would complete the cycle of operation. The invitation report can guide negotiation when a consensus can not be identified. Individual invitation staging could proceed along with a relaxation of requirements for consensus. For instance, if the first round of reviews did not generate a consensus, referees could issue invitations to others (Table 1). If the second round of reviews did not generate a consensus, referees could issue self invitations (these would be acceptances for those who had received invitations), or perhaps, direct the invitations they had already received to others. Failure on the third round would permit these previously issued invitations to serve as a coordination mechanism. That is, certain referees would have indicated a readiness to respond and others would have rejected the option of authorship unsupported by a consensus. Thus, duplication of effort could be avoided by examining the ranking of persons in terms of the invitations received and accepted, and responding accordingly. -------------------------------------------------------------------- Message Level and (Receiver): Public Private (Readers) (Mediator) Stage of Process: Publication Article Refereeing Review(1) Invitation Report Reconsideration Review(2), Withdraw, Cancel, Invite Invitation Report Reconsideration Review(3), Withdraw, Cancel, Self invitation Invitation Report Submit Article -------------------------------------------------------------------- Table 1. Message level, receiver, and type, by stage of processing (assumes no consensus reached during negotiation) New articles are requested either by the mediator or by referees, if an author and referees follow negotiations to completion. New articles must be submitted before a deadline. At the deadline, the new articles received are published. This makes them available to the readership and completes the cycle of operation for a consensus journal. Thus in the simplest case, articles are read, reviews are transmitted, invitations are issued, and new articles are submitted in a timely manner. In the extended cycle, at least a single reconsideration or negotiation stage occurs during which a target article can be cancelled by its author and during which referees can withdraw. A failure to achieve consensus leads to explicit negotiation and options such as nonconsensus invitation. The extended negotiation option makes the consensus journal more similar to a conventional journal, because there is explicit negotiation prior to the writing of an article. The quality of a consensus journal can be assessed by the degree of consensus achieved. Readers might select only those articles resulting from a consensus-based invitation, thereby controlling the quality of articles they see. Rules of dialogue The rules of operation of a consensus journal can be thought of as specifying an action system, or language game, where the actions relate to the placement of articles in a network of interconnected nodes. Participants in the game try to maximize their influence. Reputation is a crucial resource in scientific argumentation (Smolensky, Fox, King, Lewis, 1988). Participants are expected to maximize this resource. While there may be other payoffs available within a given system, such as royalty payments, this discussion assumes only reputation maximization as an individual objective. There are several opportunities for reputation enhancement in the cycle of operation. Selection as an author is a major opportunity for reputation enhancement. However, refereeing also offers significant opportunities that are not available with conventional journals. Referees can commit themselves to delivering a rebuttal to an article and thereby improve their reputation (assuming they make good on their commitment given an opportunity). If an author examines the reviews an article receives and decides to cancel it before a rebuttal is written, the referees offering rebuttals would have their reputations enhanced, without any further risk or effort. With a consensus journal, the review message can be thought of as an offer to deliver a certain type of article before the deadline. Obviously, a review message that claims a target article is erroneous, and thereby offers to deliver a rebuttal, plays a different structural role in a debate than one that criticizes an article for not being original. Thus, reviews can have a great deal of structural impact and can express a level of commitment, which would not be relevant in an environment that limits referees to a gatekeeping role. Structural Aspects With electronically published documents, it is very desirable to structure interconnections so that retrieval is facilitated and the relevance of statements becomes clear (Smolensky, Fox, King, Lewis, 1988). Thus, review messages can deal not only with the quality of an article, but also its relationship to its target article. Explicit relationships among articles becomes more necessary as the size of articles decrease and number of articles increases. With conventional journals, reviews are used to determine whether or not an article should be published. The publication decision is not dominant with electronic media, however, since distribution constraints are greatly relaxed (Quarterman, 1990, p. 259; Stodolsky, in press). Because of this, the period during which an article remains on-line assumes importance, because storage is limited. It is in this connection that the reviews of articles and the relations between articles becomes critical. In the simplest case, an article that is found incorrect by an overwhelming consensus is cancelled by its author. Failure to cancel an the article results in a continuing devaluation of the author's reputation as more and more readers come to agree with the majority. In the case of conflicting consensus positions, a rebuttal claiming that a target article is flawed is explicitly linked to the target. Failure to rebut that claim in turn has much the same effect as an overwhelming consensus that the target article is incorrect. Most interactions, therefore, take place at the knowledge frontier, as various positions are argued. These interactions generate very "bushy" argument trees, that require sophisticated navigation strategies, if large amounts of effort are not to be expended unnecessarily (Stodolsky, 1984a). The trees are thinned in the process of argumentation. Positions that are sustained remain on- line until they are thoroughly integrated into summaries or overarching theories. Decentralization A central mediator has been assumed in this description to simplify explanation. There is no reason why the calculations necessary to select new authors could not be performed decentrally. In fact, this would be necessary if readers preferred different methods of calculation for author selection. Then coordination in the selection of new authors would be shifted from consensus calculation to collection of invitations. Various types of voting rules could be applied. Authors receiving the most invitations would then be expected to submit articles. Thus, decentralization leads to an integration of the two types of invitations (consensus and individual) already discussed. The task of protecting review judgments until the deadline is reached is another required function. It is necessary, for example, because analysis of earlier submitted judgments could permit a referee submitting at the last moment to simulate a competence that did not exist, thus violating assumptions of the model. Protection can, however, be achieved decentrally using cryptography, assuming a "beacon" that emits enciphering and deciphering keys at fixed intervals (Rabin, 1983). Use of cryptography would be necessary, in any case, to ensure the authenticity of messages. Summary A consensus journal requires mechanisms for both coordination and protection. In the simplest case, a mediator can provide these. This assumes protected channels of communication and a trusted mediator. Coordination is necessary to identify consensus positions and avoid duplication of effort. Protection of reviews is necessary to ensure that assumptions of models for evaluating expertise are not violated. This protection allows valid reputation development by both authors and referees. Such reputations can then be used to ensure effective allocation of expertise. The extension of review opportunities to the entire readership vastly extends the available field of expertise. This, combined with the effective allocation of expertise and coordination that eliminates duplication of effort, provide consensus journals with a significant advantage over current mechanisms for enhancement of message quality. References Batchelder, W. H. & Romney, A. K. (1988). Test theory without an answer key. _Psychometrika_, _53_(1), 71-92. D'Andrade, R. G. (1987). Modal response and cultural expertise. _American Behavioral Scientist_, _31_(2), 194-202. Quarterman, J. S. (1990). _The matrix: Computer networks and conferencing systems worldwide_. Bedford, MA: Digital Press. Rabin, M. (1983). Transaction protection by beacons. _Journal of Computer and Systems Science_, _27_(2), 256-267. Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1986). Culture as consensus: A theory of culture and informant accuracy. _American Anthropologist_, _88_(2), 313-338. Romney, A. K. , Weller, S. C., & Batchelder, W. H. (1987). Recent applications of cultural consensus Theory. _American Behavioral Scientist_, _31_(2), 163-177. Smolensky, P., Fox, B., King, R., & Lewis, C. (1988). Computer-aided reasoned discourse or, how to argue with a computer. In R. Guindon (Ed.), _Cognitive science and its applications for human-computer interaction_. Hillsdale, NJ: Lawrence Erlbaum Associates. Stodolsky, D. (1984a). Commonalities amoung [sic] conferencing systems and their implication for marketing strategy. _Organisatoriske Fragmenter 1984_, _12_, 43-58. Stodolsky, D. (1984b, December). _Self-management of criticism in dialog: Dynamic regulation through automatic mediation_. Paper presented at the symposium Communicating and Contracts between people in the Computerized Society, Gothenburg University, Sweden. Stodolsky, D. S. (1990). Protecting expression in teleconferencing: Pseudonym-based peer review journals. _Canadian Journal of Educational Communication_, 19, 41-51. ([1989, May 9]. _Communication Research and Theory Network [CRTNET]_, No. 175 [Semi- final draft available by electronic mail from LISTSERV@PSUVM.BITNET at University Park, PA: The Pennsylvania State University, Department of Speech Communication and COMSERVE@Vm.ecs.rpi.edu at Troy, NY: Rensselaer Polytechnic Institute, Department of Language, Literature, and Communication]) Stodolsky, D. S. (in press). Archiving secure interactions [Letter]. _Psychological Science_. ([1990, May 25]. Comments on Gardner's Electronic Archive by Stodolsky. _Psycoloquy_, _1_[8].) -- David S. Stodolsky Office: + 45 46 75 77 11 x 21 38 Department of Computer Science Home: + 45 31 55 53 50 Bldg. 20.2, Roskilde University Center Internet: david@ruc.dk Post Box 260, DK-4000 Roskilde, Denmark Fax: + 45 46 75 74 01
jmaynard@thesis1.hsch.utexas.edu (Jay Maynard) (11/09/90)
That's all well and good...but what's the relevance to Usenet? Usenet isn't
even close to the journal you propose, and can't be hammered into such a
shape even if you have Mjollnir at your disposal. (For the mythology-
impaired, Mjollnir was Thor's hammer; Thor was the Norse god of thunder.)
The rest of us seem to get along just fine with Usenet as it is. What's
your beef? As I said, there's not a whole lot you can do about the level
of flamage on the net, and complicated proposals for greatly limiting the
volume of discussion certainly won't help.
--
Jay Maynard, EMT-P, K5ZC, PP-ASEL | Never ascribe to malice that which can
jmaynard@thesis1.hsch.utexas.edu | adequately be explained by stupidity.
"With design like this, who needs bugs?" - Boyd Roberts