hamid@hilbert.logiclab.cis.syr.edu (Hamid Bacha) (03/15/89)
I was deeply offended by some of the comments made by one of the referees
of a paper I sent to ICLP'89. In fact, this referee stopped short of
accusing me of plagiarizing someone else's ideas. The authors of the paper
he referred to are colleagues of mine. I am thoroughly familiar with their
work as they are with mine. The only connection between the two papers is
that both address the issue of Prolog and Content-Addressable Memory.
It is clear to me that the referee either did not bother reading the paper,
or did not understand it. He claims to be very knowledgeable in this area,
yet he went on to make some outrageous comments. (A very competent referee
rated the same paper very highly and, more important, made suggestions that
clearly show that he read the paper and understood it.)
In light of this experience, I was wondering how many other people may have
been subject to this type of refereeing, and whether there is any mechanism
in place to weed out bad referees. I would like to make it clear that I am
not trying to seek any revenge, or just trying to raise hell because my paper
got rejected. It is just that this experience opened my eyes to a potential
problem. I was just wondering whether there is any way of sorting out
incompetent referees. Or is it the case that once a referee, always a referee
no matter what kind of job one does. Do authors who have been treated unfairly
have any recourse? If so, what can they do? Should they write to the Logic
Programming Committee? Does this committee take any action against referees
that have been constantly pointed out as being unfair or incompetent?
Would this lead some authors to blaming referees for rejecting their papers
instead of the quality of their work?
I applaud the push by some members of the Logic Programming Community,
particularly E. Shapiro, toward raising the standards of the ICLP by
by accepting fewer but of higher quality papers. If so many low quality
papers were accepted in the past, does not that suggest a possible problem
with referees. I was just wondering whether anything can be done in this area.
------------------------------------------------------------------------------
Dr. H. Bacha
Coherent Research, Inc.
100 E. Washington Street
Syracuse, New York 13202torkel@nada.kth.se (Torkel Franzen) (03/16/89)
In article <1199@cmx.npac.syr.edu> hamid@hilbert.logiclab.cis.syr.edu (Hamid Bacha) writes:
* It is clear to me that the referee either did not bother reading the paper,
* or did not understand it. ...
* In light of this experience, I was wondering how many other people may have
* been subject to this type of refereeing, and whether there is any mechanism
* in place to weed out bad referees.
No doubt most people who have submitted a few papers to conferences
have had similar experiences. It is of course particularly galling
since you have no opportunity to rebut comments which you regard as
ill-conceived or incompetent. Given the flood of papers to conferences,
and the fact that it is often very hard work to make a fair evaluation of
a submitted paper, I think it's very unlikely that the situation will
change. We'll just have to grin and bear it when referees make stupid
comments and try to remember our reaction when we act as referees
ourselves.
I do have one suggestion. It is customary for the author to be known to the
referee and the referee unknown to the author. I think it should really be
the other way around. The referee shouldn't be influenced by the name and
status of the author, and he should only make comments that he is prepared
to back up. Of course such a reversal of the current practice would no doubt
make it a great deal more difficult to find referees...maurice@kulcs.uucp (Maurice Bruynooghe) (03/17/89)
Papers are not rejected because of one bad report, neither are they accepted because of one good report. Papers with at least one accept score are thorougly discussed at the meeting. The arguments of all referees are considered carefully. In case of doubt, someone present at the meeting and competent in the subject of the paper looks at the paper. There are not that many papers with not at least one "accept" score. This means many papers getting several good refereereports got rejected. Maurice Bruynooghe
lee@munnari.oz (Lee Naish) (03/20/89)
I agree with Torkel Franzen's comment that most people who have submitted conference papers have experienced bad refereeing. Often programme committee members know the referees and their peculiarities and this can be taken into account somewhat. The other thing which I advise all victims of bad refereeing to look carefully at is presentation. I have had papers rejected because referees didn't understand the paper. Here are two quote from referees reports for the paper I submitted to the ICLP: "Nothing new..." "Each tools by itself does not advance the state of the art". Neither of these assertions is correct, but (in retrospect) its not that surprising the comments were made. The reason is that presentation of the new contributions did not stand out from the descriptions of previous work. Hopefully the revised version will be better.... lee