Prose, psychopaths and persistence:
Personal perspectives on scientific publishing
David J. Pannell
School of Agricultural and Resource Economics, University of Western Australia, David.Pannell@uwa.edu.au
Last week I received a letter from a scientific journal. This was an exciting event because I was waiting to see whether they would publish a research paper that I sent them about six months ago. I opened the letter hoping for the best, but fearing the worst. The news was bad. One reviewer had a reasonably positive attitude to the paper, but the other was very negative. Some of the comments of the second reviewer are really annoying me. I can’t believe that they’ve really read the paper properly. It’s frustrating and quite disheartening, especially when I think about how much work went into the research and into writing the paper.
But deep down I know this is all perfectly normal. Publishing scientific research was not meant to be easy. It’s important, of course, for all serious researchers. It provides scrutiny, recognition, rewards and a vehicle for wide communication, but on the downside, the process is slow, heartless and somewhat random. So I’ve decided that it’s time for an exposé of the scientific reviewing process.
First, I should tell you briefly how the process works. There are hundreds of journals, each specialising in a particular research field. Once you’ve written a research paper, you choose a journal that you judge might publish it, and send the paper to the editor. The editor sends it to two or three referees or reviewers. Editors usually attempt to select referees with some expertise in the topic of the paper. In most journals, the reviewers don’t get paid anything for the papers that they review, and the authors don’t get paid either. Indeed, many journals charge authors “page charges” for the privilege of getting published.
You can only submit to one journal at a time. A common practice is to initially submit to a relatively high prestige journal with low probability of success, and if your paper is rejected, you work down the hierarchy of journals until the paper is no longer rejected.
Many journals use a system of “double-blind” reviewing, in which the identities of both authors and reviewers are kept secret by the editor. The intention is that reviewers should feel free to speak their minds without fear, and that the work should be judged on its merits, independent of the identity of the authors. While these are reasonable objectives, it doesn’t always work out. Sometimes the power of being anonymous goes to a reviewer’s head, resulting in needless aggression or negativity. Sometimes the anonymity of the author is hard to maintain, particularly in highly specialised areas with relatively few active researchers. When reviewing a paper myself, I find that I usually have a good guess at who the author is. So some journals use a system of “single-blind” reviewing, revealing the authors to the reviewers but not the other way around.
Let’s talk now about some to the problems that a researcher will encounter when trying to publish in a journal. The first is randomness. The sheer unpredictability of what reviewers will say still takes my breath away at times. It can be particularly entertaining when reviewers of the same paper are diametrically opposed in their views.
Here is an example. I submitted a paper to an Australian journal some time ago and got these two responses from the two reviewers.
Reviewer 1: ‘This paper falls into the class of papers that can be denominated as “measurement without theory”’.
Reviewer 2: ‘This paper, as it stands, is purely theoretical’.
Not surprisingly, the paper was rejected.
Sometimes, the entertainment comes from comparing the attitudes of referees for one journal with those of a subsequent journal.
I once wrote a paper on the economics of agricultural insurance. I submitted it to an American journal, and it was rejected, with highly critical responses by the reviewers.
I modified the paper slightly and submitted it to an equivalent Canadian Journal. It was accepted with minimal changes and with some enthusiasm. Subsequently that paper shared the award of the Canadian Agricultural Economics Society for their best journal article of the year.
Although the initial rejection was disappointing, I can take heart in the experiences of far more eminent researchers than myself. There are some great examples from my field of economics. For example, in 1923, an economist named Bertil Ohlin submitted to the Economic Journal a paper that introduced the factor proportions theorem to international economics. The theorem eventually earned him a Nobel Prize, but the editor of the journal, John Maynard Keynes, returned the manuscript with a blunt rejection note saying: ‘This amounts to nothing and should be refused’.
The second problem is bad or unreasonable reviewers.
If a reviewer says your research paper is insufficiently interesting or original to deserve publication, it is usually hard to argue. Such judgements are subjective and personal. Sometimes, ‘though, referees write what is obviously nonsense.
This happened to me with a paper I wrote about herbicides. Reviewer 1 quite liked the paper and made some constructive suggestions, but Reviewer 2 obviously hated the paper, primarily because “I don’t think its main point is sufficiently worth making to be the primary focus of a paper, particularly one which has some technical problems.” Despite the reviewer’s comment, no genuine technical problems were identified.
I revised the paper, and wrote a letter to the editor explaining that the supposed technical errors were not technical errors at all, and resubmitted it. He sent it off for reviewing again.
Referee 1 was happy but the second reviewer still hated it. He or she made no comment about the supposed technical problems that had been identified the previous time but introduces a brand new set of supposed technical problems, again which have no foundation.
I revised the paper again, wrote another lengthy rebuttal letter to the editor, and resubmitted again. After resubmission, the second reviewer again made no acknowledgment that the so-called errors in the paper were actually non-errors, conveniently ignored the whole issue, while re-asserting disdain for the paper in very general terms. A third reviewer brought in to adjudicate was, unfortunately, not impressed with the paper and eventually it was rejected.
Subsequently it was published in another journal with little trouble.
This was a particularly upsetting experience. I put an enormous amount of work into the two rounds of revisions to try to keep Reviewer 2 happy, but I could only feel that the reviewer was being extremely unreasonable and almost dishonest.
The other feeling I had was one of powerlessness. Even though I felt I was so obviously in the right and the reviewer so obviously wrong, I felt like my punches were landing on smoke. I had to be accountable for every tiny issue raised by the reviewer, but the reviewer could blithely ignore his or her past erroneous comments as if they had never happened, and still win the battle.
Of course, sometimes the process works, in the sense that the paper is correctly judged to be not of sufficient merit or real problems with the paper are identified, potentially saving the author from serious embarrassment.
The painful examples I’ve described did at least teach me a lot about the reviewing process, and thicken my hide for subsequent encounters. And one certainly does need a thick hide. I have found that many referees are needlessly harsh and heartless. Too many seem to take the view that their role is to demolish a paper if possible. Even when reviewers are trying to be constructive, it is very easy for an author to become disheartened at the poor reception their work has received. I believe it is very important for reviewers to consciously highlight positive aspects of papers, not just negative.
As some therapy after that experience, I wrote a little verse called “I’m the referee”. Here’s an extract.
This power's a revelation
I'm so glad it's come to me
I can be a total bastard with
I used to be a psychopath
But never more will be
I can deal with my frustrations now that
I'm a referee
I sent it to the editor, but he didn’t respond.
That story shows how much persistence it can sometimes take. And it’s not just me. Some of the most cited and celebrated papers of all time have only seen the light of day because their authors did not give up in the face of repeated rejections. A great example is George Akerlof’s wonderful 1970 paper on the economics of information. It’s called “The market for ‘lemons’”; it’s about used cars. That paper was rejected by the American Economics Review, with the comment that the journal did not publish such trivial stuff; then it was rejected by the Journal of Political Economy, who asserted the opposite: that the paper was too general to be true; then it was rejected by the the Review of Economic Studies again on the grounds that it was trivial; and finally the Quarterly Journal of Economics accepted the paper with some enthusiasm. Subsequently the paper has gone on to be one of the most widely cited papers in all of economics.
Akerlof’s understandable response to the trauma of publishing “The market for lemons” was to be discouraged about the topic and to underestimate the importance of the topic and its potential for further work. He later said, ‘its early rocky reception did have an effect on my own work. It was not until three years later when I spent 6 months on sabbatical in England that I realised that quite a few people had read the paper and even liked it. I believe I would have done follow up work on the market for lemons sooner if I had not been made to feel lucky just to have it published at all.’
The third common problem with trying to publish in a research journal is the slowness of the process.
Partly because their role is honorary, and partly because researchers are busy people, reviewers are not always quick to respond. I recently went back and checked how long the process took, for my most recent 27 submissions to journals of agricultural economics, natural resource economics and agricultural science.
The total time lag between submission and the receipt of the editor’s final decision varied from 3 months to 2.5 years. Only a quarter of my papers were accepted or rejected within 7 months, and the average for all 27 papers was 12 months.
There are other lags in the system as well: the author usually has to make changes to the paper to satisfy the referees, the editor has to look at the final version and make a decision, and then if it is accepted, it goes into a queue before being printed. For my recent papers the average total lag from initial submission to finally seeing the paper in print was almost 20 months.
It is worth acknowledging that there are often very legitimate and understandable reasons for the failure of reviewers to provide timely and high quality reviews. Editors naturally see it as desirable for reviewers to have a wide breadth of knowledge and considerable experience in publishing. But such people are inevitably very busy with their other work responsibilities, and it is certainly understandable that many of them don’t rush to undertake a task for which they get no professional recognition and usually no monetary reward, although some journals are now offering small financial payments to referees, more as a token than as a reward. They are also likely to receive a large number of papers to review. Authors can probably help themselves by recognising these realities and making sure that their paper is as short as they can make it, well written and presented, and is being submitted to an appropriate journal.
Publishing in research journals requires fortitude, resilience and persistence. So, if you are a researcher, you shouldn’t be too discouraged about the negative response of reviewers, because even the best researchers have suffered similar problems. You should not give up on a paper that you continue to believe has merit, despite being rejected, because some of the most celebrated papers have suffered through several rounds of rejection before finally being accepted. Of course do consider the possibility that your paper is, in fact, not worth persisting with, but just don’t be too quick to leap to this conclusion.
Ockham's Razor, ABC Radio National, 26/10/03: The dramas of scientific
Longer version with more stories: full paper (66K)
Selected email responses to the talk
David Pannell's home page