How to review a journal article: requirements, tips and strategies


David J. Pannell

School of Agricultural and Resource Economics, University of Western Australia

1. What the journals tell you

The notes in this section are adapted from instructions provided by the Agronomy Journal, the Journal of Consumer Research, the American Journal of Pharmaceutical Education, and the Canadian Journal of Agricultural Economics.

 The ideal review will be fair, unbiased, speedy, and confidential. The ideal reviewer will approach the paper in terms of questions such as: "Is the science good?" and "Is it understandable?" or "What is needed to make it clear?" rather than "What are all the little things that annoy me in style or presentation?"

1.1 Ethics

Volunteer reviewers and editorial board members are asked to decline from reviewing papers of any authors with whom there is a possibility or appearance of a conflict of interest. Can you can answer "yes" to one or more of the following questions?

If so, you should respond as follows: (a) if there is a clear problem with you doing the review, withdraw, or (b) if you are uncertain about whether it is appropriate, advise the editor of the situation and ask him or her to judge whether you should review the paper.

Reviewers must attempt to be impartial when evaluating a manuscript. Although it is difficult to be completely objective when assessing a paper that may not coincide with one's own beliefs or values, nevertheless, a reviewer must always strive for that goal. If a reviewer cannot separate the evaluation process from a desire to advocate a preferred theory or to reject the manuscript out-of-hand on philosophical grounds, then the reviewer should disqualify himself or herself from that review.

Do not allow the manuscript to be reproduced while in your custody. You must not use the manuscript for your personal advantage in any way. You cannot cite it or use its contents in any way until/unless it is published. If it is not published but you wish to use it, you need to contact the author (e.g. via the editor if the journal uses double-blind reviewing).

1.2 Abstract

Abstracts accompany articles in most journals, and they are often republished as printed in secondary abstracting services and journals. The abstract, therefore, should meet two requirements. A reader should be able to tell readily the value of the article and whether or not to read it completely. It also should provide the literature searcher with enough information to assess its value and to index it for later retrieval.

 According to the Agronomy Journal, the abstract should:

Different journals specify different lengths for their abstracts. Some are not long enough to do all of the points listed above. If cutting aspects out, I'd look at dropping methods, reducing objectives to a minimum, and limiting results and conclusions to absolute highlights. 

1.3 Manuscript General Content

Across all the review rounds, reviewers should strive to distinguish between what is perceived as correctable versus uncorrectable problems and between major versus minor concerns. The first round of reviews is the time for reviewers to highlight uncorrectable problems or other major concerns about a paper. It is generally inappropriate to raise them in later review rounds if they already existed in the first draft submitted. In most cases, new uncorrectable problems or new major concerns raised in later reviews should only apply to changes in a paper that have emerged through the authors' revision work.

1.4 Quality of Writing

1.5 Technical

1.6 Tables and Figures

1.7 Remember ...

1.8 What you provide to the editor

1. A brief letter addressed to the editor (not for the authorís eyes) with a specific recommendation. Different journals have slight variations in the list of possible recommendations. They are likely to be similar to the following options:

  1. Accept unconditionally. This rarely used category should be reserved for manuscripts that are virtually flawless in their content. In general, when a reviewer makes this recommendation, he or she will be regarded as having signed off on the manuscript.

  2. Accept conditionally, subject to minor revisions, according to accompanying comments. This recommendation should be made when the manuscript is judged to be quite strong and in need of only minor additions, deletions, or corrections. It does not need to be reviewed again, in your view. The editor can judge whether revisions have been made adequately.

  3. Encourage revision, according to accompanying comments. This recommendation should be used for manuscripts that have a high degree of potential for eventual publication, in addition to significant deficiencies that must be corrected. For instance, the research addresses a vital topic, but is not presented in a logical or lucid fashion, or questions may exist about some aspects of its method. This recommendation should be used when the reviewer believes that satisfactory resolution to his or her concerns is possible and that the achievement of successful resolution will result in an acceptable manuscript. Detailed Comments to the Authors are extremely important in support of this recommendation, so that the authors can conceivably answer all the concerns in a single revision. (A recommendation in this category should not be construed as a guarantee of eventual publication. In some cases, a seemingly promising manuscript will not be adequately revised to attain the quality and level of knowledge contribution required for publication.)

  4. Reject in current form, but allow resubmission of a substantially different version. The essential difference between this recommendation (reject in current form) and the option above (encourage revision) is that the current version of the manuscript is not publishable in anything approximating its present form. Instead of simply rejecting a manuscript as completely unsalvageable, this recommendation includes thoughtful advice for producing a potentially publishable, but different manuscript. This recommendation should be used sparingly. Some journals do not include it as an option. For these, you would probably switch to the next option.

  5. Reject unconditionally, because the likelihood of successful revision is remote. This recommendation is appropriate for papers that have major problems. It is viewed as having virtually no chance of ever reaching a publishable standard, at least for this journal. For example, the topic may be of minor importance; the basic conceptual development may be extremely weak or incorrect; or the empirical work may have uncorrectable defects. This category is the most common recommendation received by high prestige journals. Comments to the Authors should be especially polite in explaining the nature of the concerns, but the Comments to the Authors need not be as lengthy as in the previous categories. It is permissible and efficient to articulate only the most serious concerns, and to conserve reviewing energy for other manuscripts that stand to gain from more detailed reviewer input.

Briefly indicate your reasons for the decision in the letter to the editor. You might address issues such as the significance of the contribution, quality and thoroughness of the analysis, clarity of presentation, practical importance, originality, depth and appropriateness for the Journal.

In some cases, a reviewer may be competent to evaluate either the conceptual or methodological aspects of a paper, but not both. In such instances, the reviewer should indicate this in the cover letter to the Editor.

2. Separate comments to the author. Please ensure that comments are detailed and clear and that they are constructive in nature, even if you are recommending that the paper be rejected. Make suggestions about length, organization, tables, figures and references. Number all comments, to make it easy for the author to refer to them in correspondence with the editor.

 A quality review should note the strengths as well as the weaknesses of the manuscript. Authors need to know what they have done well and not just what they have done poorly.

The reviewer should avoid language that can be easily misconstrued or that appears condescending, including obscure or loaded vocabulary as well as humour, irony, and sarcasm. When the reviewer negatively evaluates different aspects of a paper, he or she should keep in mind that there are many ways to communicate this to the authors. For example, phrases such as "fatal flaws" or "serious mistakes" might instead be offered as "substantial concerns" or "major issues." Reviewers should be as polite and as diplomatic as they are demanding.

If the reviewer has ideas on how to improve the paper and the research, sharing those suggestions in the most precise manner possible will raise the likelihood that the authors will understand, appreciate, and utilize those ideas.

Specificity. A quality evaluation must be communicated in sufficient detail to support the reviewer's recommendation to the Editor. For instance, when alluding to previous research to uphold an assertion about some conceptual, methodological, or substantive weakness, it is imperative to provide a complete citation so the Editor and authors can locate it.

The Comments to the Authors represent the most important component of the review. They provide the rationale for the reviewer's evaluation of the manuscript, as well as suggestions for the improvement of the paper.

The Comments to the Authors should also not contain any semblance of a recommended rejection or acceptance of the manuscript. Such recommendations should be made only in the confidential Summary Letter to the Editor. It is the Editor's responsibility to make the final decision and also to decide on the best way to communicate that decision to the authors.

Comments to the Authors are generally most useful to the authors and the Editor when they begin with an overall assessment of the reviewer's reaction to the manuscript, including prominent strengths and weaknesses. This big picture is valuable in providing a context for the more detailed comments that follow. The summary is especially important in identifying the reviewer's major concerns. After the overall assessment, the detailed comments are then provided that tactfully justify the evaluation and offer constructive, specific guidance for a revision or for future research efforts. It can be useful to provide the most important of the specific comments before any minor comments on matters of detail.

2. What the journals donít tell you

2.1 Some questions you may wonder about

Why were you chosen as a reviewer?

There are several possible reasons. An article of yours may have been cited in the manuscript to be reviewed. You may have submitted an article to the journal yourself. The editor may know you or know of you from other sources. You may have been recommended to the editor by someone else.

Do you have to be an expert in the subject of the article?

No. You have to have enough expertise in related areas to be able to make sense of it, at least at a general level, but you donít necessarily have to be knowledgeable enough to, say, reproduce the research yourself. It is up to the editor to ensure that the set of reviewers covers enough areas of expertise. If there are aspects that you are unable to evaluate (e.g. you donít understand the statistical technique used) advise the editor of this when you submit your review.

Do you have to agree to do the review?

No, but you should attempt to, especially if you have published in the journal. If you just canít fit it in, decline politely, as early as possible.

What degree of quality control do reviewers provide?

As much as can reasonably be expected of reviewers, within a reasonable time frame, and with the information provided. But not more than that. Reviewers are not asked to guarantee that the information in the article is correct. They are not asked to reproduce the research, which is the only way that some errors could be detected. They are asked to assess whether it appears to be correct, judging with the best of their ability. Plenty of published articles have been found to contain errors. This is obviously better avoided if possible, but sometimes the reality is that it is impossible.

Why should you review articles?

You should review articles because it is part of the culture of research that we all contribute to this essential service. It is likely that in the long run the amount of reviewing you are asked to do will be roughly proportional to the number of articles you submit to journals.

Reviewing is also good for you. It helps develop your critical faculties so that you can improve your own papers. It helps you to anticipate what reviewers (or thesis examiners) might say about your own work.

2.2 A suggested strategy

Here is a strategy that I have found works for me.

  1. Work from a printed copy. This makes it easier to make notes on it as you go, and to read it in places where you would not take a computer.

  2. Read through the paper, making notes at the level of detail that is appropriate for that paper. The better the paper, the finer detail you should go to. For a paper that is excellent and needs only very minor revisions to be accepted, you might even note problems with punctuation and spelling, if they are not too numerous. For a paper that is terrible, you would only note major issues, or even major groups of issues. Sometimes I get hooked into rewording sentences. You should only do this if the paper doesnít need too much of it. If bits need rewriting, say so in your review.

  3. If the decision is really obvious and you are confident about your ability to judge the paper, you could write your review immediately after your first reading. Otherwise, leave it a few days or a week and come back to it. Read it again, or read problem sections again, and then write your review.

  4. When selecting your recommendation, consider the prestige of the journal and the usual standard of articles it includes. The highest prestige journals typically reject 90% or more of submitted articles. You are unlikely to be asked to review for such a journal early in your publishing career, unless you have managed to publish in one yourself. The least selective journals probably accept somewhere around 50-70% of submitted articles (after revision of course), but most journals probably accept between 20 and 50% of submissions.

  5. If you are not going to reject the paper, obtain a copy of a recent issue of the journal, or a paper from a recent issue, to check on formatting of headings, tables, figures, references. Note any problems in these areas in your review. You might also check the journalís instructions to authors at that stage, although I rarely find that useful.

  6. Write the manuscript reference number and title at the top of your review.

  7. Always number your comments. If you have multiple related comments, number them separately. This allows the editor and the authors to easily refer to your comments in further correspondence.

  8. Start by saying something positive about the paper, no matter how difficult this is. Say as many positive things as the paper deserves (or one more than that if it doesnít deserve any) before you get into criticisms.

  9. Some reviewers provide an overview of the paper. Personally, I don't think this is worth doing. The editor will read the paper too and the authors already know whatís in it!

  10. Next, note any concerns or problems at the big picture level. e.g. relating to the overall approach, the statistical methods, the interpretation of results, the quality of presentation or writing.

  11. Then, start presenting specific comments on the paper. Note the page number and the line number for each comment if possible. Start with most important or serious of the specific comments, and then include any minor issues.

  12. There is no need to write a conclusion to your review. Just stop when you run out of comments.

  13. If you have cited any literature, provide complete reference details.

Respond as quickly as you can manage. Delays in the publishing process are often very long, so itís nice not to contribute to that more than necessary.

Reviews are often between one and two pages single spaced. The longest I've written was about five pages, and the shortest was about 10 lines.

Further reading

Pannell, D.J. (2002). Prose, psychopaths and persistence: Personal perspectives on publishing, Canadian Journal of Agricultural Economics, 50(2): 101-116. Full paper (66K)

Charry AA, Murray-Prior R and Parton KA (2004). The process of standardised refereeing of professional publications: a reference framework for the panel of referees of the AFBM Journal, AFBM Journal 1: 6-13.
http://www.afbmnetwork.orange.usyd.edu.au/afbmjournal/volume1/

 

Citation: Pannell, D.J. (2006). How to review a journal article: requirements, tips and strategies, Discussion Paper, School of Agricultural and Resource Economics, University of Western Australia, http://dpannell.fnas.uwa.edu.au/reviews.htm


David Pannell home page

Copyright © 2006-10 David J. Pannell
Last revised: August 05, 2013.