Tips on Publishing and Reviewing Papers in Statistics Journals

William I. Notz is professor of statistics at The Ohio State University. He earned his bachelor’s in physics from The Johns Hopkins University and his PhD in mathematics from Cornell University. He has served as the editor of Technometrics and the Journal of Statistics Education.


Karen Kafadar is Rudy Professor of Statistics and Physics at Indiana University. She earned her bachelor’s and master’s degrees from Stanford and her PhD in statistics from Princeton. She has served as editor for JASA Reviews and Technometrics and is currently area editor for The Annals of Applied Statistics.

Our comments are influenced by our combined editorial experience as editors (at different times) of Technometrics, JASA Reviews, The Annals of Applied Statistics, and the Journal of Statistics Education. However, we both have served on the ASA Committee on Publications, so many of our comments apply to other journals as well.

Scope or Mission of Journal

Most journals have a mission statement (or editorial policy) and guidelines for authors. Read them carefully. Does your manuscript fit the mission of the journal? A journal that emphasizes applications or practical methodology is probably not a good place to submit a highly theoretical/mathematical paper. Read through some of the articles in recently published issues. Do they conform to a rigid definition-theorem-proof format? Does the mission state specifically demonstration on, or motivation by, a real data set? What do the guidelines say about the format for the abstract, key words, equations, references, etc.? If the objective of your manuscript (e.g., review of existing techniques) does not match the mission of the journal (e.g., novel methodology), almost surely the editor will reject your manuscript outright or suggest withdrawal without prejudice.

Target Audience and Readership

Many journals have a specific audience in mind. Examples and applications should be consistent with the interests of the target audience. For example, a journal targeted at engineers is probably not a good fit for a paper with examples from psychology. Similarly, a journal whose title includes some derivative of the word “Biology” (e.g., Statistics in Biopharmaceutical Research, Journal of Agricultural, Biological, and Environmental Statistics) will not be interested in a paper motivated by problems in astrophysics. A general applications journal (e.g., JASA Applications & Case Studies) may be open to all areas of application, while other journals such as Technometrics publish methodology for a wide, but definitely targeted, range of areas. A glance at some of the journal’s published articles will indicate whether the journal’s audience matches the readership you aim to reach.

General Issues

Short communications, manuscript lengths, placement of proofs, technical results, simulations, tables, figures, etc.
Generally, these issues will be spelled out in the editorial policy for the journal. If in doubt, you can send a brief inquiry to the editor. Be advised that such communications with the editor should be clear and concise. In addition to their regular jobs, the editor’s first responsibility is the review of, and decision about, current submissions. Consequently, their time for these sorts of questions is limited. In addition, long questions that describe in detail the contents of an entire manuscript and end with “Would you be interested in our work?” is basically requesting a preliminary review, the answer to which will be tentative at best. A long message may be ignored or receive a brief response (e.g., “Please see our editorial policy.”).

Brief Summary of Review Process

1. A new submission is assigned a manuscript number and its receipt is acknowledged.

2. The editor reads the manuscript and decides whether to:

a. reject the manuscript outright or determine its content to be inappropriate for the journal (“withdrawn without prejudice”)

b. encourage resubmission, but only after having addressed the editor’s issues

c. solicit the review from an associate editor (AE)

In the last case, the editor will include any additional submitted material such as appendices, a cover letter, etc., as well as a brief note. Otherwise, the editor prepares a decision letter directly.

3. Upon receipt of a manuscript, the assigned AE reads the manuscript and decides whether to:

a. recommend rejection

b. request a revision

c. send it to referees for review

In the first two cases, the AE generally will prepare an adequately detailed report for the authors that explains the shortcomings, as well as confidential comments for the editor with a recommended decision (e.g., “reject outright, do not encourage a revision” or “not yet ready for referees, but a satisfactory revision at the next stage might be”)

4. If the manuscript is sent to referees (with or without authors’ names, depending on journal policy), the request for review usually states a deadline (e.g., 4–6 weeks).

5. Each referee reads the manuscript and prepares a report for the authors and a confidential report for the associate editor. The referee’s report should not contain the recommendation; only the editor makes the final decision, and a referee report that states “the paper is acceptable as is” will put the editor in an awkward position if the AE and two other referees firmly disagree.

6. Upon receipt of all referee reports, the AE prepares an “associate editor’s report” for the authors and a confidential report for the editor that summarizes the referees’ reports and offers a recommendation. (If the referees’ reports are extremely thorough, the AE may have no further comments for the authors and will not prepare a separate report.)

7. Upon receipt of all reports from the associate editor, the editor prepares a decision letter for the authors. The decision letter and all reports from the reviewers are sent to the corresponding author.

Most journals’ goal is to complete the entire review process (from receipt of a new manuscript to sending a decision letter to the authors) in fewer than six months. This goal is not always achieved (not all referees respect review deadlines), but editors do try to encourage prompt reviews.

Some Questions and Answers

Advice about selecting an appropriate journal for submission

Should one always aim for as prestigious a journal as possible?

Many academic departments do include in their promotion criteria the reputation of the journal in which a candidate’s articles have appeared. However, some of the most prestigious journals also have the most submissions, which can mean longer review times. If publication is more important than journal reputation, the author may want to submit to a journal with shorter review times. If the author can afford to wait for a longer review time to decision, one can aim for a more “prestigious” journal. (Keep in mind that different people will assess prestige in different ways.) Regardless of expected review times, the primary consideration should be appropriateness. For example, Technometrics may enjoy a prestigious reputation, but there is little point in submitting your manuscript to this journal if it does not demonstrate the methodology via a real example from the physical, chemical, biochemical, information, or engineering sciences.

How does one choose between JASA Theory & Methods, Applied Statistics, Communications in Statistics, and other general theory/methodology journals?

The differences among these journals involve aspects of style, methodological focus, article length, emphasis on validation, etc. Currently published articles in the journal can provide some indication of appropriateness, as well as the interests of the journal’s editorial board.

How important are examples/applications that illustrate the proposed methodology for a journal?

The answer depends on the journal. For JASA-T&M, probably not very important. For Technometrics, whose mission statement explicitly indicates a real application, it is essential. Generally, demonstration of value of the methodology via a real example enhances a manuscript.

How does one decide whether an application paper should be sent to a statistics journal versus a subject-matter journal?

The answer depends on the relative emphasis on the methodology versus the application. Most subject-matter journals do not want to see the details of the statistical methodology. Similarly, while some background on the measurement process is essential, a statistics journal reader will be less interested in the details of the temperature, humidity, equipment, etc. If the targeted audience is the professional statistical society, then a statistics journal makes sense. If the targeted reader is a professional in the subject area, then the statistics part is de-emphasized.

How does one choose among the applications journals such as JASA Applications & Case Studies, Biometrics, and Technometrics?

The latter two journals are distinguished by their areas of application. The first is more broad. If you want to reach a targeted audience, then one of the latter two would be appropriate. If you want to reach a general applied statistics audience, then JASA A&CS may be appropriate.

How does one distinguish between an application paper and a methodology paper that has a substantive application?

Generally, methodology papers involve (i) novelty of proposed methodology, (ii) validation (either via theory or simulation), (iii) “illustration.” An applications-oriented paper emphasizes the application as the motivation for the methodology and carries it throughout the manuscript. The methodology may be generalizable to other situations, but the focus is on the particular application and the importance of the inferences obtained from the methodology.

Submitting articles

Is it worth putting a lot of detail and “self-promotion” into the cover letter?

Sometimes additional information can be useful to the editor, but a long cover letter that merely repeats the abstract is not necessary. Most cover letters can be short with no effect on the review process.

How important is the first draft?

The first draft is critical. A poorly written manuscript creates a negative impression that is hard to overcome in subsequent revisions, or even in subsequent manuscripts submitted by the same author. Do NOT ask the review process to write the paper for you. Submit your best work. A manuscript that is illogical and full of typographical and grammatical errors will be difficult to read and likely result in a long and unfavorable review.

Can the authors recommend reviewers to include or exclude?

Most editors accept such recommendations with a grain of salt on the assumption that recommended reviewers may be collaborators or predisposed to a favorable review, regardless of the shortcomings. Exclusions are taken seriously and usually respected.

Must previous rejections by other journals be disclosed?

No, but if you suspect the new journal may rely on the same reviewers, you may wish to disclose both this fact and a summary of the reasons for the previous rejection (and how they have since been addressed).

Review process

How can there be a disconnect between the reasonably neutral referees’ reports and a negative decision?

Referees’ reports to the authors may be less critical (in an effort to avoid discouraging the author) than the reports to the AE or editor. In such a case, the editors try to make clear the reasons for the decision based on additional comments perhaps not made clear in the reports.

How does one distinguish between an outright rejection and a rejection in which revision is discouraged (door open for revision)?

If the letter says nothing about submitting a revision, most likely it was deemed to have fatal flaws or is otherwise not suitable for the journal. In other cases, the manuscript has potential, but revision would take enormous work and the result is not guaranteed. In that case, you must decide whether you want to take a chance and try to prepare a revision, or whether you should simply resubmit to another journal. From the letters and reports:

• How negative is the decision letter?
• How negative are the reports? Is it hopeless, or will it be possible to change the reviewers’ perceptions?
• What are the reasons for nonacceptance? How much work is required to address the concerns expressed? Do you essentially have to do all new research?
• Can I convincingly address the concerns expressed in the reviews in a timely manner?

Can an author contact the editor to clarify the points in the referees’ reports before revising?

Yes, although be sure that the queries are not directed to asking if the revision will be acceptable.

Can rejections be appealed? Does it succeed? What is the best approach?

Appeals are considered at the discretion of the editor. Most editors will consider them, but the decision is unlikely to remain unchanged if the basis of the decision was “extent of contribution” or an opinion on a somewhat subjective matter. On the other hand, if all reviewers, as well as the editor, truly misunderstood the nature of the contribution, the appeal may well be justified. Even in that case, the author(s) should recognize that the contribution was not clear and make a clear (but polite) case for reconsideration. An appeal that criticizes the reviewers is not likely to be favorably received. You need to convince them to reconsider their decision so you will need convincing arguments.

Responding to reviews

Is it appropriate to respond and criticize a review (that is ‘incorrect’)?

Yes, but keep in mind that most correspondence with the editor is likely to be seen by the AE and possibly by the referee. Therefore, avoid harsh criticism and write a response that is circumspect, but clear and dispassionate (e.g., “We believe the reviewer may have misunderstood our argument and we apologize if we were unclear &helip;”

Can I complain about factual errors?

Yes, and, in fact, you should do so in your response to the editor. However, keep in mind that factual errors may not be the sole reason for a rejection.

Is it OK to tell the editor that a referee was incompetent and ask for a new referee?

Yes, but you should be sure of your ground here. The editor selected the AE for his/her expertise, and the AE selected the referees for their expertise, so you are inherently criticizing all three levels when you claim one is “incompetent.”

Should authors respond to all review comments?

Absolutely. Point-by-point responses to all the comments are essential. If you try to skip over some, the reviewer will notice and comment on the omission.

How to distinguish important comments from minor ones?

In many cases, this will be clear from the editor’s letter, AE’s letter, or the referee reports. A point that reappears in several reports is likely important. Comments phrased as suggestions may be less important. When in doubt, ask the editor.

What if some of the reviewers’ comments conflict?

Our suggestion is to respond to the comment you think is most insightful and then mention that one of the other reviewers expressed a conflicting point of view. Explain why you chose to respond in the way you did. You also can mention that if there is strong opinion that you should have responded differently, you are willing to change. You can even informally sketch out (in your cover letter) how you might do this so the reviewer with the conflicting opinion sees you thought about the comment.

Do the editors always give guidelines?

In their letter to the authors, some editors try to give guidelines concerning the revision. When in doubt, ask the editor for advice.

Do the reviewers get copies of their original reports when revisions are sent to them?

Yes.

Should authors repeat the comments in their response?

Doing so makes life easier for the reviewers. Anything you can do to make life easier for the reviewers will speed up the review. You also can refer to some comments by the numbering system used by the reviewer, especially for minor comments.

Being a referee: Writing good reviews

What do editors look for in a report? What are the components of a good report?

A useful report has cogent, well thought-out reasons for the recommendation. Imagine yourself trying to write to the authors to explain/justify your recommendation. What sort of information would you want as an author to help you understand a decision about your manuscript? Vague comments (e.g., “The paper just isn’t important.” “This has been done already.”) are not helpful. Detailed comments are especially useful when they cite chapter and verse of previous publications.

Who is responsible for checking all the technical details?

Referees are selected because they have subject matter expertise. One hopes they will catch errors, but the accuracy of the final product lies with the author(s).

How detailed should the report be?

More detail is better, provided it is relevant. However, a long delay in completing a review can be frustrating. A shorter report that provides the main criticisms may be more useful than waiting six months for a longer detailed one.

If there is a major flaw in the paper, should I bother with the rest of the paper?

You can skim it to see if there are other problems. For example, if the authors fixed the flaw, would the manuscript then be suitable for publication? Or are there further grounds for rejection?

Do I need to write down everything that is wrong with the paper?

If you have time to do so, that would be helpful, especially if you recommend revision. Detailed reports are useful for detailing all the problems with the paper. Editors often identify future AEs by the quality (and timeliness) of referee reports. If you don’t have time, focus on the most important issues, providing cogent arguments for a negative recommendation.

Should I try to edit the paper for writing style?

Most stylistic issues will be addressed in the copy-editing stage, unless they involve equations and likely may not be caught in production.

What if I, as a reviewer, am working on a similar problem? How do I handle a potential conflict of interest?

The severity of the conflict is for you to judge and disclose. Unless you are working on exactly the same problem, you likely will not need to disqualify yourself.

Is it OK to raise new issues on a revision if I didn’t pick them up on the first round?

Yes, if they are significant. The revision may have clarified some issues, which caused you to uncover new concerns.

Double-blind refereeing

Preparing double-blind copies: What is the minimum expected?

Delete any information that would easily enable a reviewer to identify you as an author. In addition to your name and address, pay attention to what you put in acknowledgments.

How do editors think it is working?

It is probably marginally helpful. Our thought is that if a reviewer really wants to figure out who the authors of a manuscript are, they can make educated guesses from the content and probably use the Internet to search for tech reports that may be versions of the manuscript or for listings in online vita. We will note that occasionally we have heard people declare they knew who the author of a manuscript was and they were wrong, so blinding does occasionally help.

Does it really help young reviewers?

We don’t know if it helps much, other than providing psychological comfort. Many reviewers actually cut young authors more slack than they would senior authors. So there may be cases in which blinding hurts a young author slightly.

Is it fair that some researchers can make their research, and thus their submitted papers, known in any number of uncontrollable ways (e.g., by giving talks and posting preprints)?

The best we can do is encourage reviewers to respect blindness and avoid trying to identify the authors.

Manuscript review times

When is it reasonable to inquire about the status of a paper?

For many journals, six months is reasonable. Inquire about the status after six months. If nothing else, it gives the editor a good reason to make inquiries of the AE. If you have concerns about whether the manuscript was received, you should check after a couple of weeks.

When is a paper considered by the journal, itself, to have been out a long time?

After six months, many editors begin to think every effort should be made to complete the review process, unless there have been problems with reviewers (and in such cases, authors should be alerted that there may be a delay). We view nine months as very long. A year should be considered unacceptable, but we do have cases that take this long and longer.

How are editors trying to shorten review times?

— Emphasizing the importance of timeliness to new AEs is important.

— Making clear that more than six months is generally unacceptable and that AEs may need to prepare a report without the benefit of other reviews and make the best recommendation they can after six or seven months.

— Emailing and phoning (nagging) reviewers if they are late. Hold reviewers to the time they promised the review.

— Providing AEs with data about their performance can help.

— Replacing poorly performing AEs. Keeping track of poorly performing reviewers and avoiding using them.

— Screening manuscripts better so poorly written manuscripts are not sent out for review.

— Requesting three or four referees for manuscripts that look like they may be tough to review. This improves the chance of getting at least one review in a timely manner.

— Encouraging AEs to let the editor know when they are going to be unusually busy.

— Identifying a few reviewers who are willing to provide quick (one or two weeks) reviews and then asking them to help with manuscripts that have been in the review process for more than six months. The main drawback is that a quick review may be brief and less insightful than would typically be the case.

What does an editor do when a referee’s report is very late or an AE becomes a “black hole”?

If a referee’s report is very late, an editor may recommend making a decision based on the reports that are available, even if this is only the report from the AE. When an AE becomes a black hole, one can stop sending the AE manuscripts and have a talk with the AE. In some cases, the problem may be temporary (the AE or referee may temporarily have heavy duties) and will correct itself.

Many editors urge AEs to let them know if this is the case so they do not send them new manuscripts. If it is clear that the situation is going to continue indefinitely, it may be best for the AE to resign. It saves authors, the editor, and the AE from a lot of aggravation and discouragement. In an extreme case, an editor may review the manuscript herself, find a second associate editor to handle the manuscript (requesting a quick review), or use a special referee who has promised to provide a quick review.

Authors should keep in mind that some delays are unavoidable. Long, technical manuscripts that are not well written can take a long time to review. AEs and referees are doing volunteer work. If their real work suddenly becomes overwhelming, they will be delayed in completing a review. Often the expectation is that the extra work will go away soon and then they will catch up. If this is the case week after week (“Next week I think I will be caught up and will be able to complete my review.”), one can suddenly find oneself with a manuscript whose report is way overdue. Sometimes, they lose track of just how much time has passed.

Also remember that statistics manuscripts (like mathematics manuscripts or theoretical physics manuscripts) are much more difficult to read and assess than manuscripts in other disciplines. Many require more than a couple of hours to read, and if a person is busy, they may have to put it aside until they have an entire day free to devote to it. Unfortunately, it is rare to have an entire day free.