Step by step guide to reviewers

            The abstract of the paper should be sent to you when you are invited to peer review. This will help you to decide whether you wish to take part in the review. It will be easier to avoid delays if you respond to invitations promptly. As well as declaring any potential conflicts of interest, it is important to detail them at this point.

Overview of the review report format

            The review report’s format differs between journals. Others take a more formal approach, while some adhere to an informal structure.

Informal Structure

            Many journals only want your “analysis of merits” as the review criteria. In this situation, you may want to become familiar with examples of prior reviews written for the journal, which the editor should be able to supply, or you may want to rely on your own developing style as you acquire experience.

Formal Structure

            A more formal approach is necessary for other journals. On occasion, you will be sent a questionnaire and asked to respond to certain questions in your evaluation. Or they can want you to use a scorecard to assess the manuscript according to several criteria. These are frequently hidden until after you check in and submit your review. Therefore, it is wise to check for any journal-specific standards and guidelines before accepting the work. If there are official guidelines, let these govern how your review is organised.

In Both Cases

            You should anticipate compiling comments to writers and perhaps private ones to editors alone, whether the reporting style clearly calls for it or not.

First Read Considerations

            When skimming, keep a pen and paper close at reach.Try to keep the following considerations in mind as they will aid in forming your general opinion:

• What is the primary issue that the study aims to address? Is it fascinating and pertinent?
• How unique is the subject? Compared to other published materials, what does it bring to the field?
• Is the paper written well? Is the writing readable and clear?
• Are the findings in line with the facts and justifications offered? Do they answer the primary query?
• Does the author have a strong case if they strongly disagree with the prevailing academic consensus? If not, what would be necessary to support their position?
• What value do tables and figures add if they are included in the paper? Do they contribute to understanding or are they unnecessary?

Spotting Potential Major flaws

While you should read the whole paper, making the right choice of what to read first can save time by flagging major problems early on.

Editors say, “Specific recommendations for remedying flaws are VERY welcome.”

Examples of possibly major flaws include:

• Drawing a conclusion that is contradicted by the author’s own statistical or qualitative evidence

• The use of a discredited method
• Ignoring a process that is known to have a strong influence on the area under study

If experimental design features prominently in the paper, first check that the methodology is sound – if not, this is likely to be a major flaw.

You might examine:

• The sampling in analytical papers
• The sufficient use of control experiments
• The precision of process data
• The regularity of sampling in time-dependent studies
• The validity of questions, the use of a detailed methodology and the data analysis being done systematically (in qualitative research)
• That qualitative research extends beyond the author’s opinions, with sufficient descriptive elements and appropriate quotes from interviews or focus groups

Major Flaws in Information

If methodology is less of an issue, it’s often a good idea to look at the data tables, figures or images first. Especially in science research, it’s all about the information gathered. If there are critical flaws in this, it’s very likely the manuscript will need to be rejected. Such issues include:

• Insufficient data
• Unclear data tables
• Contradictory data that either are not self-consistent or disagree with the conclusions
• Confirmatory data that adds little, if anything, to current understanding – unless strong arguments for such repetition are made

If you find a major problem, note your reasoning and clear supporting evidence (including citations).

Concluding the First Reading

After the initial read and using your notes, including those of any major flaws you found, draft the first two paragraphs of your review – the first summarizing the research question addressed and the second the contribution of the work. If the journal has a prescribed reporting format, this draft will still help you compose your thoughts.

The First Paragraph

This should state the main question addressed by the research and summarize the goals, approaches, and conclusions of the paper. It should:

• Help the editor properly contextualize the research and add weight to your judgement
• Show the author what key messages are conveyed to the reader, so they can be sure they are achieving what they set out to do
• Focus on successful aspects of the paper so the author gets a sense of what they’ve done well

The Second Paragraph

This should provide a conceptual overview of the contribution of the research. So consider:

• Is the paper’s premise interesting and important?
• Are the methods used appropriate?
• Do the data support the conclusions?

After drafting these two paragraphs, you should be in a position to decide whether this manuscript is seriously flawed and should be rejected (see the next section). Or whether it is publishable in principle and merits a detailed, careful read through.

Rejection After the First Reading

Even if you are coming to the opinion that an article has serious flaws, make sure you read the whole paper. This is very important because you may find some really positive aspects that can be communicated to the author. This could help them with future submissions.

A full read-through will also make sure that any initial concerns are indeed correct and fair. After all, you need the context of the whole paper before deciding to reject. If you still intend to recommend rejection, see the section “When recommending rejection.”

Before Starting the Second Read-Through

Once the paper has passed your first read and you’ve decided the article is publishable in principle, one purpose of the second, detailed read-through is to help prepare the manuscript for publication. Of course, you may still decide to reject it following a second reading.

The benchmark for acceptance is whether the manuscript makes a useful contribution to the knowledge base or understanding of the subject matter. It need not be fully complete research – it may be an interim paper. After all research is an incomplete, on-going project by its nature. The detailed read-through should take no more than an hour for the moderately experienced reviewer.

“Offer clear suggestions for how the authors can address the concerns raised. In other words, if you’re going to raise a problem, provide a solution.” (Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology)

Preparation

To save time and simplify the review:

• Don’t rely solely upon inserting comments on the manuscript document – make separate notes
• Try to group similar concerns or praise together
• If using a review program to note directly onto the manuscript, still try grouping the concerns and praise in separate notes – it helps later
• Note line numbers of text upon which your notes are based – this helps you find items again and also aids those reading your review
• Keep images, graphs and data tables in clear view – either print them off or have them in view on a second computer monitor or window

Now that you have completed your preparations, you’re ready to spend an hour or so reading carefully through the manuscript.

Doing the Second Read-Through

As you’re reading through the manuscript for a second time, you’ll need to keep in mind the argument’s construction, the clarity of the language and content.

With regard to the argument’s construction, you should identify:

• Any places where the meaning is unclear or ambiguous
• Any factual errors
• Any invalid arguments

You may also wish to consider:

• Does the title properly reflect the subject of the paper?
• Does the abstract provide an accessible summary of the paper?
• Do the keywords accurately reflect the content?
• Is the paper an appropriate length?
• Are the key messages short, accurate and clear?
• Check the Language.
• Not every submission is well written. Part of your role is to make sure that the text’s meaning is clear.

Editors say, “If a manuscript has many English language and editing issues, please do not try and fix it. If it is too bad, note that in your review and it should be up to the authors to have the manuscript edited.”

If the article is difficult to understand, you should have rejected it already. However, if the language is poor but you understand the core message, see if you can suggest improvements to fix the problem:

• Are there certain aspects that could be communicated better, such as parts of the discussion?
• Should the authors consider resubmitting to the same journal after language improvements?
• Would you consider looking at the paper again once these issues are dealt with?

On Grammar and Punctuation

Your primary role is judging the research content. Don’t spend time polishing grammar or spelling. Editors will make sure that the text is at a high standard before publication. However, if you spot grammatical errors that affect clarity of meaning, then it’s important to highlight these. Expect to suggest such amendments – it’s rare for a manuscript to pass review with no corrections.

A 2010 study of nursing journals found that 79% of recommendations by reviewers were influenced by grammar and writing style (Shattel, et al., 2010).

The Second Read-Through: Section by Section Guidance

1. The Introduction

A well-written introduction:

• Sets out the argument
• Summarizes recent research related to the topic
• Highlights gaps in current understanding or conflicts in current knowledge
• Establishes the originality of the research aims by demonstrating the need for investigations in the topic area
• Gives a clear idea of the target readership, why the research was carried out and the novelty and topicality of the manuscript

Working with editors

            An article’s acceptance or rejection depends on the recommendations from reviewers. A good peer review should meet the editor’s criteria, and this section will explain what they look for.

Reviewer’s Expertise

            If there is a specific area of the paper that the editor would want you to review, they will specify it when they invite you to review. This is due to the fact that editors may ask for reviewers who have knowledge in particular fields, such as the technique or the statistics used in the study, even though they are aware that you don’t work in the manuscript’s field. If that’s the case, it’s best to make it clear at the outset of the review that you’ll only be making comments on that particular component of the work.

            Ask the editor who invited you to assess a paper if you’re unsure of the reason. Editors would prefer that you get in touch with them if you have any questions rather than failing to answer or finish the review.

Reviewer’s Recommendation

            The recommendation can be either accept , reject , or revise mentioned with respective valid reasons.

The rating is given based on how relevant the work is related to the topic , the uniqueness and originality of the work , consistency of the conclusion , explanation by the author and the language understandability and accuracy.

            The editors are responsible for selecting the paper. This will be determined by their own reading as well as your advice and opinions. It bears emphasising that the editor’s choice won’t always coincide with your own, therefore you shouldn’t bring it up in comments to the author.

Comments to Editors and Authors

            The editor uses more than just the reviews’ comments when making a choice. In their decision letter, they frequently make reference to them.

In light of this, it will be beneficial to editors if you:

• Count the comments you make.
• Are specific about the details that, if the paper is granted the chance for revision, must be included.
• Indicate how authors can respond to any issues brought up.

Editors and authors both benefit from specific suggestions for fixing errors.

Keep in mind that it’s crucial that your comments and recommendations line up. If you’re recommending that the paper be rejected, your comments should be brief, point out the issues, and avoid being overly supportive or appearing to go against your suggestion.

If the editor appears to disagree with your suggestion or your remarks, it may place them in an unpleasant situation.

Remember that you should only use comments to editors for notes that you don’t want the writers to see. Comments to the authors, not comments to the editors, should contain any information that is crucial for the authors to know.

You will typically receive a copy of the letter that contains the editor’s judgement and any feedback from the other reviewers after the author has been informed.

For reviewing a clinical manuscript

Observe the fundamental guidelines for peer review.

There are five principles of good peer review: Content Integrity, Content Ethics, Fairness, Usefulness, and Timeliness. All sorts of manuscripts, even those with a professional or practitioner target audience, must adhere to the fundamental principles of peer review.

Considering the intended audience

            A clinically focused paper should provide the most recent evidence to support wise practise decisions. As you read, make sure the author(s) are focusing on a clinical issue as opposed to a research question. You should also consider the tone to determine if it is direct and addressed to the clinician.

You should also think about whether the publication includes diagrams or tables that are helpful in therapeutic settings, as well as reference to current, pertinent studies. Finally, evaluate whether the discussion applies the data to create a fresh perspective on clinical issues and/or treatments.

Look for a structure that is ordered          

Due to their busy schedules, clinical practitioners need organised access to the research they study. Consider how appealing the manuscript is as you review it.

• Finding the important points simple for you?
• Is it obvious which areas a busy practitioner can skip over because they are less critical?
• Is the manuscript’s structure in line with the journal’s?

Think about how it might be used in professional settings

It should be emphasised in the manuscript you are reading how it might affect practise. A strong paper will provide information on the potential implementation of the research or intervention. Information like clinically applicable screening tools and consumer- and patient-friendly education resources will also be included in a stronger manuscript.

Establish the case study’s components

If a case study is used, only the most important details should be covered. Although a physician may not always have the time to read a case study in its entirety, case studies can be a good tool to teach new concepts. Check to check if the essential information is presented and if any additional data has been included in the form of a figure or table (i.e., laboratory values, chronology of key events, photographs, etc.).

The patient and his or her family shouldn’t be recognised if the case study involves a real patient. Your review should make a clarification request if the manuscript is unclear about whether the patient is real or not.

Beware of potential conflicts of interest

Any conflicts of interest (COI) must be declared by the authors in the submissions. If an author receives payment from a business to write an article, conduct research, or put together a review, a conflict of interest (COI) may develop. It must also be disclosed if a third party (often referred to as “ghostwriting”) draughts an article that was submitted by another person.

Even if there isn’t a real conflict of interest, all disclosure information should be provided to the reviewers. Inform the editor if you believe something is missing and make note of it in your review. Additionally, alert the editor right away if you think you could have a conflict of interest with the manuscript.

Verify the safety of participants, both human and animal

All research must adhere to a set of moral requirements that safeguard both human subjects and experimental animals. Even if the institutional decision was to exempt the research from informed consent procedures, the authors must nonetheless add a statement to that effect in their publication. Notify the editor if the statement is absent because certain publications won’t accept research without it.

Reviewing Registered Reports

Background

A handful of publications offer Registered Reports, a type of empirical article where the techniques and suggested analysis are pre-registered and vetted before the research is carried out. Before data collecting starts, high-quality protocols are then tentatively accepted for publishing. With the ability to conduct exploratory (unregistered) studies and report coincidental findings, this approach is intended to reduce publication bias and research bias in hypothesis-driven research.

There are two stages to the evaluation process for registered reports. Before any data is gathered, reviewers evaluate study ideas at Stage 1 (Study Design). Reviewers take into account the entire study, including the data and interpretation, at Stage 2 (Completed Study).

Recommendations for Reviewers

Only the Introduction, Methods (including any suggested analysis), and Pilot Data will be included in Stage 1 papers (where applicable). Reviewers will be invited to rate articles at Stage 1 based on the following criteria:

• the significance of the research question(s), in publications where this is typically a requirement for admission.
• The offered hypotheses’ reasonability, logicalness, and believability.
• The technique and analysis pipeline’s validity and viability (including statistical power analysis where appropriate).
• to what extent the proposed experimental protocols and analysis pipeline can be precisely replicated in terms of clarity and level of technical information
• Whether the authors have included positive controls and quality checks as well as enough outcome-neutral tests to guarantee that the results are able to evaluate the stated hypotheses.

After Stage 1: Study Design peer review, manuscripts will either be approved, given a chance to be revised, or outright refused. Upon successful completion of the study using the pre-registered methods and analytic procedures, as well as a defensible and evidence-based interpretation of the findings, manuscripts that pass peer review will be given an in principle acceptance (IPA), which indicates that the article will be published subject to these conditions. Be aware that some journals may decide to publish both the final Stage 2 completed study and the Stage 1 Study Design after receiving an IPA, while other journals will only publish the Stage 2 final article. For more information, please refer to the individual Registered Reports rules for your publication.

            The authors will finish the manuscript, including the Results and Discussion sections, after the study is over. These Stage 2: Completed Study manuscripts will be written in a style that is more like to an ordinary article. The reviewers will then receive the article again and be tasked with rating:

• Whether the data can test the hypotheses put forth by the authors by meeting the deemed outcome-neutral conditions (such as quality checks, positive controls).
• If the Introduction, justification, and stated hypotheses are the same as the Stage 1 proposal that was approved (required).
• Whether the authors followed the registered experimental protocols precisely.
• if the authors’ unregistered post hoc analyses are justified, informative, and methodologically sound.
• Whether the data support the writers’ conclusions.

However, authors are not required to do so unless further post hoc tests are required to meet one or more of the Stage 2 review requirements. Stage 2 reviewers may propose that authors disclose additional post hoc tests on their data. Please be aware that editorial decisions will be made based on data-supported conclusions (even if they are negative findings) and adherence to the approved protocols and experimental design in Stage 1 rather than novelty and perceived importance of results.