4 ways to get higher quality peer review comments

Does the quality of the peer review comments your journal receives vary more than it should?

Well-thought-out reviewer comments aside, many editorial teams find that they too often receive comments that are:

  • Meandering and difficult to interpret
  • Sparse or lacking the level of detail needed to be constructive
  • Hyper-critical in incidental areas while missing the bigger picture

Such unfocused or insufficient reviewer comments can create snags in peer review and cause frustration for all parties involved. Many editorial teams struggle with wanting to ensure that they’re providing authors with robust feedback but feeling like reviewer comment quality is outside of their control. While you can’t guarantee that your journal will receive top-notch reviewer comments all of the time, there are some steps your editorial team can take to improve reviewer comment quality.

Use a peer review feedback form

One of the best steps editorial teams can take to improve the quality of reviewer comments their journal receives is to require all reviewers to complete a standardized feedback form. When reviewers are left to fashion comments from a set of instructions, no matter how thorough and well-formatted they may be, the likelihood of some reviewers misinterpreting or skimming over expectations is high. However, using a required, standard feedback form journals can ensure that all reviewers address the same key manuscript areas while deterring reviewers from giving tangential feedback. Reviewer feedback forms work best when they are automated using peer review software, so reviewers literally can’t submit comments without answering all of the necessary questions.

Find the right balance of feedback form questions and make them specific

The results of your journal’s reviewer feedback form will only be as good as its design. Once your journal has a feedback form set up, track the quality of responses submitted and make adjustments to the questions as needed to improve feedback outcomes.

The best feedback forms have a balanced number of questions that are easy to interpret. Aim to provide enough questions to adequately guide reviewers but not so many that you begin to overwhelm them. Remember, reviewer fatigue is real! Make questions specific so reviewers understand the goal of each question—this includes writing questions clearly and formatting questions to reflect the level of feedback needed. For example, journals can use open-ended questions for substantive feedback and Likert Scale questions for high-level assessments and recommendations.

Some common feedback form flaws to avoid are:

  • Requiring nonessential questions: As noted, it’s important to have a balanced number of feedback form questions. Avoid filling up your form with questions you don’t need.
  • Combining questions: Make sure that each question in your feedback form is only about one thing. When questions touch on multiple areas it’s more likely for reviewers to submit unclear or partial responses.
  • Not asking for a publication recommendation: While the decision to accept a manuscript, reject it, or ask for revisions is ultimately up to your editors, asking reviewers for a direct recommendation can help ensure you’re interpreting their comments correctly.

It’s a good idea to also include an open-ended response field for comments to the editor at the end of your feedback form. This will ensure that reviewers are able to comment on all aspects of the manuscript that they think are necessary, even if your form doesn’t address them all. Over time, this field can help you to keep improving your feedback form questions by revealing any important assessment areas you’ve missed.

Limit revise and resubmit rounds

In addition to taking steps to improve initial reviewer comments, journals should set clear parameters for revise and resubmit rounds to ensure subsequent comments remain helpful and on track. As explained by former managing editor of Aztlán Journal of Chicano Studies, Wendy Laura Belcher, reviewer feedback can become less focused over multiple rounds of revisions and review. Belcher said that a common problem that editors should look out for is reviewers picking apart new areas of a submission in each round of review. This can turn into a frustrating cycle for authors and reviewers who just want to fulfill their respective expectations. Journals can avoid these situations by setting clear parameters for which aspects of the manuscript reviewers should be commenting on as well as a limit to the number of rounds of revision a manuscript can go through.

Help reviewers improve

Finally, your journal can help reviewers grow and improve the reviewer comments you receive by giving reviewers insight into the quality of their feedback. Journals can either provide reviewers with direct feedback on their comments, noting what was most useful and whether any comments were unnecessary or confusing, or they can send reviewers copies of author decision letters to provide insight into how their comments were interpreted and relayed to the author. Additionally, journals working with more early-career reviewers may want to provide some training materials such as a guide to writing constructive reviewer comments.

Source: Scholastica


3 Steps to Ensure Your Journal Receives Punctual Peer Reviews

Journal editors spend much of their time working to build out a network of possible peer reviewers for new submissions. It can be difficult to find scholars within a journal’s subject area, especially for niche publications, who are able and willing to provide regular peer reviews. As a result, most editors are constantly on the hunt for new reviewers. After searching for and securing reviewers for a manuscript, the last situation that an editor wants to be in is having one or more of those reviewers go silent.

Unresponsive reviewers can cause significant delays in a journal’s time to publication, creating stress for editors trying to get out their next issue on time and frustrating authors who are hoping to get a decision as soon as possible. What can editors do to avoid sending review assignments and hearing crickets? It can be difficult to predict whether a reviewer might become unresponsive. However, there are ways for editors to try to avoid such situations. Below we outline 3 steps you can take.

1. Check your journal’s past reviewer data before sending a review request

As you build out a reviewer database for your journal, one of the best things you can do to ensure timely reviews is to keep track of all your journal’s past reviewer activity. This can most easily be achieved via peer review software. Many systems, like Scholastica, will automatically track your journal’s reviewer activity with no added work on your part. However, if you’re not yet using a peer review system you can start tracking some reviewer stats in a spreadsheet.

Among the primary reviewer stats your journal should track are:

  • Average days for completing a review assignment
  • Pending review requests from your journal
  • Currently late reviews
  • Number of completed reviews

From the above stats you can start to glean insights into which reviewers you should reach out to and which you may want to wait on or even remove from your list. If you find that a reviewer already has a late or pending review, you’ll quickly know not to reach out to them until those assignments are addressed. Conversely, if you find that a reviewer has completed one or more reviews in a timely manner and has not declined a review invitation recently, that reviewer is likely a good candidate to contact.

Keeping a record of reviewer activity is especially important for journals with many editors, a large reviewer pool, or both. If you have multiple editors pulling from the same reviewer list without any log of reviewer activity, you’re likely to encounter more attrition in review requests because there will be a higher likelihood of editors reaching out to the same go-to reviewers too frequently and potentially turning them off from working with your journal. Even for journals with one managing editor selecting reviewers, it’s unlikely that, that editor will be able to recall each reviewer’s history with the journal off hand. Having a place for the managing editor to find reviewer data will help them avoid spending hours searching through email chains to figure out when a reviewer was last contacted and how they responded.

In order to ensure consistent data, editors should aim to incorporate peer reviewer tracking into their workflows as seamlessly as possible. The more manual steps you have to take to track reviewer activity, the more likely your editors will be to forget steps, leading to incomplete or inaccurate data. With the right peer review software, you can track reviewer activity without adding any extra steps for your team. For example, editors using Scholastica enable automatic reviewer activity tracking as soon as they invite reviewers to their journal via our system.

2. Have a set peer review timeline

Once editors have identified reviewers to reach out to, one of the most tasking parts of the peer review process can be waiting for them to acknowledge and respond to the review request. Review requests can sometimes get buried in scholars’ inboxes leading to days or even weeks of delay before they send a reply.

In order to avoid extensive wait times for reviewers to reply to invitations, one of the best things editorial teams can do is develop an established timeline for review requests. The timeline should account for one or more review reminders sent at designated times and then a final cutoff point for the reviewer to either respond to the invitation or be assumed unavailable.

Dianne Dixon, Managing Editor of International Journal of Radiation Biology piloted this approach to review requests and has seen great success. Her journal’s timeline includes sending an initial review request, sending a reminder four days later, sending a final reminder four days after that, and then finally removing the reviewer from the list after letting them know that she realizes they are likely unable to accept. In this closing email, Dixon asks reviewers to please let her know if they find they are able to review the manuscript. She said using this series of emails with a cutoff point for review responses has decreased delays in her journal’s peer review process.

3. Use automated reviewer reminders

After reviewers have accepted an assignment and agreed to a review deadline, it’s important for editors to periodically check in with them to ensure the review doesn’t fall off their radar. One of the best ways to do this is to send reminder emails at regular intervals.

Editors can try to block out time in their schedules to send review reminder emails, but with so many tasks to keep track of this can often become a bit of a chore. This is another area where peer review software can step in. Many software systems will enable editors to set up automatic weekly or bi-weekly reviewer assignment reminders, which editors can schedule to start sending as the assignment deadline approaches. It’s also a good idea to set up automatic late review reminder emails, that way journals can know late reviewers will be contacted as soon as they miss an assignment – possibly before the assigned editor even realizes.

Despite automation sometimes connoting a sense of detachment, it’s important for editors to consider how automated emails can actually help make journal communication more personal. Automated review reminders help journals stay in constant contact with reviewers and free up editors’ time for sending more thorough responses to specific reviewer questions among other benefits.

Overall emphasize the importance of reviewer communication

There is always the chance that a reviewer will accept an assignment with every intention of completing it on time but then become preoccupied with other obligations and fall behind on their deadline. Such cases are unpredictable for both the journal and reviewers.

Often when reviewers suddenly become unresponsive, the situation can be solved by encouraging reviewers to communicate if their circumstances have changed. Reviewers may be hesitant to go back on their promise, so it’s important for editors to remind them that in cases where they simply can’t complete an assignment on time the best course of action is to make that known.

The Committee on Publication Ethics’ (COPE) “Ethical Guidelines for Peer Reviewers“ stresses the importance of reviewers acknowledging if they are no longer able to complete an assignment. It states that reviewers should “always inform the journal promptly if your circumstances change and you cannot fulfill your original agreement or if you require an extension.” Journals can point reviewers to these guidelines or simply remind them in review requests that the journal encourages reviewer updates, even if it means reviewers having to decline an invitation they previously accepted.

Source: Scholastica