5 ways to increase peer review transparency to foster greater trust in the process

Another Peer Review Week is upon us, and this year’s theme, “Trust in Peer Review,” comes at a particularly apt time. The COVID-19 pandemic and other recent global crises have magnified the role of peer review as a mechanism to ensure research quality while, in many ways, putting trust in peer review to the test. Pressures on journal editorial teams and volunteer referees to peer review larger volumes of papers at record speeds have raised concerns in and outside of academia about the potential for higher levels of human error in an inherently imperfect process.

The pandemic aside, since peer review became part of scholarly publishing (a mere ~60 years ago in the 350+ year history of journals!), concerns about flawed peer review processes, overburdened reviewers, and reviewer biases have put peer review on shaky ground. Studies in the early 2010s revealing the ongoing replication crisis in many disciplines marked a particularly apprehensive time in peer reviews’ history. However, despite some discontent with the current state of peer review, surveys conducted in recent years, including Publons’ 2018 “Global State of Peer Review“ report, have revealed that most researchers want to work to improve peer review rather than replace or bypass it. Across academia, stakeholders in scholarly journal publishing have responded by placing greater emphasis on vetting and improving peer review practices, as exhibited by last year’s Peer Review Week, themed “Quality in Peer Review.” As stakeholders work to fortify peer review, the question now is — what can be done to build up and, in many cases, rebuild trust in the process?

With the curtain pulled back on peer reviews’ bumps and bruises for all to see, arguably, one of the best courses of action to address its limitations is to readily acknowledge them and work to make peer review practices more transparent. In this post, we round up five ways to increase peer review transparency that could help foster greater trust in the process.

Putting research questions and methods before findings

One of the primary concerns about the efficacy of peer review expressed in recent years is that the predominant peer review model — where scholars design studies, conduct them, write reports, and then receive feedback — may be contributing to the reproducibility crisis by hindering the publication of null and negative results. During a keynote address for the 2019 International Society of Managing and Technical Editors Conference (ISMTE), Brian Nosek, Executive Director for the Center for Open Science, discussed how in the “publish or perish” research culture dominated by the Journal Impact Factor and other citation-based incentive systems, tidily packaged positive research outcomes are often favored by journals over negative or inconclusive ones. As a result, scholars may be compelled to, intentionally or not, introduce research spin into their work.

To address concerns about the potential for research spin and biases against null and negative results, the Registered Reports publishing format has been gaining ground in recent years. In the Registered Reports format, peer review is split into two parts:

  1. An initial peer review of the study concept and design, used to make an acceptance or rejection determination
  2. Peer review of the finished paper — this review stage is solely to assess the quality of the research contents, not the nature of the findings

Notably, the journal Royal Society Open Science recently initiated an expedited pre-registration process for coronavirus-related submissions to help scholars avoid following false leads. Journals interested in implementing the Registered Reports publishing format can find many helpful resources on The Center for Open Science website.

Employing more open peer review practices

Once studies have been completed and peer review is underway, another trust concern many have expressed is a lack of transparency around the robustness of journal peer review, the rationale behind publication decisions, and the identities of those doing the reviewing. Journals can address high-level uncertainties about the nature of their peer review processes by providing detailed peer review policies on their publication websites. As for making the recommendations and/or identities of peer reviewers more transparent, some journals are now experimenting with more open peer review practices. Definitions and levels of open peer review can vary substantially, but the term generally refers to one or a combination of the following practices:

  • Publishing review reports alongside journal articles to make reviewer recommendations open
  • Making author and reviewer identities open to both parties and readers
  • Soliciting public peer review comments in addition to or in lieu of invited reviews

Proponents of open peer review practices argue that they promote accountability and allow for reviewer recognition. Of course there are also arguments on the opposite side of the spectrum in favor of blinded peer review to prevent skewed referee reports. For example, some argue when reviewers know the identities of authors, there is the potential for them to make recommendations based on implicit biases. As exhibited by the various definitions of open peer review, there is no one-size-fits-all approach. So this is an area where journals can and likely should weigh the pros and cons of different models as they apply to their discipline and particular publication to determine the best course of action for them.

Developing shared peer-review standards and taxonomies

Another fundamental aspect of increasing peer review transparency and ultimately trust in the process is developing more universal peer-review standards and nomenclature (to help with hurdles in communicating peer review policies like the many definitions of “open peer review”). STM recently launched a “Working Group on Peer Review Taxonomy” to address the definitional component of peer review standards that journal publishers should keep on their radar, the first draft of which is available here. Another recent example of developing shared publishing definitions that can help increase research transparency and, consequently, trust in peer review is the CRediT – Contributor Roles Taxonomy initiative, which introduces naming conventions for different types of author roles.

In addition to working to develop shared peer review taxonomies, scholarly publishing stakeholders have been exploring new methods of normalizing and expressing journal publishing standards, including via the TOP Factor. Launched by the Center for Open Science in February 2020, TOP Factor is a new journal assessment system based on the Transparency and Openness Promotion (TOP) Guidelines, which consists of eight publishing standards to improve research transparency and reproducibility. TOP Factor has the potential to foster a uniform shared framework for implementing and demonstrating adherence to journal publishing best practices, which would increase peer review transparency and make it easier to compare publisher processes and norms. The TOP Guidelines also promote FAIR data principles and open data, which could help to foster a more self-correcting research environment.

Facilitating the sharing of review reports across journals

Finally, a mounting trust concern to be addressed is the increasing peer-review burden faced by many scholars. As the rate of articles published across disciplines continues to increase, many worry that reviewers will struggle to keep up, potentially leading to more research errors slipping through the cracks. Adding another plot twist — in reality, a lot of the review work placed on scholars is redundant because they are being asked to vet papers that have already undergone peer review elsewhere.

One possible solution to help alleviate the peer review pressures scholars face is the Manuscript Exchange Common Approach (MECA), an initiative launched by the National Information Standards Organization (NISO) in May 2018. MECA aims to develop a framework for transferring manuscripts and review reports between different peer review systems. According to the MECA website sharing review reports between journals could help save an estimated “15 million hours of researcher time [that] is wasted each year repeating reviews.” In the same vein, the C19 Rapid Review Initiative, a large-scale collaboration among 20 publishers to improve the efficiency of coronavirus-related research processing, is piloting a version of review report sharing by requiring reviewers to consent to have their identities and review reports shared among participating publishers’ and journals.

Putting it all together

The fact that peer review is imperfect, like any human endeavor, is no secret or surprise. Working to make all aspects of the peer review process more transparent could help safeguard against inevitable human errors. At the same time, it could expand the potential for more widespread assessment of peer review approaches and outcomes, something that scholars argue is needed to improve and increase trust in peer review.

Source: https://blog.scholasticahq.com/

Linkedin   
Whatsapp   

Guidelines for Book Review

The Journal for ReAttach Therapy and Developmental Diversities (JRTDD) will publish book reviews on major books across fields and sub- fields covered by the journal.

  • If you are interested in reviewing a book for the Journal, please send an expression of interest to the Editor-in-chief on journaljrtdd@gmail.com.
  • Authors and publishers should also contact the Editor if they would like their books to be reviewed in the journal.

The JRTDD seeks reviews that assess a book’s strengths and weaknesses and locate it within the current field of scholarship. A review should not simply be a listing of contents, though its overall organization and emphasis are up to the individual reviewer. Reviewers should avoid lists of minor imperfections (e.g. misplaced commas) but should not hesitate to draw attention to serious editorial problems and errors of fact or interpretation. It is also helpful if reviewers indicate for which audiences and libraries the book seems appropriate. The Book Review Editors reserve the right to edit for content and length. In summary, book reviews should be timely and objective and should consider the following:

  • The intended audience for the book and who would find it useful
  • The main ideas and major objectives of the book and how effectively these are accomplished
  • The soundness of methods and information sources used
  • The context or impetus for the book – political controversy, review research or policy, etc
  • Constructive comments about the strength and weaknesses of the book

Book reviews for the Journal for ReAttach Therapy and Developmental Diversities should be should be double spaced, using a standard sized font Times New Roman. All material, including long quotes, should be double-spaced. Please use minimal style formatting in your document. Reviews of single books should be between 750-1000 words.

The review should begin with a hanging indent paragraph for each book in the review that includes its title, subtitle, author/s or (editor/s), and publishing information in the following format:

All subsequent paragraphs should be indented. There are no footnotes in the Journal for ReAttach Therapy and Developmental Diversities book reviews. References to texts not under review should be parenthetical only. Also, please type your name and institution, exactly as you wish it to be published, at the end of the review. For example: Frederik Johnson, University of Amsterdam, The Netherlands.

The following information should be given about the book being reviewed at the start of each review:

  • Author/Editor Name, Book Title, Publisher, Year of Publication, ISBN: 000-0-00-000000-0, Number of Pages, Price
Linkedin   
Whatsapp   

4 ways to get higher quality peer review comments

Does the quality of the peer review comments your journal receives vary more than it should?

Well-thought-out reviewer comments aside, many editorial teams find that they too often receive comments that are:

  • Meandering and difficult to interpret
  • Sparse or lacking the level of detail needed to be constructive
  • Hyper-critical in incidental areas while missing the bigger picture

Such unfocused or insufficient reviewer comments can create snags in peer review and cause frustration for all parties involved. Many editorial teams struggle with wanting to ensure that they’re providing authors with robust feedback but feeling like reviewer comment quality is outside of their control. While you can’t guarantee that your journal will receive top-notch reviewer comments all of the time, there are some steps your editorial team can take to improve reviewer comment quality.

Use a peer review feedback form

One of the best steps editorial teams can take to improve the quality of reviewer comments their journal receives is to require all reviewers to complete a standardized feedback form. When reviewers are left to fashion comments from a set of instructions, no matter how thorough and well-formatted they may be, the likelihood of some reviewers misinterpreting or skimming over expectations is high. However, using a required, standard feedback form journals can ensure that all reviewers address the same key manuscript areas while deterring reviewers from giving tangential feedback. Reviewer feedback forms work best when they are automated using peer review software, so reviewers literally can’t submit comments without answering all of the necessary questions.

Find the right balance of feedback form questions and make them specific

The results of your journal’s reviewer feedback form will only be as good as its design. Once your journal has a feedback form set up, track the quality of responses submitted and make adjustments to the questions as needed to improve feedback outcomes.

The best feedback forms have a balanced number of questions that are easy to interpret. Aim to provide enough questions to adequately guide reviewers but not so many that you begin to overwhelm them. Remember, reviewer fatigue is real! Make questions specific so reviewers understand the goal of each question—this includes writing questions clearly and formatting questions to reflect the level of feedback needed. For example, journals can use open-ended questions for substantive feedback and Likert Scale questions for high-level assessments and recommendations.

Some common feedback form flaws to avoid are:

  • Requiring nonessential questions: As noted, it’s important to have a balanced number of feedback form questions. Avoid filling up your form with questions you don’t need.
  • Combining questions: Make sure that each question in your feedback form is only about one thing. When questions touch on multiple areas it’s more likely for reviewers to submit unclear or partial responses.
  • Not asking for a publication recommendation: While the decision to accept a manuscript, reject it, or ask for revisions is ultimately up to your editors, asking reviewers for a direct recommendation can help ensure you’re interpreting their comments correctly.

It’s a good idea to also include an open-ended response field for comments to the editor at the end of your feedback form. This will ensure that reviewers are able to comment on all aspects of the manuscript that they think are necessary, even if your form doesn’t address them all. Over time, this field can help you to keep improving your feedback form questions by revealing any important assessment areas you’ve missed.

Limit revise and resubmit rounds

In addition to taking steps to improve initial reviewer comments, journals should set clear parameters for revise and resubmit rounds to ensure subsequent comments remain helpful and on track. As explained by former managing editor of Aztlán Journal of Chicano Studies, Wendy Laura Belcher, reviewer feedback can become less focused over multiple rounds of revisions and review. Belcher said that a common problem that editors should look out for is reviewers picking apart new areas of a submission in each round of review. This can turn into a frustrating cycle for authors and reviewers who just want to fulfill their respective expectations. Journals can avoid these situations by setting clear parameters for which aspects of the manuscript reviewers should be commenting on as well as a limit to the number of rounds of revision a manuscript can go through.

Help reviewers improve

Finally, your journal can help reviewers grow and improve the reviewer comments you receive by giving reviewers insight into the quality of their feedback. Journals can either provide reviewers with direct feedback on their comments, noting what was most useful and whether any comments were unnecessary or confusing, or they can send reviewers copies of author decision letters to provide insight into how their comments were interpreted and relayed to the author. Additionally, journals working with more early-career reviewers may want to provide some training materials such as a guide to writing constructive reviewer comments.

Source: Scholastica

Linkedin   
Whatsapp   

JRTDD list of reviewers has been updated

Dear readers,

We want to inform you that the JRTDD list of reviewers has been updated on February 5th. There are 55 reviewers from 18 different countries. In this occasion I would like to thank all of them. We hope for further collaboration and also we expect new reviewers.

JRTDD Editor-in-chief

Linkedin   
Whatsapp   

Peer Review Week 2019 in Macedonia

Dear readers,

Peer Review Week 2019 is already started from today 16 to 20 September 2019, with activities taking place online, on social media and at events across the globe.

This year’s theme of “Quality in peer review” will celebrate all initiatives aimed at improving peer review quality, and the peer review process. We want to hear everyone’s perspectives on quality in peer review, from early career researchers to senior academics, authors, reviewers, editors, medical charities, policymakers, funders, publishers and citizens.

In Macedonia, Macedonian Association of Medical Editors will organize event on September 19th. The event is entiled: PEERREVIEW – BASIC PRINCIPLE IN SCIENTIFIC COMMUNICATION will held in Macedonian Academy of Science and Arts from 13:00 to 15:00.

AGENDA

13:00 – 13:20 – Quality of peer review -basic process in publishing 
Prof. Dr. Gordana Ristovska
13:20 – 13:40 Prevention and elimination of plagiarism
Prof. Dr. Doncho Donev
13:40 – 14:00 Ethics of peerreview
Prof. Dr. Vladimir Trajkovski
14:00 – 15:00 Discussion and conclusions

JRTDD Editor-in-chief

Linkedin   
Whatsapp