Reducing bias in peer review

Peer review is the cornerstone of the scholarly publishing system, contributing to the scientific enterprise by supporting good-quality research and weeding out flawed manuscripts. As rigorous as the process is, it is prone to reviewer biases—explicit and implicit—which could compromise objectivity. While biases cannot be eliminated completely, journals and publishers have been taking some concrete steps to reduce bias in peer review.

1. Including underrepresented groups in the peer reviewer pool

A survey conducted by COPE indicates diversity, equity, and inclusion (DEI) being key in peer review, suggesting that initiatives to increase DEI can play a role in reducing bias. A Publons report highlighted underrepresentation of women in peer review and dominance of reviewers from established regions around the world in the peer review process, indicating potential gender and geographical bias. A more diverse pool of reviewers offers an opportunity to obtain different perspectives, improve the quality of peer review, and reduce bias in the process. Maintaining a database of reviewers from diverse groups and implementing policies to encourage participation of people from emerging regions and underrepresented groups can help journals in their diversity initiatives. Several organizations have proposals in place to encourage this; for instance, editors-in-chief at the Journal of Biological Chemistry plan to provide their associate editor with summary reports of personalized, geographical and gender distribution of the peer reviewers they invite. They have also taken steps to address the lack of diversity in the peer review community by increasing the number of underrepresented groups in junior associate editor positions.

2. Experimenting with the peer review model

One of the ways the scholarly community is trying to address bias in peer review and improve transparency is by experimenting with different peer review models. The premise is that transparency can increase accountability and eventually reduce bias in peer review. Therefore, open peer review has become a key part of the open science movement. In a survey conducted by Nature with its referees, 63% of respondents suggested publishers should experiment with alternative forms of peer review, and over half of them indicated that the process could be more transparent. In 2020, Nature started an initiative where authors were offered the option to have their accepted manuscripts published along with anonymized peer review reports and their responses to the review comments. An open peer review model was also adopted by several journals, including BMJ Open and medical journals in the BMC series. Cross-review is another option for increasing review accountability, giving reviewers the chance to see and comment on other reviews before the editor makes a final decision. This process can place specific checks to reduce unfair, overly critical, or inappropriate reviewer comments.

3. Maintaining anonymity in peer review

While increasing transparency is one way to reduce bias, another approach is double-blind peer review. A study suggests that early career researchers tend to prefer double-blind peer review as it can reduce bias against authors with less experience, female authors, or authors from minority groups. A survey also suggests that is a preferred model, with respondents viewing it as the most effective when compared with single-blind, post-publication, and open peer review. Double-blind peer review can also help reduce potential reviewer bias associated with other aspects of author identity, such as their status/reputation in the field or their affiliations. Furthermore, masking reviewers’ identities can make it easier for reviewers to comment on an article objectively without fear of offending any of the authors or institutions involved in the research. Every peer review model has its merits, and it’s important to consider them before adopting a model. For instance, while double-blind peer review can help in reducing certain biases, an open model can promote greater transparency. Triple-blind and quadruple-blind peer review models have also been proposed as a way to anonymize authors’ details during review.

4. Building an internal mechanism to evaluate peer review reports

A study suggests that oversight of reviewer reports by associate editors can help journal editors have control over the peer review process. Adding a step to form a structured peer process by assigning an associate editor or expanding their role to vet, tone down, or blacklist peer reviewers who make biased statements can be a helpful strategy. Some Wiley journals use this approach, where associate editors review comments provided by reviewers.

5. Addressing biases through training

While the above steps could help reduce bias in peer review, a more direct way to address implicit bias is through peer review training. Incorporating modules to promote a better understanding of implicit biases and how to identify them can lead to increased self-awareness among reviewers. The Canada Research Chairs Program (CRCP) has an interactive training unit focused on addressing bias in peer review. The American Heart Association (AHA) encourages their peer reviewers to go through their unconscious bias training program. Efforts by the community to support such training programs can be a right step towards reducing bias in peer review.

6. Exploring automation tools to support peer review

Use of AI tools in the peer review process has been proposed as a means to potentially identify biases in human decision-making by using data analysis and AI to replicate parameters of the cognitive functions implemented in peer review. A study published in Humanities and Social Sciences Communications evaluated an AI tool that was trained on 3,300 peer-reviewed conference papers. The tool was able to predict the outcome of human peer review based on superficial aspects such as formatting, word distribution, and readability, which potentially highlights biases in the decision-making process. However, the authors of this study share concerns that the existing sociocultural biases within peer review may be incorporated into the algorithms trained on data from potentially biased reviews. This can inadvertently result in technology-promoted prejudice against specific groups of authors. Further investigations are required before AI can be a viable solution to root out bias in peer review.

Opinions on how to reduce bias in the peer review process may differ; however, there is consensus on the need to weed out bias from scholarly publishing. The initiatives taken to address this problem may be challenging to implement, but with cooperation from key stakeholders in the scholarly community, they can mitigate bias more effectively in the long term.

Nisha Nair

Leave a reply

Your email address will not be published. Required fields are marked *