Open Peer Review Module

Open Scholar coordinated a consortium of five partners to develop an open source module that can be installed on institutional repositories to enable overlay open peer review.

The problem

Research productivity is increasing at an unprecedented rate. Technological innovations, a surge in available computing power, and the ease with which digital information is stored and communicated is helping researchers to cross experimentation boundaries, to increase data availability, and to facilitate the transfer of knowledge. As a result, traditional research is being transformed into a dynamic and globally interconnected effort where ideas, tools and results can be made instantly accessible to the entire academic community. Institutional and multidisciplinary open access repositories like Zenodo play a crucial role in this emerging landscape by enabling immediate accessibility to all kinds of research output.

One important element still missing from open access repositories, however, is a quantitative assessment of the hosted research items that will facilitate the process of selecting the most relevant and distinguished content. Common currently available metrics, such as number of visits and downloads, do not reflect the quality of a research product, which can only be assessed directly by peers offering their expert opinion together with quantitative ratings based on specific criteria.

Our Solution

With the financial support of OpenAIRE, Open Scholar coordinated a consortium of five partners to develop an open source module that can be installed on existing institutional or other repositories to  enable open peer review. Digital research works hosted in these repositories can thus be evaluated by an unlimited number of peers that offer not only a qualitative assessment in the form of text, but also quantitative measures that are used to build the work’s reputation. Importantly, this evaluation system is open and transparent. By open we mean that the full text of the peer reviews are publicly available next to the original research work. By transparent we mean that the identity of the reviewers is disclosed to the authors and to the public. In our model, openness and transparency are two elemental aspects that are necessary to address the issue of biased or non-expert opinions, which is inherent in the anonymous peer review model.

The module also comes with a reviewer reputation system based on the assessment of reviews themselves, both by the community of users and by other peer reviewers. This allows a sophisticated scaling of the importance of each review on the overall assessment of a research work, based on the reputation of the reviewer.

The central idea of this project is to capitalise on the existing infrastructure offered by open access repositories and take the necessary steps to convert them into functional evaluation platforms. This can be accomplished by the addition of an overlay peer review service that will allow expert scholars to provide qualitative and quantitative evaluations for all published and unpublished research works. This complementary evaluation process that can run in parallel to traditional journal peer review will:

  • enable the peer review of any research work deposited in a repository, including data, code and monographs.
  • provide novel metrics for the quantitative assessment of research quality.
  • create a sophisticated reputation system for reviewers.
  • allow the weighting of reviews based on the quality of previous reviewer contributions.
  • facilitate the selection of relevant content from digital repositories by distinguishing material that has been validated by reviewers using tags and advanced search filters.
  • engage the research community in an open and transparent dialogue over the soundness and usefulness of any research work.
Resources

OPRM was developed as a plugin for repositories using the DSpace software package, but is designed in a way that facilitates adaptation to other repository software suites such as EPrints and Invenio.

The module is currently installed on two Spanish repositories:

  1. The institutional repository of the Spanish National Research Council (DIGITAL.CSIC)
  2. The repository of the Spanish Oceanographic Institute (e-IEO)

The source code is available at: //github.com/arvoConsultores/Open-Peer-Review-Module

Project final report: OPRM_final_report

Use case report and user feedback (in Spanish): //digital.csic.es/handle/10261/167425

Our project partners