Requiring reviewers to justify their ratings with helpful written comments, not just submit a number, measurably improves the reliability of peer ratings.
Patchan, M. M., Schunn, C.D., Clark, R. (2017). "Accountability in peer assessment: examining the effects of reviewing grades on peer ratings and peer feedback." Studies in Higher Education, 43(12), 2263–2278.
A rating alone tells an author almost nothing they can act on, and it's easy for a reviewer to submit a number without doing the underlying evaluative work. Patchan, Schunn, and Clark asked whether holding reviewers accountable for the quality of their written comments — not just their numeric ratings — changes how reliable and valid those ratings turn out to be.
The researchers examined how reviewing grades — assessments of the reviewer's own performance as a reviewer, based on the helpfulness of their comments — affected the peer ratings and feedback reviewers subsequently produced.
Requiring students to give helpful comments, and grading them on it, improved rating reliability. Accountability for the quality of the feedback itself, not just the numeric score, is what drives more careful, trustworthy peer evaluation.
This is the direct research basis for a feature many online peer-review systems, Peerceptiv included, build into their reviewing-grade algorithms: reviewers are evaluated not just on submitting a rating, but on the accuracy and helpfulness of the comments backing it up. If a workplace peer-review process only asks for a score with no accountability for the reasoning behind it, this research says the ratings that come out of it will be less trustworthy than they could be.
Chris Schunn co-authored this study, which underpins the accountability mechanism built into Peerceptiv's reviewer-grading algorithm. He is a Professor of Psychology, Learning Sciences and Policy, and Intelligent Systems at the University of Pittsburgh and a Senior Scientist at the University's Learning Research and Development Center (LRDC), where he has directed research projects backed by more than $80M in federal grants. As Peerceptiv's Chief Learning Scientist, his research directly shapes how the platform structures reviewer prompts, rubrics, and feedback workflows.
