Applying Social Judgment Theory to better understand what peer reviewers pay attention to when evaluating fellowship grant proposals

CLICK HERE TO BOOK

Abstract

While research has revealed startling inconsistencies and lack of agreement in reviewers’ evaluations of grant proposals (Pier et al., 2018), the reasons why such inconsistencies arise remain unclear. In this talk, I will present results from the first phase of the “Towards Outstanding Research Reviews” (TORR) project” (TORR) project, which aims to understand how individual peer reviewers use and combine information in written grant proposals to make their recommendations for funding. Informed by the Social Judgment Theory (SJT) framework (Hammond, 1977), we conceived reviewers’ recommendations as resulting from the application of a professional “judgement policy” and we first sought to infer what research quality factors expert reviewers pay attention to while reviewing a proposal for funding in a semi-naturalistic setting. To do so, we recorded think aloud protocols from 7 reviewers who were re-examining a grant application for the Wellcome Trust’s Humanities and Social Sciences Fellowship funding scheme, which they had recently reviewed. The think aloud protocols were adapted from the Critical Decision Making method of interviewing (Klein, 1989) and the Cognitive Event Analysis framework (Steffensen et al., 2016). Following a content analysis of the transcripts, we identified a total of 56 possible quality criteria mentioned by reviewers, which were organised around 5 areas of quality. Next, we conducted a survey where a further 28 expert reviewers were asked to rank the quality criteria in order of importance. Contrary to earlier findings suggesting lack of consensus between peer-reviewers, we uncovered ten quality factors which were consistently rated as most important for assessing the quality of a fellowship application. Upon presenting the factors identified, I will discuss implications advancing the understanding of peer-review judgements, the peer-reviewers’ dilemmas we uncovered through this process, as well as the challenges and opportunities involved in meta-research studies.

References

-           Hammond, K.R. et al. (1977) Social judgment theory: Applications in policy formation. In Human judgment and decision processes in applied settings (Kaplan, M. F. and Schwartz, S., eds), pp. 1–29, Academic Press.

-           Klein, G.A. et al. (1989) Critical decision method for eliciting knowledge. IEEE Transactions on Systems, Man, and Cybernetics 19, 462–472

-           Pier, E.L. et al. (2018) Low agreement among reviewers evaluating the same NIH grant applications. Proceedings of the National Academy of Sciences 115, 2952–2957

-           Steffensen, S.V. et al. (2016) Cognitive events in a problem-solving task: a qualitative method for investigating interactivity in the 17 Animals problem. Journal of Cognitive Psychology 28, 79– 105.

The Speaker

Gaëlle Vallée-Tourangeau is Professor of Behavioural Science at Kingston University, London.