Abstract:

Epistemological differences fuel continuous and frequently divisive debates in the social sciences and the humanities. Sociologists have yet to consider how such differences affect peer evaluation. The empirical literature has studied distributive fairness, consensus, and the norm of universalism, but neglected the content of evaluation and how epistemological differences affect perception of fairness in decision-making. The normative literature suggests that evaluators should overcome their epistemological differences by "translating" their preferred standards into general criteria of evaluation. However, little is known about how procedural fairness actually operates in panels, and more specifically about how agreements are reached in the face of epistemological diversity. Drawing on 81 interviews with panelists serving on five multidisciplinary fellowship competitions in the social sciences and the humanities, we show that: 1) Evaluators generally draw on four epistemological styles to make arguments in favor of and against proposals. These are the constructivist, comprehensive, positivist, and utilitarian styles. 2) Although the comprehensive style is favored, there is considerable diversity in the epistemological styles used in the panels we studied; 3) Peer reviewers define a fair-decision making process as one in which panelists engage in "cognitive contextualization," that is, use epistemological styles most appropriate to the field or discipline of the proposal under review. These findings challenge the normative literature that associates procedural fairness with the use of generalizable criteria of evaluation.