How do Civil Servants Update? Constraints to Evidence Based Decision Making


Michael Callen, Adnan Khan, Asim Khwaja, and Asad Liaqat. In Preparation. “How do Civil Servants Update? Constraints to Evidence Based Decision Making”.


A primary purpose of development research is to generate evidence that policy makers believe in and that changes their priors in a way that leads to the take-up of this evidence in policy making. We present some findings that suggest this process is fraught with problems. 

First, civil servants have inaccurate beliefs about quantities of interest for policy making. This is worrying since they have been going through courses explicitly designed to give them a broad policy training. It could, however, be argued that they don't need these quantities in their day to day lives. Even if that is the case, when we generate evidence, we hope that the policy maker has some baseline understanding of the issues we are going to present evidence on, but if they are very ill-informed, we need a lot more work on framing the issue than we currently do. 

Second, we show that two key elements of development research are not aligned with how policy makers respond to evidence: (i) Research produces large N evidence, for the sake of being right but also for the sake of being able to convince people. Policy makers do not differentially update in large N evidence as much as they should, which puts this assumed purpose of large N research in doubt, (ii) Research is done with high internal validity in some areas and it is expected that policy makers in other areas (that are not too different) will believe this evidence and use it for policy making. We find, however, that policy makers update more in small N evidence from their area than from large N evidence from other areas, indicating that perhaps if the goal is to convince the policy maker we should be investing in small N research everywhere rather than large N research in some places, once we have a reasonable prior that a policy works.

Last updated on 01/17/2018