Recently NCG member, and the Program Director of The William and Flora Hewlett Foundation's Global Development and Population program, Ruth Levine took to the Foundation's blog to reflect on the process she and her staff use to identify quality research for funding purposes.
We are not staffed like a research funding institution, and we cannot count on journals’ quality assurance processes to vet all of the final products. So we have to figure out how to judge research quality, from proposal to finished product. That’s not so easy.
Research quality is a concept with many dimensions: Is it relevant? Are the choices about how to collect and analyze data appropriate, and are the methods applied correctly? Are the findings communicated in ways that work for technical and for policy audiences?
As grant makers with deep knowledge of the fields in which we work, we’re in a pretty good position to assess relevance of the questions and accessibility of the findings. It’s far harder, though, to figure out if the sampling design is sound, or if the statistical methods are the right ones, and are used correctly. We don’t have time to read every research paper our grantees produce, and I’m pretty sure they don’t want their program officer asking them a lot of questions about statistical power, endogenous variables and fixed-effects modeling. But we do have to find ways to assess the soundness of the research.
Be sure to read Ruth's full blog post for her list of a few ways they assess research quality.