Lai Ma, Junwen Luo, Thomas Feliciani, Kalpana Shankar, How to evaluate ex ante impact of funding proposals? An analysis of reviewers’ comments on impact statements, Research Evaluation, rvaa022, https://doi.org/10.1093/reseval/rvaa022
Abstract
Impact statements are increasingly required and assessed in grant applications. In this study, we used content analysis to examine the ‘comments on impact’ section of the postal reviews and related documents of Science Foundation Ireland’s Investigators’ Programme to understand reviewers’ ex ante impact assessment. We found three key patterns: (1) reviewers favoured short-term, tangible impacts, particularly commercial ones; (2) reviewers commented on process-oriented impact (formative) in a more concrete and elaborate manner than on outcome-oriented impact (summative); and (3) topics related to scientific impacts were widely discussed even though the impact section was to be used for evaluating economic and societal impacts. We conclude that for ex ante impact assessment to be effective, funding agencies should indicate the types of impact expected from research proposals clearly instead of a general ‘wish list’ and that more focus should be put on process-oriented impact than outcome-oriented impact.
This is an advance article so its not open access but return to the Research Evaluation web page when the article is published, and it will be available to everyone and will have a complete citation.
Using 261 reviewer reports from Science Foundation Ireland the authors try to answer the following questions
- How do reviewers evaluate impact statements in grant applications without evidence or proof of impact?
- What do reviewers perceive as impact in grant applications?
- Do they evaluate impact statements based on what kinds of impact will be achieved or how impact will be achieved?
- If there are so many uncertainties and inconsistencies in ex post impact assessments, how can we evaluate impact ex ante fairly and objectively in the context of grant applications?
The results are summarized in the abstract. What is also interesting are the conclusions arising from this research.
- Clarify the purposes and criteria for evaluating potential impact, especially when funding allocations can have significant implications for knowledge production and solving important problems such as poverty and the climate crisis.
- Process-oriented impacts are more appropriate in ex ante impact assessment for they reduce uncertainties and randomness in evaluation, on the one hand, and they prompt plans and activities to achieve impacts, on the other.
- Further analyses of peer reviews of different types of funding programmes and the decision-making process of peer reviewers are needed to improve the fairness, transparency, and efficiency of impact assessments.
But what is not stated and equally important is the need to train reviewers in assessing impacts and the opportunity to re-define the “peer” in peer reviewer. Both these are predicated on some observations by the authors that imply (but would need to be tested by interviews with the reviewers) that scientific peers are good at reviewing science. They are not good at reviewing plans to achieving socioeconomic impact.
But here’s my “thing” about the limitation in assessing impact ex ante as presented in this paper. The authors return to the following observation, “ex ante impact assessment are judgements based on prediction, because reviewers evaluate the outputs and outcomes of a research study which has yet to take place, and because serendipity is an important element in realizing impact.”
Let me get this right…we can’t easily/accurately assess something that hasn’t happened. But isn’t this exactly what reviewers do when reviewing the scientific components of an application? The experiments haven’t happened yet, but peer reviewers can assess the track record of the research team, the infrastructure/equipment in place, the access to data/resources, the methods to be undertaken. So why is this a barrier for ex ante impact assessment and not a barrier for ex ante research assessment? Impact “peers” can review the impact track record of applicants (a mandatory component for NHMRC Australia applications), the partnerships in place, access to resources/audiences/end users and the KMb/KT methods to be undertaken.
The authors’ objectives, research and conclusions are good, but they miss this important point. The issue isn’t that it can’t happen but that the wrong people are being asked to do the ex ante impact assessment.
Questions for brokers
- All these are important but order them in priority for funders from highest to lowest: create better guidance documents for applicants; train academic peers to review impact; use impact experts to review impact.
- What guidance would you give to applicants about presenting short term (by the end of the funded grant period) and longer term (beyond the funded grant period) impacts in their grant applications?
- Academic researchers are expected to participate on peer review committees as part of their job. What incentives might be provided to impact practitioners or patients and community reviewers for whom serving on grant review panels is not part of their jobs?
Research Impact Canada is producing this journal club series to make evidence on KMb more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article (when it is available). Then come back to this post and join the journal club by posting your comments.