Yaqub, O. Malkov, D., and Siepel, J. (2023) How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework, Research Evaluation, 32(2). Pages 273–285, https://doi.org/10.1093/reseval/rvad019
Abstract
Although ex post evaluation of impact is increasingly common, the extent to which research impacts emerge largely as anticipated by researchers, or as the result of serendipitous and unpredictable processes, is not well understood. In this article, we explore whether predictions of impact made at the funding stage align with realized impact, using data from the UK’s Research Excellence Framework (REF). We exploit REF impact cases traced back to research funding applications, as a dataset of 2,194 case–grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex ante and ex post impact. We find that impact claims in these case–grant pairs are often congruent with each other, with 76% showing alignment between anticipated impact at funding stage and the eventual claimed impact in the REF. Co-production of research, often perceived as a model for impactful research, was a feature of just over half of our cases. Our results show that, contrary to other preliminary studies of the REF, impact appears to be broadly predictable, although unpredictability remains important. We suggest that co-production is a reasonably good mechanism for addressing the balance of predictable and unpredictable impact outcomes.
I really like this paper since it provides evidence to counter all those who think ex ante impact statements in grant applications are nothing more than fictional wish lists or the more politely named “imagined impact” in this article. This paper presents the evidence that good impact planning (ie a good knowledge mobilization plan) aligns with eventual impact. And this alignment is enhanced when co-production is employed as a knowledge mobilization method.
How did they do this? It is in the abstract. This could only have been done in the UK where impact case studies in the Research Excellence Framework can be paired with associated grant applications. In 2020 the UK funding councils removed dedicated pathways to impact strategies favouring inclusion of impact in the research descriptions so note that this only involves case-grant pairs from before 2020.
Enhancing alignment by using co production (cited in 54% of case-application pairs) as a knowledge mobilization method makes sense since the end beneficiaries are at least consulted and preferably engaged throughout the research process. The authors state, “Active co-production of knowledge with an end-user is not only a pathway to impact, but also one that may drive alignment between researchers and stakeholders.” We have known this but now it is placed in context of grant applications and can serve as rationale for grant applicants to take their knowledge mobilization strategies seriously and get them done in a timely fashion (ie not 48 hours before the grant application deadline!).
If you are interested how KMb York supports impact strategies in grant applications see this paper we wrote in 2016.
See figure 2 for a summary of the data. Bottom line: 84% of the topics between application and case had alignment at least to some extent and stakeholders were aligned at least to some extent in 88% of the case-application pairs.
Here’s a statement that made me think: “Exact topic identification occurs when the same technology or research output is predicted in the funding application and subsequently appears in the impact case. This is perhaps more in line with traditional, more linear models of impact. It is certainly part of the underlying logic behind the introduction of
‘pathways to impact’ statements.”
One thing we know (or assume) to be true is that there is rarely – if ever – a straight line between research and impact. It is always iterative with frameworks having double headed arrows in every diagram. Since writing a pathway to impact section is knowledge mobilization planning does this mean that planning for knowledge mobilization activities creates a more linear pathway?
Does good knowledge mobilization planning help to straighten this pathway to impact?
Makes you go hmmmm…..
Questions for brokers:
- Does good knowledge mobilization planning help to straighten this pathway to impact?
- Strengths and weaknesses to dedicated impact strategy in a grant application or embedding impact in the research section?
- How can you use this instead of just telling a researcher to take the impact section of their grant applications seriously?
Research Impact Canada is producing this journal club series to make evidence on KMb more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other people interested in knowledge mobilization. Read this open access article. Then come back to this post and join the journal club by posting your comments on our LinkedIn.