Yaqub, O., Malkov, D., and Siepel, J. (2023) How unpredictable is research impact? Evidence from the UK’s Research Excellence Framework. Research Evaluation. 32: 273-285. https://doi.org/10.1093/reseval/rvad019.
Abstract
Although ex post evaluation of impact is increasingly common, the extent to which research impacts emerge largely as anticipated by researchers, or as the result of serendipitous and unpredictable processes, is not well understood. In this article, we explore whether predictions of impact made at the funding stage align with realized impact, using data from the UK’s Research Excellence Framework (REF). We exploit REF impact cases traced back to research funding applications, as a dataset of 2,194 case–grant pairs, to compare impact topics with funder remits. For 209 of those pairs, we directly compare their descriptions of ex ante and ex post impact. We find that impact claims in these case–grant pairs are often congruent with each other, with 76% showing alignment between anticipated impact at funding stage and the eventual claimed impact in the REF. Co-production of research, often perceived as a model for impactful research, was a feature of just over half of our cases. Our results show that, contrary to other preliminary studies of the REF, impact appears to be broadly predictable, although unpredictability remains important. We suggest that co-production is a reasonably good mechanism for addressing the balance of predictable and unpredictable impact outcomes.
The article itself does a good job of articulating why this research is needed. “Researchers’ concerns often centred on the inability to foresee research impacts a priori: ‘It is impossible to predict the outcome of a scientific piece of work and, no matter what framework it is that you want to apply, it will be artificial and come out with the wrong answer—because if you try to predict things, you are on a hiding to nothing.”
Basically, researchers think it is impossible to predict impact ex ante. Not so, according to this research. The authors compared claims of impact and engagement of stakeholders (their words) between grant applications and REF impact case studies using 209 grant application/case study pairs. This seemed to be more consistent (61%) in applications from Arts and Humanities Research Council-funded cases greater than the 36% of Economic and Social Science Research Council-funded cases. Science & technology facilities (39%) and life sciences (27%) were also less frequently matched.
Co production was coded in 54% of the sample pairs. Applications to Arts and Humanities Research Council cited co-production 61% of the time (note the same % alignment with stakeholders) while applications for science & technology facilities cited co-productions only 22% of the time. “Our findings also show that end-users of research are often identifiable at the funding stage of research, highlighting the importance of building longer-term relationships between researchers and the stakeholders who have interest in their work.”
Just one comment on incentives for co-production. The authors state, “to this end, incentivizing co-production through impact…is a way of encouraging academics to engage. In a co-production mode, we therefore expect researchers to explicitly identify research outputs as being co-produced with stakeholders, either identified in the bidding stage or the impact case.” What incentives exist for co-production partners? The academic researcher gets the grant. The academic institution gets the REF funding. Aren’t they forgetting the partner in co-production?
Bottom line: “our analysis shows that research impact can, to a measurable extent, be explicitly anticipated before research is funded, at the topic and stakeholder level.” Looks like all the complaining about predicting impact being impossible is…well…wrong. At least 61% wrong for arts and humanities, 76% wrong overall.
This is important for Canada and other countries that require some form of impact statement in grant applications but do not follow through with impact assessment. We can now have some confidence that the effort undertaken by researchers (and those who support them) for describing future impacts, especially when underpinned by co-production methods, at least helps create the conditions where impact might occur.
Questions for brokers:
- From the perspective of a co-production partner, what are the incentives for engaging in research collaborations?
- From the abstract 76% of application-case study pairs show alignment between impact at application and case study. Is this compelling enough to make researchers stop complaining about impact in grant applications?
- Why are arts and humanities more easily able to identify impacts and partners ex ante?
Research Impact Canada is producing this journal club series to make evidence on KMb more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other people interested in knowledge mobilization.