Greenhalgh, T., Raferty, J., Hanney, S., & Glover, M. (2016). Research impact: A narrative review. BMC Medicine, 14(78), 1-16. https://bmcmedicine.biomedcentral.com/articles/10.1186/s12916-016-0620-8
Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base. Its mechanisms are complex and reflect the multiple ways in which knowledge is generated and utilised. Much progress has been made in measuring both the outcomes of research and the processes and activities through which these are achieved, though the measurement of impact is not without its critics. We review the strengths and limitations of six established approaches (Payback, Research Impact Framework, Canadian Academy of Health Sciences, monetisation, societal impact assessment, UK Research Excellence Framework) plus recently developed and largely untested ones (including metrics and electronic databases). We conclude that (1) different approaches to impact assessment are appropriate in different circumstances; (2) the most robust and sophisticated approaches are labour-intensive and not always feasible or affordable; (3) whilst most metrics tend to capture direct and proximate impacts, more indirect and diffuse elements of the research-impact link can and should be measured; and (4) research on research impact is a rapidly developing field with new methodologies on the horizon.
This article is an important review of some of the dominant frameworks for research impact assessment (RIA) and points the way to some new arrivals on the scene. Anything by Trish Greenhalgh is good so this was an easy pick for this month’s journal club.
The frameworks review Payback, Research Impact Framework, Canadian Academy of Health Sciences (CAHS), monetisation, societal impact assessment and UK Research Excellence Framework. Context is interesting here as I had never heard about monetisation or societal impact assessment and this review is lacking the Knowledge to Action Cycle which is dominant in Canada.
Bottom line: if you want to do it well it really doesn’t matter what framework you choose. It will be time consuming, require dedicated skills and, therefore, be expensive.
If you don’t want to dive into the deep end of the RIA pool, the research impact framework was developed by and for academics interested in measuring themselves. “As such, it is a ‘light touch’ checklist intended for use by individual researchers who seek to identify and select impacts from their work “without requiring specialist skill in the field of research impact assessment”. Apart from this, all others were labour intensive.
A couple of things that stand out that aren’t in the abstract (which is a great summary of the take home information):
• The SPRIT Action Framework (a newer model) employs a logic model (as most do), but unusually, the ‘logic model’ focuses not on the research but on the receiving organisation’s need for research. I find this compelling because impact is a function of our industry partners making products, our government partners developing policies and our community partners delivering social services. It is true that clinical research or education research in the classroom can make an impact on immediate patients or students but the opportunity for this research is to scale beyond the single clinic or classroom. To assess the impact of research you *must* engage those who are using it as well as those producing it. This is the basis of the co-produced pathway to impact which would be interesting to compare against these models (as I have already done).
• Contribution mapping is a new approach in RIA. “In this model, the focus is shifted from attribution to contribution – that is, on the activities and alignment efforts of different actors (linked to the research and, more distantly, unlinked to it) in the three phases of the research process (formulation, production and extension”. Contribution mapping aligns with a performative paradigm (column 6 in table 1 in the article). For more on contribution analysis see a previous journal club post.
Questions for brokers:
1. What framework is the basis for your research impact assessment approach? Does it focus on the producers of research or the users of research?
2. Are you a researcher (or a research administrator) using Researchfish to capture the evidence of impact? Why are you capturing the evidence of impact from a researcher during or at the end of a research study when the researcher isn’t the one making the impact (see above) and the impact hasn’t usually occurred by the end of the study?
3. Are you an experience research impact assessor? If not who are collaborating with that has these skills? Where can you go to build your skill set in research impact assessment?
Research Impact Canada is producing this journal club series to make evidence on knowledge mobilization more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.