Assessing Health Research and Innovation Impact: Evolution of a Framework and Tools in Alberta, Canada

Graham, K.E.R., Langlois-Klassen, D., Adam, S.A.M., Chan, L., & Chorzempa, H. (2018) Assessing health research and innovation impact: Evolution of a framework and tools in Alberta, Canada. Frontiers in Research Metrics and Analysis, 3, 1-14. https://doi.org/10.3389/frma.2018.00025

Abstract

Publicly funded research and innovation (R&I) organizations around the world are facing increasing demands to demonstrate the impacts of their investments. In most cases, these demands are shifting from academically based outputs to impacts that benefit society. Funders and other organizations are grappling to understand and demonstrate how their investments and activities are achieving impact. This is compounded with challenges that are inherent to impact assessment, such as having an agreed understanding of impact, the time lag from research to impact, establishing attribution and contribution, and consideration of diverse stakeholder needs and values. In response, many organizations are implementing frameworks and using web-based tools to track and assess academic and societal impact. This conceptual analysis begins with an overview of international research impact frameworks and emerging tools that are used by an increasing number of public R&I funders to demonstrate the value of their investments. From concept to real-world, this paper illustrates how one organization, Alberta Innovates, used the Canadian Academy of Health Sciences (CAHS) impact framework to guide implementation of its fit-for-purpose impact framework with an agnostic international six-block protocol. The implementation of the impact framework at Alberta Innovates is also supported by adopting emerging web-based tools. Drawing on the lessons learned from this continuous organizational endeavor to assess and measure R&I impact, we present preliminary plans for developing an impact strategy for Alberta Innovates that can be applied across sectors, including energy, environment and agriculture, and may possibly be adopted by other international funders.

This is another article from the impact assessment team at Alberta Innovates who wrote a 2012 article presenting their adaptation of the impact assessment framework from the Canadian Academies of Health Sciences. What I love about their work is they write as practitioners working in the space of impact assessment. Academic theory is crucial to our work but so is sharing the knowledge of practitioners actually working in the impact trenches.

This article presents an improvement of the CAHS framework as presented in the earlier article and adds to it a six-step protocol for research impact assessment. It also presents some answers to the question: why do I need a framework?

• impact frameworks provide a common language and approach to inform the systematic and standardization of data collection needed to conduct impact assessment and provide a model of the pathways to impact that allow funders to convey progress to longer term, and ultimate impacts.
• frameworks help organizations articulate their impact story—and the stories of those they fund—by providing a tangible structure for organizing evidence about the progress to, and achievement of, results along various pathways to impact.

These are great reasons for needing a framework but Alberta Innovates doesn’t just produce a framework they actually use it, something I find academic researchers don’t always do. We have a plethora of impact frameworks and even though they are frequently cited they are much less frequently implemented as shown by another journal club post.

There are a couple of key changes to their previously published impact assessment framework:

1. listening to stakeholders as a necessary precursor to identifying the needs of the research and innovation system
2. including three nested circles of impact categories:
   1. stakeholder groups including those who will use the research evidence
   2. types of impact: policy, products, practice, process
   3. informing decisions about innovations

These are similar to some of the recently reported changes to the impact assessment framework of the Canadian Health Services and Policy Research Alliance also reported in 2018. Since Alberta Innovates is part of the Canadian Health Services and Policy Research Alliance these enhancements to the CAHS framework are not surprising.

They also present their six-step process of research impact assessment:

1. understand the context
2. identify the purpose of the assessment: advocacy, allocation, analysis, accountability
3. define indicators of success
4. develop the design, methods, and data collection
5. communicate and use the findings
6. manage the assessment

Implementing their own impact assessment framework, they show that in 2016-2017 they supported 2,795 researchers, trainees and staff who produced 1,175 scientific articles, 251 influences on policy/practice, 42 medical products, 121 improvements in health outcomes and 21 instances of IP/licensing.

Questions for brokers:

1. Why are impact frameworks so frequently created and so infrequently used in practice?
2. A linear logic model underpins this framework. Is this helpful or does it create an artificiality since research to impact is not a linear process?
3. Creating a plan for impact assessment is the same thing as creating a knowledge translation plan. See here and discuss.
4. Is the return on investment in 2,795 researchers: about expected; amazing; underwhelming?

ResearchImpact-RéseauImpactRecherche is producing this journal club series as a way to make the evidence and research on knowledge mobilization more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read the open access article. Then come back to this post and join the journal club by posting your comments.