David Phipps (RIR-York) and Julie Bayley (Coventry University)
The National Institute for Health Research (UK) and the Association of Medical Research Charities held an impact forum on April 27, 2016. We were invited to kick at some of the popular impact pathways by asking five critical questions.
Deux établissements britanniques, le National Institute for Health Research et l’Association of Medical Research Charities, organisaient un forum sur l’impact de la recherche, le 27 avril dernier. Nous avions été invités à poser cinq questions délicates afin de déboulonner les moyens populaires de mettre la recherche en action.
The presentations and discussions started NIHR and ARMC on a path to a shared vision of understanding impact of health research on patients. This was not their first shared event. On October 2015 they co-sponsored an event titled “Getting the Most Out of Clinical Research” – which aimed to instigate more collaborative working between charities and the NIHR; examine areas where the charity sector and NIHR can work together and identify issues that require further work”. Clearly health research impact was one issue demanding some closer attention.
In preparing for this presentation, Mark Taylor, Director of Impact (such a cool title), at NIHR, asked for a critical appraisal of impact pathways to help government funders as well as medical research charities (large and small) take a critical approach to impact planning and impact assessment.
Julie Bayley (Coventry University and my ACU Fellowship partner) and I started from the premise that impact is a permanent feature of the research landscape whether through centralized research assessment exercises (such as the Research Excellence Framework in the UK) or as a function of the research grant application. We then developed five questions to guide critical appraisal of the pathways. Our questions and the pathways we appraised [Knowledge to Action Cycle (KTA), Payback Model, Co-produced Pathway to Impact (CPPI)] are in our presentation posted on slide share.
For the sake of argument we will put aside the fact that KTA isn’t a pathway to impact but a conceptual framework; however, it is mis-used as a pathway (especially in Canada) so we will continue that tradition of mis-use for the purposes of critical appraisal.
With so many ways to conceptualise impact, it can be difficult to determine how best to draw from these models in practice. So we devised a set of five questions to guide thinking, and identified how each model ‘scores’ in each category. The questions and scorecard (graded A to F) are:
- Does your impact pathway accommodate and enable collection of evidence for patient benefit?
In health research it is critically important that impact be patient centric and be measured as a benefit to patients, their caregivers and families. None of the pathways did this explicitly although each included an evaluation stage where patient benefit could be assessed. CPPI had “citizens served” as an ultimate impact so this could be read to be patients in a health research paradigm.
KTA: B
Payback: B
CPPI: B+
- Does your impact pathway support engagement of end users (incl. patients, policy, service providers) throughout?
This is hugely important because academic researchers only make an impact beyond the academy by engaging stakeholders and collaborating with end users and partners throughout the research process. KTA does not have any allowance for collaborators beyond the research space, in fact is explicitly separates the knowledge creation “funnel” from the action cycle. Payback doesn’t explicitly separate knowledge production from its use but it also doesn’t explicitly engage partners. CPPI is built on co production between researchers and partners throughout the pathway.
KTA: F
Payback: C
CPPI: A
- Does your impact pathway Work at the level of the project, the program, the organization, the system?
This is of particular interest to the NIHR/AMRC collaboration on impact since they are seeking a common language that works for the +£1B NIHR and the small medical research charities that may be investing $£1-2M. KTA isn’t a pathway and doesn’t work for any single organization. A citation analysis of papers “using” KTA cited that the original authors of KTA never expected a single organization to operationalize the entire pathway. Payback is the basis for the Albert Innovation Health Solutions impact assessment framework that is used at the level of the research project and the research funding agency. CPPI has been used as a model for a single research project, and $4-6M/year Networks of Centres of Excellence such as NeuroDevNet as illustrated in our presentation.
KTA: F
Payback: A
CPPI: A
- Does your impact model enable planning by providing general logic informing specific adaptation?
An impact pathway is just a model. It is just the start of your impact journey. Impact must be planned with specific goals, specific activities, and indicators specific to the research in question. The pathway provides the general logic but it must also guide specific implementation. All three models are logic models that describe activities from research to impact, all three get a B…they would get an A if they could simplify the specific implementation but that’s the job of the pathway user be it a researcher, research user or a research funder.
- Does your impact model drive uptake/adoption?
Researchers don’t make products, their industry partners do. Researchers don’t make public policies, their government partner do. Researchers don’t usually deliver social services their community partners do. Therefore research must be taken up and adopted by partners from the private, public and non-profit sectors to have an impact on end users (i.e. patients). All three pathways have a dissemination moment. KTA has a transfer point from the knowledge creation stage to the action cycle (at the bottom of the funnel). Payback and CPPI both explicitly include dissemination; however, in CPPI dissemination is in collaboration with partners/end users so is more an engaged exchange instead of a unidirectional transfer.
KTA: B
Payback: B
CPPI: A
Our goal of the presentation was to give the attendees these questions to take away as they begin to engage with impact pathways. We did not make any recommendations as to which pathway to choose as that will be influenced by deliberations ongoing between NIHR, AMRC and their stakeholders. It will also be influenced by the local context of the research and impact in question. But remember, choosing a pathway that makes sense for a particular context is just the beginning. Every pathway needs to be made specific and evaluated by collecting evidence on indicators that are specific to each stage of the pathway.