Progressing Research Impact Assessment: A ‘Contributions’ Approach

Morton, S. (2015). Progressing research impact assessment: A ‘contributions’ approach. Research Evaluation, 24(4), 405-419. doi:10.1093/reseval/rvv016
There is an increasing interest in demonstrating the outcomes from research for the purposes of learning, accountability, or to demonstrate the value of research investments. However, assessing the impact of social science research on policy and practice is challenging. The ways in which research is taken up, used, and reused in policy and practice settings means that linking research processes or outputs to wider changes is difficult, and timescales are hard to predict. This article proposes an empirically grounded framework for assessing the impact of research—the Research Contribution Framework. A case study approach was adopted to explore the nature of research impact and how it might be assessed. Findings were used to design, develop, and test a framework to assess the contribution of research to relevant areas of policy and practice and to articulate wider benefits. The framework has been adapted from contribution analysis, using the idea of ‘contribution’ to help explain the ways research is taken up and used to influence policy and practice. The framework allows for a focus on the roles of research users, and examines both processes and outcomes. It is argued that this approach gets round some of the common problems in assessing impact. It provides a method of linking research and knowledge exchange to wider outcomes whilst acknowledging and including contextual factors that help or hinder research impact. It is practical, balancing robustness with feasibility. It is adaptable for a wide range of content, types of impact assessment, and purposes.
This paper builds on a KMb journal club post of an earlier publication of Sarah Morton. That article described the critical roles that non-academic research users play in mediating impacts of research beyond the academy. It also introduced the logic model under pinning Sarah’s knowledge mobilization/knowledge exchange practice that knowledge exchange activities lead to: research uptake – research use – research impact. This logic model is based on models of research use that are interactive between researchers and research users through a process of engagement “involving many actors interacting and communicating over time”.
Take away 1: Research impact is not a function of dissemination *to* research users but an ongoing engagement *with* research users. Recall the earlier journal club post that speaks of the critical role of research users in mediating research impact.
This paper describes the Research Contribution Framework (RCF) that builds on this logic model to create a narrative of research impact. Sarah quotes literature indicating that “there is general agreement in the utility of a case study approach for assessing impact in order to capture the context-specific and variable nature of impact and…interviews were often the most useful source of information…” The RCF is a method of data collection, often through interviews with stakeholders, to generate the evidence that informs the narrative of research impact.
There is a brief discussion of the research vs the researcher and can we attribute change to a particular research project and not to all the other aspects of the researcher who may be influencing the beliefs, attitudes and practices of the non-academic partners. See a recent LinkedIn discussion on this topic and weigh in with your opinions.
In discussing when to use RCF and impact assessment Sarah points out that impact assessment is built in from the planning stage when the links between how research is done, who is engaged, and its potential impact are all considered at the start…however, impact assessment is often carried out retrospectively…” See a recent blog post about how prospective knowledge mobilization planning is ex-ante research impact assessment with impact being the dependent variable and knowledge mobilization being the independent variable (I barely know what this means but evaluators understand this!). Employing RCF at the beginning of your knowledge exchange/mobilization intervention will allow you to more effectively capture the data during the conduct of the project and provide you with more reliable evidence to inform your narrative. Employing RCF as a planning tool, not just an evaluation tool, allows knowledge brokers to treat RCF as a living document, updating and refining on a periodic basis as new evidence emerges and contexts inevitably change.
Take away 2: knowledge mobilization planning shouldn’t be a snapshot in time but an ongoing process of planning, review, assessment and planning.
This is a great paper in that it provides the frameworks and even the questions to ask your stakeholders so you can collect the evidence to inform your impact narrative. The gap comes in discussion of contextual analysis. “The process for applying the RCF is: 1) to conduct contextual analysis…” and then later when assessing final outcomes, “From contextual analysis, where might research be/have been useful?” Contextual analysis is important at the front and the back ends of RCF but there is no explanation of contextual analysis in the paper. In subsequent correspondence Sarah acknowledged that contextual analysis remains under developed as a method. She told me, “The contextual analysis is the least developed – and that isn’t just me – more broadly throughout complexity theory, the idea that context matters but we don’t really have great tools for understanding it.”
Sarah discusses the counterfactual – what might have happened had the research not been conducted or the researcher not been engaged with the end user and is this discussion even helpful. This is what I do: ask the research user partner if the change would have happened without the research input. It doesn’t tell the whole story but it does address the counterfactual. In a co-production paradigm the answer is almost always, “No, this wouldn’t have happened without the input of the research(er).” However, Sarah acknowledges that RCF is very much about showing *how* research contributed to a change decision and not an assessment of whether if contributed or not.
Take away 3: stay in touch with your non-academic research partners not only to collect the RCF evidence but to determine if you even made a difference outside of the academy.
The tables and figures are immensely helpful, especially Table 1 that presents sample questions to ask of stakeholders and Table 3 that presents indicators for knowledge exchange/mobilization processes.
Check out the second figure for a seizure inducing example of how the process isn’t linear over time even though our diagrams create a linear flow from research to uptake to use to impact. See an earlier KMb journal club post discussing how linear may be ok. The figure is a mess because our work is messy. But especially when working in a system of research impact projects the progress of the system, not of any individual project, must be linear as projects will progress to impact or become abandoned.
And I am very much looking forward to reading, the following article from Sarah’s references:

  • Bannister & O’Sullivan (2013) Knowledge mobilisation and the civic academy: The nature of evidence, the roles of narrative and the potential of contribution analysis. Contemporary Social Science, 8(3), 249-262.

Thanks for this reference, Sarah!
Questions for brokers:

  1. Is it reasonable to differentiate the impact of research compared to a researcher? If we are trying to articulate the impact of a university’s research is this distinction even helpful? Is the researcher more than just the sum of his/her research projects?
  2. Even though contextual analysis remains under developed we are all engaged in informal analysis that helps us understand the context of our knowledge mobilization interventions. What are you doing to understand your context? More importantly, what are you doing to understand the context of research use of your non-academic partners?
  3. Contribution: does the language of contribution overcome issues of attribution? We know that research is only one factor that might contribute to a change. Does changing our language help or do we need to change our practice as well?

ResearchImpact-RéseauImpactRecherche (RIR) is producing this journal club series as a way to make evidence on KMb more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.