E.A. Jensen, S. Noles, MS Reed, and P. Lang. (2025) How can a research program influence public policy? Evaluating a decade of research impact using an evidence-based theory of change. Environmental Science & Policy. 171: 104182 https://doi.org/10.1016/j.envsci.2025.104182.
Abstract
This study proposes the systematic empirical development of a theory of change for both planning and evaluating research impact on public policy. This use of theory of change is underdeveloped as a retrospective research impact evaluation method, capable of covering long timespans. Here, we assess impact at the level of a UK water research program with a 10-year timeframe. We developed a program-level theory of change from overlapping data sources. This involved integrating inside-out and outside-in perspectives on impact processes through survey, interview and focus group-style participatory workshops to triangulate a model of policy influence. The result was a triangulated theory of change, refined through an iterative process. This method offers an adaptable evaluation framework that can also provide a robust basis for planning future research impacts. The findings underscore the importance of considering multiple perspectives and evidence sources in understanding research impact pathways, contributing to more effective and impactful research strategies in policy domains. The study’s 10-year scope also shows the potential for evidence-based theories of change to explain some of the long-term, complex dynamics that enable research to influence policy.
Kudos to the authors. They appear to be from project management, research and policy spaces. Not all academics writing about what other people have done. The authorship embodies some of the learnings from the research. This article is interesting because it developed a theory of change (ToC) following extensive empirical research. Many ToC are hypothetical, created at the planning stages “often armchair exercises”. This project developed their ToC after 10 years of funding to map what contributed to the project’s impacts. The multi method approach to data collection included:
- Online fact-finding interviews with research program representatives
- Online scoping survey with project researchers
- Online structured qualitative interviews with external project partners and beneficiaries
- Online participatory workshops with external partners and beneficiaries
- Online participatory workshop with research program staff
Everything was online. There were activities with researchers, partners and beneficiaries. Well done.
Check out the ToC in Figure 1. You will need a magnifying glass if you printed it out but note the inclusion of impact activities, enabling factors and intermediate outcomes. This is a 10-year project and there are intermediate outcomes. There is mention in the discussion about the length of time it takes for research to make an impact. See here for a journal club article. Nonetheless there are good examples of changes in policy and practice and the use of research evidence. Perhaps the longer-term outcomes will be benefits of those policies and practices on end beneficiaries.
The empirical evidence collected in the five activities above helped the authors identify how much different aspects of the project had an impact on impact. The top ranked contributor was the research program’s project management expertise contributing 81% to project impacts. Other aspects included researcher/policy collaborations during the project (78%), practical value of research findings (77%) perceived credibility of the research program and value of research findings (71%), making research findings accessible (69%), policy awareness and communication skills of researchers (60%). At the bottom of the pack was previous collaborations between the researchers and policy organizations (46%).
Observation #1: assuming the people doing the project management are the ones doing this research is it a surprise that they are ranked the most significant?
Observation #2: Previous collaborations contribute the least. What does this say about trust being an enabling factor?
Finally there are some conclusions about the use of research evidence in policy contexts.
“One project manager voiced their frustration, stating that “some of them I perceive don’t do anything with the information […]. That’s very frustrating […] to try and realize impact and communicate the project and get wider engagement.” … “Several policy organization representatives in senior roles pinpointed their struggle to fulfill project steering group responsibilities due to a lack of resources. They shared that they “just haven’t had the resources […] to actually get involved” This suggests that the use of evidence at the partner/user side is limited. They might be involved in co-creating the research but their ability to use the research is limited.
The authors state, “They also emphasize the capacity for co-produced research and impact to contribute towards less obvious impact on relationships, knowledge sharing, culture shifts and capacity-building for future collaborative research”.
“Co-produced research and impact”. I have been thinking a lot about co-produced impact recently. If we co-produce the research but we don’t help co-produce the impact, then we are not helping overcome barriers for use and implementation of research evidence.
Questions for brokers:
- Everything was online. How might some in person activities have benefitted data collection? Or is all online good enough?
- Previous collaborations contribute the least. What does this say about trust being an enabling factor?
- Co-produced impact vs co-produced research. Discuss.
Research Impact Canada is producing this journal club series to make evidence on KMb more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other people interested in knowledge mobilization. Read this open access article. Then come back to this post and join the journal club by posting your comments on our LinkedIn.