Canadian Health Services and Policy Research Alliance (2018). Making an impact: A shared framework for assessing the impact of health services and policy research on decision-making. https://docs.wixstatic.com/ugd/5adc92_3ae941eaedb04ab4a66b6f83f98a479d.pdf?platform=hootsuite&mc_cid=47f94af269&mc_eid=ffbad33983
No Abstract.
There is a ton in this report all directed towards assessing the impacts of health services and policy research (HSPR). It builds on the impact assessment framework of the Canadian Academies of Health Sciences that was published in 2009. The CAHS framework was the basis for the impact assessment work of Alberta Innovates and is being considered as an impact framework by many provincial health funders in Canada.
And now the Canadian Health Services and Policy Research Alliance has improved on the CAHS framework and provided two additional elements to help assess the impact of HSPR on decision making. The three components to impact assessment are:
Component 1 – The HSPR Informing Decision-Making Pathways to Impact
Component 2 – Informing Decision-Making Indicators, Methods, and Data Collection Tools
Component 3 – Communicating Impact
Component 1: The Pathway to Impact Key Improvements to the CHAS impact framework include: 1) the explicit engagement among researchers, end users and decision makers throughout the process and 2) the plan, do, study, act cycle of decision making. The latter was implied in the CAHS framework but not the former. As one author of the co produced pathway to impact that embeds both features I am pleased to see these refinements of the CAHS framework.
Component 2: Indicators The authors started with a literature review and worked with an Indicator Review Panel to come to the final set of indicators that are broken down by stage of the logic model (component 1). The authors provide 3 pages of potential sources of data and methods of indicator data collection. This is a huge contribution. And yet in even more detail is Appendix A, five pages of indicators aligned by stage of the logic model with reflection on data sources, availability of data and levels of aggregation.
Component 3: Communicating This is perhaps the least developed of the three components. The authors discuss narratives, case studies, score cards/dashboards (the later very briefly). Appendix B presents the CHSPRA impact narrative template.
But herein lies the gap. Table 2 presents the core set of 12 indicators and 11 of them include “survey policy/decision maker”. The suite of three components could be enhanced by providing guidance to these surveys and interviews that form a large core of data collection. Research Impact Canada is piloting a tool to collect the evidence of impact that includes not only a case study template, but an interview guide derived from contribution analysis. We have completed three pilots of this tool in York’s Knowledge Mobilization Unit and Kids Brain Health Network and we will be writing it up soon enough. Our experience to date is that an interview guide increases consistency of evidence collection (potentially facilitating assessment across projects) and increases efficiency of evidence collection. While some reports indicate each impact case study in the 2014 REF cost £7,000 of person time our experience reduces that to about $1,200, a ten-fold reduction.
The other gap is the lack of prospective impact planning. There is one mention of using the framework to prospectively plan for impact assessment (something I have compared to knowledge translation planning) but this remains a missing piece that would complete advice to those seeking to maximize the impact of HSPR in Canada.
Questions for brokers:
- Five pages of indicators: amazing treasure trove or indicator overload?
- The authors discuss the need to turn the framework into a specific pathway to impact for each project. The conceptual framework is nice but don’t we need examples of how this has been used in practice?
- Review the scoring of impact pathways that included both the CPPI and the Alberta Innovates framework. How do these enhancements to the CHSPRA logic model affect the score previously assigned to the Alberta framework?
ResearchImpact-RéseauImpactRecherche (RIR) is producing this journal club series as a way to make evidence on KMb more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.