#ShitDavidSays About Impact #7: If Impact Occurred but No One Was There to Measure It… / #ShitDavidSays About Impact, no 6 : s’il y a un impact, mais que personne n’est là pour le mesurer…

If impact occurred but no one was there to measure it did anything ever really happen? In this 7th and final post in this series, David speaks about the importance of assessing research impacts because if we don’t how can we demonstrate the value of research to end beneficiaries? He points out the irony of asking researchers to report on impacts in end of grant reports.
Si la recherche produit un impact, mais que personne n’est là pour le mesurer, est-ce qu’on peut dire qu’il a vraiment eu lieu ? Dans ce 7e et dernier billet de la série, David parle de l’importance d’évaluer les impacts : comment prouver la valeur des recherches aux utilisateurs finaux si l’on n’a rien mesuré ? Il souligne aussi l’ironie qu’il y a à demander aux chercheurs de faire état des impacts dans leurs rapports finaux.

In Canada, we are developing a culture of creating impacts. This is evident through grant applications that require a knowledge translation (CIHR, health charities), knowledge mobilization (SSHRC) or commercialization (NSERC) strategy. SSHRC also requires an outcomes statement that is a prediction of the difference the funded project will make on Canadians. As identified by the Canadian Federation of Humanities and Social Sciences, this could be an impact on scholarship and training as well as an impact on the economy, society and culture or public policy.
But if we don’t assess the impacts beyond scholarship and training how can we fulfil these obligations in our grant applications?
The UK Research Excellence Framework (REF) 2014 collected 6,679 case studies of research impact and assessed them by panels of academic and non-academic experts in 36 Units of Assessment (i.e. examples of impact arising from history were not compared to impacts arising from chemistry). Research on the REF identified 3,709 unique pathways to impact.
Let me say that again. 6,679 stories of impact and 3,709 different ways to make impact.
That’s right. There is no cookie cutter approach to either creating or assessing impact. In my closing address to C2UExpo 2017, I pointed out how hard it is not only to plan for impact but also to assess impact but being hard was no excuse not to do either. I observed that we didn’t give researchers tenure so they could do something easy.
Get out there an assess the impact of your research because if you don’t then did you ever make a difference to anyone other than your academic colleagues? The UK has done it for the whole post secondary system, surely there’s something we can do in Canada?
Well yes there is, thank you for asking.
If you are effectively planning your impact strategy you are therefore also planning your impact assessment. Knowledge translation planning is ex ante research impact assessment. If you plan your impact you are inherently identifying the processes needed to move from research to impact including the indicators and data sources that will be the evidence of impact. Adapting a generic logic model of research impact (like the co produced pathway to impact) to your specific case will help guide your efforts for impact planning and impact assessment.
Research Impact Canada (RIC) is piloting a research impact assessment tool adapted from the REF. This tool provides a semi structured interview guide that creates consistency for collecting the evidence of impact and a case study template to create consistency for expressing the evidence of impact.
But here’s the thing about impact assessment…recall who is actually making the impact? SSHRC’s 2013 evaluation of their knowledge mobilization programs showed that it was primarily the partners of SSHRC funded projects that had the evidence of impact, not the researchers. This makes sense since it is the partners, not the researchers, who are making the products (industry), developing the policies (government) or delivering the social services (community) that have an impact on local and global citizens.
You need to use the interview guide in the RIC tool to gather the evidence of impact from partners and end users. And you need to do that long after the project has finished since the impact hasn’t usually happened within the course of a funded research project.
HEY FUNDERS…if we need to collect the evidence of impact from partners long after the project has ended why do you always ask researchers to report on impacts in their end of grant reports?
To return to the question in the title of this post, if a partner uses the evidence produced in a research project to help make impact but no one was there to collect the evidence of impact long after the grant ended then did anything ever really happen?
Go into the forest and see if that tree really did fall.