Netherland’s Research Impact Assessment Exercise / Exercice d’évaluation de l’impact de la recherche aux Pays-Bas

The UK has the Research Excellence Framework. Australia launched the Engagement and Impact Assessment exercise. And the Netherlands has the Standard Evaluation Protocol. Canada can learn from these and from the Research Impact Canada network as we implement our own tool for research impact assessment.

Le Royaume-Uni s’est doté d’un cadre pour l’excellence en matière de recherche, le Research Excellence Framework. L’Australie a mis en place un exercice d’évaluation de la participation et de l’impact dans ce domaine, l’Engagement and Impact Assessment. Et les Pays-Bas disposent d’un protocole d’évaluation normalisé, le Standard Evaluation Protocol. Le Canada peut tirer des enseignements de ces modèles et exploiter le Réseau Impact Recherche qui existe déjà au pays afin de mettre en œuvre son propre outil d’évaluation de l’impact de la recherche.

There is increasing global interest in creating socioeconomic impacts from academic research. National networks such as Research Impact Canada and the National Alliance for Broader Impacts (US) invest in methods to create impacts but neither have national systems of impact assessment. The UK and Australia have national research impact assessment (RIA) exercises but no formal structures to create impact.

And new for me is the Netherland’s RIA process called the Standard Evaluation Protocol (no snappy title points for the Dutch…maybe it suffered in translation). Every six years NLD research institutions are required to self-assess and present to external committees on the research from the past six years and plans for the subsequent six years. The assessment returns a rating of unsatisfactory, good, very good or excellent. The submission, the committee report and the institutional response are posted on line creating public accountability.

The assessment reviews research quality, relevance to society and viability (the “extent to which the organization is equipped for the future”).

For readers of this blog the relevance to society will be of greatest interest. Go straight to Appendix D Table D1 which provides a selection (not an exhaustive list) of indicators for societal impact:

Demonstrable products: reports (for example for policymaking); articles in professional journals for non-academic readers; instruments, infrastructure, datasets, software tools or designs that the unit has developed) for societal target groups; outreach activities, for example lectures for general audiences and exhibitions.

Use of products: Patents/licences: use of research facilities by societal parties; projects in cooperation with societal parties; contract research

Marks of recognition: public prizes; valorisation funding; number of appointments/positions paid for by societal parties; membership of civil society advisory bodies

The SEP submissions are reviewed by committee assessing the narratives of research quality and societal relevance. This is similar to the REF. A significant difference is the committee review happens as a site visit to the submitting unit. This face to face element of the assessment creates greater opportunities for evaluation than an arm’s length committee assessing a submission as in the REF.

What is also similar to the REF and the Australian pilot is that the method and the indicators are predicated on the academic research institution describing the impact of the research. But we know that it isn’t the researchers who are making the products, developing the policies or delivering the services that have an impact. Research partners from the private, public and non-profit sectors make the products, policies and services are the ones making the impact. Yet we ask the research institution to step in and tell someone else’s story of impact. That’s ok so long as the indicators come from the non-academic partners; however, the indicators in the SEP all of which are academic centric.

How long before Canada jumps on the research impact assessment (RIA) bandwagon? Alberta Innovates is implementing the Canadian Academies of Health Sciences’ RIA framework. The co-produced pathway to impact is being implemented by some of the Networks of Centres of Excellence including Kids Brain Health Network, MEOPAR, AllerGen, Cell Can and PREVNet who helped conceptualize the pathway. However, these are pathways that help to guide the progress from research to impact. They are not research impact assessment protocols.

Research Impact Canada is undertaking an RIA pilot which we riffed off the REF as explained in Mobilize This! on April 12, 2017. We have used our RIA tool on one example of impact from York’s Knowledge Mobilization Unit. Based on that experience we revised the interview questions we derived from Sarah Morton’scontribution analysis. We are revising the guidelines and will develop it as an RIA tool that can be used along your pathway to impact, not just ex post research impact assessment (at the end).

When Canada is ready for a national impact assessment process we will be ready with a validated tool. But Canada, please call us first. Let us help you develop a Canadian research impact assessment exercise.

Leave a Comment