Wróblewska, M.N. (2025) One size fits all? A comparative review of policy-making in the area of research impact evaluation in the UK, Poland and Norway, Research Evaluation, Volume 34, 2025, rvaf010, https://doi.org/10.1093/reseval/rvaf010.
Abstract
The Impact Agenda, introduced with Research Excellence Framework 2014 (REF), constituted a revolution in research evaluation in the UK. ‘Research impact’ (impact of scholarly work outside of academia) became one of three profiles under which research quality is evaluated. This shift in the British evaluation system was followed, and often emulated, by policy-makers around the world. Among them are Norway and Poland. In 2015–18, Norway experimented with impact evaluation using an REF-style impact case study model. It took a light-handed approach, not tying the exercise to funding. Poland has copied elements of the REF verbatim, embedding them within an evaluation framework which is linked funding. The article offers a perspective on impact evaluation regulations adopted in the three countries. There are several analogies between them, including definitions of impact, use of case studies as the basis for evaluation, structure of the impact template, use of English as the language of evaluation, and expert/peer review model of evaluation. They differ when it comes to the mode of introduction of the exercise (gradual vs. shift), aims of the exercise, and level of transparency of the policy-making and evaluation process. The main goal of this paper is to provide a comprehensive overview of the three approaches to impact evaluation against the backdrop of the respective broader science systems. It also provides first inroads into two fundamental questions: (1) How does the articulation of research impact change depending on the goals of the exercise and the broader academic and social context; and (2) How do the effects of the exercise differ from one national context to another?
This article looks at impact from a national perspective comparing impact assessment exercises from UK, Norway and Poland. It looks at similarities and differences and explains how the impact assessment policies and procedures evolved. This is of interest to anyone who is thinking about impact assessment in their country. Note to Canadians: we don’t do this. This article will be key learning before we start discussing this. Another interesting article looked at the evolution of impact assessment in UK and Australia and how both depended on each other in an interactive fashion.
The article refers to the Research Excellence Framework (REF) in UK as a protype centralized research impact assessment exercise and describes it as the gold standard. It is neither gold nor standard but it has been around the longest, is most embedded in the country’s research ecosystem and is the one that all other countries look to for inspiration and learning. Read this book if you want a look at some of the “issues” as the impact agenda has arisen in the UK – ie it isn’t golden as I mention above.
There are similarities and differences articulated “There are several analogies between them, including definitions of impact, use of case studies as the basis for evaluation, structure of the impact template, use of English as the language of evaluation, and expert/peer review model of evaluation. They differ when it comes to the mode of introduction of the exercise (gradual vs. shift), aims of the exercise, and level of transparency of the policy-making and evaluation process” However, one key difference I want to focus on is Norway not tying the results to funding.
My opinion, not in the article, is that when you reward an assessment exercise with money (and there is £2B – yes billion invested each year in the UK based on success in REF) unintended consequences will occur such as gaming the system and shifting staff functions. I like the approach Norway takes by doing assessment but not rewarding with money. As the article indicates this is similar to the Netherlands Standard Evaluation Protocol.
Check out Figure 1 for a timeline of the evolution of research impact assessment in the three countries.
One key take away for me is that any assessment needs to be grounded in the local research policy context. “Performance-based research funding systems ‘need to be examined in their national contexts to understand their motivations and design’ and that rather than discus best practice scholars should aim to provide ‘the basis for mutual learning among countries’” I like that there are no “best practices”. It also means it would be hard for a consultant to bail into a country and support this work without being embedded in the research system.
The article also says (well, not quite like this) that assessments should not reward only impact unicorns. “One existing study looking at Norway’s approach to impact evaluation stressed that the exercise, like its British predecessor, focused too much on ‘showcasing extraordinary impact’, hence unusual or participially striking cases of impact, rather than the more typical, ‘normal’ impacts which arise from the simple fact of embeddedness of disciplines in broader society.”
It is so important that assessment doesn’t drive behaviour. Impact needs to happen as part of a healthy research ecosystem. Assessment comes along every few years and describes what impact has happened. It shouldn’t drive the process.
Questions for brokers:
- Are you looking for an impact unicorn? Why should a national impact be any more valuable than a local impact especially if your university is focused on anchoring your local community.
- Money or no money? If not money, why would anyone do this work? Hint: there is no money for impact assessment in Canada and yet there are 45 members in Research Impact Canada all wanting to do it better.
- Canadian brokers: do we need impact assessment? Why or why not?
Research Impact Canada is producing this journal club series to make evidence on KMb more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other people interested in knowledge mobilization. Read this open access article. Then come back to this post and join the journal club by posting your comments on our LinkedIn.