Sivertsen, G. and Meijer, I. (2020) Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?, Research Evaluation, 29(1): 66-70, https://doi.org/10.1093/reseval/rvz032
Abstract
Societal impact of research does not occur primarily as unexpected extraordinary incidents of particularly useful breakthroughs in science. It is more often a result of normal everyday interactions between organizations that need to create, exchange, and make use of new knowledge to further their goals. We use the distinctions between normal and extraordinary societal impact and between organizational- and individual-level activities and responsibilities to discuss how science–society relations can better be understood, evaluated, and improved by focusing on the organizations that typically interact in a specific domain of research.
This paper took a while to sink in and realize the difference between normal and extraordinary impacts. There are definitions, of course
- We define normal societal impact as the results of active, productive, and responsible interactions between (units of) research organizations and other organizations according to their purposes and aims in society.
- In contrast to normal impact, we define extraordinary societal impact as more rare incidences where traditional and typical or new and untypical interactions between science and society have unexpected widespread positive or negative implications for society.
Let me add some translation
- Normal impact should happen as a matter of “normal” interactions between individuals but primarily organizations on both sides of the research producer/research user relationship.
- Extraordinary impact rarely occurs and is more a result of unintended accidents and is predicated on the action of individuals rather than organizations.
Got it yet? No? Keep at it because as these become unpacked some interesting observations arise.
- REF (ref.ac.uk) [for those new to this space this is the UK research excellence and impact assessment exercise] tends to pick up extraordinary impacts and is, by design, focused on one side (the researcher) of the researcher/user interaction.
- This approach (normal impact) could provide evaluations to learn from. As a bonus, the four problems mentioned above (causality, attribution, internationality, and time scale) will become less important for the analysis.
Now #2 is interesting. A researcher normally interacting with a research user reduces the effects of causality, attribution, internationality (only works if the two are in different countries) and time scale. I see this in our practice at KMb York where we support co-produced research as our primary method of knowledge mobilization. When a researcher and research users “normally” work together any impact is caused by the research, is attributable to the research and the research user is embedded and able to quickly take up the research evidence and implement it into policies, practices and services that ultimately have an impact on the lives of end beneficiaries.
This paper recommends that impact assessment should include formative and summative elements. “If the purpose of an evaluation is to be formative more than summative (not only assuring value for money but improve by learning from experience and advice), and societal impact is studied as an interaction, both sides of the interaction should be able to learn from the evaluation.” REF is summative and is only interested in a description of impacts and the evidence that links (extraordinary) impacts (according to this paper) to the research. The paper concludes with four recommendations
- Focus on normal impact rather than extraordinary impact, explore how to better organize activities that support daily interactions
- Focus on relationships and interactions not the ultimate impact (for more on this check out outcome mapping as a method)
- Understand that different fields of research will have different purposes for interactions
- Apply an organization level perspective. “An organizational-level evaluation may focus on how well the systematic interaction is taken care of in the strategies, infrastructures, management, incentives and rewards, and daily life of the organizations”
Questions for brokers
- How would you change REF to assess the organizational supports for normal impact?
- How do you get normal vs extraordinary impact? Can you explain it to a non-specialist friend?
- Attribution…is it such a big deal? For more info check here.
Research Impact Canada is producing this journal club series to make evidence on KMb more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.