ISRIA Statement: Ten-Point Guidelines for an Effective Process of Research Impact Assessment

Adam, P., Ovseiko, P.V., Grant, J., Graham, K.E.A., Boukhris, O.F., Dowd, A-M., Balling, G.V., Christensen, R.N., Pollitt, A., Taylor, M., Sued, O., Hinrichs-Krapels, S., Solans‐Domènech, M., Chorzempa, H., & the International School on Research Impact Assessment (ISRIA). (2018). ISRIA statement: Ten-point guidelines for an effective process of research impact assessment. Health Research Policy and Systems, 16:8, 1-16. https://doi.org/10.1186/s12961-018-0281-5

Abstract

As governments, funding agencies and research organisations worldwide seek to maximise both the financial and non-financial returns on investment in research, the way the research process is organised and funded is becoming increasingly under scrutiny. There are growing demands and aspirations to measure research impact (beyond academic publications), to understand how science works, and to optimise its societal and economic impact. In response, a multidisciplinary practice called research impact assessment is rapidly developing. Given that the practice is still in its formative stage, systematised recommendations or accepted standards for practitioners (such as funders and those responsible for managing research projects) across countries or disciplines to guide research impact assessment are not yet available.

In this statement, we propose initial guidelines for a rigorous and effective process of research impact assessment applicable to all research disciplines and oriented towards practice. This statement systematises expert knowledge and practitioner experience from designing and delivering the International School on Research Impact Assessment (ISRIA). It brings together insights from over 450 experts and practitioners from 34 countries, who participated in the school during its 5-year run (from 2013 to 2017) and shares a set of core values from the school’s learning programme. These insights are distilled into ten-point guidelines, which relate to (1) context, (2) purpose, (3) stakeholders’ needs, (4) stakeholder engagement, (5) conceptual frameworks, (6) methods and data sources, (7) indicators and metrics, (8) ethics and conflicts of interest, (9) communication, and (10) community of practice.

The guidelines can help practitioners improve and standardise the process of research impact assessment, but they are by no means exhaustive and require evaluation and continuous improvement. The prima facie effectiveness of the guidelines is based on the systematised expert and practitioner knowledge of the school’s faculty and participants derived from their practical experience and research evidence. The current knowledge base has gaps in terms of the geographical and scientific discipline as well as stakeholder coverage and representation. The guidelines can be further strengthened through evaluation and continuous improvement by the global research impact assessment community.

This is another piece from Kathryn Graham at Alberta Innovates and this time with her international colleagues from the International School on Research Impact Assessment (ISRIA) that ran from 2013 to 2017. Other journal club posts from this group include a post on Alberta’s implementation of the CAHS Impact Assessment Framework and its more recent evolution.

The consensus statement in this article arises from the collective experience of +450 participants in ISRIA events from 34 countries. It provides 10 steps to guide development and implementation of research impact assessment exercises.

Very helpful is a brief analysis of the state of the art of impact assessment in: Europe, UK, Spain, Netherlands, US, Australia, Canada and references to examples in Guatemala, Hong Kong, Argentina, Brazil, Qatar, Iran and Indonesia.

The 10 principles of research impact assessment are:

1.     Analyse your context: Context analysis helps understand the internal and external environment in which research takes place and is being assessed.

2.     Reflect continuously on your purposes: Continuous reflection on the purposes of RIA and one’s relationship to the research being assessed helps refine the assessment questions and methodology. The 4A purposes of RIA include advocacy, accountability, analysis and allocation (there is a helpful description of each of these four in the article)

3.     Identify stakeholders and their needs: Identifying and analysing stakeholders and their needs helps prioritise stakeholder interests, develop engagement strategies and determine RIA requirements.

4.     Engage with key stakeholders early on: Engaging with stakeholders early and throughout the process of RIA can help ensure the social robustness of RIA and make real advances in how science is shaped.

5.     Choose conceptual frameworks critically: Conceptual frameworks can support RIA by reducing the complexity of the phenomenon under investigation for the purposes of data collection, organisation and analysis.

6.     Use mixed methods and multi-data sources: RIA is best approached using a combination of mixed methods and a variety of data sources.

7.     Select indicators and metrics responsibly: The misuse of quantitative indicators and metrics can lead to gaming and unintended negative results.

8.     Anticipate and address ethical issues and conflicts of interest: Anticipating and addressing such ethical issues and conflicts of interests can help maximise the social value of RIA.

9.     Communicate results through multiple channels: A comprehensive and diversified communication strategy can facilitate effective translation of RIA results into practice.

10.  Share your learning with the RIA community: For RIA to continue developing its methods and grow its evidence base, it is important for scholars and practitioners to share their learning with the RIA community of practice.

New to me, there is an interesting matrix for stakeholder analysis (#4 step), the Mendelow Matrix. It helps identify how to engage with each category of stakeholder based on their interest and power.

 

Questions for brokers:

1. As mentioned in a blog post from 2015, planning your knowledge translation is similar to planning for impact assessment. In fact, KT planning is ex ante research impact assessment. Look at the 10 elements of the ISRIA consensus statement. How many of these also apply to KT/impact planning?

2. Frameworks: which framework are you using in your impact planning and why? See a previous journal club about citing a framework but not actually using it.

3. Review the Mendelow Matrix and consider where your stakeholders are aligned against power and interest. Why are you spending effort on group A: those with little interest and little power?

Research Impact Canada (RIC) is producing this journal club series as a way to make evidence on KMb more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.