Harland, K. and O’Connor, H. (2015) Broadening the Scope of Impact. http://www.smalladvancedeconomies.org/wp-content/uploads/SAEI_Impact-Framework_Feb_2015_Issue2.pdf
Forward (this report does not have an abstract)
This report is prepared by the Science and Innovation group of the Small Advanced Economies Initiative (SAEI). The SAEI brings officials and experts from New Zealand, Israel, Singapore, Denmark, Finland, and Ireland together to consider policy issues of common interest where the perspective of small size influences policy choices. The initiative, which was started by New Zealand in 2012, operates under three broad streams: science and innovation, economics, and global issues affecting small advanced economies. The Secretariat for the science and innovation stream and for coordinating the entire initiative is housed in the Office of the Chief Science Advisor to the Prime Minister of New Zealand.
The work presented here has been jointly led by Science Foundation Ireland and the SAEI Secretariat and has the following aims and objectives:
- To provide a common language for the concepts surrounding impact and impact assessment across countries
- Sharing lessons in implementation of impact evaluation, particularly at the allocation stage
- Enabling improvements in the future analysis of impact through ideas for tools and metrics to gather better information across all impact areas
- Providing opportunity for the use of common metrics to enable cross-country comparisons in future analyses
There is much in this report especially since it is from the Small Advanced Economies Initiative (SAEI): New Zealand, Ireland, Israel, Denmark, Finland and Singapore. I will focus on the thinking about ex ante research impact assessment, but some other gems presented:
- Definitions of impact: scholarly yes, but how do you define impact beyond scholarship? See Figure 1.6 for classifications of different types of impact
- How does the level of research impact assessment (individual, team, system) affect your methods and feasibility of assessment?
- Outcomes vs impacts: especially the important moments of dissemination and uptake – a model very similar to the co-produced pathway to impact (CPPI) which we developed in 2014 and published in 2016, a case of convergent evolution!
- The purpose of your assessment: accountability, advocacy, analysis, allocation
- Challenges of impact assessment: attribution, time scale, non-linear (a challenge of the figure and accompanying Box 1 which the CPPI addresses)
And now to the presentation of ex ante impact assessment. See an earlier post where I conclude that planning for impact (ie in a grant application) is ex ante research impact assessment. By developing an impact strategy for a grant application, you are conducting an ex ante (= at the beginning vs ex post = at the end) impact assessment. This report digs into this connection between impact planning and impact assessment in much greater detail and illustrates that with examples from the SAEI countries.
Heads up research administrators:
From the perspective of assessing impact strategies in multidisciplinary grant applications of 4-10 years and with budgets of 1-10M Euros per year the article presents perspectives on the following:
- How do countries select the themes of the grant program – consulting end users to determine research priorities is a key part of the CPPI?
- When should assessment of impact occur: at the proposal selection stage (ie in the application) as well as throughout the conduct of the funded research program?
- Does ex ante impact assessment happen in concert with, in parallel to or after the assessment of research excellence?
- What counts as evidence for impact strategies and impact track records?
Check this against the last impact strategy you wrote or reviewed:
An impact statement will at a minimum demonstrate that applicants have awareness of a pathway to achieving impact, including the following:
- demonstrated understanding of the beneficiaries and users of their research
- understanding of how to engage with and transfer knowledge to these beneficiaries, or how to progress the research to the next stage of development
- commitment to maximising the impact of their proposed research for the benefit of the economy and society
- awareness of potential timescales at a high level
Another example of convergent evolution: see this link for a tool we developed a couple of years ago for research administrators to guide their assessment of impact strategies (sorry…their ex ante impact assessment plans!) in grant applications.
Finally, see the table on page 24 for options on who should review impact strategies: an oversight board, scientific experts or impact experts. One guess who I think should be reviewing…
Questions for brokers:
- Will you write an impact strategy (i.e. undertake ex ante impact assessment) differently if it is to be reviewed in concert with, in parallel to or after the assessment of research excellence?
- What are the strengths and weaknesses of the different categories of impact assessors (table on page 24)?
- Review the CPPI. What does it offer to the model of outcomes and impacts presented in this report?
Research Impact Canada is producing this journal club series as a way to make evidence on KMb more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments