Governing by narratives: REF impact case studies and restrictive storytelling in performance measurement

Governing by narratives: REF impact case studies and restrictive storytelling in performance measurement

Bandola-Gill, J. and Smith, K.E. (2021). Governing by narratives: REF impact case studies and restrictive storytelling in performance measurement, Studies in Higher Education, https://doi.org/10.1080/03075079.2021.1978965

ABSTRACT

Performance assessment is permeating increasingly diverse domains of higher education, even in areas previously perceived to be too complex and idiosyncratic to quantify. The UK’s attempts to assess ‘research impact’ within the Research Excellence Framework (REF) are illustrative of this trend and are being closely monitored by several other countries. A fundamental rationale for employing narrative case studies to assess impact within REF, rather than taking a (less resource intensive) quantified approach, was that this would allow for the variation, complexity and idiosyncrasy inherent in research impact. This paper considers whether this promise of narrative flexibility has been realised, by analysing a combination of REF impact case study reports and interviews and focus group discussions with actors involved in case study production. Informed by this analysis, our central argument is that the very quality which allows narratives to govern is their ability to standardise performance (albeit whilst retaining a degree of flexibility). The paper proposes that REF impact case studies position narratives of impact as technologies of governance in ways that restrict the ‘plot line’ and belie the far more complex accounts held by those working to achieve research impact. This is partly because, as research impact becomes institutionalised within universities’ measurement infrastructures, higher education institutions become impact gatekeepers, filtering out narratives that are deemed overly complex or insufficiently persuasive, while perpetuating particular approaches to recounting tales of impact that are deemed likely to perform well. Crucially, these narratives not only describe impact but actively construct it as an auditable phenomenon.

This article has me thinking a lot, so this journal club post is longer than the usual 450 words.

This article is mainly critiquing the use of narrative case studies in the UK Research Excellence Framework (REF), how they were chosen to not constrain the creative story telling of impact but have evolved into a management/evaluation tool that does exactly the opposite. It constrains descriptions of impact into four archetypes with standardized narrative features (plot, moral and heroes), privileges impact narratives with a quantitative (especially financial) conclusions and leads to the exclusion of stories of impact that do not easily conform.

The four impact archetypes are:

1. Problem-solving: Research is offering solutions to existing policy and practice problems
2. Tool building: Research is providing specific tools to support policy and practice
3. Reframing: Research is reframing existing account of problems and/or solutions
4. Public engagement: Research is facilitating learning and deliberation across various stakeholders

But these four are not equally weighted. Public engagement was seen by institutions to be the least compelling story to tell so impact case studies featuring public engagement were not selected by institutions for submission as often because they were harder to fit into the model of a successful REF case study. There is “widespread uncertainty among academics and impact assessors about the relationship between public engagement and research impact, and a sense that (with some discipline specific exceptions) focusing on public engagement is a risky route to impact.” So much for the creative arts, humanities and some fundamental sciences. This illustrates that there is societal impact that is not easily captured in the REF, see #1 below.

Some implications from this paper

1. REF impact ≠ societal impact. Institutions choose types of impact that can fit easily into REF style. “It was notable that several of the academics we interviewed differentiated between meaningful research impact and ‘REF impact’, which was positioned as impact narrated for a bureaucratic exercise.” I will come back to this REF impact ≠ societal impact repeatedly.
2. In REF impact case studies, the academic is the central character in the creation of impact. This essentially is a lie. Most impact happens distal to the research and is supported by industry making a new/improved product, government developing a better public policy or community organizations delivering social services.
   • This marginalizes the contributions to the creation of impact by research partners, stakeholders and people with lived/living experiences.
   • Sidebar just to poke the bear: the “impact” of educational research in the classroom or nursing research in the clinic (as examples) is more proximal to the research but the students or patients are actually data in the published study. And while they have received benefit from being research participants, the research is not describing broader societal impacts arising from the educational or nursing practice being taken up and implemented beyond the individual study. These may be eligible as REF impacts but are they societal impacts (see #1)?
3. The structure of the REF case study and the need to conform to an easily documented plot creates artificial links of causality from research to impact that do not reflect the messy reality of impact. “This assumption of causality was in striking contradiction to the way participants perceived realities of achieving impact. Regardless of the target for impact (e.g. policy, industry, culture or society), routes to achieving impact were consistently articulated as being more complex, unpredictable and serendipitous than REF case studies suggest.” As in #1 there is REF impact and then there is broader societal impact.
   • How can you document serendipity or report on the “impact of accident”?
4. Institutions are the ones limiting the impact in case studies and they are responding to REF as “they adapt their strategies, communication styles and internal structures to respond to external performance assessment systems” including considering impact in career progression, management structures and creating impact offices to help construct impact case studies.
   • This latter observation has a corollary that these impact offices are not supporting the creation of broader societal impact, but they are documenting REF impact. See #1.
   • Institutions choose which case studies to submit to the REF. They choose the case studies that best fit the model (plot, hero, moral with academic as hero) and then they support impact in those case studies including investing funds to accelerate impact. They are supporting REF impact not broader societal impacts. See #1.
5. “In impact case studies, this seemed to require proof of achieving a specific (always positive) change on the basis of research.” Impacts are always positive in REF? What about in real life where academic research has left legacies of trauma for Indigenous people (for example)? Try submitting that to a REF impact panel!

Is supporting REF impact but nor broader societal impact the same thing as REF promoting extraordinary impacts at the expense of ordinary impacts? See this earlier journal club post.

One quote from a respondent speaks to risks of vulnerability and marginalization in institutions administering the REF. “[We] had meetings and workshops and whatever where you could go along to learn how to write your impact assessment, and then they selected from the submitted impact assessments. […] It was used as a lever of power and influence by our administrators.” This paper does not reflect more deeply but I have heard that traditionally marginalized groups of researchers, particularly women, are under-represented in case studies chosen by institutions for submission. For more on this see this post on the LSE blog and a paper in Social Sciences & Humanities Communications.

Questions for brokers:

1. There is evidence that institutions submitting to the REF recapitulate systemic marginalization of female scholars. Do you think the same might hold true for scholars from other equity seeking groups including racialized, disabled, Indigenous and LGBTQ+ scholars?
2. Winston Churchill is quoted as saying “democracy is the worst form of Government except for all those other forms that have been tried from time to time”. Is REF the impact assessment equivalent of democracy?
3. Poke the bear: While REF-able, the subjects of applied research who receive positive benefits from their participation in research projects such as clinical trials are only beneficiaries of research not examples of societal impact. Discuss.
4. REF impacts are always positive impacts. Is this another example of how REF impact ≠ societal impacts which may be positive or negative?

Research Impact Canada is producing this journal club series to make evidence on KMb more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.