Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014

Reichard, B., Reed, M.S., Chubb, J. et al. Writing impact case studies: a comparative study of high-scoring and low-scoring case studies from REF2014. Palgrave Commun 6, 31 (2020). https://doi.org/10.1057/s41599-020-0394-7

Abstract

This paper reports on two studies that used qualitative thematic and quantitative linguistic analysis, respectively, to assess the content and language of the largest ever sample of graded research impact case studies, from the UK Research Excellence Framework 2014 (REF). The paper provides the first empirical evidence across disciplinary main panels of statistically significant linguistic differences between high- versus low-scoring case studies, suggesting that implicit rules linked to written style may have contributed to scores alongside the published criteria on the significance, reach and attribution of impact. High-scoring case studies were more likely to provide specific and high-magnitude articulations of significance and reach than low-scoring cases. High-scoring case studies contained attributional phrases which were more likely to attribute research and/or pathways to impact, and they were written more coherently (containing more explicit causal connections between ideas and more logical connectives) than low-scoring cases. High-scoring case studies appear to have conformed to a distinctive new genre of writing, which was clear and direct, and often simplified in its representation of causality between research and impact, and less likely to contain expressions of uncertainty than typically associated with academic writing. High-scoring case studies in two Main Panels were significantly easier to read than low-scoring cases on the Flesch Reading Ease measure, although both high-scoring and low-scoring cases tended to be of “graduate” reading difficulty. The findings of our work enable impact case study authors to better understand the genre and make content and language choices that communicate their impact as effectively as possible. While directly relevant to the assessment of impact in the UK’s Research Excellence Framework, the work also provides insights of relevance to institutions internationally who are designing evaluation frameworks for research impact.

Sometimes we need research to tell us what we already know and should already be doing. But judging by this analysis of REF 2014 case studies it is clear that some who are in charge of writing these case studies are not practicing what the rest of us already know. Some of their conclusions:

  1. Highly rated case studies provided specific, high magnitude and well-evidenced articulations of significance and reach.
    • Canada doesn’t use evidence as a verb
    • “where reach was more limited geographically, many high scoring case studies used context to create robust arguments that their reach was impressive in that context” (= excellent impact can be local)
  2. Highly rated case studies used distinct features to establish links between research (cause) and impact (effect).
    • Accomplished by using attributional phrases such as cited in, used to and resulting in.
    • One gap is the lack of knowledge mobilization efforts to create the links between research and impact. Attesting and providing evidence that impact has occurred is necessary. Adding how it occurred will create greater causal links between research and impact. Research Impact Canada has overcome this limitation with an adaptation of the REF case studies. See the RIC guidelines, interview guide and case study guidelines in English and French (forthcoming in a book chapter).
  3. Highly rated case studies were easy to understand and well written.
    • Well, of course
  4. Highly rated case studies were more likely to describe under-pinning research fundings rather than research processes.
    • Just like the REF guidance says
  5. One nit I will pick is the authors state that this is a new genre of writing defined as “clear and direct, and often simplified”…. I will point to early work we did on writing clear language ResearchSnapshots of published articles. Clear writing is certainly a genre, but it isn’t new.

Questions for brokers:

  1. Evidence is a noun not a verb: discuss.
  2. Did any of these findings surprise you?
  3. What will you do to ensure you employ the four conclusions in your next impact case study?

Research Impact Canada is producing this journal club series to make evidence on knowledge mobilization more accessible to knowledge brokers and to create online discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.

© Research Impact Canada. All rights reserved. No part of this site may be reproduced without our written permission.