Inglis, M. Gadd and Stokoe, E. (2024) What is a high-quality research environment? Evidence from the UK’s research excellence framework. Research Evaluation, 00:1-16. https://doi.org/10.1093/reseval/rvae010
Abstract
As part of the UK university sector’s performance-related research funding model, the ‘REF’ (Research Excellence Framework), each discipline-derived ‘Unit of Assessment’ must submit a statement to provide information about their environment, culture, and strategy for enabling research and impact. Our aim in this paper is to identify the topics on which these statements focus, and how topic variation predicts funding-relevant research environment quality profiles. Using latent Dirichlet allocation topic modelling, we analysed all 1888 disciplinary ‘unit-level’ environment statements from REF2021. Our model identified eight topics which collectively predicted a surprisingly large proportion—58.9%—of the variance in units’ environment scores, indicating that the way in which statements were written contributed substantially to the perceived quality of a unit’s research environment. Assessing research environments will increase in importance in the next REF exercise and the insights found through our analysis may support reflection and discussion about what it means to have a high-quality research environment.
Heads up, this article has nothing to do with knowledge mobilization and little to do with research impact but it is helpful to expand our scope (haha – see below) once in a while.
This is a very UK focused article since it analyses Research Environment sections of the UK’s Research Excellence Framework (REF). The REF is a system wide assessment of research excellence, impact and environment that happens every 6 (ish) years in the UK. This is critically important to the UK research system as the REF is used, in part, to allocate £2 billion (ish) annually. A university’s success in one REF assessment locks in its funding for the next 6 (ish) years. This is not the place to go down a REF rabbit hole but this huge incentive creates lots of unintended consequences. Moving on….
Anything by Lizzie Gadd and colleagues is worth a read. Check out her work on the SCOPE framework for research assessment. Nothing to do with this article but a valuable contribution to the scholarship and practice of research assessment.
This article is titled “what is a high quality research environment” but it would more accurately be titled “what is a high quality research environment section in REF submissions”. The authors evaluated the environment sections of REF 2021 submissions to analyze the characteristics of successful research environment statements using topic modelling. They downloaded 1888 unit level environment statements and analyzed a total of 18 million words. Much scholarship on REF has been on research quality and research impact so it is good to see some analysis of research environment statements.
Their analysis focused on eight topics: Internal Structure of Research Units, Career Development and EDI, Immature Research Environments, Staff Ways of Working, REF-Focused Research Strategies, Exemplifications of Staff Ways of Working, Industry Partners and Funding, and ECR Development. And then the authors discuss each in detail and draw conclusion on what to do and what not to do when considering drafting an environment submission to the REF. But here’s the thing, “All the analyses we have reported in this paper are correlational in nature…Given this, care must be taken before assuming that the relationships we have reported are causal.”
But so what for those of us who will never draft a research environment section for REF? At some point we will be writing about the research environment at our institutions whether in a strategic research plan or a report to Senate etc so the findings here can help. I also think that the eight topics presented in this article can be things to think about when actually designing a healthy research environment, REF notwithstanding. But be a critical reader. For example, if applying this beyond REF environment sections I would expand industry partners to include all partners but that is a REF success technique not a recommendation to exclude research partnerships with anyone other than industry.
And a quick note: the UK use the word “staff” to mean academic faculty. In Canada university “staff” are understood to be professional staff.
Questions for brokers:
- If you are in the UK how might use this in REF 2029 for the new People, Culture and Environment section?
- If you are not in the UK think about the research environment at your institution and how you might use this article in your next research strategy or report.
- No matter where you are how do the eight topics help you think about your research environment?
Research Impact Canada is producing this journal club series to make evidence on knowledge mobilization more accessible to knowledge brokers and to facilitate discussion about research on knowledge mobilization. It is designed for knowledge brokers and other parties interested in knowledge mobilization.