McLean, R.K.D., Carden, F., Aiken, A.B. et al. Evaluating the quality of research co-production: Research Quality Plus for Co-Production (RQ + 4 Co-Pro). Health Res Policy Sys 21, 51 (2023). https://doi.org/10.1186/s12961-023-00990-y
Abstract
Background: Co-production is an umbrella term used to describe the process of generating knowledge through partnerships between researchers and those who will use or benefit from research. Multiple advantages of research co-production have been hypothesized, and in some cases documented, in both the academic and practice record. However, there are significant gaps in understanding how to evaluate the quality of co-production. This gap in rigorous evaluation undermines the potential of both co-production and co-producers.
Methods: This research tests the relevance and utility of a novel evaluation framework: Research Quality Plus for Co-Production (RQ + 4 Co-Pro). Following a co-production approach ourselves, our team collaborated to develop study objectives, questions, analysis, and results sharing strategies. We used a dyadic field-test design to execute RQ + 4 Co-Pro evaluations amongst 18 independently recruited subject matter experts. We used standardized reporting templates and qualitative interviews to collect data from field-test participants, and thematic assessment and deliberative dialogue for analysis. Main limitations include that field-test participation included only health research projects and health researchers and this will limit perspective included in the study, and, that our own co-production team does not include all potential perspectives that may add value to this work.
Results: The field test surfaced strong support for the relevance and utility of RQ + 4 Co-Pro as an evaluation approach and framework. Research participants shared opportunities for fine-tuning language and criteria within the prototype version, but also, for alternative uses and users of RQ + 4 Co-Pro. All research participants suggested RQ + 4 Co-Pro offered an opportunity for improving how co-production is evaluated and advanced. This facilitated our revision and publication herein of a field-tested RQ + 4 Co-Pro Framework and Assessment Instrument.
Conclusion: Evaluation is necessary for understanding and improving co-production, and, for ensuring co-production delivers on its promise of better health.. RQ + 4 Co-Pro provides a practical evaluation approach and framework that we invite co-producers and stewards of co-production—including the funders, publishers, and universities who increasingly encourage socially relevant research—to study, adapt, and apply.
Heads up this article is about evaluating research co-production quality. It is not about evaluating research impact although my guess is that doing good research co-production will produce good research impacts.
This article presents a new evaluation framework for co-produced research. The project itself is co-produced with academic authors joined by non-academic co-authors from funders, governments, health authorities, insurance agency and one consultant. Bravo.
The authors felt a new evaluation method was needed as there is lots of theory and practice-based evidence but a lack of empirical research into evaluation for co-production. They worked from the Research Quality + method which has already been covered in this journal club.
“The RQ + approach introduced three tenets for holistic research quality evaluation, and we use these tenets as our conceptual building blocks. These tenets include: (1) context matters for any evaluation of research, (2) research quality is a multi-dimensional, values-driven concept, and (3) evaluations of research must be empirical and systematic, not only based on peer opinion.”
Figure 2 shows the adaptation to RQ+4CoPro which was previously published including a rubric for the three contextual factors (the “tenets” above) and quality dimensions and sub-dimensions. Unfortunately, they are not linked to this article so you’ll have to dig into the references to find it (hint…it’s here). This study was a field test of the RQ+4CoPro tool. They identified 18 dyads where each dyad member had a co-produced research project. Each member of the dyad used the tool to evaluate the other dyad member’s project. The authors then used interviews to assess the evaluators’ opinions of the tool.
One thing to keep in mind: the average time to conduct the evaluation was 3.91 hours. This is not a quick and dirty evaluation.
Some conclusions from the evaluation of the evaluators
- “There was a unanimous (18/18) response from participants that the RQ + 4 Co-Pro Framework was an important and timely development in the realm of co-production research.
- All participants endorsed RQ+ 4 Co-Pro as relevant for a broad range of research co-production projects.
- All participants reported the Framework was useful for the evaluation of research co-production.”
Finally, the authors recommend using the tool before, during and after the co-produced research project. They present good reasons to do it in all three stages. In addition, doing it before gives you a baseline to compare during and after the project.
Questions for brokers:
- What would RQ+4CoPro look like if it was to evaluate the quality of research impact instead of (as well as?) research quality?
- 3.91 hours: is this long, short or about right? Why?
- Do you have the skills needed to evaluate research and impact or will you need to either build or hire specific evaluation expertise?
Research Impact Canada is producing this journal club series to make evidence on knowledge mobilization more accessible to knowledge brokers and to facilitate discussion about research on knowledge mobilization. It is designed for knowledge brokers and other parties interested in knowledge mobilization.