Assessing the Impact of Healthcare Research: A Systematic Review of Methodological Frameworks

Cruz Rivera, S., Kyte, D.G., Aiyegbusi, O.L., Keeley, T.J., & Calvert, M.J. (2017). Assessing the impact of healthcare research: A systematic review of methodological frameworks. PLoS Medicine 14(8): e1002370. https://doi.org/10.1371/journal.pmed.1002370

Abstract

Background: Increasingly, researchers need to demonstrate the impact of their research to their sponsors, funders, and fellow academics. However, the most appropriate way of measuring the impact of healthcare research is subject to debate. We aimed to identify the existing methodological frameworks used to measure healthcare research impact and to summarise the common themes and metrics in an impact matrix.

Methods and findings: Two independent investigators systematically searched the Medical Literature Analysis and Retrieval System Online (MEDLINE), the Excerpta Medica Database (EMBASE), the Cumulative Index to Nursing and Allied Health Literature (CINAHL+), the Health Management Information Consortium, and the Journal of Research Evaluation from inception until May 2017 for publications that presented a methodological framework for research impact. We then summarised the common concepts and themes across methodological frameworks and identified the metrics used to evaluate differing forms of impact. Twenty-four unique methodological frameworks were identified, addressing 5 broad categories of impact: (1) ‘primary research-related impact’, (2) ‘influence on policy making’, (3) ‘health and health systems impact’, (4) ‘health-related and societal impact’, and (5) ‘broader economic impact’. These categories were subdivided into 16 common impact subgroups. Authors of the included publications proposed 80 different metrics aimed at measuring impact in these areas. The main limitation of the study was the potential exclusion of relevant articles, as a consequence of the poor indexing of the databases searched.

Conclusions: The measurement of research impact is an essential exercise to help direct the allocation of limited research resources, to maximise research benefit, and to help minimise research waste. This review provides a collective summary of existing methodological frameworks for research impact, which funders may use to inform the measurement of research impact and researchers may use to inform study design decisions aimed at maximising the short-, medium-, and long-term impact of their research.

I usually present a journal club article that can help our knowledge mobilization practices. I’m not certain what this paper adds. It is a review of impact frameworks for healthcare research. As the abstract says 24 unique frameworks were identified for five categories of impact and 80 different metrics. The authors state, “whilst previous researchers have summarised existing methodological frameworks and impact case studies they have not summarised the metrics for use by researchers, funders, and policy makers.” And they claim to “provide a simplified consolidated resource”.

The 24 frameworks might be “unique” but they are not distinct. Seven of the 24 frameworks are based on the payback method. And as the authors point out, “the Payback Framework drew upon a literature review and was refined through a case study approach. Arguably, this approach could be considered inferior to other methods that involved extensive stakeholder involvement.” So, is this a critique of all seven frameworks even if they subsequently engaged stakeholders?

Also, I wouldn’t call 80 metrics simplified. It is useful to have these 80 metrics named but the paper gives no guidance on how or when to use them. And while all 24 frameworks are compared along several parameters in table 1 and figure 2 the authors don’t draw much in the way of conclusions. Impact practitioners need more from impact researchers. It would have been nice to know those conditions/situations which demand one framework over another.

And finally, there are contradictions in Figure 4 which shows a “pathways to research impact diagram”. This pathway is depicted longitudinally with short term impacts at the left and broader economic impacts on the right. Makes sense except that the patents granted are depicted as broader socioeconomic impacts and are absent as a dissemination event under short term impacts. A granted patent does nothing more than disseminate the information in the patent. Only when a company sells product based on that patent is there an economic impact. “Collaborative research with industry” is identified as a mid term outcome but “research contracts and income from industry” are also mentioned in socioeconomic impacts. Which is it?

The paper may be useful background for those writing other research articles but not so much for practitioners.

Questions for brokers:

  1. Why is an issued patent considered by many as a metric of innovation when all it does is disseminate the invention.?
  2. When does innovation occur: when something is invented or when something is produced and sold?
  3. What are you looking for from research on impact frameworks?

Research Impact Canada (RIC) is producing this journal club series as a way to make evidence on KMb more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments.