Scarrow, G., Angus, D., & Holmes, B. (2017). Reviewer training to assess knowledge translation in funding applications is long overdue. Research integrity and Peer Review, 2(13), pp. 1-7. https://doi.org/10.1186/s41073-017-0037-8
Abstract
Background: Health research funding agencies are placing a growing focus on knowledge translation (KT) plans, also known as dissemination and implementation (D&I) plans, in grant applications to decrease the gap between what we know from research and what we do in practice, policy, and further research. Historically, review panels have focused on the scientific excellence of applications to determine which should be funded; however, relevance to societal health priorities, the facilitation of evidence-informed practice and policy, or realizing commercialization opportunities all require a different lens.
Discussion: While experts in their respective fields, grant reviewers may lack the competencies to rigorously assess the KT components of applications. Funders of health research—including health charities, non-profit agencies, governments, and foundations—have an obligation to ensure that these components of funding applications are as rigorously evaluated as the scientific components. In this paper, we discuss the need for a more rigorous evaluation of knowledge translation potential by review panels and propose how this may be addressed.
Conclusion: We propose that reviewer training supported in various ways including guidelines and KT expertise on review panels and modalities such as online and face-to-face training will result in the rigorous assessment of all components of funding applications, thus increasing the relevance and use of funded research evidence. An unintended but highly welcome consequence of such training could be higher quality D&I or KT plans in subsequent funding applications from trained reviewers.
I like this article because I spend so much time helping researchers craft impact strategies (knowledge mobilization/translation plans) only to (not infrequently) hear that it was only the scientific and training components of the application that were discussed at review. As this article points out, peer reviewers are usually scientific peers with no training in KT, so they will focus on what they know. However, if one goal (not the only goal) of research funding is to develop new knowledge and translate that knowledge into benefits for society then it is incumbent upon funders to train reviewers to also provide a critical review of the impact strategies as well as the intellectual merit of the applications.
The article points to steps being taken as identified in the literature:
- Providing resources for reviewers such as the Guide for Assessing Health Research Knowledge Translation (KT) Plans, tips for reviewers from past chairs and reviewers of KT applications
- Embedding specific review criteria for KT in funding applications
- Providing an overview of KT as part of peer review panel orientation
- Having KT experts on peer review panels and research users on panels as context experts to assess the merit of proposed research: its relevance to, potential uptake by, and impact on targeted stakeholders
The article then proceeds to critique each of these and, “reviewer training or education in KT, tailored to the specific requirements of the funding program, is critical to ensuring a rigorous review of KT plans and funding of excellent applications, thereby improving the likelihood that research evidence will be used to inform policy and practice.”
One article cited showed that only 9% of reviewers ever received training in grant review – just grant review, not even mentioning review of impact strategies. The same article showed that 64% would take training if it were offered (the remaining 36% clearly not being challenged by a lack of self esteem!). It seems like grant review is something scholars are just expected to know.
The authors point to some considerations about training for KT strategies
- The type of KT being reviewed: dissemination and co-production are different methods and may need distinct training
- The review panel’s level of knowledge and expertise in KT, including the chair: some reviewers are experienced in KT and some aren’t. Training should reflect different stages of KT sophistication
- The resources available: what is optimal and what is feasible are often not the same thing.
In summary, the key informants felt that reviewers do need a solid understanding of what good KT plans should contain and guidelines are not sufficient for this purpose. Training should therefore be required. Training should be delivered just in time and should recognize existing levels of expertise. Having an expert on every panel is an alternative but is not be feasible given the small pool of available experts.
Questions for brokers:
- Have you been a KT/Impact expert on a grant review panel? What were your experiences? Would you do it again?
- If the best funder advice for applicants is generic (i.e. your impact strategy for water management in remote communities will be different than your impact strategy for commercializing a medical device) how can training for review be made more specific for the grant competition?
- Isn’t learning about impact like learning to swim? You can’t learn to swim by watching a webinar. You need to get in the pool (but with a buddy and a life guard). Similarly, can you learn about impact from a webinar or do you actually need to learn by doing?
ResearchImpact-RéseauImpactRecherche is producing this journal club series as a way to make the evidence and research on knowledge mobilization more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read the open access article. Then come back to this post and join the journal club by posting your comments.