Building a Framework for the Evaluation of Knowledge Translation for the Canadian Network for Observational Drug Effect Studies

Sketris, I.S., Carter, N., Traynor, R.L., Watts, D., & Kelly, K. (2019). Building a framework for the evaluation of knowledge translation for the Canadian Network for Observational Drug Effect Studies. Pharmacoepidemiology and Drug Safety. 1–18. https://doi.org/10.1002/pds.4738

Abstract

Purpose: The Canadian Network for Observational Drug Effect Studies (CNODES), a network of pharmacoepidemiologists and other researchers from seven provincial sites, provides evidence on the benefits and risks of drugs used by Canadians. The Knowledge Translation Team, one of CNODES’ four main teams, evaluates the impact of its efforts using an iterative and emergent approach. This article shares key lessons from early evaluation phases, including identifying stakeholders and their evaluation needs, choosing evaluation theories and approaches, and developing evaluation questions, designs, and methods appropriate for the CNODES context.

Methods: Stakeholder analysis was conducted using documentary analysis to determine key contextual factors and research evidence needs of decision maker partners and other stakeholders. Selected theories and frameworks from the evaluation and knowledge translation literature informed decisions about evaluation design and implementation. A developmental approach to evaluation was deemed appropriate due to the innovative, complex, and ever‐changing context.

Results: A theory of change, logic model, and potential evaluation questions were developed, informed by the stakeholder analysis. Early indicators of program impact (citation metrics, alternative metrics) have been documented; efforts to collect data on additional indicators are ongoing.

Conclusion: A flexible, iterative, and emergent evaluation approach allows the Knowledge Translation Team to apply lessons learned from completed projects to ongoing research projects, adapt its approaches based on stakeholder needs, document successes, and be accountable to funders/stakeholders. This evaluation approach may be useful for other international pharmacoepidemiology research networks planning and implementing evaluations of similarly complex, multistakeholder initiatives that are subject to constant change.

This article describes development and implementation of an evaluation framework for the knowledge translation aims of a multi centre research network providing applied research outputs to decision makers, primarily in governments. The article describes multiple stakeholders and multiple goals using multiple inputs. The article is positioned as a lesson for other similarly complex and distributed research networks.

The article describes the following steps in development of their evaluation framework:

  1. Stakeholder analysis: primarily using document review to understand the context in which CNODES operates
  2. Indicator development: Indicators for implementation and delivery and early outcomes of CNODES’ knowledge translation activities were relatively straightforward to identify and track (citation and altmetrics). Establishing indicators and collecting data on longer term outcomes and broader impacts are less straightforward and require multiple indicators and methods to maximize value for a variety of stakeholders. The take away from this is that quantitative indicators are gathered upstream and qualitative indicators are measured downstream in the evaluation. One thing they didn’t describe is the data system used to store the data gathered as part of quantitative and qualitative data collection.
  3. Knowledge translation logic model
  4. Knowledge translation theory of change
  5. Proof of concept evaluation: assesses the rationale for the model, the key characteristics of the model and the organizational structure of the model

Whatever else you might or might not take away from the article, this sequence of events presents a logical step to follow. To this list I would add a first step: understand the purpose of the evaluation. The authors describe their three-fold purpose but haven’t included it in this stepwise process description.

Here’s one observation: they did stakeholder analysis by document review. The article describes lots of engagement with stakeholders from stakeholders submitting queries, to integrated KT by engaging stakeholders throughout the process and finally disseminating to stakeholders. With such close association with stakeholders why didn’t they engage stakeholders in the stakeholder analysis?

Check out Figure 1 Knowledge Translation Logic Model. It identifies three functions: knowledge production, knowledge mobilization and knowledge translation. Knowledge production I understand but I don’t see the difference between mobilization and translation. I am not one to get stuck in definitional dystopia, but definitions are needed here because the authors are using them distinctly whereas many impact practitioners (including myself) use these interchangeably depending if they are operating in a health (translation) or social science (mobilization) context.

Questions for brokers:

  1. Document driven stakeholder analysis vs stakeholder engaged analysis: describe the strengths and weaknesses of each approach.
  2. Knowledge mobilization vs knowledge translation: do you care? If so, why?
  3. What is the relationship between a logic model and a theory of change?

Research Impact Canada is producing this journal club series as a way to make evidence on KMb more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments