Belcher, B. and Palenberg, M. (2018) Outcomes and Impacts of Development Interventions: Toward Conceptual Clarity. Am J. Eval. 39(4) 478-495. https://journals.sagepub.com/doi/10.1177/1098214018765698
Abstract
The terms “outcome” and “impact” are ubiquitous in evaluation discourse. However, there are many competing definitions that lack clarity and consistency and sometimes represent fundamentally different meanings. This leads to profound confusion, undermines efforts to improve learning and accountability, and represents a challenge for the evaluation profession. This article investigates how the terms are defined and understood by different institutions and communities. It systematically investigates representative sets of definitions, analyzing them to identify 16 distinct defining elements. This framework is then used to compare definitions and assess their usefulness and limitations. Based on this assessment, the article proposes a remedy in three parts: applying good definition practice in future definition updates, differentiating causal perspectives and using appropriate causal language, and employing meaningful qualifiers when using the terms outcome and impact. The article draws on definitions used in international development, but its findings also apply to domestic public sector policies and interventions.
I am on record as being happy to leave definitions to others. I am busy enough doing the job of impact and rely on academics who have the mandate to pick apart the minutiae of definitions. I resist getting stuck in definitional dystopia. That’s why I am pleased that Brian Belcher and Markus Palenberg care more than I do about definitions. Don’t get me wrong. I appreciate the need for definitional clarity. I also appreciate that someone else will do the work.
The paper does not give a “once and for all” definition of outcomes and impacts. Rather the paper reviews many definitions from multinational bodies such as UN agencies, OECD and some national bodies such as CIDA (not Global Affairs Canada). The paper identifies elements that are present in these definitions and make recommendations for the rest of us as we work with these terms.
An important departure point for definitions are those that start from the “intervention” and map forward (the intervention perspective) and those that start from the observed change and map backwards (the system perspective). This is consistent with Sandra Nutley’s work from Research Unit for Research Utilization who wrote in Using Evidence (2007) about being able to forward track and backwards track impact assessments.
One of the main ambiguities in the definitions of outcomes and impacts is the temporal nature and the causal nature. Must outputs precede outcomes and must there be outcomes before impacts? And must these happen sequentially in short, medium and long terms? There is not consensus among the definitions reviewed.
One important distinction in the definitions from Results4Development is, In R4D, “results language needs to cover both the research and the development domain, and the terms outcome and impact have their own meaning in each. Since research is not a core element in other international development schemes. For those of us working in research impact we need to consider both scholarly and broader (ie socioeconomic) impacts.
The authors provide advice when considering your own definitions:
- Only use defining elements that add clarity
- Do not use defining elements for which all values are possible
- Clearly differentiate outcomes and impact from each other and clearly define them in relation to other terms
Another piece of advice is relevant to research impact planning (=ex ante impact assessment). “We find that describing functional types of results in definitions holds much potential. In this case, the type of change is characterized concretely, which greatly enhances specificity. This can be done by developing a theory of change with key expected/intended outcomes, including reasonable end-of-program outcomes, that is, outcomes that could reasonably be expected to occur within the time and resource constraints of the intervention under consideration.”
But in the meantime, the best I can do is consider their advice about a theory of change that describes the “stuff” one expects to happen along the way as well as at the end.
Questions for brokers:
- Has this article helped your practice or are you waiting – as am I – for definitive definitions?
- Has lack of definitive definitions impeded your practice? I don’t feel it has impeded mine but then I am not the one trying to define the impacts of my research.
- Do you think outputs must precede outcomes and those outcomes must precede impacts? Check out this post that says “it depends”.
Research Impact Canada is producing this journal club series as a way to make evidence on KMb more accessible to knowledge brokers and to create on line discussion about research on knowledge mobilization. It is designed for knowledge brokers and other knowledge mobilization stakeholders. Read this open access article. Then come back to this post and join the journal club by posting your comments