Abstract:

The basic ideas behind contribution analysis were set out in 2001. Since then, interest in the approach has grown and contribution analysis has been operationalized in different ways. In addition, several reviews of the approach have been published and raise a few concerns. In this article, I clarify several of the key concepts behind contribution analysis, including contributory causes and contribution claims. I discuss the need for reasonably robust theories of change and the use of nested theories of change to unpack complex settings. On contribution claims, I argue the need for causal narratives to arrive at credible claims, the limited role that external causal factors play in arriving at contribution claims, the use of robust theories of change to avoid bias, and the fact that opinions of stakeholders on the contribution made are not central in arriving at contribution claims.

Keywords: causal factors, causal narratives, contribution analysis, contribution claims, contributory causes, theories of change

This ahead of print version may differ slightly from the final published version.

The use of theories of change (ToCs) is a hallmark of sound evaluation practice. As interventions have become more complex, the development of ToCs that adequately unpack this complexity has become more challenging. Equally important is the development of evaluable ToCs, necessary for conducting robust theory-based evaluation approaches such as contribution analysis (CA). This article explores one approach to tackling these challenges through the use of nested actor-based ToCs using the case of an impact evaluation of a complex police-reform program in the Democratic Republic of Congo, describing how evaluable nested actor-based ToCs were built to structure the evaluation.

It is frequently assumed and not contested that evaluation should play a significant role in budgeting and, more specifically, in expenditure reviews. This article argues otherwise: that evaluation is neither fit nor designed to play such a role. Rather, if there is a desire by budget officials for credible evidence on the performance of interventions, then they need to invest in a different form of evaluation, namely, expenditure evaluations, separate and distinct from ministry-based evaluation.

Although theories of change are frequently discussed in the evaluation literature and there is general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what it shows, how it can be represented, and how it can be used. This article outlines models for theories of change and their development that have proven quite useful for both straightforward and more complex interventions. The models are intuitive, flexible, and well-defined in terms of their components, and they link directly to rigorous models of causality. The models provide a structured framework for developing useful theories of change and analyzing the intervention they represent.

Models for theories of change vary widely as do how they are used. What constitutes a good or robust theory of change has not been discussed much. This article sets out and discusses criteria for robust theories of change. As well, it discusses how these criteria can be used to undertake a vigorous assessment of a theory of change. A solid analysis of a theory of change can be extremely useful, both for designing or assessing the designs of an intervention as well as for the design of monitoring regimes and evaluations. The article concludes with a discussion about carrying out a theory of change analysis and an example.

Although theories of change are frequently discussed in the evaluation literature and there is general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what it shows, how it can be represented, and how it can be used. This article outlines models for theories of change and their development that have proven quite useful for both straightforward and more complex interventions. The models are intuitive, fl exible, and well-defined in terms of their components, and they link directly to rigorous models of causality. The models provide a structured framework for developing useful theories of change and analyzing the intervention they represent.

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each contributor provides a personal viewpoint, based on his or her own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that Canadian evaluation has matured in many ways since 2003 (when a first panel scan was conducted): professional designation is a reality; the infrastructure is stronger than ever; organizations are more focused on results. Still, evaluation is weakened by lacunas in advanced education and professional development, limited resources, lack of independence, rigidity in evaluation approaches, and lack of self-assessment. While the demand for evaluation and evaluators appears on the rise, the supply of evaluators and the financial resources to conduct evaluations are not. The collective definition of the field of evaluation still lacks clarity. There is also reassurance in looking toward the future. With increased appetite for evaluation, evaluators could make a real difference, especially if evaluators adopt a more systemic view of program action to offer a global understanding of organizational effectiveness. The implementation of a Certified Evaluator designation by CES is a major opportunity to position evaluation as a more credible discipline.

As many reviews of results-based performance systems have noted, a weak evaluative culture in an organization undermines attempts at building an effective evaluation and/or results management regime. This article sets out what constitutes a strong evaluative culture where information on performance results is deliberately sought in order to learn how to better manage and deliver programs and services. Such an organization values empirical evidence on the results it is seeking to achieve. The article outlines and discusses practical actions that an organization can take to build and support an evaluative culture.

Results, and more particularly outcomes, are at the centre of public management reform in many jurisdictions, including Canada. Managing for outcomes, including setting realistic outcome expectations for programs, and credibly reporting on what was achieved are proving to be challenges, perhaps not unexpectedly, given the challenges faced in evaluating the outcomes of public programs. This article discusses how the use of results chains can assist in setting outcome expectations and in credibly reporting on the outcomes achieved. It introduces the concepts of an evolving results-expectations chart and of telling a performance story built around the program's results chain and expectations chart.