John Mayne

Special Issue

Using Actor-Based Theories of Change to Conduct Robust Evaluation in Complex Settings

Authors:

This ahead of print version may differ slightly from the final published version.

The use of theories of change (ToCs) is a hallmark of sound evaluation practice. As interventions have become more complex, the development of ToCs that adequately unpack this complexity has become more challenging. Equally important is the development of evaluable ToCs, necessary for conducting robust theory-based evaluation approaches such as contribution analysis (CA). This article explores one approach to tackling these challenges through the use of nested actor-based ToCs using the case of an impact evaluation of a complex police-reform program in the Democratic Republic of Congo, describing how evaluable nested actor-based ToCs were built to structure the evaluation.

Special Issue

Linking Evaluation to Expenditure Reviews: Neither Realistic nor a Good Idea

Authors:
Pages:
316-326

It is frequently assumed and not contested that evaluation should play a significant role in budgeting and, more specifically, in expenditure reviews. This article argues otherwise: that evaluation is neither fit nor designed to play such a role. Rather, if there is a desire by budget officials for credible evidence on the performance of interventions, then they need to invest in a different form of evaluation, namely, expenditure evaluations, separate and distinct from ministry-based evaluation.

Fall

Théories du changement : comment élaborer des modèles utiles

Authors:
Pages:
174-201

Although theories of change are frequently discussed in the evaluation literature and there is general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what it shows, how it can be represented, and how it can be used. This article outlines models for theories of change and their development that have proven quite useful for both straightforward and more complex interventions. The models are intuitive, flexible, and well-defined in terms of their components, and they link directly to rigorous models of causality. The models provide a structured framework for developing useful theories of change and analyzing the intervention they represent.

Theory of Change Analysis: Building Robust Theories of Change

Authors:
Pages:
155-173

Models for theories of change vary widely as do how they are used. What constitutes a good or robust theory of change has not been discussed much. This article sets out and discusses criteria for robust theories of change. As well, it discusses how these criteria can be used to undertake a vigorous assessment of a theory of change. A solid analysis of a theory of change can be extremely useful, both for designing or assessing the designs of an intervention as well as for the design of monitoring regimes and evaluations. The article concludes with a discussion about carrying out a theory of change analysis and an example.

Fall

Useful Theory of Change Models

Authors:
Pages:
119-142

Although theories of change are frequently discussed in the evaluation literature and there is general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what it shows, how it can be represented, and how it can be used. This article outlines models for theories of change and their development that have proven quite useful for both straightforward and more complex interventions. The models are intuitive, fl exible, and well-defined in terms of their components, and they link directly to rigorous models of causality. The models provide a structured framework for developing useful theories of change and analyzing the intervention they represent.

Spring

The Lay of the Land: Evaluation Practice in Canada in 2009

Authors:
Pages:
1-49

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each contributor provides a personal viewpoint, based on his or her own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that Canadian evaluation has matured in many ways since 2003 (when a first panel scan was conducted): professional designation is a reality; the infrastructure is stronger than ever; organizations are more focused on results. Still, evaluation is weakened by lacunas in advanced education and professional development, limited resources, lack of independence, rigidity in evaluation approaches, and lack of self-assessment. While the demand for evaluation and evaluators appears on the rise, the supply of evaluators and the financial resources to conduct evaluations are not. The collective definition of the field of evaluation still lacks clarity. There is also reassurance in looking toward the future. With increased appetite for evaluation, evaluators could make a real difference, especially if evaluators adopt a more systemic view of program action to offer a global understanding of organizational effectiveness. The implementation of a Certified Evaluator designation by CES is a major opportunity to position evaluation as a more credible discipline.

Fall

Building an Evaluative Culture: The Key to Effective Evaluation and Results Management

Authors:
Pages:
1-30

As many reviews of results-based performance systems have noted, a weak evaluative culture in an organization undermines attempts at building an effective evaluation and/or results management regime. This article sets out what constitutes a strong evaluative culture where information on performance results is deliberately sought in order to learn how to better manage and deliver programs and services. Such an organization values empirical evidence on the results it is seeking to achieve. The article outlines and discusses practical actions that an organization can take to build and support an evaluative culture.

Fall

A Commentary on Shepherd’s “In Search of a Balanced Canadian Federal Evaluation Function: Getting to Relevance”

Authors:
Pages:
47-50

Spring

Audit and evaluation in public management: challenges, reforms, and different roles

Authors:
Pages:
11-45

Audit and evaluation play important roles in public management. There can be confusion and debate, however, over what each of these functions cover and indeed what roles they should play. This article reviews and compares each in relation to public management and the key challenges each face in today's public sector. Audit and evaluation should play different roles in public management and provide different information on the performance of public sector organizations, each playing to its own strengths.

Fall

Performance studies: the missing link?

Authors:
Pages:
201-208

Typically, a good measurement strategy to support results-based management includes both ongoing performance measures and periodic evaluations. It is argued that this is a too limited set of measurement tools, resulting, not infrequently, in less useful and costly ongoing performance measures. It is proposed that, in addition to ongoing performance measures and periodic evaluations, an alternative measurement tool called a performance study should be used in many situations, and further, that in a number of circumstances, performance studies should replace specific ongoing performance measures.