Navigating expectations, values and context: a Canadian evaluator abroad
This article uses the story of a multi-year project conducted in four countries in east central and southeastern Europe to illustrate the challenges faced by an evaluator working in a social, political, and cultural context different from her own. Situations involving divergent expectations and unexpected events created circumstances where the evaluator had to shift roles and approaches to ensure the evaluation could remain on course. The evaluation was deemed problematic at various stages of implementation and more successful at other times. In 2010, 6 years after its completion, questions linger as to the degree the evaluation was indeed "successful."
Successful or not: it depends on your frame of reference
This article describes an evaluation that was judged to be unsuccessful from the point of view of key program stakeholders. This was due to the fact that the evaluation did not support program advocates who had much to gain from positive evaluation findings. We argue that, although the knowledge needs of stakeholders must be taken into account, the integrity of evaluation practice, consistent with codes of professional behaviour, should guide the conduct of all aspects of what evaluators do.
Je me souviens... de t'avoir trop longtemps cherché
In its simplest expression, the purpose of program evaluation is to explore the logic that links a set of related activities to their expected results. Hence, program evaluation serves to determine the extent to which the logic of a program has unfolded as expected. A prerequisite to this assessment is to have a program (evaluand) that is adequately defined. This article explores the potential impacts of having an ill-defined or incomplete definition of a program that is about to be evaluated. It discusses some strategies that evaluators can explore to avoid what may well turn out to be a fatal flaw in what is already a rather challenging research process.
Management challenges in M&E: thoughts from South Africa
This impressionistic narrative focuses on experiences in setting up South Africa's first government monitoring and evaluation (M&E) unit. The high political expectations for the program required that M&E provide a progress overview and diagnosis of the ambitious land reform program. Key contextual factors that influence success and that may pose challenges are considered in this account, which shows both how strategies had to be changed to address varied demands and the mixed results these produced. Based on the practical experiences, a set of suggestions are provided to emerging evaluators as they sharpen their evaluation acumen.
Learning from evaluation misadventures: the importance of good communication
Good communication is a key factor affecting the success of evaluation in many ways. This article presents four examples from a variety of government and international organizations illustrating communication-gone-wrong, and often evaluation management-gone-wrong. The article discusses what can be learned from these situations in order to provide for more effective communication in other evaluation contexts, and thereby enhancing the relevance, quality, and use of evaluation. This article highlights the implications of these experiences for effective project management of overall evaluation functions as well as of specific evaluations.
Reflections over 25 years: evaluation then, now, and into the future
Réflexions autour des 25 ans de la Revue Canadienne d'Évaluation de Programme : l'évaluation au passé, au présent, et au futur
Guest Editor's Remarks: As I recall – or how to take advantage of less-than-successful evaluation experiences / Un mot de la rédactrice invitée : « Je me souviens » ou comment tirer profit d'expériences d'évaluation qu'on ne peut qualifier de succès
Evaluation, valuation, negotiation: some reflections towards a culture of evaluation
Although the evaluation of public policies is a subject of growing interest in Latin America, there are problems with the design and implementation of evaluations, as well as with the limited use of their results. In many cases, the evaluations have more to do with generating descriptions and less with assessing these activities and using those assessments to improve planning and decision making. These points are explored in a case study of the evaluation of a rural development program in Argentina, emphasizing the process of negotiation and consensus building between the evaluators and the official in charge of approving the evaluation report. The lessons learned from the experience point to the generation and consolidation of a culture of evaluation in the region.
L'influence de l'évaluation sur les suites des projets d'expérimentation: l'exemple des projets d'informatisation au Québec
The dissemination and sustainability of many health-care IT projects in Quebec have hardly been influenced by their evaluation, even when the evaluation was participative, pluralist, comprehensive, formative, and user-focussed. This article aims to improve our understanding of why that has been the case and to suggest courses of action and reflection. Human and organizational factors and other factors related to the evaluation itself are considered. To increase the impact of evaluation on project outcomes, the authors recommend knowing the project better, adding to the action plan usually intended to promote knowledge and learning, and further developing frameworks for integrating practices and theories to improve our understanding of the conditions for their use in action.
Successful evaluation management: engaging mind and spirit
The key to a successful or unsuccessful evaluation is often the quality of the relationship between evaluators and their clients. This article examines the social rather than technical aspects of evaluation and how this can affect the outcome. The author discusses the critical interactions and responsibilities of the evaluation manager and the evaluation practitioner when an evaluation is contracted out. Practice guidance is offered in how to manage an evaluation contract to a successful conclusion.
Chronique d'une évaluation ratée
The evaluation presented in this article combines a number of the problems outlined in a textbook example: confusion between "control" and "evaluation," insufficient resources, overly ambitious goals, and tension between stakeholders, a situation that led to disregard for all the principles enumerated in the French evaluation society charter: pluralism, distance, competency, respect for individuals, transparency, responsibility, and opportunity. The lessons learned described in this article highlight the need to follow those principles, both on the part of the agency commissioning the evaluation and the evaluator, the importance of ethical issues, and the fragile balance that allows evaluations to be implemented. Finally, the author questions the usefulness of recourse to arbitration proceedings.
Program evaluation without a client: the case of the disappearing intended users
Evaluators know they are supposed to identify and engage with stakeholders. What happens when the client has a very narrow concept of the meaning of evaluation? What happens when the primary stakeholders, including the client, disappear just as the evaluation gets started? First, it is important to acknowledge the challenge, then develop a strategy to negotiate the scope of the evaluation and to broaden the community of stakeholders. Divergent pathways are explored to facilitate use of the evaluation findings in such settings.
La figure de l'évaluateur, mi-ingénieur mi-bricoleur
This article describes an evaluation experience in health promotion. Caught in a wide range of representations, challenges, and expectations, the evaluator adapts her practice to provide a quality evaluation to address the users' needs. The authors analyse the evaluation process with a focus on the French Evaluation Society's principles and suggest adjustments specific to the field of health promotion. The evaluator's ability to take on several roles and to adapt to the context is fundamental here.