CES Online Learning Goes Live: Welcome to the CES e-Institute!

Volume 19, 2004 - Spring

The lay of the land: evaluation practice in Canada today

Authors :
Pages :
143-178

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each of the contributors provides a personal viewpoint, based on their own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that evaluators possess skills that other professions do not offer; they are social and ecc researchers versed in using empirical data collection and analysis methods to provide a strong factual foundation for program and policy assessment. However, program evaluation has not acquired an identity of its own and, in fact, has tended to neglect key evaluation issues and to lose emphasis on rigour. Today's program evaluation environment is dominated by program monitoring, the lack of program evaluation self-identity, and insufficient connection with management needs. But evaluation is not without opportunities — resultsbased and outcome-based management, advocacy and partnership efforts, individual training and development, and bridging between program management and policy development represent some. But first, evaluators must self-define to communicate to others what their specific contribution is likely to be. The article concludes with implications for the practice of evaluation in Canada and the blueprint of a workplan for evaluators individually and collectively, in their organizations and in their professional association.

Toward a best practice for evaluating the impact of government programs on job creation

Authors :
Pages :
1-29

This article reviews and analyzes the results of a literature search for studies evaluating the impact of government programs on job creation. Issues that complicate the interpretation of any impact measurement method are identified. Advantages and disadvantages for each methodology are described. It is concluded that no single methodology allows for a complete assessment. The use of a combination of user/company surveys and user-selected case studies is recommended. If a financial analysis is required, a modified cost/benefit analysis should be included. The weaknesses of each of these methodologies are minimized by the introduction of complementary methods.

Reporting on outcomes: setting performance expectations and telling performance stories

Authors :
Pages :
31-60

Results, and more particularly outcomes, are at the centre of public management reform in many jurisdictions, including Canada. Managing for outcomes, including setting realistic outcome expectations for programs, and credibly reporting on what was achieved are proving to be challenges, perhaps not unexpectedly, given the challenges faced in evaluating the outcomes of public programs. This article discusses how the use of results chains can assist in setting outcome expectations and in credibly reporting on the outcomes achieved. It introduces the concepts of an evolving results-expectations chart and of telling a performance story built around the program's results chain and expectations chart.

Unnatural selection: an examination of the Fraser Institute's performance index for the Donner Canadian Foundation Awards for Excellence in the Delivery of Social Services

Authors :
Pages :
61-96

Using Darwinian metaphors, this article examines a relatively new quantitative performance evaluation tool for Canadian nonprofits: the Fraser Institute's Performance Index for the Donner Canadian Foundation Awards for Excellence in the Delivery of Social Services. Several aspects of the Index, including strategic management, outcome monitoring, and income independence, are analyzed in light of their ability to accurately evaluate excellence in performance and their potential unsuitability for many non-profits. A theme throughout this discussion is the interaction of the "business-like" approaches favoured by the Index and the fundamentally different nature of non-profit perspectives. Particular attention is paid to the difference between organizations that share a profit motive and non-profits that have diverse organization-specific missions. Context-specific mission-driven approaches to non-profit performance evaluation are recommended.

Analyser le degré d'intégration de l'approche écologique dans les programmes de promotion de la santé: le cas des programmations de réduction de tabagisme de deux directions de la santé publique québécoises

Authors :
Pages :
97-123

For many years, the WHO has promoted the implementation of ecological strategies to improve public health, focusing, in particular, on various social and environmental health determinants and settings. Although that approach has raised interest among public health professionals, the conceptual tools to direct its development and evaluation are still lacking. To fill the gap, this article presents a tool for the assessment of the integration of the ecological approach in programs. To illustrate its potential, the smoking reduction programs implemented by two public health departments in Quebec were evaluated using the application, which allows an assessment of the ecological dimension.

An empirical study of building the evaluation capacity of K–12 site-managed project personnel

Authors :
Pages :
125-141

This article examines the effects of professional development, including formal workshops and ongoing consultation, on the evaluation capacity of K–12 school faculty and administrators who were conducting evaluations of 17 site-managed projects. Changes in the faculty's and administrators' (a) attitudes toward evaluation, (b) self-confidence as evaluators, and (c) assessments of their capabilities as evaluators were examined. School personnel's attitudes toward evaluation did not improve, but their self-confidence as evaluators and their assessment of their evaluation capabilities both showed improvement. The conclusions buttress the argument that, with training and the assistance of experienced evaluators, school personnel can build their evaluation capacity. A number of limitations in study design and data are noted.