Greg Mason

Fall

Assessing the value for money of active labour market programming for persons with disabilities

Authors:
Pages:
21-38

Treasury Board of Canada's new policy on evaluation and its accompanying directive have placed increased pressure on those conducting federal evaluations to not only quantify the impacts of programming but also make measurable assessments of their value. However, making accurate statements about the value for money of programming can be difficult during evaluations. A number of technical and practical challenges can make common approaches infeasible. This article discusses a recent assessment of the value for money undertaken during the evaluation of the Canada-Manitoba Labour Market Agreement for Persons with Disabilities. It demonstrates a number of approaches that can be used to overcome some of the most common barriers to the assessment of value for money in evaluations.

Fall

BOOK REVIEW: R. Pawson. (2006). Evidence-based Policy: A Realist Perspective

Authors:
Pages:
268-270

Spring

The Lay of the Land: Evaluation Practice in Canada in 2009

Authors:
Pages:
1-49

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each contributor provides a personal viewpoint, based on his or her own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that Canadian evaluation has matured in many ways since 2003 (when a first panel scan was conducted): professional designation is a reality; the infrastructure is stronger than ever; organizations are more focused on results. Still, evaluation is weakened by lacunas in advanced education and professional development, limited resources, lack of independence, rigidity in evaluation approaches, and lack of self-assessment. While the demand for evaluation and evaluators appears on the rise, the supply of evaluators and the financial resources to conduct evaluations are not. The collective definition of the field of evaluation still lacks clarity. There is also reassurance in looking toward the future. With increased appetite for evaluation, evaluators could make a real difference, especially if evaluators adopt a more systemic view of program action to offer a global understanding of organizational effectiveness. The implementation of a Certified Evaluator designation by CES is a major opportunity to position evaluation as a more credible discipline.

Spring

Evaluation in the context of the Social Union Framework Agreement: a case study of the national child benefit

Authors:
Pages:
139-158

The Social Union Framework Agreement (SUFA) and specific programs such as the National Child Benefit (NCB) represent joint government delivery of programming, and present many challenges for evaluators. Aside from attribution (which this paper argues is not really the central issue), the essential problem faced in the evaluation of these federal-provincial-territorial initiatives is that programming is becoming both more complex and heterogeneous. The concepts of joint planning and information sharing demand a high level of cooperation among program sponsors. A test of the rationale and effectiveness for agreements such as SUFA will be whether participating governments support detailed evaluations and performance measurement.

Spring

The lay of the land: evaluation practice in Canada today

Authors:
Pages:
143-178

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each of the contributors provides a personal viewpoint, based on their own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that evaluators possess skills that other professions do not offer; they are social and ecc researchers versed in using empirical data collection and analysis methods to provide a strong factual foundation for program and policy assessment. However, program evaluation has not acquired an identity of its own and, in fact, has tended to neglect key evaluation issues and to lose emphasis on rigour. Today's program evaluation environment is dominated by program monitoring, the lack of program evaluation self-identity, and insufficient connection with management needs. But evaluation is not without opportunities — resultsbased and outcome-based management, advocacy and partnership efforts, individual training and development, and bridging between program management and policy development represent some. But first, evaluators must self-define to communicate to others what their specific contribution is likely to be. The article concludes with implications for the practice of evaluation in Canada and the blueprint of a workplan for evaluators individually and collectively, in their organizations and in their professional association.

Spring

Value-for-money analysis of active labour market programs

Authors:
Pages:
1-29

Accountability requirements by central agencies in government have imposed expectations on management to show results for resources used — in other words, "value for money." While demonstrating value for money means showing that the program has relevance and a rationale and that the program logic and theory make sense, the core of value for money lies in showing that a program is cost-effective. Unfortunately, many public programs and policies do not provide quantifiable outcomes, and this limits conclusions on value for money. However, labour market training programs are amenable to cost-effectiveness analysis (CEA), provided that the evaluation methodology meets certain conditions. This article reviews CEA in the context of labour market training, especially programs directed to eccally disadvantaged groups. After reviewing the data availability and the analytical methods commonly used to support value-for-money analysis of training programs, the authors present several practice improvements that would increase the "value" and validity of value-for money analysis.

Spring

Recent Advances in Questionnaire Design for Program Evaluation

Authors:
Pages:
73-94

Evaluation uses questionnaires as a central data-gathering technique, yet researchers often appear unaware of recent developments in questionnaire design. This article reviews issues beyond the creation of standardized questions and the basic rules researchers find useful in data collection. These elementary guidelines remain robust for much evaluation research and should not be abandoned hastily. However, rapid change in the theory underlying questionnaire design has important implications for evaluation. Three themes illustrate these changes. First, magnitude scales and their use in client satisfaction scales show how response categories can improve individual questions. Second, decision theory sees respondents as selecting "correct" responses from a portfolio of potential answers. In this view, the answer to a question is conditioned by the values held by the respondent and his or her perception about the risks of revealing true feelings. Third, in some cases it is possible to cast the entire questionnaire into a framework that replicates how choices are made by respondents. In one application, the questionnaire simulates the decision-making of a consumer in the market place. Each of these three themes is applied to problems evaluators face in data collection involving surveys and questionnaires.

Spring

Coping With Collinearity

Authors:
Pages:
87-93

Collinearity is very common in linear regression. The common methods for diagnosing the disturbance , such as evaluating parameter instability when variables are removed from the specification are only suggestive. Recent developments are reviewed which assist in diagnosing collinear disturbances. These include condition indexes and variance proportions decompositions and are available in a number of statistical packages. Some corrective strategies are also examines. In general, it is not correct to simply drop variables from the specification unless they are redundant to the program logic and underlying theory.