M. Theresa Hunter and James C. McDavid

University of Victoria
Abstract: University education in evaluation is an important support for the professionalization of the field. Using internet sources and direct contacts with 105 Canadian university departments, our study develops a Canada-wide inventory of graduate courses and programs, with emphasis on evaluation. Comparisons are made with inventories of American graduate evaluation programs undertaken between 1980 and 2018. Although Canada has about half as many multi-course graduate programs in evaluation as were found in the United States in 2018, the relative number of Canadian programs is disproportionately greater, considering the two countries’ population sizes. Canada also has a more diverse range of disciplines offering evaluation education programs.
Keywords: Canada–US comparisons, evaluation education, inventory, professionalization

Contribution analysis (CA) is a theory-based approach that has become widely used in recent years to conduct defensible evaluations of interventions for which determining attribution using existing methodologies can be problematic. This critical review of the literature explores contribution analysis in detail, discussing its methods, the evolution in its epistemological underpinnings to establishing causality, and some methodological challenges that are presented when CA is applied in practice. The study highlights potential adaptations to CA that can improve rigour, and describes areas where further work can strengthen this useful evaluation approach.

A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each contributor provides a personal viewpoint, based on his or her own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that Canadian evaluation has matured in many ways since 2003 (when a first panel scan was conducted): professional designation is a reality; the infrastructure is stronger than ever; organizations are more focused on results. Still, evaluation is weakened by lacunas in advanced education and professional development, limited resources, lack of independence, rigidity in evaluation approaches, and lack of self-assessment. While the demand for evaluation and evaluators appears on the rise, the supply of evaluators and the financial resources to conduct evaluations are not. The collective definition of the field of evaluation still lacks clarity. There is also reassurance in looking toward the future. With increased appetite for evaluation, evaluators could make a real difference, especially if evaluators adopt a more systemic view of program action to offer a global understanding of organizational effectiveness. The implementation of a Certified Evaluator designation by CES is a major opportunity to position evaluation as a more credible discipline.

The main thesis of this article is that program evaluation and program evaluators have largely missed out on the movement, now into its second decade, to make performance measurement the centrepiece of public sector management and accountability. If these developments are not strategically faced by evaluators, program evaluation runs the risk of becoming less and less relevant to public sector and nonprofit organizations. Evaluators, divided by epistemological and methodological differences, have collectively not been willing to embrace professionalization as a way to reclaim territory lost to the audit and management consulting professions. It is essential that evaluators develop and implement strategies that are intended to create a professional practice space that includes designing and implementing performance measurement systems. Evaluators are well-positioned to do so, if they act collectively.

Program evaluation has undergone a major transformation in the British Columbia Government during the past five years. Although conventional program evaluations continue to be conducted in some ministries, the dominant trend is towards performance measurement. The recently passed Budget Transparency and Accountability Act mandates strategic and business planning, performance measurement and public reporting. Nine issues are identified in this article that need to be addressed if public performance reporting is to mature into sustained performance management in the BC government. Given the resource scarcity and the organizational and political culture, it will be very challenging to implement performance-based public accountability in BC.

This article suggests an important role for program evaluation skills in contracting out government services. Contracting out is being increasingly viewed as a practical and efficient alternative to governmental production of services. Key steps in the contracting out process offer opportunities to utilize program evaluation skills. Further, because contractors are held at arms length, program managers who monitor and evaluate contracts can do so without the conflict of interest problems arising when they are involved in evaluations of their own programs.

This article summarizes the findings from a summative quasi-experiment that examined the impacts of radar cameras on traffic speed in Vancouver. Two arterial streets were included in the evaluation. Knight Street was subjected to a two-month intervention wherein a radar camera was set up periodically and police officers recorded both the speed and licence numbers of vehicles photographed by the camera. Traffic tickets were then mailed to the registered owners of speeding vehicles. Granville Street served as the comparison street for the quasi-experiment. Traffic speeds were measured electronically before, during, and after the intervention using an induction loop buried in the pavement on each street. ARIMA analysis of average daily speeds and percentage of vehicles exceeding the posted speed limit indicated there were significant reductions in both variables on Knight Street, whereas speeds on Granville tended to remain constant. Although these findings offer support for policies that would promote the use of radar cameras, the summative thrust of this evaluation does not allow us to clearly distinguish radar camera impacts from those attributable to the officers who actually implemented this technology.

The National Council of the Canadian Evaluation Society (CES) recognized a need to designate a professional status for the practice of evaluation for individuals who meet appropriate criteria. A consortium of experienced CES members developed an Action Plan with policy options based on (a) a literature review, (b) consultations with relevant professional organizations, (c) knowledge and experience brought by consortium members, and (d) the 2005 Survey of Evaluation Practice and Issues in Canada. The Action Plan recommended three successive levels of professional designation, each with progressively more demanding criteria. Out of this plan, the CES adopted the Credentialed Evaluator designation.