Answering the Why Question in Evaluation: The Causal-Model Approach
Theorists and researchers have urged that evaluation test the causal assumptions about why an intervention should work to achieve its goals. Though the terminology is different, these writers agree that the assumed pathway from program to ultimate outcome be made explicit and tested empirically. The causal model approach, because it is geared toward providing evidence on why a program worked or failed, is advocated whenever time, money, and experience permit. After defining casual-model evaluation, an example from the crime prevention literature is used to contrast the approach with traditional evaluations. Benefits, limitations, and other issues are also discussed.
Towards Sustainability of Human Services: Assessing Community Self-Determination and Self-Reliance
Self-determination and self-reliance (SDR) are concepts that lie at the very core of human development and thus are pertinent to group and community development. SDR implies the ability and skill to negotiate with external environments to influence the terms of interdependence. Our challenge as evaluators has been to identify core elements that are reflective of the transition from dependency to self-reliance. A further challenge has been to develop a process of investigation that has relevance across various communities. To meet this challenge, we have identified an approach to monitoring and assessing the processes and outcomes associated with the movement of communities from dependency to self-determination and self-reliance. Also, we present a Self-Determination/Reliance (SDR) model constructed on the essential SDR dimensions of accountability, decision making, information, knowledge and skills, and resource mobilization. Critical movement along each of these dimensions indicates progress toward sustainability of community-based human service projects, programs, and initiatives.
The Structure of Stages in The Evaluation Cycle: An Event Sequence Analysis
From time to time, research on evaluation issues deals with a sequence of events or episodes which an analyst would prefer to examine in its entirety rather than event by event. However, quantitative methods for comparing sequences of events or activities and analyzing their similarity among populations and samples have rarely been applied in social research, This article describes the tasks undertaken during the cycle of evaluation of housing programs by CMHC prior to 1997 and analyses the major evaluation products in terms of the similarity of constituent tasks. A brief introduction to combinatorial methods for measuring similarities among whole character sequences is presented and new research applications are proposed. The analysis uses a new program for sequence alignment analysis called ClustaIG, which is a modification of the ClustalX program used in molecular biology.
Mieux comprendre l'impact par l'analyse de l'implantation : Le cas d'un programme de prévention chez des travailleurs avec ordinateur
This article presents three methods used to evaluate an ergc preventive training program for VDT workers: an epidemiological method to measure effects and two qualitative methods of work analysis based on ergc and psychodynamic approaches. Combining different sources of data aimed at an understanding of the impact of the program in a given context and led to a better understanding of the use of training to introduce preventive practices in the workplace and discussion of quantitative measurements and findings. Finally, the complementary data allowed a more documented assessment of the program and indicated possible improvements.
The Redesign of Advanced Patrol Training for Police Constables in Ontario: Making Use of Evaluation to Maximize Organizational Effectiveness and Efficiency
The use of evaluation to maximize the effectiveness and efficiency of the Ontario Police College in the delivery of a refresher training program for police officers is described. Confronted by a growing backlog of patrol officers requiring refresher training and ongoing financial constraints, in 1997 the College, in co-operation with Ontario police services, initiated a comprehensive refresher training needs analysis. A multi-stage, multi-method strategy was employed in the research. The results of the needs analysis were used to redesign the curriculum, format, and delivery of the Advanced Patrol Training (APT) refresher course for patrol constables. As a result of the APT redesign, the number of police officers receiving refresher training increased by an average of 657% annually (effectiveness), while costs associated with the APT course decreased by 50% (efficiency). The evaluation, design, and delivery of the APT course has become a model for other police training programs in Ontario.
Contributions of Evaluation Research to the Development of Community Policing in a Canadian City
This article reviews evaluation research on community policing (CP) in the City of Windsor, including the Windsor Police Service's utilization of evaluation evidence. It examines five studies of the initial year of two CP patrol units, one beginning in 1990 and the other in 1992. The studies included surveys of the CP-related attitudes and experiences of constables, community residents, and managers of small business, as well as time series analyses of crime trends. The results provided favourable process and outcome evidence. By 1995, more than half the city population was serviced by CP units, and follow-up time series analyses reconfirmed the earlier encouraging crime trends findings. However, in contrast to the positive beginnings, a 1997 survey of the service's membership revealed no strong support for CP, including an apparent division among management in attitudes about CP. Recent CP developments in the service are considered.
Lessons Learned from Establishing a Research and Evaluation Unit for a Regional Health Authority
The Canadian health care establishment changed rapidly during the 1990s as several provinces adopted regionalization to reduce costs and improve the efficiency and effectiveness of health service delivery. In Winnipeg, Manitoba, the Winnipeg Community and Long Term Care Authority (WCA) is one of two such local health authorities. The WCA established a Research and Evaluation Unit (REU) to support the effective monitoring and evaluation of dozens of WCA programs and services. Regional health authorities face several challenges with respect to how to evaluate their diverse programs and services. Using the WCA for illustrative purposes, this practice note provides suggestions for addressing these challenges.
Focusing on Inputs, Outputs, and Outcomes: Are International Approaches to Performance Management Really So Different?
The focus of performance information appears, on the surface, to differ among Canadian, Australian, and US federal governments. While these countries emphasize different aspects of performance, their federal guidelines on performance measurement share important common ground - the logic model. A performance logic model helps clarify the linkages between inputs, activities, and process and outputs, short- and long-term outcomes, and impacts. The model assists both analysts and managers to articulate the cause-effect theory of a program or service and should answer the fundamental questions about WHY an initiative exists, WHAT the expected outcomes are, WHO the program or service will reach and HOW it will be delivered. There is a tendency for organizations to focus on measurement before first describing the logic of their enterprise. International practice suggests that Canadian, Australian, and US approaches all promote the understanding of program logic before measurement. Such an understanding will be key to the successful implementation of performance management initiatives in each of these countries.