Volume 11, 1996 - Fall

Service User Input: Fact or Fiction? The Evaluation of the Trauma Program, Department of Rehabilitation, Sault Ste. Marie, Ontario

Authors :
Pages :
111-126

This study explores the current trend or incorporating service user data into program evaluations. Using a case Study approach and citing three years of evaluation data of the Trauma Program at Sault Ste, Marie General Hospital, northern Ontario, the author challenges the traditional assumptions that promote service user data in program evaluations. The main conclusion is that although service user data is touted as being important, its political importance far outweigh,; its evaluation value Slid utility in particular with hard-to-access samples. Implications are directed toward evaluators, administrators, and policy makers.

Coming Tto Grips with Changing Canadian Health Care Organizations: Challenges for Evaluation

Authors :
Pages :
127-148

Canadian health care organizations are in the midst of' rapid change. Reduced federal transfer payments have necessitated radical cost containment measures, including megers reconstructing downsizing, and outsourcing. At the same time, health care organizations are undergoing a transition in approach to management and accreditation, from quality assurance (QA) to continuous quality improvement (CQI). This article describes that transition, examines the status quo of evaluation activities in both QA-based and CQI-based organizations, and discusses some of the opportunities and challenges facing evaluators in the health care sector.

Community Health Centers. Evaluation Issues and Approaches

Authors :
Pages :
149-158

Community health centers present a challenge for evaluators because of their multiple and overlapping programs, variable client base, culturally diverse clientele, and non traditional approach to health and wellness. This article provides an overview of the evaluation strategy adopted by a rural southwestern Ontario community health center to overcome these challenges. The purpose of the evaluation was to obtain a broad picture of the center's internal dynamics, processes, and efficacy following its first two years of operation.

Peer Assessment of Evaluation Products: Practice in the Department of National Defence

Authors :
Pages :
159-166

Although evaluation reports produced in the Department of National Defence have been subject to peer assessment since 1972, limited evidence to support this practice was found in the evaluation literature. Peer assessments are used to ensure that evaluation standards have been met, to correct gaps in reasoning and errors, and to solicit insights from reviewers.

Using a logic model to focus health services on population health goals

Authors :
Pages :
167-174

This article describes an application of program logic models to help regional health authorities focus attention and efforts on health outcome goals in British Columbia. The article emphasizes the value of graphic modeling techniques such as logic models in identifying the presumed contribution to regional health status of health services and other health determinants. It argues that logic models facilitate overall governance of health care services by creating performance-monitoring frameworks for both short-term and long-term outcome objectives.

Why All This Evaluation? Theoretical Notes and Empirical Observations on the Functions and Growth of Evaluation, with Denmark as an Illustrative Case

Authors :
Pages :
1-34

This article demonstrates that theoretical debates and developments in the area of organizational analysis can be a fruitful source of inspiration for evaluation utilization research. The article considers two sets of questions. One concerns the societal function of evaluation, the other its historical development. Three sets of well-established lenses, ground by competing views of organizations and organizational behavior, are used to address these concerns. The theories in question see organizations as rational systems, political systems, or cultural systems. The utilization of policy evaluation in Danish political-administrative practice is used as an illustrative empirical case. The article further demonstrates that inspiration from organizational analysis will bring us no closer to a unified understanding of what evaluation utilization is. On the contrary, it shows that debates on and controversies over evaluation utilization belong to the order of things, and that this order entails multiple realities depending on one's theoretical point of departure.

Getting On and Off the Policy Agenda: A Dualistic Theory of

Authors :
Pages :
35-60

This article deals with the impact of program evaluation on policy making and, more specifically, on policy making with regard to home care and services for the elderly in Quebec. Three important evaluations have influenced policy making in this area since 1983. The article proposes a "policy communities" framework to explain why and how these evaluations either influenced or failed to influence policy. It suggests that program evaluation strategies should reflect the political context in which evaluation takes place, and that the utilization of program evaluation results depends to some extent on program evaluators' ability to participate effectively in the policy processes in question.

The Use of Holistic Versus Analytic Scoring for Large-Scale Assessment of Writing

Authors :
Pages :
61-110

This article discusses a variety of issues associated with the use of holistic and analytic scoring methods in large-scale student assessments. The article reviews the education literature on the subject, describes the results of a recent Saskatchewan study comparing the results of the two scoring approaches, and discusses some of the study's implications for future large-scale assessments.

Evaluating the Outcomes of Literacy Training: A Feasibility Study

Authors :
Pages :
87-110

Challenges inherent in evaluating literacy training programs include strict confidentiality policies, the reluctance to "test" students, irregular meeting schedules, and the limited attention span of low-literate students. Guided by interviews and focus groups, this project explored the feasibility of tutor-based assessments in three domains: cognitive skills, functional literacy, and literacy-related self-confidence. Over a period of one and a half years, the assessment battery was returned by 38 of 67 tutors from two literacy programs-one highly structured and curriculum based, the other learner centered. Level of formal education and program placement were found to be important indicators of functional performance and confidence. Length of time the student had been in the program, meanwhile, was unrelated to functional performance and inversely related to ratings of self-confidence. Recommendations for further evaluation of literacy programs are based on our findings regarding utility of the various measures, feasibility for administration, and acceptability to students, tutors, and program administrators.