Fourth-Generation Evaluation

Naturalistic program evaluation strategies are gaining acceptance within the evaluation profession. They are especially useful in the examination of the transaction, or process, components of programs. One of their disadvantages is that they demand extensive quantities of evaluator time and effort which are difficult to arrange. Guba refers to evaluation conducted within the naturalistic paradigm (as distinct from the use of naturalistic techniques) as fourth-generation evaluation. This article describes an evaluation study in the health field that attempted to stay within the naturalistic paradigm while mitigating some of the drawbacks of the methodological process of naturalistic evaluation.

Utilization Isn't Everything

This article examines an unanticipated problem associated with the successful implementation of the "utilization-focused" method of evaluation described by Patton. The study involved the development and evaluation of a suicide screening instrument for use by the Alberta Solicitor General's Department. To maximize the possibility that these results would be useful, a utilization-focused approach was adopted. We will describe the evolution of the project over a series of stages, from identifying stakeholders to preparing a final report. Although this method improved utilization of results, the enthusiasm generated created a Hawthorne effect, making it difficult to meet the study's original objective of assessing the validity of the screening. We discuss the process rather than the content of this evaluation, and the problems encountered, in a way that highlights some of the inevitable trade-offs involved in applied research.

Needs Assessment: Broadening The Perspective On Its Utility And Timing

Needs assessment, a tool for program planning, involves collecting data from service agencies, key informants, and potential clients, and drawing inferences from indicators of community needs. As portrayed in the literature, this approach to evaluation should only be used prior to program development. Both the utility and the timing of the needs assessment approach have been restricted by this portrayal, as case studies will illustrate. Needs assessments should be conducted not only in the initial planning stage but periodically after a program has been implemented. In conjunction with formative evaluations, periodic needs assessments can be useful in the on-going process of program modification and in planning for future services delivery.

Plenary address to the 1988 Canadian Program Evaluation ConferenceL toward an evaluation which serves politics by arriving in time and, even, early


Quantifying Qualitative Decision Trees For Multi-Respondent Decision-Making: Theory And A Program Evaluation Application

A method for quantifying qualitative decision trees is presented and applied within a program evaluation setting. The approach combines Likert type scales with decision trees and develops a mathematical curve based on the associated set of means and standard deviations. Depending upon where the respondent's result plots, one of six different decision conditions are suggested.

Risk Analysis And Program Evaluation

Evaluation studies by the Government of Canada follow standard procedural guidelines but have been retrospective and idiosyncratic in their methodologies. Therefore, they do not enable comparisons of the likely future effectiveness of competing programs. In contrast, the World Bank uses a combination of benefit/cost analysis and risk analysis to produce standard "bottom line" measures of program effectiveness. The latter approach makes evaluation studies more useful in decision making.

On The Use And Misuse Of Input-Output Based Impact Analysis In Evaluation

Estimates of ecc activity generated and jobs created that are derived using input-output analysis are often presented in program evaluations and confused with the benefits of the resulting from the program. Two such cases are presented as examples. We argue that for two main reasons this type of analysis constitutes a misuse of input-output analysis. First, input-output estimates generated using the Keynesian closed versions of models are biased upwards because they ignore the price and financial feedbacks that tend to reduce multipliers in macroecc models. Second, and more important, it is inappropriate to consider induced effects resulting from a particular program in isolation, because such effects can only be properly considered in the aggregate at the level of overall stabilization policy. In this paper we contend that cost-benefit analysis, with its assumption of full employment, is the most appropriate tool for analyzing the benefits resulting from particular programs. Input-output analysis should be confined to providing estimates of the industrial or regional breakdown of the direct impact of a program or of the employment impacts of program spending. It should not be sed to generate Keynesian multipliers.

Professionalism And School-Based Program Evaluation

Participating in program evaluations is a new part of the classroom teacher's role. If teachers are to embrace the opportunities provided by this participation, they need to understand the process of program evaluation and to appreciate its potential to empower them in their professional lives. This article discusses these issues in the context of Saskatchewan's current curriculum change and offers an illustration of one instance where participating in a proper evaluation study did enhance the professionalism of a teacher.