The Threat Of Evaluation In Research And Development

Program evaluation, as practised by the evaluation community, is seldom performed on research and development (R&D) programs. Scientists and research managers are threatened by formal program evaluation, and therefore do not initiate or support evaluations. This threat is due to three major factors inherent in the scientific comunity or in organizations in general: the mythology of science, ignorance, and the fear of discovery. The evidence given here suggests that each factor consists of several issues, some bring the myth that research programs cannot be evaluated, that managers and scientists do not known about evaluation methods, and that they are afraid of program changes and outside evaluators. The current situation is not expected to change without a fundamental shift within the R&D community itself.

The Uses of Research: A Case Study In Research And Policy

Three case studies of the use of research and evaluation results in a provincial ministry of education are presented. The cases are frames in terms of Cousins and Leithwood (1986) review of the research utilization literature. The factors affecting utilization in each case are analyzed. The cases point to the need for a stronger political orientation in examining research use in organizations. Implications for the study of utilization and for the conduct of useful research are drawn.

C'est pas grave, c'est rien que vos nerfs L'Évaluation d'un programme de prévention géré par le milieu

Its just your nerves a prevention program based on information-sharing regarding women's use of minor tranquilizers and alcohol, is sponsored by Health and Welfare Canada. It is available both in English and in French («C'est pas grave, c'est rien que vos nerfs»). This latter version has been distributed to 144 agencies throughout Quebec and was subsequently evaluated a year later. A survey on utilisation patterns aimed at assessing implementation showed that the program was differently used (from extended use to no use) from one site to another and from one region to another. Context was determines in explaining variations in use. An impact evaluation confirmed the quality and appropriateness of the program, in particular when it is used within the context of a long term program and extended by follow-up activities. From this research, it appears that evaluation of the implementation can be helpful to monitor the development of innovative and milieu-based programs, specially with regard to diffusion and promotion, training and monitoring of expected and unexpected results.

Tensions In Trans-cultural Native Education Programs: Hurdles For The Sensitive Ealuator

Trans-cultural native education programs are defined as those which are designed to prepare members of these minority groups to function in the majority culture. These programs operate under a number of tensions of constraints which must be considered during program evaluations. In this paper several tensions which arose during the evaluations of three programs are examined and discussed in the light of what is currently known about cross-cultural evaluations.

The Collinearity Problem in Regression-Discontinuity Models

The article addresses the issue of collinearity in regression-discontinuity models, as raised by a recent book review in this Journal. It examines the causes of collinearity between the constant term in the model and a dichotomous explanatory variable that indicated participation in a program being evaluated. It shows that removal of the constant does not solve the collinearity problem but in fact may bias the estimated program impact. Collinearity can truly be overcome only by collecting more data (of an appropriate kind).

Coping With Collinearity

Collinearity is very common in linear regression. The common methods for diagnosing the disturbance , such as evaluating parameter instability when variables are removed from the specification are only suggestive. Recent developments are reviewed which assist in diagnosing collinear disturbances. These include condition indexes and variance proportions decompositions and are available in a number of statistical packages. Some corrective strategies are also examines. In general, it is not correct to simply drop variables from the specification unless they are redundant to the program logic and underlying theory.

The Evolution of the Canadian Forces' Alcoholism Rehabilitation Program Evaluation System


Program Evaluation And Health Policy: Nix, Knack And Nexus

Results of program evaluations in the health field are often ignored or under utilizes by health policy decision-makers and by program managers. The paper discusses several reasons for this phnon. It is suggested that if program evaluators are to make a positive contribution to changing the design and operation of health programs and the nature and specifics of health policies, they must be cognizant of the reasons for the lack of use of program evaluations to date and be willing to modify their program evaluations accordingly.

Health Care Evaluation in a Government Agency: Goals, Organization, and Software

The demands on the Manitoba Health Services Commission to provide timely data to assist government, health boards, planners and service providers have been increasing. Users have wanted a system to provide easy manipulation of data to address questions of cost containment without compromising quality of care, of difference in utilization across areas, and of program evaluation. Such problems are similar across provinces, as are the data sets available to address them -- hospital admissions/separations, physician claims, personal care home records, and so forth. The Manitoba Health Services Commission is setting up an Information Centre to instruct potential end users on data base availability, restrictions on access, software, and so on. Users will be running the Health Applications System (HAS) in combinations with the well-known Statistical Analysis System (SAS) to do their own data analysis. Analysts will be able to address issues singled out b Statistics Canada as top priority information needs: health care utilization, hospital morbidity, and mortality. An effort will be made to feed back information to hospitals and providers; the critical task is to use available analytic tools to help provide constructive direction to the health care system.

Fundamental Methodologies In Identifying Values In The Evaluation Of Health Programs

The evaluation of programs that deliver medical care begins with a determination of the value a patient places on the outcomes of the treatment. This paper describes several strategies drawn from the fields of psychology and eccs that identify these values or preferences and attempt to measure them. Techniques for ranking, preference scaling, time-trade-off, and incorporating risk into a measure of utility of a treatment are explored.

Evaluation and Hospital Funding

Increasing hospital costs in both Canada and the United States forces a re-examination of the funding system incentives. The current funding based on standards gives administrators and planners the wrong incentives while a funding system based on cost-outcome or cost-effectiveness analysis would change the incentives towards shorter stays and thus less pressure on capital funding.