Building an Evaluative Culture: The Key to Effective Evaluation and Results Management
As many reviews of results-based performance systems have noted, a weak evaluative culture in an organization undermines attempts at building an effective evaluation and/or results management regime. This article sets out what constitutes a strong evaluative culture where information on performance results is deliberately sought in order to learn how to better manage and deliver programs and services. Such an organization values empirical evidence on the results it is seeking to achieve. The article outlines and discusses practical actions that an organization can take to build and support an evaluative culture.
A Promising New Approach to Eliminating Selection Bias
This article presents a creative and practical process for dealing with the problem of selection bias. Taking an algorithmic approach and capitalizing on the known treatment-associated variance in the X matrix, we propose a data transformation that allows estimating unbiased treatment effects. The approach does not call for modelling the data, based on underlying theories or assumptions about the selection process, but instead calls for using the existing variability within the data and letting the data speak. We illustrate with an application of the method to Italian Job Centres.
La fidélité de l'implantation d'un programme d'intervention pour des adolescentes agressées sexuellement: une mesure par observation systématique
Reported here is a first study of the implementation fidelity group intervention program for sexually abused adolescent girls, consisting of 18 sessions centered on 9 themes. Through systematic observation of the intervention protocol used with two groups of adolescents, the case study describes implementation fidelity and its evolution. Results show that the first group outperformed the second in terms of implementation adherence. Levels of participation were similar between groups but quality appears to have improved for the second group. The activities implemented lasted longer than planned in both groups. Results are discussed in light of the factors that promote fidelity and the debate between fidelity and adaptation.
Interrater Reliability in Content Analysis of Healthcare Service Quality Using Montreal's Conceptual Framework
This study examines the usefulness of the Montreal Service Concept framework of service quality measurement, when it was used as a predefined set of codes in content analysis of patients' responses. As well, the study quantifies the interrater agreement of coded data. Two raters independently reviewed each of the responses from a mail survey of ambulatory patients about the quality of care and recorded whether or not a patient expressed each concern. Interrater agreement was measured in three ways: the percent crude agreement, Cohen's kappa, and the coefficient of the generalizability theory. We found all levels of interrater code-specific agreement to be over 96%. All kappa values were above 0.80, except four codes associated with rarely observed characteristics. A coefficient of generalizability equal to 0.93 was obtained. All indices consistently revealed substantial agreement. We empirically showed that the content categories of the Montreal Service Concept were exhaustive and reliable in a well-defined content-analysis procedure.
Coûts de transaction et politiques agri-environnementales
This article proposes an analytical framework and method for measuring transaction costs associated with the application of French agri-environmental policies. Transaction cost theory attempts to take into consideration the effect of costs involved in drawing up and securing contracts for ecc organization and efficiency. Originally developed to understand forms of industrial organization, it has proven relevant in the assessment of public policies, in particular territorial and environmental policies involving a large number of players, often with diverging interests, which raises important issues of information asymmetry. The proposed method is applied to two highly different initiatives, one fairly standard («grass premium») and the other localized («local farming contract»). The diverse forms transaction costs may take for the various players involved (government departments, professional organizations, farmers) are shown, as is the usefulness of analysing these costs for an accurate evaluation of the effectiveness of the policies.
The Downward Trend of Survey Response Rates: Implications and Considerations for Evaluators
Rapidly declining response rates and the associated threat of nonresponse bias call into question the validity of data obtained through telephone surveys, a tool often used in evaluation. This article explores changes in nonresponse bias over time by examining three data points (1991, 1996, and 2002) from an annual household telephone survey conducted by the University of Alberta's Population Research Lab. Results demonstrate a substantial decline in response rates accompanied by an increasing level of bias in variables related to respondent education. Implications of these results are investigated through regression analyses and suggest that declining representation of individuals with less education could significantly impact a variety of survey variables, thus creating opportunity for opinions of the more educated to become more heavily weighted in evaluation results. In turn, such results could be used to inform government policies and programs in ways that advantage the educated middle class.
An Illustration of a Methodology to Maximize Mail Survey Response Rates in a Provincial School-Based Physical Activity Needs Assessment
Two mail surveys were conducted as a province-wide needs assessment to examine the opportunities for, barriers to, and participation in physical activity in Ontario elementary and secondary schools. Dillman's Tailored Design Method (TDM) was used to maximize the quality of responses and the response rate. Both surveys entailed five mailings to key informants from randomly selected schools. The response rate among the 599 elementary and 600 secondary schools was 85% and 79%, respectively. This article discusses how the TDM strategies (i.e., strategies to establish trust, increase perceived rewards, and decrease perceived costs among key informants) were used to yield high response rates, examines the response rates after each mailing, and shares some lessons learned, which will be useful to researchers who are considering using the TDM in surveys designed to develop and evaluate programs.
BOOK REVIEW: L. Bickman & D.J. Rog (Eds.). (2009). The SAGE Handbook of Applied Social Research Methods (2nd ed.)
COMPTES RENDUS DE LIVRES: Michael Quinn Patton. (2008). Utilization-Focused Evaluation (4e éd.)
BOOK REVIEW: D.L. Streiner & S. Sidani (Eds.). (2010).When Research Goes Off the Rails: Why It Happens and What You Can Do About It.