Editor's Remarks / Un mot du rédacteur
Preparing the editor’s remarks for CJPE 30(2) has brought a big smile to my soul. This issue is jam-packed with gems. Covering a wide range of hot and less hot topics, the issue has more than a little bit of something in it for all readers from the many corners of the evaluation community in Canada and abroad. [...]
Useful Theory of Change Models
Although theories of change are frequently discussed in the evaluation literature and there is general agreement on what a theory of change is conceptually, there is actually little agreement beyond the big picture of just what a theory of change comprises, what it shows, how it can be represented, and how it can be used. This article outlines models for theories of change and their development that have proven quite useful for both straightforward and more complex interventions. The models are intuitive, fl exible, and well-defined in terms of their components, and they link directly to rigorous models of causality. The models provide a structured framework for developing useful theories of change and analyzing the intervention they represent.
Meeting at the Crossroads: Interactivity, Technology, and Evaluation Utilization
Increasing the Utility of AAMC Canadian Graduation Questionnaire Data for Program Evaluation: Subscale Development
The AAMC Canadian Graduation Questionnaire (CGQ) is widely used for evaluating undergraduate MD programs; however, the analysis of individual items limits its usefulness. To aid in the management and interpretation of the CGQ for use in program evaluation, this study combined items from the sections on clinical learning experiences, physician competencies, and student services into scales and examined their internal structure and reliability. Factor analyses conducted on data from 517 undergraduate medical students supported combining the items into 15 scales. Two examples illustrate how the scales can be used to evaluate student experiences for different cohorts over time and across sites.
Challenges in Evaluating a Prototype Project in a Large Health Authority: Lessons Learned
A developmental evaluation was undertaken to evaluate a prototype test of a new model of perinatal healthcare across acute maternity, public health, and primary care in two hospitals in a large health authority. The project was initiated to bridge gaps in care across the acute and community settings to ensure a seamless perinatal healthcare journey for women. The objective of the evaluation was to support the prototyping process by providing data to inform decisions as the prototype was developed and by documenting decisions as they were made. Thi s article explores challenges faced during the evaluation, including unfamiliarity of the health sector with prototype projects and their inherent uncertainty, a disconnect between the rapid pace of a prototype project and bureaucratic hurdles of working within a large organization, and high leadership turnover throughout the project. How these challenges were addressed, and the lessons learned for future evaluations, are discussed.
Implementation and Evaluation of an Evidence-Based Treatment of Disruptive Behaviour within a Children's Mental Health Program
This practice note details the synchronous implementation and developmental evaluation of a novel model of care that integrates an evidence-based treatment of children with disruptive behaviour within a community-based children's mental health centre in Canada. The overall objective of the project was to evaluate the impact and viability of the integrated program. A multidisciplinary team used a "knowledge to action" framework to guide implementation activities and a developmental evaluation approach to support learning and innovation, as well as measure program success. Key insights and lessons learned with regards to building staff capacity and appreciation for evaluation, balancing clinical and evaluation practices, and sustaining evaluation procedures are discussed. Suggestions for future similar efforts within a children's mental health context are provided.
Inspiring Future Program Evaluators through Innovative Curriculum for Undergraduates
Instruction in program evaluation is challenging given the inherent interdisciplinary nature of the field. As well, there is no one discipline typically dedicated to evaluation training, and few formal programs and university course offerings exist. Despite these limitations, training and education at the postsecondary level continues to be vital in supporting the professionalization of program evaluation, especially as it is a requirement for credentialing. The current article presents an innovative project comprising both education and hands-on training of program evaluation practices for undergraduate students. The project involved in-class lectures targeting specific program evaluation competencies and a program evaluation assignment in an upper-level undergraduate psychology course. Students were asked to develop a logic model and identify psychometrically sound evaluation measures based on an existing community organization's program or on a theoretical example. At the end of the course, students ( N = 58) completed surveys to assess their achieved evaluation competencies and experience with program evaluation. Overall, students gained evaluation-specific skills and knowledge, and the assignment was successful in promoting interest in program evaluation as a discipline. It is our hope that the current project can support faculty to integrate program evaluation in engaging and meaningful ways into their own curriculum.
Évaluation de la qualité d'un partenariat dans le cadre de la mise en place d'une intervention intersectorielle de type école, famille et communauté
The partnership represents a principle of action at the heart of public policies of various kinds. It can be defined as a formal association between organizations that decide to cooperate to achieve a common goal. The formative evaluation of quality conditions of the collaborative action is useful to support partners in improving their practice. This article describes first a tool available to achieve a collective diagnosis by members of a partnership investing voluntarily in an evaluation process and then presents a concrete application of it. It concludes by exposing some knowledge and lessons learned from the experience of the evaluation of a partnership for a comprehensive and integrated cross-sectoral school-community intervention.
Book Reviews: Trevisan, M. S., & Walser, T. M. (2015) Evaluability assessment: Improving evaluation quality and use. Thousand Oaks, CA: Sage. 181 pages. ISBN 978-1452282442.
Book Reviews: Trevisan, M. S., & Walser, T. M. (2015) Evaluability assessment: Improving evaluation quality and use. Thousand Oaks, CA: Sage. 181 pages. ISBN 978-1452282442.
Book Reviews: Donaldson, S. L. (Ed.). (2013). The future of evaluation in society: A tribute to Michael Scriven. Charlotte, NC: Information Publishing. ISBN 978-1-62396-451-1.
Book Reviews: Donaldson, S. L. (Ed.). (2013). The future of evaluation in society: A tribute to Michael Scriven. Charlotte, NC: Information Publishing. ISBN 978-1-62396-451-1.