Toward Learning from Change Pathways: Reviewing Theory of Change and Its Discontents
Abstract: The concept of Theory of Change (ToC) is well established in the evaluation literature, underpinning substantial research and practice eff orts. However, its ability to facilitate learning has been increasingly debated. The objective of this paper is to identify, characterize, and evaluate concerns over the use of ToCs based on a review of relevant studies. Seven concerns are found: distinguishing ToCs from other evaluation approaches, conceptual vagueness, under-developed ToCs, undercontribution to theoretical knowledge, uncertainty in stakeholder engagement, neglecting context, and overlooking complexity. Priority areas for improvement include integrating context and complexity throughout the ToC process, contributing theoretical knowledge, and engaging stakeholders as appropriate.
This is the advances online verison.
Dancing between “Zoom In” and “Zoom Out”-Perspectives to Evaluate Social Innovation Labs
Missing in Action: Indigenous Knowledge Systems in Evaluation of Comprehensive Community Initiatives
Abstract: Comprehensive community initiatives (CCIs) coordinate social and structural change across multiple community sectors and represent promising approaches to complex social problems. Nowhere is this more relevant than for Indigenous children and families. However, strategies to evaluate initiatives to enhance Indigenous well-being must be meaningful to Indigenous communities. A review of literature regarding evaluation of CCIs identified various principles, case studies, methodologies, and methods grounded in Western ways of knowing and approaches to research. Research that engages with Indigenous-led comprehensive community initiatives is needed to enhance evaluation practices for CCIs that enables resurgence of Indigenous traditions and worldviews.
This is the advanced online version
Qualitative Comparative Analysis in Mixed Methods Research and Evaluation. Reviewed by Debbie Gowensmith, University of Denver/ Groundswell Services, Inc
Leila C. Kahwati & Heather L. Kane. (2019).
Qualitative Comparative Analysis in Mixed Methods Research and Evaluation.
Mixed Methods Research Series: Vol. 6. Los Angeles, CA: SAGE.
Paperback, 289 pages. (ISBN: 978-1- 5063-9021-5)
Though evaluators often want to understand the combination of complex conditions that lead to specific outcomes, many traditional statistical tests merely yield information about the significance of discrete variables rather than how specifi c confi gurations of variables lead to an outcome. For these complex situations, qualitative comparative analysis (QCA) is a method worth considering. QCA was initially developed in the late 1980s by Charles C. Ragin (now at the University of California-Irvine), who nestled set theory logic and Boolean algebra within a deep knowledge of cases in order to assess which combinations of conditions were necessary or sufficient contributors to outcomes. In this book, author-evaluators Leila C. Kahwati and Heather L. Kane provide a primer on QCA with explanation and examples of its application in mixed methods evaluation, including recent developments in the method since its creation.
This is the advanced online version
The Use and Benefits of Evaluation Framework Modules at the Canadian Foundation for Healthcare Improvement: Engaged Capacity Building and Collaborative Program Evaluation Planning
Abstract: The Canadian Foundation for Healthcare Improvement (CFHI) shares its experience using a collaborative approach to developing evaluation frameworks. CFHI introduced a series of four participatory workshop modules where technical and content experts co-design evaluation frameworks. This method leverages the expertise within the evaluation team to build organization-wide evaluation capac-ity. This practice note describes the modules, their delivery, and lessons learned. Participants’ feedback suggests that evaluation modules are an eff ective strategy to strengthen the relationship and communication between evaluation experts and program staff .
This is the advanced online version.
Anticipating and Addressing Stakeholders’ Stereotypes of Evaluation
Abstract: It is likely that program and policy stakeholders hold biases and stereo-types about both evaluators and evaluation.
Evaluators can challenge stakeholders’ negative stereotypes through communication strategies and intentional word choice, but first they must recognize the existence of stakeholders’ stereotypes.
Th is Practice Note describes an instructional activity designed to help evaluators and participants in evaluator education experiences hone their communication skills, with a focus on describing evaluative perspectives, processes, and tools in a way that is accessible to stakeholders.
The process helps evaluators both to be better aware of the stereotypes and biases that stakeholders are likely to possess and to provide language to help challenge negative and/or inappropriate perceptions. The Practice Note also provides a list of common biases about evaluation, and language to help evaluators anticipate and address them.
This is the advanced online version.
Application of an Evaluation Framework for Extra-Organizational Communities of Practice: Assessment and Refinement
Effective Data Visualization: The Right Chart for the Right Data.
Second Edition. Thousand Oaks,
CA: SAGE. Paperback, 352 pages.
(ISBN: 978-1-5443-5088-2)
Stephanie D. H. Evergreen. (2019).
The Data Visualization Sketchbook. Thousand Oaks, CA: SAGE. Spiral, 136 pages. (ISBN: 978-1-5443-8469-6) Reviewed by Lisa M. P. O’Reilly MPA, CE Stephanie Evergreen has a solid reputation as an expert on data visualization (dataviz) and reporting in evaluation. Her blog (https://stephanieevergreen.com/ blog) is on my list of resources for how to make a phonebook of data into a focused visual. Her most recent book on dataviz, the second edition of Eff ective Data Visualization: The Right Chart for the Right Data, and its companion Th e Data Visualization Sketchbook are the “in real life” (IRL) version of her blog. Much of the book is a repurposing of work published online. As is the case in her blog, Evergreen’s tone in the book is light and clear. We are her colleagues with whom she is sharing something she has figured out. There are helpful asides and words of encouragement that endear her to us and make the text an enjoyable read. (Note: To get the most from this resource, read with MS Excel open, prefer-ably while online, and play along. Readers will finish with a file of useful example visuals. Thanks Stephanie!)
This is the advanced online version
Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System
Abstract: This article examines the differences between approaching an evaluation problem from a program perspective and doing so from a systems perspective.
The terms program , systems , systems thinking, and systems concepts are first defined. Then, using an actual evaluation of a cardiac care system, it is shown how initial investments in a program theory approach were deemed inadequate to account for the influence of external factors on patient outcomes.
It was decided that a systems thinking approach was more appropriate for evaluating the interactions between several agencies comprising the cardiac care system.
It is then shown how System Evaluation Theory (SET) was used to systematically apply different systems concepts to define and evaluate the cardiac care system.
The discussion compares and contrasts the program and system evaluation approaches, noting the conditions under which each is more appropriate. It concludes by noting scope and cost diff erences between the two approaches.
This is the advanced online version.