Sustainability Analysis of Intervention Benefits: A Theory of Change Approach
Abstract: Theories of change (ToCs) are models of how an intervention is expected to bring about changes in results of interest—the benefits. Robust ToCs spell out the conditions under which the intervention should “work,” that is, bring about or contribute to the desired benefits. But rarely does the ToC address issues of the future sustainability of the benefits from the intervention. The article discusses how ToCs can adopt a sustainability lens and, along with the intervention, become sustainability- aware, by undertaking a sustainability analysis, exploring the sustainability of the various key ToC elements, and seeking to identify and support sustainability mechanisms.
This is the advances online verison.
Toward Learning from Change Pathways: Reviewing Theory of Change and Its Discontents
Abstract: The concept of Theory of Change (ToC) is well established in the evaluation literature, underpinning substantial research and practice eff orts. However, its ability to facilitate learning has been increasingly debated. The objective of this paper is to identify, characterize, and evaluate concerns over the use of ToCs based on a review of relevant studies. Seven concerns are found: distinguishing ToCs from other evaluation approaches, conceptual vagueness, under-developed ToCs, undercontribution to theoretical knowledge, uncertainty in stakeholder engagement, neglecting context, and overlooking complexity. Priority areas for improvement include integrating context and complexity throughout the ToC process, contributing theoretical knowledge, and engaging stakeholders as appropriate.
This is the advances online verison.
Anticipating and Addressing Stakeholders’ Stereotypes of Evaluation
It is likely that program and policy stakeholders hold biases and stereotypes about both evaluators and evaluation. Evaluators can challenge stakeholders’ negative stereotypes through communication strategies and intentional word choice, but first they must recognize the existence of stakeholders’ stereotypes. This Practice Note describes an instructional activity designed to help evaluators and participants in evaluator education experiences hone their communication skills, with a focus on describing evaluative perspectives, processes, and tools in a way that is accessible to stakeholders. The process helps evaluators both to be better aware of the stereotypes and biases that stakeholders are likely to possess and to provide language to help challenge negative and/or inappropriate perceptions. The Practice Note also provides a list of common biases about evaluation, and language to help evaluators anticipate and address them.
Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System
Dancing between “Zoom In” and “Zoom Out” Perspectives to Evaluate Social Innovation Labs
Social Innovation (SI) Labs are increasingly gaining attention as a specifi c class of social innovation promising to effect systems change. Evaluation of SI Labs is in its infancy, and so there is a limited track record of systematic evidence and learning to support the practice of SI Labs. This practice note shares insights and learning from the process of evaluating a certain model of SI Labs grounded in the “zoom in–zoom out” approach. This represents a hybrid approach that combines elements of developmental evaluation with elements of other evaluation approaches to enable the interplay between two complementary perspectives: a more in-depth look into each phase of the Lab (“zoom in”) and a broader look at the Lab in its entirety (“zoom out”), and its contribution to a more established place-based strategy.
Missing in Action: Indigenous Knowledge Systems in Evaluation of Comprehensive Community Initiatives
Comprehensive community initiatives (CCIs) coordinate social and structural change across multiple community sectors and represent promising approaches to complex social problems. Nowhere is this more relevant than for Indigenous children and families. However, strategies to evaluate initiatives to enhance Indigenous well-being must be meaningful to Indigenous communities. A review of literature regarding evaluation of CCIs identified various principles, case studies, methodologies, and methods grounded in Western ways of knowing and approaches to research. Research that engages with Indigenous-led comprehensive community initiatives is needed to enhance evaluation practices for CCIs that enables resurgence of Indigenous traditions and worldviews.
The Use and Benefits of Evaluation Framework Modules at the Canadian Foundation for Healthcare Improvement: Engaged Capacity Building and Collaborative Program Evaluation Planning
The Canadian Foundation for Healthcare Improvement (CFHI) shares its experience using a collaborative approach to developing evaluation frameworks. CFHI introduced a series of four participatory workshop modules where technical and content experts co-design evaluation frameworks. This method leverages the expertise within the evaluation team to build organization-wide evaluation capacity. This practice note describes the modules, their delivery, and lessons learned. Participants’ feedback suggests that evaluation modules are an eff ective strategy to strengthen the relationship and communication between evaluation experts and program staff .
Review: Leila C. Kahwati & Heather L. Kane. (2019). Qualitative Comparative Analysis in Mixed Methods Research and Evaluation.
Dancing between “Zoom In” and “Zoom Out”-Perspectives to Evaluate Social Innovation Labs
Review: Stephanie D. H. Evergreen. (2019). Eff ective Data Visualization: The Right Chart for the Right Data. Second Edition.
Stephanie Evergreen has a solid reputation as an expert on data visualization (dataviz) and reporting in evaluation. Her blog (https://stephanieevergreen.com/ blog) is on my list of resources for how to make a phonebook of data into a focused visual. Her most recent book on dataviz, the second edition of Eff ective Data Visualization: The Right Chart for the Right Data, and its companion Th e Data Visualization Sketchbook are the “in real life” (IRL) version of her blog.
Toward Learning from Change Pathways: Reviewing Theory of Change and Its Discontents
Application of an Evaluation Framework for Extra-Organizational Communities of Practice: Assessment and Refinement
Communities of practice (CoPs) are groups of people who work together on an ongoing basis and share knowledge and expertise. CoPs exist both within and outside of organizations, although extra-organizational CoPs have received less evaluation attention. The primary objective of this study was to assess the applicability of a multi-level, multiple-value evaluation framework for extra-organizational CoPs. Qualitative interviews were conducted with an extra-organizational CoP— the Canadian Community of Practice in Ecosystems Approaches to Health (CoPEH-Canada). The evaluation framework oriented both the member interview guide and the deductive content analysis. Th e findings showed that the evaluation framework was sufficiently comprehensive to capture the values generated. Following refl ection on these findings, challenges in its application and suggested revisions to the framework are provided; also discussed are limitations and strengths, evaluation research next steps, and the opportunities for future applications.
Sustainability Analysis of Intervention Benefits: A Theory of Change Approach
Theories of change (ToCs) are models of how an intervention is expected to bring about changes in results of interest—the benefits. Robust ToCs spell out the conditions under which the intervention should “work,” that is, bring about or contribute to the desired benefits. But rarely does the ToC address issues of the future sustainability of the benefits from the intervention. The article discusses how ToCs can adopt a sustainability lens and, along with the intervention, become sustainability-aware, by undertaking a sustainability analysis, exploring the sustainability of the various key ToC elements, and seeking to identify and support sustainability mechanisms.
Missing in Action: Indigenous Knowledge Systems in Evaluation of Comprehensive Community Initiatives
Abstract: Comprehensive community initiatives (CCIs) coordinate social and structural change across multiple community sectors and represent promising approaches to complex social problems. Nowhere is this more relevant than for Indigenous children and families. However, strategies to evaluate initiatives to enhance Indigenous well-being must be meaningful to Indigenous communities. A review of literature regarding evaluation of CCIs identified various principles, case studies, methodologies, and methods grounded in Western ways of knowing and approaches to research. Research that engages with Indigenous-led comprehensive community initiatives is needed to enhance evaluation practices for CCIs that enables resurgence of Indigenous traditions and worldviews.
This is the advanced online version
Qualitative Comparative Analysis in Mixed Methods Research and Evaluation. Reviewed by Debbie Gowensmith, University of Denver/ Groundswell Services, Inc
Leila C. Kahwati & Heather L. Kane. (2019).
Qualitative Comparative Analysis in Mixed Methods Research and Evaluation.
Mixed Methods Research Series: Vol. 6. Los Angeles, CA: SAGE.
Paperback, 289 pages. (ISBN: 978-1- 5063-9021-5)
Though evaluators often want to understand the combination of complex conditions that lead to specific outcomes, many traditional statistical tests merely yield information about the significance of discrete variables rather than how specifi c confi gurations of variables lead to an outcome. For these complex situations, qualitative comparative analysis (QCA) is a method worth considering. QCA was initially developed in the late 1980s by Charles C. Ragin (now at the University of California-Irvine), who nestled set theory logic and Boolean algebra within a deep knowledge of cases in order to assess which combinations of conditions were necessary or sufficient contributors to outcomes. In this book, author-evaluators Leila C. Kahwati and Heather L. Kane provide a primer on QCA with explanation and examples of its application in mixed methods evaluation, including recent developments in the method since its creation.
This is the advanced online version
The Use and Benefits of Evaluation Framework Modules at the Canadian Foundation for Healthcare Improvement: Engaged Capacity Building and Collaborative Program Evaluation Planning
Abstract: The Canadian Foundation for Healthcare Improvement (CFHI) shares its experience using a collaborative approach to developing evaluation frameworks. CFHI introduced a series of four participatory workshop modules where technical and content experts co-design evaluation frameworks. This method leverages the expertise within the evaluation team to build organization-wide evaluation capac-ity. This practice note describes the modules, their delivery, and lessons learned. Participants’ feedback suggests that evaluation modules are an eff ective strategy to strengthen the relationship and communication between evaluation experts and program staff .
This is the advanced online version.
Anticipating and Addressing Stakeholders’ Stereotypes of Evaluation
Abstract: It is likely that program and policy stakeholders hold biases and stereo-types about both evaluators and evaluation.
Evaluators can challenge stakeholders’ negative stereotypes through communication strategies and intentional word choice, but first they must recognize the existence of stakeholders’ stereotypes.
Th is Practice Note describes an instructional activity designed to help evaluators and participants in evaluator education experiences hone their communication skills, with a focus on describing evaluative perspectives, processes, and tools in a way that is accessible to stakeholders.
The process helps evaluators both to be better aware of the stereotypes and biases that stakeholders are likely to possess and to provide language to help challenge negative and/or inappropriate perceptions. The Practice Note also provides a list of common biases about evaluation, and language to help evaluators anticipate and address them.
This is the advanced online version.
Application of an Evaluation Framework for Extra-Organizational Communities of Practice: Assessment and Refinement
Effective Data Visualization: The Right Chart for the Right Data.
Second Edition. Thousand Oaks,
CA: SAGE. Paperback, 352 pages.
(ISBN: 978-1-5443-5088-2)
Stephanie D. H. Evergreen. (2019).
The Data Visualization Sketchbook. Thousand Oaks, CA: SAGE. Spiral, 136 pages. (ISBN: 978-1-5443-8469-6) Reviewed by Lisa M. P. O’Reilly MPA, CE Stephanie Evergreen has a solid reputation as an expert on data visualization (dataviz) and reporting in evaluation. Her blog (https://stephanieevergreen.com/ blog) is on my list of resources for how to make a phonebook of data into a focused visual. Her most recent book on dataviz, the second edition of Eff ective Data Visualization: The Right Chart for the Right Data, and its companion Th e Data Visualization Sketchbook are the “in real life” (IRL) version of her blog. Much of the book is a repurposing of work published online. As is the case in her blog, Evergreen’s tone in the book is light and clear. We are her colleagues with whom she is sharing something she has figured out. There are helpful asides and words of encouragement that endear her to us and make the text an enjoyable read. (Note: To get the most from this resource, read with MS Excel open, prefer-ably while online, and play along. Readers will finish with a file of useful example visuals. Thanks Stephanie!)
This is the advanced online version
Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System
Abstract: This article examines the differences between approaching an evaluation problem from a program perspective and doing so from a systems perspective.
The terms program , systems , systems thinking, and systems concepts are first defined. Then, using an actual evaluation of a cardiac care system, it is shown how initial investments in a program theory approach were deemed inadequate to account for the influence of external factors on patient outcomes.
It was decided that a systems thinking approach was more appropriate for evaluating the interactions between several agencies comprising the cardiac care system.
It is then shown how System Evaluation Theory (SET) was used to systematically apply different systems concepts to define and evaluate the cardiac care system.
The discussion compares and contrasts the program and system evaluation approaches, noting the conditions under which each is more appropriate. It concludes by noting scope and cost diff erences between the two approaches.
This is the advanced online version.