Volume 35, 2020 - Fall

Toward Learning from Change Pathways: Reviewing Theory of Change and Its Discontents

Authors :

 Abstract: The concept of Theory of Change (ToC) is well established in the evaluation literature, underpinning substantial research and practice eff orts. However, its ability to facilitate learning has been increasingly debated. The objective of this paper is to identify, characterize, and evaluate concerns over the use of ToCs based on a review of relevant studies. Seven concerns are found: distinguishing ToCs from other evaluation approaches, conceptual vagueness, under-developed ToCs, undercontribution to theoretical knowledge, uncertainty in stakeholder engagement, neglecting context, and overlooking complexity. Priority areas for improvement include integrating context and complexity throughout the ToC process, contributing theoretical knowledge, and engaging stakeholders as appropriate.

VIEW TEXT

This is the advances online verison.

Dancing between “Zoom In” and “Zoom Out”-Perspectives to Evaluate Social Innovation Labs

Authors :
 
United Way Greater Toronto
 
Abstract: Social Innovation (SI) Labs are increasingly gaining attention as a specifi c
class of social innovation promising to effect systems change. Evaluation of SI Labs
is in its infancy, and so there is a limited track record of systematic evidence and
learning to support the practice of SI Labs. This practice note shares insights and
learning from the process of evaluating a certain model of SI Labs grounded in the
“zoom in–zoom out” approach. This represents a hybrid approach that combines
elements of developmental evaluation with elements of other evaluation approaches
to enable the interplay between two complementary perspectives: a more in-depth
look into each phase of the Lab (“zoom in”) and a broader look at the Lab in its entirety
(“zoom out”), and its contribution to a more established place-based strategy.
 
 
 
This is the advanced online version.

Missing in Action: Indigenous Knowledge Systems in Evaluation of Comprehensive Community Initiatives

Authors :

 Abstract: Comprehensive community initiatives (CCIs) coordinate social and structural change across multiple community sectors and represent promising approaches to complex social problems. Nowhere is this more relevant than for Indigenous children and families. However, strategies to evaluate initiatives to enhance Indigenous well-being must be meaningful to Indigenous communities. A review of literature regarding evaluation of CCIs identified various principles, case studies, methodologies, and methods grounded in Western ways of knowing and approaches to research. Research that engages with Indigenous-led comprehensive community initiatives is needed to enhance evaluation practices for CCIs that enables resurgence of Indigenous traditions and worldviews.

VIEW TEXT

This is the advanced online version

 

Qualitative Comparative Analysis in Mixed Methods Research and Evaluation. Reviewed by Debbie Gowensmith, University of Denver/ Groundswell Services, Inc

Leila C. Kahwati & Heather L. Kane. (2019).

Qualitative Comparative Analysis in Mixed Methods Research and Evaluation.

Mixed Methods Research Series: Vol. 6. Los Angeles, CA: SAGE.

Paperback, 289 pages. (ISBN: 978-1- 5063-9021-5) 

Though evaluators often want to understand the combination of complex conditions that lead to specific outcomes, many traditional statistical tests merely yield information about the significance of discrete variables rather than how specifi c confi gurations of variables lead to an outcome. For these complex situations, qualitative comparative analysis (QCA) is a method worth considering. QCA was initially developed in the late 1980s by Charles C. Ragin (now at the University of California-Irvine), who nestled set theory logic and Boolean algebra within a deep knowledge of cases in order to assess which combinations of conditions were necessary or sufficient contributors to outcomes. In this book, author-evaluators Leila C. Kahwati and Heather L. Kane provide a primer on QCA with explanation and examples of its application in mixed methods evaluation, including recent developments in the method since its creation.

VIEW TEXT

This is the advanced online version

The Use and Benefits of Evaluation Framework Modules at the Canadian Foundation for Healthcare Improvement: Engaged Capacity Building and Collaborative Program Evaluation Planning

Authors :

Abstract: The Canadian Foundation for Healthcare Improvement (CFHI) shares its experience using a collaborative approach to developing evaluation frameworks. CFHI introduced a series of four participatory workshop modules where technical and content experts co-design evaluation frameworks. This method leverages the expertise within the evaluation team to build organization-wide evaluation capac-ity. This practice note describes the modules, their delivery, and lessons learned. Participants’ feedback suggests that evaluation modules are an eff ective strategy to strengthen the relationship and communication between evaluation experts and program staff .

VIEW TEXT

This is the advanced online version.

 

 

Anticipating and Addressing Stakeholders’ Stereotypes of Evaluation

Authors :

Abstract: It is likely that program and policy stakeholders hold biases and stereo-types about both evaluators and evaluation.

Evaluators can challenge stakeholders’ negative stereotypes through communication strategies and intentional word choice, but first they must recognize the existence of stakeholders’ stereotypes.

Th is Practice Note describes an instructional activity designed to help evaluators and participants in evaluator education experiences hone their communication skills, with a focus on describing evaluative perspectives, processes, and tools in a way that is accessible to stakeholders.

The process helps evaluators both to be better aware of the stereotypes and biases that stakeholders are likely to possess and to provide language to help challenge negative and/or inappropriate perceptions. The Practice Note also provides a list of common biases about evaluation, and language to help evaluators anticipate and address them.

VIEW TEXT

This is the advanced online version.

 

Application of an Evaluation Framework for Extra-Organizational Communities of Practice: Assessment and Refinement

Authors :
Abstract: Communities of practice (CoPs) are groups of people who work together
on an ongoing basis and share knowledge and expertise. CoPs exist both within and
outside of organizations, although extra-organizational CoPs have received less
evaluation attention. The primary objective of this study was to assess the applicability
of a multi-level, multiple-value evaluation framework for extra-organizational
CoPs. Qualitative interviews were conducted with an extra-organizational CoP—
the Canadian Community of Practice in Ecosystems Approaches to Health (CoPEHCanada).
The evaluation framework oriented both the member interview guide and
the deductive content analysis. Th e findings showed that the evaluation framework
was sufficiently comprehensive to capture the values generated. Following refl ection
on these findings, challenges in its application and suggested revisions to the framework
are provided; also discussed are limitations and strengths, evaluation research
next steps, and the opportunities for future applications.
 
 
This is the advanced online version

Effective Data Visualization: The Right Chart for the Right Data.

Authors :

Second Edition. Thousand Oaks,

CA: SAGE. Paperback, 352 pages.

(ISBN: 978-1-5443-5088-2)

Stephanie D. H. Evergreen. (2019).

The Data Visualization Sketchbook. Thousand Oaks, CA: SAGE. Spiral, 136 pages. (ISBN: 978-1-5443-8469-6) Reviewed by Lisa M. P. O’Reilly MPA, CE Stephanie Evergreen has a solid reputation as an expert on data visualization (dataviz) and reporting in evaluation. Her blog (https://stephanieevergreen.com/ blog) is on my list of resources for how to make a phonebook of data into a focused visual. Her most recent book on dataviz, the second edition of Eff ective Data Visualization: The Right Chart for the Right Data, and its companion Th e Data Visualization Sketchbook are the “in real life” (IRL) version of her blog. Much of the book is a repurposing of work published online. As is the case in her blog, Evergreen’s tone in the book is light and clear. We are her colleagues with whom she is sharing something she has figured out. There are helpful asides and words of encouragement that endear her to us and make the text an enjoyable read. (Note: To get the most from this resource, read with MS Excel open, prefer-ably while online, and play along. Readers will finish with a file of useful example visuals. Thanks Stephanie!)

VIEW TEXT

This is the advanced online version

Comparing and Contrasting a Program versus System Approach to Evaluation: The Example of a Cardiac Care System

Authors :

Abstract: This article examines the differences between approaching an evaluation problem from a program perspective and doing so from a systems perspective.

The terms program , systems , systems thinking, and systems concepts are first defined. Then, using an actual evaluation of a cardiac care system, it is shown how initial investments in a program theory approach were deemed inadequate to account for the influence of external factors on patient outcomes.

It was decided that a systems thinking approach was more appropriate for evaluating the interactions between several agencies comprising the cardiac care system.

It is then shown how System Evaluation Theory (SET) was used to systematically apply different systems concepts to define and evaluate the cardiac care system.

The discussion compares and contrasts the program and system evaluation approaches, noting the conditions under which each is more appropriate. It concludes by noting scope and cost diff erences between the two approaches.

VIEW THE TEXT

This is the advanced online version.