Evaluation in Transition: The Promise and Challenge of South - South Cooperation

In order for the evaluation field to ensure its salience over the next decade, high-profile areas of work have to be sought where evaluative practices are underexplored or undervalued yet can help inspire and accelerate urgently needed transformations while also advancing evaluation theory and practice. This article highlights one such opportunity, offered by South-South cooperation (SSC), an increasingly powerful force in international development yet overshadowed by the frameworks, narratives, and approaches of North-South cooperation (NSC), better known as international development cooperation or development aid. The values, principles, achievements, and challenges that define SSC are seldom discussed at evaluation events or in the evaluation literature, and development evaluation continues to be shaped largely by theories from the Global North and by North-South interactions, despite the growing prominence of SSC and the Global South in world affairs. This article is dedicated to creating awareness of the current situation by highlighting how more intensive engagement with the unique aspects and underexplored opportunities of SSC might help shift paradigms and practices in evaluation, especially but not only in the Global South.

Pour que le domaine de l’évaluation assure sa pertinence au cours de la prochaine décennie, il faudra cerner des secteurs de travail où les pratiques d’évaluation sont sous-explorées ou sous-évaluées, mais pourraient tout de même aider à inspirer et à accélérer des transformations dont le besoin se fait urgemment sentir, tout en continuant à faire progresser la théorie et la pratique de l’évaluation. Le présent article souligne une telle occasion, offerte par la coopération Sud-Sud, un mouvement qui a le vent dans les voiles dans le domaine du développement international, mais qui reste dans l’ombre des cadres, des trames narratives et des approches de la coopération nord-sud, mieux connue sous le nom de coopération au développement international ou aide au développement. Les valeurs, les principes, les réalisations et les défis qui définissent la coopération Sud-Sud font rarement l’objet de discussions lors d’activités d’évaluation ou dans la recherche sur l’évaluation; et l’évaluation dans le domaine du développement continue à être largement façonnée par les théories du Nord et par les interactions nord-sud, malgré l’importance croissante de la coopération Sud-Sud et du Sud global dans les affaires mondiales. Le présent article est consacré à la sensibilisation à la situation actuelle en mettant en lumière la façon dont un engagement plus intensif à l’égard des aspects uniques et des possibilités sous-explorées de la coopération sud-sud pourrait aider à modifier les paradigmes et les pratiques en évaluation, particulièrement, mais non seulement dans le Sud.


Introduction: Transforming Evaluation Practice for “Business Unusual”

The articles in this thematic segment are all in themselves “unusual” in relation to the usual business of the CJPE. Rather than reporting on empirical work or methodological innovations, this collection of articles considers ways in which evaluation practice could now be transformed to contribute to efforts that support “Business Unusual.” I am therefore delighted, and most grateful to the CJPE for having agreed to their publication. The driving motivation for coordinating this collection has been prompted by my reflection on more than 30 years of experience in doing and commissioning evaluations in the public health sector. The power and dynamics of neoliberalist socio-political economics have been a major influence on what has been, and still is, evaluated, the motivation behind such evaluations, and their ultimate use. It will become obvious that, in developing this segment, I am far from being alone in recognizing that a fundamental change is needed, and urgently.

Conclusion: Transforming Evaluation Practice for “Business Unusual”

More than ever before, the idea of returning to “business as usual” is being challenged. Over the last few years, we have witnessed unprecedented health and socio-economic impacts from climate change-induced catastrophes: extensive wildfires, floods, droughts, and disease transmission. There are new, emerging infectious diseases already present in certain regions of the world which have spread rapidly to a large proportion of the population. The various Severe Acute Respiratory Syndrome (SARS) epidemics of recent years are one such example. But the most recent SARS COV-2 (referred hereafter by its more popular name of COVID-19) has been a pandemic that continues to have devastating impacts on the world’s social, political, economic, education, and especially public health systems as we experience distinctly different COVID pandemics around the world.

Beverly Parsons, Lovely Dhillon, and Matt Keene (Eds.). (2020). Visionary Evaluation for a Sustainable, Equitable Future . Charlotte, NC: Information Age, 276 pages.

In stark contrast to the imminent dystopia often depicted in today’s media,Visionary Evaluation for a Sustainable, Equitable Future brings the reader some much-needed optimism. Through engaging, Socratic dialogue, readers are ushered into 2030, a year the authors have designated the Anthropocene, where anthropogenic ecological and climate change have nearly reached a point of no return on Earth. Coincidentally, this benchmark coincides with the Sustainable Development Goals’ (SDGs) conclusion, in which collective learning, knowledge sharing, and capacity building are expected to contribute to stronger evaluation systems and practices. This alignment is reflected in introductory chapters (Chapters 1–3) which set the stage using self-reflective interchange, in an attempt to find solutions to the world’s most difficult challenges. Challenges such as the trend toward mass extinction, technological advancement and artificial intelligence, and environmental sustainability and equity are explored in detail, using collaborative evaluation and transnational partnership as potential means to overcome these obstacles for the planet and its peoples in this new epoch.

Roots and Relations: Celebrating Good Medicine in Indigenous Evaluation

Wuliit Wchapikal (wool-leet which-ah-pee-kul): Good Medicine! That is what the new “Roots and Relations” (R&R) permanent section in CJPE intends to be. The purpose of the R&R journey is to honour our lineage, grow our kinship, and sustain our intergenerational legacies of Indigenous wisdom and practices in evaluation. R&R will sacredly hold and protect traditional knowledge, respect and assert sovereignty, provide a nurturing space for Indigenous voices, and celebrate Indigenous innovations in evaluation.

The R&R logo was conceptualized by Larry Bremner CE, FCES (he/him, Métis) and Dr. Nicole Bowman (Lunaape/Mohican) with traditional Knowledge Keepers and a graphic artist who is a traditional Longhouse participant and a citizen of the Oneida Nation. The Oneida digital artist says, “The Roots and Relations image is symbolic of our creation coming together. I added celestial trees, earthly flowers, and medicine plants. I’ve added the sun, moon, water, and stars. The face at the bottom is representative of our ancestors. The angular designs are representative of everything that grows here on earth.”

A Multi-Stage Approach to Qualitative Sampling within a Mixed Methods Evaluation: Some Refections on Purpose and Process

Abstract:  We share experiences from a mixed methods evaluation in rural India  that combines a randomized controlled trial (RCT) of 400 villages with embedded case studies in four villages. Specifcally, we present two lessons from the multi-stage  sampling approach adopted to select the four case-study villages, which frst prioritized key-informant observations regarding intervention status in order to shortlist locations and subsequently used data from the RCT’s baseline survey to select the final sample. In doing so, we highlight how large-scale mixed methods program  evaluations in education can go beyond questions of “what works” to answering those of “how,” “why,” and “why not.”

Trena M. Paulus and Jessica N. Lester. (2021). Doing Qualitative Research in a Digital World. Thousand Oaks, CA: Sage. VitalSource. Paperback, 376 pages.


Editor’s Remarks

The diversity, vitality, and adaptability of our evaluation community shine through in this issue of the CJPE. I am pleased to introduce a new section, titled “Roots and Relations: Celebrating Good Medicine in Indigenous Evaluation.” Larry Bremner and Nicole Bowman, who have graciously accepted to lead us as section editors, outline their vision for the section, which opens up new opportunities to Indigenous students, scholars, practitioners, policymakers, and evaluators to share their work in different ways. This is an important step in decolonizing our evaluation and academic practices, and the entire team is grateful to those who will contribute to this space for years to come.

Next, a thematic segment edited by Marlène Laeubli Loud draws lessons learned from evaluating during a pandemic, and how our approaches and processes must adapt to “Business Unusual” in this era of societal and environmental changes. Contributions to this segment include a paper from Zenda Ofir on the role of South-South collaboration in shifting traditional narratives in evaluation, a paper from Louise Gallagher and Zenda Ofir on the rights of nature as a framing construct for evaluation practice, and a paper from Adam Hejnowicz and Scott Chaplowe on data science and its potential to further evaluative inquiry. These topics are certain to resonate with evaluators around the world.

The Canadian Journal of Program Evaluation- Editor's Remarks

Practice Notes from a Participatory Impact Evaluation of a Leadership Development Program for People Living with HIV

Interest in participatory evaluation and other collaborative inquiry approaches has risen substantially over the past few decades. However, there appears to be a lack of practical information about using and applying participatory evaluation approaches on the ground. This Practice Note uses a participatory impact evaluation of a leadership development program for people living with HIV and findings from a meta-evaluation of this work to describe the participatory evaluation approach adopted, challenges and lessons learned related to conducting a participatory evaluation, and some key factors and implications to consider for maximizing the success of future participatory evaluations.

L’intérêt pour l’évaluation participative et d’autres méthodes d’enquête collaborative a considérablement augmenté au cours des dernières décennies. Cependant, il semble y avoir un manque d’informations pratiques concernant l’utilisation et l’application des approches d’évaluation participative sur le terrain. Cette note sur la pratique utilise une évaluation d’impact participative du programme de développement du leadership pour les personnes vivant avec le VIH et les résultats d’une méta-évaluation de ce programme pour décrire la méthode d’évaluation participative adoptée, les difficultés rencontrées et les enseignements tirés de la réalisation d’une évaluation participative, et certains facteurs et implications clés à prendre en compte pour maximiser le succès des futures évaluations participatives.

Evaluation Advisory Groups: Considerations for Design and Management

Cultivating Cultural Competence

Cultural competence is a complex and contested notion. Yet it remains integral to working with difference in the context of evaluation practice. Given its status in evaluation practice, the field’s commitment to cultural competence prompts the need for further interrogation and reconsideration. Accordingly, this article explores the establishment and conceptualization of cultural competence. Potential challenges to cultural competence are also examined. In consideration of these challenges, an alternative framework is offered based on the philosophy of Emanuel Levinas. This work aims to support the evaluation community’s ability to work with cultural diversity, a vital aspect of evaluation practice.

La compétence culturelle est une notion complexe et contestée. Pourtant, elle reste un élément incontournable de la pratique de l’évaluation. Compte tenu de ce contexte, l’engagement du domaine envers la compétence culturelle entraîne la nécessité de s’interroger davantage sur le sujet et de le considérer à partir de nouveaux points de vue. Le présent article explore l’établissement et la conceptualisation de la compétence culturelle. Des défis potentiels liés à la compétence culturelle sont aussi examinés. Compte tenu de ces défis, un nouveau cadre est offert, qui s’appuie sur la philosophie d’Emanuel Levinas. Ce travail vise à appuyer la capacité qu’a la communauté de l’évaluation de travailler dans un contexte de diversité culturelle, ce qui constitue un aspect vital de la pratique de l’évaluation.

kawiyahîtamik kesi wîcehtâsôk : (To Examine in Order to Support/Redirect)

Recently, human-service providers across Canada have made conscious efforts toward reconciliation through Indigenizing programming. However, while the delivery of programs has shifted, how they are evaluated remains rooted in Western ideologies and methodologies. In response to the tension created by using Western evaluation methods for assessing Indigenous-designed programs, we have developed an Indigenous program evaluation framework based in nêhiyaw (Cree) teachings and co-created by Elders and Knowledge Keepers. We use an illustrative example to demonstrate how an appropriately developed Indigenous program evaluation framework leads to more comprehensive, accurate, and meaningful data collection, evaluation, and recommendations.

Récemment, les agences de santé et de services sociaux à travers le Canada ont fait des efforts conscients de réconciliation grâce à l’adaptation de leurs programmes aux besoins des communautés autochtones. Cependant, alors que la prestation des programmes a changé, la façon dont ils sont évalués reste enracinée dans les idéologies et méthodologies occidentales. En réponse à la tension créée par l’utilisation des méthodes d’évaluation occidentales pour évaluer les programmes conçus par les communautés autochtones, nous avons élaboré un cadre d’évaluation des programmes autochtones fondé sur les enseignements nêhiyaw (cris) et co-créé par les aînés et les gardiens du savoir. Nous utilisons un exemple illustratif pour démontrer comment un cadre d’évaluation de programme autochtone bien développé mène à une collecte de données, une évaluation et des recommandations plus complètes, précises et significatives.

Catching the Wave: Harnessing Data Science to Support Evaluation’s Capacity for Making a Transformational Contribution to Sustainable Development

In this article we set ourselves three challenges: first, to examine the potential of the data revolution to aid the transformational change required to achieve sustainable development goals second, and the ability of evaluation to contribute to greater engagement with data science technologies; and third, the capacity of data science to further evaluation as an innovative and progressive field of inquiry. We also discuss the political, economic, cultural, and ethical challenges that data science presents to sustainable development and evaluation. We conclude that data science and evaluation can enhance each other to address the key development challenges of our time.

Dans cet article, nous nous sommes donnés trois défis : premièrement, examiner le potentiel de la « révolution des données » d’appuyer les changements transformationnels requis pour atteindre les objectifs de développement durable; deuxièmement, la capacité de l’évaluation de contribuer davantage grâce à un engagement accru envers les technologies de la science des données; et troisièmement, la capacité de la science des données de faire progresser l’évaluation comme champ d’études novateur et progressif. Nous discutons aussi des défis politiques, économiques, culturels et éthiques posés par la science des données à l’égard du développement durable et de l’évaluation. Nous arrivons à la conclusion que la science des données et l’évaluation peuvent s’améliorer mutuellement pour répondre aux principaux défis en matière de développement de notre époque.

The Rights of Nature: An Emerging Transformation Opportunity for Evaluation

“Building back better” in the post-COVID-19 Anthropocene era requires novel ideas and ways of working to truly challenge “business as usual” and contribute to urgently needed systems transformations. This article invites post-normal evaluation professionals to engage with the concept of the Rights of Nature, a generative form of institutional innovation that recognizes ecosystems and natural communities as entities that have an independent right to exist and flourish that can be enforced under legal or social norms. Pathways are suggested to use evaluation as values-driven practice to reflect on and encourage human-nature relationships founded on mutual dependence, cooperation, and synergy.

Pour "reconstruire en mieux" à l’ère de l’Anthropocène post-COVID-19, il faut de nouvelles idées et méthodes de travail qui nous permettent véritablement de remettre en question le "business as usual" et de contribuer aux transformations systémiques nécessaires et urgentes. Cet article invite les professionnels de l’évaluation de l’ère « post-normale » à s’engager dans le concept des droits de la nature, une forme générative d’innovation institutionnelle qui reconnaît les écosystèmes et les communautés naturelles comme des entités ayant un droit indépendant à l’existence et à l’épanouissement qui peut être appliqué en vertu de normes juridiques ou sociales. Des pistes sont suggérées pour utiliser l’évaluation comme une pratique axée sur les valeurs afin de réfléchir aux relations entre l’humain et la nature, fondées sur la dépendance mutuelle, la coopération et la synergie, et de les encourager.

A Multi-Stage Approach to Qualitative Sampling within a Mixed Methods Evaluation: Some Reflections on Purpose and Process

We share experiences from a mixed methods evaluation in rural India that combines a randomized controlled trial (RCT) of 400 villages with embedded case studies in four villages. Specifically, we present two lessons from the multi-stage sampling approach adopted to select the four case-study villages, which first prioritized key-informant observations regarding intervention status in order to shortlist locations and subsequently used data from the RCT’s baseline survey to select the final sample. In doing so, we highlight how large-scale mixed methods program evaluations in education can go beyond questions of “what works” to answering those of “how,” “why,” and “why not.”