Volume 34, 2019 - Fall

Editor’s Remarks

Authors :
 
I am pleased to present this issue of CJPE, which contains two theme segments as well as several practice notes and book reviews. Th e first theme segment, which includes an article by Catherine Fallon in French and an article by Lisa Birch and Steve Jacob in English, was guest edited by Jacob and focuses on democracy and evaluation. The second theme segment features a series of keynote addresses on reconciliation and culturally reponsive evaluation from the 2018 CES National Conference and was guest edited by Nan Wehipeihana. Our keynote speakers have generously put their addresses into writing in order to reach all CES members and other evaluators around the world. I would like to take this opportunity to thank Nicole Bowman-Farrell, Larry Bremner, Debbie Delancey, Kate McKegg, and Nan Wehipeihana for their contributions. In addition to these two segments, this issue also features an important update on contribution analysis by John Mayne, an innovative approach to partnership evaluation by Angèle Bilodeau and Gillian Kranias, an inventory of Canadian evaluation education programs by Theresa Hunter and James McDavid, and an evaluation exemplar by Catherine Fréchette-Simard, Jonathan Bluteau and Isabelle Plante. This issue also includes a research and practice note by Jeremy Acree on the collaborative approaches to evaluation principles, and one by Christopher Cook and collaborators on theatre-based evaluation. Finally, we include two book reviews. I thank our authors for their continued submissions and contributions and hope that this issue provides thought-provoking material for all of our readers.
 
Isabelle Bourgeois Editor-in-Chief
 

2018 Keynote Panel Members on Reconciliation and Culturally Responsive Evaluation— Rhetoric or Reality?

Authors :
Pages :
229-330

Introduction to Articles Prepared by CES Calgary

Debbie DeLancey

2018 CES Conference Co-Chair
When selecting a theme for the 2018 Canadian Evaluation Society Annual Conference, the co-chairs wanted to build on the CES Board’s 2016 motion in response to the Truth and Reconciliation Commission’s Calls to Action, committing to including reconciliation in its values, principles, and practices, with a specifi c commitment to strengthen the promotion of, and support for, culturally responsive evaluation. We wanted a theme that would challenge evaluators to explore what it means to truly work as collaborative partners and allies, not only with Indigenous governments and communities but also with other populations and communities that have traditionally been disempowered.

Revisiting Contribution Analysis

Authors :
Pages :
171-191

Abstract:

The basic ideas behind contribution analysis were set out in 2001. Since then, interest in the approach has grown and contribution analysis has been operationalized in different ways. In addition, several reviews of the approach have been published and raise a few concerns. In this article, I clarify several of the key concepts behind contribution analysis, including contributory causes and contribution claims. I discuss the need for reasonably robust theories of change and the use of nested theories of change to unpack complex settings. On contribution claims, I argue the need for causal narratives to arrive at credible claims, the limited role that external causal factors play in arriving at contribution claims, the use of robust theories of change to avoid bias, and the fact that opinions of stakeholders on the contribution made are not central in arriving at contribution claims.

Keywords: causal factors, causal narratives, contribution analysis, contribution claims, contributory causes, theories of change

 

Self-Evaluation Tool for Action in Partnership: Translation and Cultural Adaptation of the Original Quebec French Tool to Canadian English

Authors :
Pages :
192-206

Self-Evaluation Tool for Action in Partnership: Translation and Cultural Adaptation of the Original Quebec French Tool to Canadian English

Angèle Bilodeau
Université de Montréal

Gillian Kranias
Health Nexus

Abstract: This article presents the translation and cultural adaptation, into Canadian English, of the Outil diagnostique de l’action en partenariat, a tool widely used to support the practice of partnerships since its creation in French, in Quebec, in 2008. The theoretical foundations and properties of the original tool are presented, followed by a summary of methodological guidelines and a description of the process and results. The methodology involved an expert committee to formulate the English tool and verify its equivalence with the original, and a pretest with target users. Th is rigorous procedure ensures equivalence of the translated tool and its cultural adaptation to the intended users.

Keywords: action in partnership, action network, inter-sectoral action, partnership self-evaluation tool, translation-adaptation of measurement tools

 

Comparison of Canadian and American Graduate Evaluation Education Programs

Authors :
Pages :
207-234

M. Theresa Hunter and James C. McDavid

University of Victoria
Abstract: University education in evaluation is an important support for the professionalization of the field. Using internet sources and direct contacts with 105 Canadian university departments, our study develops a Canada-wide inventory of graduate courses and programs, with emphasis on evaluation. Comparisons are made with inventories of American graduate evaluation programs undertaken between 1980 and 2018. Although Canada has about half as many multi-course graduate programs in evaluation as were found in the United States in 2018, the relative number of Canadian programs is disproportionately greater, considering the two countries’ population sizes. Canada also has a more diverse range of disciplines offering evaluation education programs.
Keywords: Canada–US comparisons, evaluation education, inventory, professionalization

 

Évaluation de la mise en oeuvre du programme d’intervention In vivo chez les jeunes de 11–12 ans d’une école primaire québécoise présentant un problème de comportements intériorisés

Authors :
Pages :
235-254

Une étude exploratoire

Université du Québec à Montréal

Abstract: Internalized disorders are the most current mental health issue during childhood and adolescence. This study sought to assess the implementation of In vivo, an elementary school program aiming to treat anxiety with stress management strategies. The study is based on a mixed method design and collected data from the students experiencing the program (n=4, 11-12 y. o.), their parents (n=4), and other school staff members involved (n=7). Th e findings of the study show that the program is relevant in this specifi c context and also highlight the obstacles and facilitators to implementation.
Keywords: anxiety, program evaluation, elementary school, prevention, stress, internalized disorders.

Dramatizing Learning, Performing Ourselves: Stories of Theatre-Based Evaluation in Vancouver’s Downtown Eastside

Authors :
Pages :
255-271

UBC Learning Exchange, University of British Columbia

Abstract: Voices UP!, a play developed and performed collectively with participants from Vancouver’s Downtown Eastside, illustrates the use of theatre as an innovative evaluation method for data collection, analysis, and knowledge translation. Th is artful process can convey complex, experiential evaluation findings and create engaging opportunities for learning, while building relationships and skills among participants. In this article we describe the creation process utilized for this theatre-based evaluation project, as well as guiding principles and lessons learned for evaluators who may want to engage in similar theatre-based participatory work.
Keywords: arts-based evaluation, knowledge translation, participatory evaluation, research-based theatre

 

Applying the Collaborative Approaches to Evaluation (CAE) Principles in an Educational Evaluation: Reflections from the Field

Authors :
Pages :
272-281

University of North Carolina at Greensboro

Abstract: This practice note presents reflections on the application of the collaborative approaches to evaluation (CAE) principles used as a guide in planning, implementing, and managing a collaborative evaluation in a higher educational setting. Th e reflection is based the evaluation of a technology integration program intended to enhance K–12 teacher preparation in a school of education at a public university in the southeast United States. The evaluation was conducted during a one-year period by the author and a diverse team of novice and experienced evaluators. Discussion of the principles and their influence on collaborative practice are based on an analysis of evaluator reflections, meetings with stakeholders, and a culminating interview with stakeholders that were recorded and documented throughout the evaluation. Key takeaways from our reflection and analysis highlight the ways in which the CAE principles encourage reflection, the emphasis of some principles based on the specificities of context, and challenges applying the principles that emerged throughout the evaluation.
Keywords: CAE principles, collaborative evaluation, education, evaluation practice, refl ective practice

 

Démocratisation et effi cience sont-elles antinomiques au niveau local?/The generalized use of new public management tools, in the name of the effi ciency and effectiveness of public action

Authors :
Pages :
282-302

Démocratisation et effi cience sont-elles antinomiques au niveau local?Le cas des réformes managériales des administrations publiques locales en Wallonie1

Université de Liège

Abstract:

The generalized use of new public management tools, in the name of the effi ciency and effectiveness of public action, forces us to question once more the relationship between democracy and administration. At a local level, it is possible to reanalyze the politics-administration-citizens triangle, as it evolves following the appropriation of new strategic management tools and participation mechanisms. Th e study of the emergence of these new mechanisms in the Walloon region of Belgium highlights new means of exercising power and public action and bring into question their involvement in the logic of democratic reinforcement.

Keywords: public administration, strategic management, participation, local government

“Deliverology” and Evaluation: A Tale of Two Worlds

Authors :
Pages :
302-328

Université Laval

Abstract: In recent years, the new political governance, a partisan model that contributes to a permanent campaign, gained ground in public organizations. In this new context, “deliverology” is portrayed as an innovative method to help governments implement new policies and deliver on election promises. This article presents the similarities and diff erences that exist between “deliverology” and evaluation. Is deliverology really something new or is it another case of old wine in a new bottle? Is deliverology a substitute for or, instead, a complement to institutionalized evaluation? To what extent does new political governance (exemplified by deliverology and performance measurement) undermine evidence-based decision making? What is the value-added of deliverology? These questions are addressed through a critical reflection on deliverology and its value-added in Canada, where evaluation became institutionalized in many departments and agencies under the influence of results-based management, promoted by the advocates of new public management over four decades.
Keywords: deliverology, election promises, evaluation, new political governance, results-based management

 

​Introduction to Articles Prepared by CES Calgary

Authors :
Pages :
329-330
 

2018 Keynote Panel Members on Reconciliation and
Culturally Responsive Evaluation—
Rhetoric or Reality?
Debbie DeLancey
2018 CES Conference Co-Chair

When selecting a theme for the 2018 Canadian Evaluation Society Annual Conference, the co-chairs wanted to build on the CES Board’s 2016 motion in response to the Truth and Reconciliation Commission’s Calls to Action, committing to including reconciliation in its values, principles, and practices, with a specifi c commitment to strengthen the promotion of, and support for, culturally responsive evaluation. We wanted a theme that would challenge evaluators to explore what it means to truly work as collaborative partners and allies, not only with Indigenous governments and communities but also with other populations and communities that have traditionally been disempowered.

Creating New Stories: The Role of Evaluation in Truth and Reconciliation

Authors :
Pages :
331-342

Proactive Information Services Inc.

Abstract: This paper describes the origins of the Truth and Reconciliation Commission of Canada, with the focus on how evaluators and their professional associations can contribute to truth and reconciliation. At the professional association level, the actions that the Canadian Evaluation Society has taken in committing itself to incorporating truth and reconciliation into its values, principles, and practices are highlighted. At the individual level, evaluators are challenged to refl ect on their practice. As storytellers, evaluators have been complicit in telling stories that, while highlighting the damaging legacy of residential schools, have had little influence on changing the status quo for Indigenous peoples and communities. The need to reconsider who should be telling the stories and what stories should be told are critical issues upon which evaluators must refl ect. The way forward also needs to include a move toward a more holistic view, incorporating the interaction between human and natural systems, thus better refl ecting an Indigenous, rather than a Western, worldview. The imperative for evaluators, both in Canada and globally, to see Indigenous peoples “as creators of their own destinies and experts in their own realities” is essential if evaluation is to become “a source of enrichment . . . and not a source of depletion or denigration.”
Keywords: competency domains, Indigenous approaches, reconciliation, relationality, residential schools, storytelling, Tribal Critical Theory, truth

 

Nation-to-Nation Evaluation: Governance, Tribal Sovereignty, and Systems Thinking through Culturally Responsive Indigenous Evaluation

Authors :
Pages :
343-356

University of Wisconsin – Madison / Bowman Performance Consulting

Abstract: This paper was presented as part of the opening plenary panel at the 2018 Canadian Evaluation Conference in Calgary, Alberta, on May 27, 2018. Th rough telling the origin stories of First Nations/Indigenous people and Western evaluation colleagues, we can begin to understand the history and practical applications for advancing the truth through evaluation. The Doctrine of Discovery is rarely told as part of the Western canon of history or contemporary evaluation practice. Th ere are significant and negative cultural, human rights, and social impacts that have deep institutional and systemic roots that continue to cause harm to First Nations/ Indigenous populations throughout the world. To change centuries of old negative outcomes and impacts, we must understand our personal origin stories and the origin stories embedded within evaluation. Governance, policy, and evaluation can work as transformative levers for professional and sustained change if systems, critical and Indigenous theories, and methods are utilized. This paper offers origin stories of First Nations and colonial nations as a historical perspective and a new Tribal Critical Systems Theory to change contemporary Nation-to-Nation evaluation practices.
Keywords: culturally responsive evaluation, First Nations, government evaluation, Indigenous evaluation, Native American, sovereignty, systems evaluation

 

White Privilege and the Decolonization Work Needed in Evaluation to Support Indigenous Sovereignty and Self-Determination

Authors :
Pages :
357-367

The Kinnect Group

Abstract: This paper builds on a keynote paper presented at the 2018 Canadian Evaluation Society annual conference by Kate McKegg, a Pākehā, non-Indigenous evaluator from Aotearoa, New Zealand. Kate reflects on the concept and implications for Indigenous people of white privilege in colonized Western nations. She discusses some of the ways in which white privilege and its consequences play out in the field of evaluation, perpetuating colonial sentiments and practices that maintain and reinforce inequities and injustice and potentially threaten the social justice aspirations of the field. Kate argues that those with white privilege have much work to do, unpacking and understanding their privilege if they are to have any chance of playing a role in deconstructing and dismantling the power structures that hold colonizing systems in place. She suggests that for evaluators to be effective allies for Indigenous sovereignty and self-determination, they must undertake ideological, cultural, emotional, and constitutional work. This work will be tough and scary and is not for the faint hearted. But it is vital to unlocking the potential transformation that can come from just and peaceful relationships that affirm and validate Indigenous peoples’ ways of knowing and being.
Keywords: allies, colonization, evaluation, Indigenous, power, white privilege

 

Increasing Cultural Competence in Support of Indigenous-Led Evaluation: A Necessary Step toward Indigenous-Led Evaluation

Authors :
Pages :
368-384

Research Evaluation Consultancy Limited

Abstract: This paper builds on a keynote paper presented at the 2018 Canadian Evaluation Society annual conference by Nan Wehipeihana, an Indigenous (Māori) evaluator from Aotearoa New Zealand. Nan defines Indigenous evaluation as evaluation that is led by Indigenous peoples; has clear benefi ts for Indigenous peoples; has Indigenous people comprising most of the evaluation team; is responsive to tribal and community contexts; and is guided and underpinned by Indigenous principles, practices, and knowledge. She argues for Indigenous led as a key criterion for Indigenous evaluation, with no assumed or automatic role for non-Indigenous peoples unless by invitation. She outlines a range of tactics to support the development of Indigenous evaluators and Indigenous evaluation and presents a model for non-Indigenous evaluators to assess their practice and explore how power is shared or not shared in evaluation with Indigenous peoples, as a necessary precursor to increasing control of evaluation by Indigenous peoples.
Keywords: control, culturally responsive evaluation, decision making, Indigenous evaluation, Indigenous-led, Kaupapa Māori, power

 

Book Reviews / Comptes rendus de livres

Authors :
Pages :
385-387

Anne Vo and Christina A. Christie (Eds.). (2015).

Evaluation Use and Decision Making in Society: A Tribute
to Marvin C. Alkin. Charlotte, NC: Information Age
Publishing. Paperback, 190 pages.
(ISBN: 978-1-68123-004-7)
Reviewed by Margaret Schultz Patel , University of Denver/ Schultz Patel Evaluation
Compared to other branches of research, the field of evaluation is uniquely concerned with producing actionable information. Without this charge, it is not clear what would distinguish evaluation from other types of applied research. Despite this seemingly simple focus, seasoned evaluators know all too well that there are oft en many barriers to shepherding evaluations toward achieving this fi nal purpose. The volume under review contains the musings of seasoned evaluators and evaluation theorists grappling with these barriers from many diff erent perspectives.

Book Reviews / Comptes rendus de livres

Authors :
Pages :
388-390

Scott G. Chaplowe and J. Bradley Cousins. (2016).
Monitoring and Evaluation Training: A Systematic Approach . Thousand Oaks, CA: SAGE. Hardcover, 439 pages. (ISBN 978-1-4522-8891-8)
Reviewed by Michele Tarsilla , UNICEF, West and Central Africa, Regional Offi ce, Dakar, Senegal

When I first looked at the title of this book, I confess that I had mixed feelings. First, the book’s focus on “monitoring and evaluation” seemed a bit too broad: monitoring and evaluation certainly complement each other; however, their respective specificities risk being neglected when the “M” and the “E” are bundled together. Next, the book’s focus on “training” seemed too narrow to tackle the complexity of the M&E capacity realms. Specialized literature confi rms that training by itself is only one of the many strategies one could pursue to strengthen M&E capacity. I have argued for years that the excessive attention given to training (a phenomenon that I refer to as the objectification of capacity) has undermined efforts made thus far to strengthen national M&E capacity. To the contrary, it is through interventions and strategies geared toward capacity strengthening at not only the individual but also the organizational and systemic levels that sustainable and contextually relevant Evaluation Capacity Development (ECD) occurs (Tarsilla, 2014).