Why a Focus on Integration and Complex Mixed Methods Evaluation Designs?: Introducing this Special Issue of the Canadian Journal of Program Evaluation

Mixed Methods Design in Evaluation, by Donna Mertens

Mixed Methods Design in Evaluation by Donna Mertens is a useful, well-organized book that explores the numerous applications and benefits of mixed methods approaches. In the words of the volume editors, after examining published evaluation examples, Mertens “concludes with prompts that engage the reader in thinking about how mixed methods can improve social inquiry” (p. xiv). Mertens advocates that mixed methods in evaluation “strengthen the credibility of evaluation findings” (p. 3) and that they “have the potential to be responsive in a dynamic way that is reflective of the complexity found in most evaluation contexts” (p. 177). In this book, Mertens emphasizes the integration of mixed methods in evaluation throughout the text and, using numerous examples, demonstrates diverse strategies for situating mixed methods within various evaluation paradigms.

Pat Bazeley. (2018). Integrating Analyses in Mixed Methods Research. Sage. Paperback, 328 pages.

“Effective integration doesn’t suddenly happen at the end of the project; it derives from a conversation that began as initial purposes and designs were being formulated, that then continued throughout the project” (p. 278). This quote illustrates the essential role of integration in mixed methods research and orients the reader to the practical approach advanced in the book for understanding the complex world from multiple viewpoints. The author, Pat Bazeley, defines integration as “the extent that different data elements and / or varied strategies for analysis of those elements are combined in such a way as to become interdependent (a two-way process) in reaching a common theoretical or research goal” (p. 10).

Integrating Analyses in Mixed Methods Research is organized into three parts, containing a total of twelve chapters, and provides insights into the meaning of integration, strategies for integration, and suggestions for maximizing analyses.

Michael D. Fetters. (2020). The Mixed Methods Research Workbook: Activities for Designing Implementing, and Publishing Projects. Sage. Paperback, 293 pages.

In The Mixed Methods Research Workbook, Michael D. Fetters has developed a practical resource for scholars, graduate students, and individuals with an interest in mixed methods research (MMR). Since many of the concepts presented are essential to all types of human research, the workbook would make an excellent required reading for an introductory graduate-level MMR course. The workbook is intricately scaffolded, with engaging lessons covering vital areas of MMR training—literature review, ethics, publishing, validity, grant applications, and philosophy worldviews. In particular, the integration of mixed methods is a key feature of the workbook. Fetter’s pedagogical intentionality is apparent, and the organizational structure and textual features render this a useful text for a range of audiences and worthy of keeping on your bookshelf for future reference. Common threads include collaborating with peers and illustrating concepts with stories drawn from a range of fields. Recent and relevant in-text citations solidify the evidence-based research used in developing this workbook and guiding mixed methods projects.

 

Editor's Remarks

I am pleased to introduce this special issue of the CJPE , guest edited by Dr. Cheryl N. Poth and Dr. Michelle Searle. This special issue makes an important contribution to advancing our understanding of evaluation methodology by focusing on complex mixed methods designs. The contributions outlined in this issue recognize the inherent complexity of educational and social interventions and propose concrete examples and solutions for integrating mixed methods into evaluative design, to leverage the strengths of both qualitative and quantitative approaches, and to benefit from the interaction and integration of both types of methodologies.

Closing Commentary: The Contributions of Innovations in Integration in Complex Mixed Methods Evaluation Designs in This Issue and Beyond

Special Issue: Integration in Complex Mixed Methods Evaluation Designs

Special Issue: Integration in Complex Mixed Methods Evaluation Designs

Responding to the Evaluation Capacity Needs of the Early Childhood Field: Insights from a Mixed Methods Community-based Participatory Design

For this special issue, our article provides an illustrative example of data integration in a complex mixed methods design. It advances three mixed insights about the evaluation capacity needs of the early childhood field generated by using a mixed methods community-based participatory research (MMCBPR) design. We integrated quantitative findings from an evaluation capacity needs assessment tool with qualitative themes from community consultations. Findings highlight the pressing need for practices aimed at enhancing individual capacity through skills-based training, shifting organizational learning culture through resource investment, and cultivating interdependencies between organizational evaluation culture and individual evaluation capacity through engagement and collaborations.

Advancing a Competency-Based Mixed Methods Tool to Assess a Course-Based Service-Learning Model Integrating Real-World Experience

Given increasingly complex societal issues such as global aging, evaluators must develop essential competencies responding to the messiness of reality. This practice note advances an argument for the inclusion of service learning as a form of experiential learning that engages learners in hands-on practice with guided reflection, and shares a competency-based mixed methods (MM) tool for generating learning evidence in a course-based service-learning model. I aligned key design elements with pedagogical approaches with evaluation competencies. Competency-based guidance is critical to ensure effective assessment and quality course delivery. I conclude with lessons learned for integration of MM within program evaluation education.

Instructions to Authors

SUBMISSION GUIDELINES AND EVALUATION CRITERIA

Developing a Comprehensive Mixed Methods Evaluation to Address Contextual Complexities of a MOOC

Massive open online courses (MOOCs) are gaining popularity, yet they are rarely evaluated using mixed methods that consider the complexity of participants’ demographics, geographic spread, and MOOC design and curricula. In this article, we critically reflect on evaluating the Mixed Methods International Research Association (MMIRA) MOOC. A literature review on MOOCs and how they are evaluated is presented along with a logic model. After a description of the MMIRA MOOC and the multi-phase mixed methods evaluation design, the logic model is used to reflect on conducting a mixed methods evaluation. We conclude with the challenges experienced when evaluating amidst a MOOC’s complexities.

Peer Reviewers for Volume 36 and Manuscripts Submitted in 2021 / Examinateurs des manuscrits du volume 36 et des manuscrits soumis en 2021

Special Issue: Integration in Complex Mixed Methods Evaluation Designs / L’intégration des méthodes mixtes complexes en évaluation

A Multi-Stage Approach to Qualitative Sampling within a Mixed Methods Evaluation: Some Reflections on Purpose and Process

We share experiences from a mixed methods evaluation in rural India that combines a randomized controlled trial (RCT) of 400 villages with embedded case studies in four villages. Specifically, we present two lessons from the multi-stage sampling approach adopted to select the four case-study villages, which first prioritized key-informant observations regarding intervention status in order to shortlist locations and subsequently used data from the RCT’s baseline survey to select the final sample. In doing so, we highlight how large-scale mixed methods program evaluations in education can go beyond questions of “what works” to answering those of “how,” “why,” and “why not.”

Nous faisons part d’expériences découlant d’une évaluation par méthodes mixtes en Inde rurale qui combinait un essai clinique randomisé (ECR) de 400 villages avec des études de cas intégrées dans quatre villages. Plus précisément, nous présentons deux leçons tirées de l’approche d’échantillonnage à plusieurs degrés adoptée pour sélectionner les quatre villages de l’étude de cas, qui a d’abord priorisé les observations d’informateurs clés concernant l’état de l’intervention afin de présélectionner les emplacements, puis utilisé les données de l’enquête de base de l’ECR pour sélectionner l’échantillon final. Ce faisant, nous soulignons la manière dont les évaluations de programmes à grande échelle par méthodes mixtes dans le domaine de l’éducation peuvent aller au-delà de questions de « ce qui fonctionne » pour répondre à des questions de « comment », « pourquoi » et « pourquoi pas ».

Integrating Data to Evaluate a Global Health Grand Challenge

This article describes the integrated, mixed methods (MM) design used to evaluate the Saving Lives at Birth ([email protected]) program. [email protected] is a multi-stakeholder, donor-supported global health initiative to tackle maternal and neonatal mortality via innovation. Since [email protected]’s launch in 2011, the program has supported 116 innovations through 147 awards around the globe. The evaluation for this large and complex program included a largely retrospective MM design aligned with principles of evaluating complexity. This paper highlights these MM evaluation strategies and integration dimensions employed to complete the [email protected] evaluation that could inform future evaluations of portfolio-level global health programs.

 

Cet article décrit les méthodes mixtes (MM) intégrées employées lors de l’évaluation du programme Sauver des vies à la naissance ([email protected]). [email protected] est une initiative mondiale pour la santé, soutenue par des donateurs, visant à lutter contre la mortalité maternelle et néonatale grâce à l’innovation. Depuis le lancement de [email protected] en 2011, le programme a appuyé 116 innovations par l’intermédiaire de 147 subventions à travers le monde. L’évaluation de ce programme vaste et complexe a compris un cadre de MM, largement rétrospectif, conforme aux principes d’évaluation de la complexité. Le présent article souligne ces stratégies d’évaluation par MM et les dimensions d’intégration utilisées pour compléter l’évaluation [email protected] qui pourraient éclairer les évaluations futures des programmes du portefeuille de santé mondiale.

Leveraging Mixed Methods Designs for Promoting Evaluation and Evaluation Capacity Building

Limited evaluation capacity, power dynamics, and resource constraints act as organizational barriers that inhibit evaluations in higher education contexts. Evaluation capacity promotes evaluation and minimizes the impact of these barriers. Embedding a synergistic combination of qualitative and quantitative methods within an evaluation and evaluation capacity building (ECB) initiative was our attempt to address these organizational barriers. In this practice note, we illustrate how MM designs catalyze evaluative thinking that promotes evaluation and supports ECB efforts. Our use and integration of MM strategies to build evaluation capacity and infuse evaluation into organizational culture informs higher education evaluation capacity-building initiatives.

Des capacités organisationnelles en évaluation limitées, des dynamiques de pouvoir et des contraintes en matière de ressources agissent comme des obstacles organisationnels qui entravent les évaluations de programmes de formation postsecondaires. Les capacités organistionnelles en évaluation soutiennent les activités d’évaluation et minimisent l’impact de ces obstacles. En intégrant une combinaison synergique de méthodes qualitatives et quantitatives à une initiative d’évaluation et de renforcement des capacités en évaluation (RCÉ), nous avons tenté de surmonter ces obstacles organisationnels. Dans la présente note sur la pratique, nous illustrons la façon dont les conceptions de MM catalysent la pensée évaluative qui promeut l’évaluation et soutient les efforts de RCÉ. Notre utilisation et notre intégration des stratégies de gestion de l’information en vue de renforcer les capacités en évaluation et d’intégrer l’évaluation à la culture organisationnelle sous-tendent les initiatives de renforcement des capacités en évaluation dans le secteur postsecondaire.

Learning and Leading: Integrating Mixed Methods in a Collaborative Approach to Educational Evaluation

This practice note describes the benefits of integrating mixed methods in a collaborative approach to evaluation with school districts and community partners in southwestern Ontario. We discuss the ways in which the integration of qualitative and quantitative data generated a multi-faceted perspective about a new mental health professional role as a complex educational phenomenon. The insights related to recognizing the role of context, supporting the use of mixed methods with stakeholders, and responding to evolving needs related to youth mental health and well-being in schools highlight the practical value of using mixed methods in a collaborative approach to evaluation.

La présente note sur la pratique décrit les avantages d’intégrer des méthodes mixtes à une approche collaborative pour l’évaluation avec des districts scolaires et des partenaires communautaires dans le sud-ouest de l’Ontario. Nous discutons des façons par lesquelles l’intégration de données qualitatives et quantitatives génère une perspective multidimensionnelle sur le nouveau rôle des professionnels de la santé mentale comme phénomène éducationnel complexe. Les propositions liées à la reconnaissance du rôle que joue le contexte, à l’appui de l’utilisation de méthodes mixtes avec des intervenants, et à la réponse aux besoins en pleine évolution en matière de santé mentale chez les jeunes et de bien-être dans les écoles mettent en relief la valeur pratique de l’utilisation de méthodes mixtes dans le cadre d’une approche collaborative en évaluation.

A Framework to Combine Mixed Methods Integration and Developmental Evaluation to Study Complex Systems

This article considers the opportunities and challenges of mixed methods integration within the context of conducting developmental evaluations with diverse stakeholders. The article proposes a four-part planning framework that explicitly maps the eight principles of developmental evaluation to key questions about integration: Why integrate? What will be integrated? When and how will integration occur? The framework is used to explore and compare two large-scale evaluations that employed mixed methods integration. Ultimately, the work illustrates a need to balance data collection with the momentum of moving initiatives forward, check assumptions about stakeholder co-creation, and cultivate integration literacy.

L’objectif principal de cet article est d’examiner les possibilités et défis de l’intégration de méthodes mixtes dans le contexte de la conduite d’évaluations dans une optique de développement avec diverses parties prenantes. L’article propose un cadre de planification en quatre parties pour les équipes de recherche: Pourquoi intégrer? Qu’est-ce qui sera intégré? Quand et comment l’intégration aura-t-elle lieu? Le cadre est utilisé pour explorer et comparer deux évaluations à grande échelle qui ont cherché à utiliser l’intégration des méthodes mixtes. En fin de compte, le travail montre la nécessité d’équilibrer la collecte de données et l’élan de faire avancer les initiatives, de vérifier les hypothèses sur la co-création des parties prenantes et la nécessité de cultiver la culture de l’intégration.

Announcement from Roots and Relations: Celebrating Good Medicine in Indigenous Evaluation

The “Roots and Relations” (R&R) will sacredly hold and protect traditional knowledge, respect and assert sovereignty, provide a nurturing space for Indigenous voices, and celebrate Indigenous innovations in evaluation. Larry Bremner CE, FCES (he/him, M tis) and Dr. Nicole Bowman (Lunaape/Mohican) will facilitate the processes of developing and upholding these foundational elements of R&R. Th is section will honour our lineage, grow our kinship, and sustain our intergenerational legacies of Indigenous wisdom and practices in evaluation. Roots and Relations will be a welcoming place for Indigenous graduate students, curious and passionate Indigenous community members, Indigenous practitioners, First Nations and other Indigenous policymakers, leaders, and scholars on Kukuna Ahkuy (Mother Earth) to share their important work.

 

View Text