Christina A. Christie

Special Issue

Meta-Modeling: A Theory-Based Synthesis Approach

Authors:

This ahead of print version may differ slightly from the final published version.

Research synthesis has become an increasingly popular approach for summarizing primary research. In the past two decades, interest in mixed methods reviews has steadily grown, followed, more recently, by an increased attention to theory-based syntheses. This article advances and illustrates a practical application of meta-modeling—a mixed methods, theory-based synthesis approach. The proposed methodology combines meta-analytic and qualitative comparative techniques in developing a program theory—a meta-model—of how and why a program works. As the article illustrates, meta-modeling provides for a structured and transparent synthesis approach for building program theories across existing studies.

Fall

Latent Profiles of Evaluators' Self-Reported Practices

Authors:
Pages:
225-254

Presented are the results of a study using latent profile analysis to describe the self-reported practices of 138 evaluators. Four classes emerged and were labeled (a) indistinct pattern of practice, (b) method-focused, (c) user-focused, and (d) robust pattern of practice. Evaluators in the "indistinct pattern of practice" class had mean item responses closest to zero relative to the other three classes, suggesting relatively weak associations with the practices described by the study instrument. The "method-focused" class had strong and distinct preferences for using particular methods. The "user-focused" class placed high importance on the role of the evaluator as facilitator, and was concerned with attaining a high level of stakeholder/participant involvement. The "robust pattern of practice" class evaluators showed a high degree of rigor in their reported patterns of practice, specifically as they are prescribed by evaluation theorists. The four profiles were distinguished by years of evaluation experience, degree attainment, and internal/external evaluator status.

Special Issue

Perceptions of Evaluation Capacity Building in the United States: A Descriptive Study of American Evaluation Association Members

Authors:
Pages:
37-30

This article offers a descriptive picture of American Evaluation Association (AEA) members' attitudes and perceptions related to evaluation capacity building (ECB). For this study, we analyzed data that were originally collected in the spring of 2006 from 1,140 AEA members in the United States on evaluation use. The current study is an attempt to add to the ECB knowledge base by describing respondents' views concerning (a) the importance of ECB as an evaluation approach, (b) the role of evaluators in undertaking ECB-related activities, (c) ECB-related factors that influence use, and (d) the extent to which evaluation activities foster organizational learning and change outcomes. Respondents are largely familiar with ECB and agree that building evaluation capacity is a role of the evaluator. Linkages between organizational learning and ECB were supported. Learning-focused organizational outcomes were rated more favourably than change-focused organizational outcomes.

Spring

Advancing Empirical Scholarship to Further Develop Evaluation Theory and Practice

Authors:
Pages:
1-18

Good theory development is grounded in empirical inquiry. In the context of educational evaluation, the development of empirically grounded theory has important benefits for the field and the practitioner. In particular, a shift to empirically derived theory will assist in advancing more systematic and contextually relevant evaluation practice, as well as lead to the development of contingency theories that specify the conditions under which particular evaluation practices are optimal. In contrast to the historical outside-in stance, empirical research on the field of evaluation must acknowledge practitioners as "knowers," allowing for unique insights into the intersection of theory and practice.

Fall

The language of evaluation theory: insights gained from an empirical study of evaluation theory and practice

Authors:
Pages:
33-45

Broad concern for language issues in evaluation has been limited in comparison to other social science disciplines. In this article, some occasions of definitional or conceptual confusion with evaluation theory language are identified that emerged during a study conducted by Christie. We suggest that much of the language we use to describe evaluation practice is steeped in theoretical terminology, which may limit the utility of the language. We also argue that theoretical language ought to be used with great care, with attention to the subtleties and nuances of terms, for there may be unexpected confusion or ambiguity in the field about the terms we routinely use. A research agenda is offered, suggesting that it would be both an informative as well as a useful task for us to learn more about the everyday "folk theories" of the field and the vernacular used to describe them.

Spring

The Delphi technique as a method for increasing inclusion in the evaluation process

Authors:
Pages:
105-122

The question of how best to integrate the views of underrepresented and marginalized groups in the evaluation process is of critical importance to many evaluation theorists and practitioners. In this article the Delphi technique, a method used to achieve consensus on a set of issues with the participation of all interested parties without incident or confrontation that could compromise the validity of collected data, is offered as a procedure for enhancing marginalized group participation in the evaluation process. Demonstrated by a case example, the Delphi technique is used to help ensure that all relevant stakeholders have a voice and that sometimes-silenced voices have equal influence. As a result, it is suggested that this technique lends itself to implementation with social justice evaluation models. The benefits of and lessons learned when using the Delphi technique to promote marginalized group participation and representation in evaluations are discussed.