Evaluation capacity building (ECB) is a domain of inquiry and practice that has captured the eye and the imagination of many members of our field. Some of our colleagues work as practicing evaluation trainers or have some commitment to capacity building in their evaluation practice. Others have contributed to our understanding of ECB theory through reflective papers and books. And there are colleagues and students of evaluation who have conducted empirical research in the area. Of course, there are also those—the numbers may be small—who engage with all three spheres of interest. We are pleased to present in this special edition of the Journal a collection of empirical studies on the integration of evaluation into theorganizational culture. Motivation for this theme is multi-fold.
First,not just in the domain of ECB but more generally in our field, it can be argued that there is a lack of empiricism in our profession. We wanted to add the empirical knowledge base by way of strengthening it andproviding insight for future direction and practice.
Second, much of our own work has been related to evaluation as a potential lever fororganizational change. (By “our,” I mean myself, university-based colleagues, practice-based colleagues, and students.) Organizational ECB is naturally part and parcel of that general domain of inquiry.
Third, through the published literature, conferences, and personal connections, we are increasingly aware of other colleagues who are either doing research on ECB or supervising students working in the area.
Fourth, although we do see the occasional empirical study published in this general area—and this is becoming increasingly frequent—we have not yet seen a collection of empirical studies published as one package. Therehave been other collections published with a focus on ECB, but they have not been entirely empirical. Finally, our editorial team has had a commitment to publishing research on evaluationover the last number of years, and this special edition provided an opportunity to push further on that agenda.
Understanding Organization Capacity for Evaluation: Synthesis and Integration
The special issue is devoted to the examination of organizational capacity for evaluation and evaluation capacity building (ECB) through empirical inquiry. The compilation consists of two quantitative surveys of evaluators and seven single or multiple case studies across a broad array of organizations in a diverse contexts (e.g., east-central Ontario, California, Hawaii, Minnesota, and Israel). In this final article, the authors look across the collection of studies to identify emerging themes and trends with implications for ECB. The emergent themes are defining ECB; conceptualizing ECB outcomes; organizational context; ECB implementation issues; and enabling factors and barriers to organizational evaluation capacity development.
Organizational Capacity to Do and Use Evaluation: Results of a Pan-Canadian Survey of Evaluators
Despite increasing interest in the integration of evaluative inquiry into organizational functions and culture, the availability of empirical research addressing organizational capacity building to do and use evaluation is limited. This exploratory descriptive survey of internal evaluators in Canada asked about evaluation capacity building in the context of organizational characteristics (learning, support structures), evaluative activity and use, and variables that mediate use. We received a total of 340 usable responses to an online survey. This article provides a descriptive account of the findings with a cursory look at differences across respondent role, organization type, and self-reported perceived level of evaluation knowledge. Results showed a pattern of moderately high ratings of organizational learning and support functions, the extent to which evaluation is being conducted and used, and stakeholder involvement in evaluation. Some differences across respondent roles, organization type, and evaluation knowledge were observed. Results are discussed in terms of an agenda for future inquiry.
Perceptions of Evaluation Capacity Building in the United States: A Descriptive Study of American Evaluation Association Members
This article offers a descriptive picture of American Evaluation Association (AEA) members' attitudes and perceptions related to evaluation capacity building (ECB). For this study, we analyzed data that were originally collected in the spring of 2006 from 1,140 AEA members in the United States on evaluation use. The current study is an attempt to add to the ECB knowledge base by describing respondents' views concerning (a) the importance of ECB as an evaluation approach, (b) the role of evaluators in undertaking ECB-related activities, (c) ECB-related factors that influence use, and (d) the extent to which evaluation activities foster organizational learning and change outcomes. Respondents are largely familiar with ECB and agree that building evaluation capacity is a role of the evaluator. Linkages between organizational learning and ECB were supported. Learning-focused organizational outcomes were rated more favourably than change-focused organizational outcomes.
Evaluation Capacity Building in the Schools: Administrator-Led and Teacher-Led Perspectives
The purpose of this article is to describe evaluation capacity building using an immersion approach in two schools: one with an administrator-led process and one with a teacher-led process. The descriptions delineate conceptual, developmental, and sustainability aspects of capacity building through the perspectives of the teachers, principals, and evaluation specialists. The immersion approach to evaluation capacity building highlights the distinctions between developing capacity for using evaluation information and the capacity to conduct evaluations, and provides insight into the use of direct and indirect approaches to building capacity. The two cases also showcase the roles played by ownership in developing evaluation capacity, especially sustainability.
The Road to Evaluation Capacity Building: A Case Study from Israel
We present an empirical case study of an Evaluation Capacity Building (ECB) initiative in a school in Israel. First, we tell the story of the school's 10-year journey toward the successful integration of evaluation through ECB. Then we examine the case according to King's (2002) four elements of ECB: teachers, students, the curriculum, and the context. We present four concrete examples of evaluations conducted at the school illustrating how these elements influenced the results of ECB. We follow this presentation with a discussion of the case in relation to other ECB checklists. After thorough and systematic investigation, we believe that through this case we can learn more about the factors that contribute to or inhibit the successful outcome of an ECB effort in schools.
Participatory Evaluation as Seen in a Vygotskian Framework
In participatory evaluations of K–12 programs, evaluators develop school faculty's and administrators' evaluation capacity by training them to conduct evaluation tasks and providing consultation while the tasks are conducted. A strong case can be made that the capacity building in these evaluations can be examined using a Vygotskian approach. We conducted participatory evaluations at 9 Hawaii public schools and collected data on the extent to which various factors affected participating school personnel's learning about program evaluation. After the evaluations were completed, a trained interviewer conducted standardized interviews eliciting the participating school personnel's opinions about the methods and effects of the capacity building. Two reviewers used codes representing Vygotskian concepts to categorize the interview results. We present the results of the coding and provide conclusions about the value of using a Vygotskian framework to examine capacity building in participatory evaluations.
Informing Evaluation Capacity Building Through Profiling Organizational Capacity for Evaluation: An Empirical Examination of Four Canadian Federal Government Organizations
According to the literature published on the topic, the development of an organization's capacity to do and use evaluation typically follows four stages: traditional evaluation, characterized by externally mandated evaluation activities; awareness and experimentation, during which organizational members learn about evaluation and its benefits by participating in a number of evaluation-related activities; evaluation implementation, the stage at which the role of evaluation is more clearly defined in the organization; and evaluation adoption, which occurs when evaluative inquiry becomes a regular and ongoing activity within the organization through the allocation of continued financial and human resources. In this article we argue that this perspective is oversimplified and that it is essential to understand the complexity of an organization's evaluation capacity in order to better understand how it might proceed with evaluation capacity building (ECB). We present an analysis of four Canadian federal government organizations' self-assessment of their organizational evaluation capacity using a profile conceptual framework developed as part of our larger study. We then integrate the resulting multidimensional profiles of observed levels of organizational evaluation capacity with the aforementioned stages of ECB to provide added value in thinking about the direction of organizational ECB.
Insights into Evaluation Capacity Building: Motivations, Strategies, Outcomes, and Lessons Learned
Evaluation capacity building (ECB) is a topic of great interest to many organizations as they face increasing demands for accountability and evidence-based practices. While many evaluators are engaged in evaluation capacity building activities and processes with a wide variety of organizations, we still know very little about whose capacity is being built, what strategies are being used, and the overall effectiveness of these efforts. To explore these issues, a research study was conducted with 15 organizations that have been involved in ECB efforts during the last few years. The findings reported in this article are part of a larger study, and represent interviews with 25 evaluators and 13 clients (n = 38), who have facilitated and supported an organization's ECB effort. We specifically focus on the participants' motivations for engaging in ECB, the teaching and learning strategies used to facilitate capacity building, their perceived outcomes of this effort, and their lessons learned.
A Bumpy Journey to Evaluation Capacity: A Case Study of Evaluation Capacity Building in a Private Foundation
Evaluation capacity building (ECB) processes were explored through a case study of the ECB efforts of a private foundation that had solid financial resources, intelligent staff, an internal evaluation officer, and access to prominent evaluation consultants. The overarching research question was: What are the factors that affect the ECB process in an organization? The research design included in-depth interviews, surveys, document analysis, and observations involving a sample of the foundation staff and external consultants. This article illuminates the complexity of the organization's experiences with building evaluation capacity and provides insights on what must be considered when developing a viable evaluation function in organizations.
Using Evaluation Capacity Building (ECB) to Interpret Evaluation Strategy and Practice in the United States National Tobacco Control Program (Ntcp): A Preliminary Study
The Office on Smoking and Health (OSH) of the Centers for Disease Control and Prevention (CDC) supports state programs for the prevention and control of tobacco use through the National Tobacco Control Program (NTCP). OSH provides the NTCP with expert guidance and technical assistance on tobacco use control and disease surveillance as well as evaluation of tobacco control programs. These services fit national health goals and provide data to inform national and state policy making and program planning. However, the NTCP's delivery of services, achievement of goals, and evaluation of efforts is hindered by fluctuations in dedicated state funds to support tobacco use prevention and control programs. To maximize effort and resources, evaluation capacity building (ECB) is a strategy for strengthening evaluation services, program efficiency, and program effectiveness, that is, program improvement. This article interprets NTCP using an ECB frame to learn the utility of this approach for making suggestions for structural and practice changes that lead to program improvement.