Volume 34, 2019 - Spring

Community, Theory, and Guidance: Benefits and Lessons Learned in Evaluation Peer Mentoring

Authors :
Pages :
9

This ahead of print version may differ slightly from the final published version.

The majority of evaluation practitioners begin their career in allied fields and stumble into evaluation. As such, university offerings and evaluation professional development sessions have become increasingly popular. As the field continues to professionalize and new mentoring programs emerge, empirical work examining teaching and training in evaluation has gained traction. However, little is known about the role that opportunities such as mentoring play in evaluation training. The purpose of this article is to explore the expected and unexpected benefits of our experiences as participants in an evaluation mentoring program, lessons learned, and logistical and structural promoters of success in peer mentoring.

Principles, Approaches, and Methods for Evaluation in Indigenous Contexts: A Grey Literature Scoping Review

Authors :

This ahead of print version may differ slightly from the final published version.

This article describes findings from a scoping review of the grey literature to identify principles, approaches, methods, tools, and frameworks for conducting program evaluation in Indigenous contexts, reported from 2000–2015 in Canada, the United States, New Zealand, and Australia. It includes consultation with key informants to validate and enrich interpretation of findings. The fifteen guiding principles, and the approaches, methods, tools, and frameworks identified through this review may be used as a starting point for evaluators and communities to initiate discussion about how to conduct their evaluation in their communities, and which approaches, methods, tools, or frameworks would be contextually appropriate.

Perceptions of the Use of an Outcome Monitoring Tool in a Clinical Psychology Training Centre: Lessons Learned for Performance Measurement

Authors :

This ahead of print version may differ slightly from the final published version.

The purpose of this study was to examine the perceptions of the Outcome Questionnaire (OQ) following its implementation in a university-based psychological services training centre. Participants were doctoral-level student clinicians (n = 49), clinical supervisors (n = 17), and clients (n = 24). Data was collected through surveys, semi-structured interviews, and focus groups. Findings indicated that the majority of clinicians used the OQ to monitor outcomes and the majority of stakeholders perceived it as useful. However, the extent to which the information provided by the OQ was being used was variable. Lessons learned for implementation of performance measurement systems within mental health services are discussed.

Évaluation du processus d’implantation de l’approche axée sur les forces en s’appuyant sur la théorie de la diffusion de l’innovation

Authors :

This ahead of print version may differ slightly from the final published version.

Most policies in the area of mental health recommend the promotion of evidence-based practices. The force-based approach is recognized as a practice that can contribute to the recovery of people with severe mental disorders. However, several studies have raised numerous implementation challenges that limit its scope. An evaluation of the implementation process was conducted with stakeholders who received training on the strength-based approach. The results indicate that training is not sufficient and that institutional commitment is required to ensure the quality of the implementation.

Evaluation Literacy: Perspectives of Internal Evaluators in Non-Government Organizations

Authors :

This ahead of print version may differ slightly from the final published version.

While there is an abundance of literature on evaluation use, there has been little discussion regarding internal evaluators’ role in promoting evaluation use. Evaluation can be undervalued if context is not taken into consideration. Evaluation literacy is needed to make evaluation more appropriate, understandable, and accessible, particularly in non-government organizations (NGOs) where there is a growing focus on demonstrable outcomes. Evaluation literacy refers to an individual’s understanding and knowledge of evaluation and is an essential component of embedding evaluation into organizational culture. In recognition of the value of the internal perspective, a small exploratory exercise was undertaken to reveal internal evaluator roles and ways of engaging with colleagues around evaluation. The exercise examined a key question: What is the role of evaluation literacy in internal evaluation in the non-government sector? Three Australian auto-narrative examples from internal evaluators highlight evaluation literacy and locate it among the multiplicity of roles required for optimal evaluation uptake. Analysis of the narratives revealed the underlying issues affecting evaluation use in NGOs and the skills needed to motivate and enable others to access, understand, and use evaluation information. Responding to the call for expanded research into internal evaluation from a practice perspective, the authors hope that the findings will stimulate a wider conversation and further advance understanding of evaluation literacy.

Perceived Facilitators and Barriers to Evaluative Thinking in a Small Development NGO

Authors :

This ahead of print version may differ slightly from the final published version.

The Global Goals come with challenging implications for non-governmental organizations (NGOs) in international development and their capacity for high-quality evaluation practice and evaluative thinking. NGOs are pressured to work efficiently, be accountable to donors and beneficiaries, and demonstrate impact. They must also critically examine the underlying assumptions behind their work, or else the sustainability of their work becomes jeopardized. Using previously collected evaluation data from a small NGO in water-based development, this paper highlights perceived facilitators and barriers to evaluative thinking and where they might occur in the evaluation process for an NGO constrained by time and resources.

Secure Data Handling: An Essential Competence for Evaluators

Authors :

This ahead of print version may differ slightly from the final published version.

Since it is paramount that the rights and welfare of evaluation participants and stakeholders be respected, we argue that the abilities and knowledge necessary to appropriately safeguard data ought to be considered an essential competence for evaluators. Building from past contributions, and in consultation with research ethics and data security experts from our home institution, recommended practices in the collection, handling, and storage of evaluation data were identified. A three- dimensional framework for secure data handling was developed, considering type of information handled, harm posed by a potential confidentiality breach, and corresponding steps to securing confidential information.

Scaling up Programs: Reflections on the Importance of Process Evaluation

Authors :

This ahead of print version may differ slightly from the final published version.

For more than a decade, policy makers and funding agencies have been focused on identifying innovative and successful programs and bringing them to scale. Evaluators play an important role in these scaling efforts by helping to document what works and by monitoring program implementation. They can also monitor the replication of taking innovative programs to scale. In this research and practice note, we reflect on our evaluation experiences with a public-private partnership designed to scale up a health and wellness program within a large, urban school district at ten elementary schools. In doing so, we highlight the importance of conducting a process evaluation at the beginning of the program to ensure that the program is being implemented as intended. We also describe how these early evaluation findings helped to improve the program during its second year.

Evaluating Youth Drop-In Programs: The Utility of Process Evaluation Methods

Authors :

This ahead of print version may differ slightly from the final published version.

In North America, neighbourhood youth centres typically offer essential community-based programs to disadvantaged and marginalized populations. In addition to providing pro-social and supportive environments, they provide a host of educational and skill-development opportunities and interventions that build self-esteem, increase positive life relationships and experiences, and address social determinants of health. However, evaluators of such centres often have to work with moving changes in temporal components (i.e., service users, services, programs, and outcomes) that are unique and idiosyncratic to the mandate of the centre. Although there is an abundance of research on youth programs in general, there is a void in the literature on drop-in programs specifically, which this study aims to address. The lack of empirical research in this area inhibits knowledge about the processes of these centres. For this reason, the article concludes that process evaluation methods may be effectively used to substantiate the practice skills, knowledge, and managerial competencies of those responsible for program implementation.

20 Years Later: Reflections on the CES Student Evaluation Case Competition

Authors :

This ahead of print version may differ slightly from the final published version.

Since 1996, the Canadian Evaluation Society (CES) has held an annual case competition for college and university students. By 2016, a total of 1,132 students had participated. An online questionnaire was sent to 768 participants with available email addresses; eight additional participants entered the study after viewing an online posting. The questionnaire was completed by 112 former participants (response rate: 14%). Findings suggest that participating in the case competition was a positive experience that led to an appreciation of evaluation, increased teamwork skills, and provided stronger résumés. Some indicated that participating influenced their choice of evaluation as a focus for their career.