Guest Editors' Remarks/Un mot des rédacteurs invités
No matter how you land: challenges of a longitudinal multi-site evaluation
In an earlier article, we described the mindset and process for implementing and conducting a multi-site study. In this article, we take the perspective of the multi-site study's coordinating centre. Using the Community Mental Health Evaluation Initiative as a case study, we focus on four major aspects of the initiative — data collection and management, the evaluated programs, partnerships, and knowledge transfer. We discuss a variety of challenges that we faced in relation to these activities during the course of our longitudinal multi-site study and how we met them — both those actions that were met with success and those that were not.
A comprehensive evaluation framework for mental health consumer/survivor organizations: values, conceptualization, design, and action
This article provides a framework for the evaluation of mental health consumer/survivor organizations that consists of four main components: (a) participatory processes, (b) conceptualization of the activities and outcomes at the individual and systems levels of these organizations, (c) the combination of quantitative and qualitative methods for examining activities and outcomes, and (d) dissemination and action. We assert that these components form a comprehensive and holistic framework for evaluating mental health consumer/survivor organizations; we illustrate how these components can be put into action through a case study of four mental health consumer/survivor organizations in Ontario; and we reflect on the lessons that we have learned in conducting this evaluation.
A benefit for everyone: family-researcher collaboration in the mental health field
This article outlines the benefits of family member involvement at all stages of mental health research. As an integral aspect of a longitudinal study of family self-help/mutual aid organizations, family members participated as advisors, researchers, and knowledge translators. Their participation served to improve the research design, inform the research questions, enhance the quality of the data, and assist in knowledge translation. Reciprocal benefits to the family members include increased research capacity, self-esteem and empowerment, financial remuneration, and a sense of ownership in the research. Innovative methodologies that consider families as full participants in all aspects of ongoing program evaluation research are recommended.
Creating inclusive workplaces: employing people with psychiatric disabilities in evaluation and research in community mental health
People with psychiatric disabilities face barriers to employment both in the larger community and within the mental health system itself. Strategies used to affirmatively employ people with psychiatric disabilities as research personnel in an evaluation of community mental health services are described in this article. Our goal was to address two critical issues in the provision and evaluation of mental health services: the importance of meaningful work and productivity in the lives of people with psychiatric disabilities, and obtaining valid and reliable data regarding the effectiveness of community mental health services. A three-phase methodology for developing affirmative employment opportunities is presented, consisting of three components: affirmative planning, affirmative support, and affirmative rigour and method. The methodology is intended as a guide to assist evaluators and researchers in fulfilling the vision of the Canadian Human Rights Act and the Employment Equity Act.
Conducting evaluation research with hard-to-follow populations: adopting a participant-centred approach to maximize participant retention
Longitudinal designs are effective for the evaluation of innovative social programs, but attrition can be a significant problem, particularly with hard-to-follow populations such as persons who are homeless. Tracking strategies for locating participants are essential, but retaining participants requires anticipating and addressing participants' needs at every stage of the research. A strategy that emphasizes appropriate interviewer characteristics, the relationship between interviewer and participant, and participants' "investment" in the research is critical. In other words, evaluation researchers can improve the retention of even hard-to-follow study participants by adapting research design and procedures to be "participant-centred." An example is given of a program evaluation in Ottawa, Ontario, that implemented strategies to adapt to the needs of persons with severe mental illness and a history of homelessness.
Inside the black box: challenges in implementation evaluation of community mental health case management programs
Fidelity measurement is an evolving field in mental health case management program evaluation. This article presents an exploratory study in which two separate fidelity measures, the Dartmouth Assertive Community Treatment Scale (DACTS) and the Key Component Profiles (KCP), were used to assess structure and process elements of three mental health case management programs. The programs were studied because they all provided services to seriously mentally ill inner city populations and shared a common context for practice. However, one program followed the Assertive Community Treatment (ACT) model, the other two were Intensive Case Management (ICM) programs, and one of the ICM programs formed a significant partnership with a home care agency for service delivery. The extent to which the DACTS and KCP were able to measure the structure and process similarities and differences of the programs is examined. The results provide information for evaluators on the possible strengths and limitations of each fidelity tool in differentiating various elements of the case management models and reinforce the importance of assessing program fidelity from a multi-dimensional perspective.
The use of the Mulh Community Ability Scale as a program evaluation tool
The Mulh Community Ability Scale (MCAS) is a standardized measure of functioning of people with mental illness living in the community. This study assessed feasibility and utility of the MCAS for routine outcome monitoring of clients enrolled in intensive community support programs. Burden related to training and administration was assessed and scale properties were evaluated, using data collected from a multisite community mental health evaluation initiative. Minimum effort was required to achieve good inter-rater reliability and administer the measure. The validity of the MCAS for measuring change in community functioning in new program clients was supported. However, MCAS ratings for ongoing clients were stable. Future studies can assess the relevance of the MCAS for measuring improvement in this client group.
The experience of developing a package of instruments to measure the critical characteristics of community support programs for people with a severe mental illness
A major challenge for program evaluators is to develop and use measures of program implementation that are both comprehensive and generalizable across programs. The goal is to understand how measures of program characteristics are related to outcomes and, ultimately, to improve program effectiveness. In this brief report, we describe our experience in developing a package of instruments to measure critical characteristics of community support programs for people with a severe mental illness. We highlight three primary methodological challenges encountered: ensuring comprehensiveness, attaining a common understanding of program boundaries, and incorporating multiple perspectives. Potential solutions are discussed. We conclude that, despite these challenges, the goal of model-free measures of program characteristics is essential to enable meaningful program comparisons and to substantiate important process- outcome links.
Planning for multi-site ethics review
This article is a brief report that draws on the literature and interviews with CMHEI investigators to explore the challenges of obtaining research ethics board approval for multi-site studies. It goes on to suggest how the complexity of multi-site approval could be addressed in terms of study budget, staff time, and study timelines.
valuation des coûts des services de soutien en santé mentale communautaire
Despite the high level of expressed interest in assessing costs of health and social services in program evaluation, the application of methodologies to estimate costs still remains relatively rare. The research note describes the application of a methodology developed in England to comprehensively cost health and social services. Using an assessment of services and supports consumed by 90 clients of a community mental health agency in Ottawa, Ontario, the different stages of the methodology are described. Results show the average annual total costs of services and supports used by the clients to be $24,878 per year ($68 per day), with a wide range variability associated with total costs of services for the different clients. The challenges of using this costing methodology in program evaluation are discussed.