Rigour and Feasibility in Tobacco Control Evaluation: Toward a Successful Reconciliation
The challenge of reconciling scientific rigour with feasibility is central to the goals of policy and program evaluation for tobacco control. Evaluations conducted in various settings are held to high standards of performance, and must also be considered feasible by program authorities and stakeholders. This article describes three recent examples from the field of tobacco control. Issues of context, relevance, and stakeholder participation in planning evaluation designs are central to successful reconciliation. To affect tobacco control evaluation positively, reconciliation between the goals of rigour and feasibility needs to occur on two levels: between evaluator and stakeholders, and within the evaluation plan.
Evaluating school-based tobacco control programs and policies: an opportunity gained and many opportunities lost
In the domain of school-based tobacco control, there is a large gap between the research being performed and the type of research needed to inform school-based prevention activities. The Tobacco Module of the School Health Action Planning and Evaluation System is a data collection and feedback system designed to support school-based tobacco control planning and evaluation. Despite numerous opportunities to evaluate interventions and natural experiments that have been occurring within schools, school boards, provinces, and even nationally within the last few years, a lack of systematic data collection means that the impact of these initiatives has not been robustly evaluated.
Can tobacco control advocacy work be evaluated?
Tobacco control advocates have played an important role in Ontario's tobacco control efforts. As with other advocacy work, their efforts have been mostly unexamined. Based on Ontario's tobacco control advocacy work, this article highlights the challenges of evaluating the work of advocates (e.g., having multiple aims, multilayered structures, shifting time frames, measuring longterm goals, dealing with external factors, having goals change at midstream, having different approaches working simultaneously) and summarizes some theories and tools (e.g., Bellwether Methodology, Agenda Setting Theory, Context-in approaches, Pathways of Influence, and logic models) that have been used to conduct effective advocacy evaluations.
Advocacy Evaluation Theory as a Tool for Strategic Conversation: A 25-year Review of Tobacco Control Advocacy at the Canadian Cancer Society
Advocacy evaluation has been considered difficult, even in specific and uncomplicated situations. This article describes a short evaluative review of a series of tobacco control policy advocacy activities involving multiple parties over 25 years. Despite its obvious limitations, the effort suggests that the retrospective development of a structured contribution analysis and "key event" stories from documentary and key informant information can generate compelling circumstantial evidence for impact. Perhaps more importantly, the study was found to provide what appear to be valuable insights and lessons for the planning and management of future advocacy initiatives.
Challenges and Approaches to Evaluating Comprehensive Complex Tobacco Control Strategies
Challenges to evaluating comprehensive complex strategies revolve around addressing comprehensiveness, attribution, and complexity. The latter requires attention to synergies amongst interventions, feedback loops, and other forms of nonlinearity. The article reviews and assesses how well six approaches to evaluating strategies succeed in dealing with these challenges, including one developed in light of the initial review. None of the approaches offer ideal solutions to the challenges of complexity. The quantified logic model approach suggests the need to simplify and refrain from trying to assess all causal chains in complex strategies. Intervention path contribution analysis, an approach under development, explores the possibilities of using contribution analysis to validate evaluative propositions developed from literature, program theory, and incomplete evaluative information. Greater understanding of synergies, feedback loops, and nonlinearity in general requires accumulation of knowledge over time from thoughtful strategy evaluations under a variety of contexts.
Monitoring Tobacco Control Outcomes: The Unintended Consequences of Target Selection
We explore key lessons arising out of target-setting agendas in tobacco control. Two real-world examples are described that illustrate the pitfalls of choosing an omnibus target for a comprehensive strategy: (a) a target to reduce wholesale cigarette sales in Ontario and (b) a First Ministers' target to reduce current smoking. Changing contexts brought about by shifts in illicit tobacco sales made it problematic to interpret success in both cases. The discussion draws attention to key considerations in setting targets, including unintended consequences, data quality, ideal number, and the importance of context such as stakeholder roles in target selection and reporting.
Common measures to advance the science and practice of population intervention for chronic disease prevention: The promise and two early experiences in tobaco control
Pertinent evidence to inform population interventions for chronic disease prevention is sparse. The use of common measures across multiple jurisdictions is a promising approach to study "natural experiments" that can dually advance research/knowledge development and evaluation/practice improvement for population intervention. Early experiences with provincial tobacco control strategies and North American quitlines reveal the importance of (a) sustained collaboration across research, evaluation, policy, and practice communities; (b) honouring different perspectives; and (c) stable institutional support for the creation and implementation of common measures. The promise of common measures will be better understood as mature examples of their use are explored.
Utilité de l'évaluation de l'évaluabilité des politiques gouvernementales de lutte contre le tabagisme: l'expérience québécoise des centres d'abandon du tabagisme
The government's tobacco control policies are often multifaceted and large-scale in nature. The process and outcome evaluation of such programs generally requires a significant investment in time, money, and effort. A critical but often neglected step prior to conducting a successful evaluation is the evaluability assessment. This tool is a method for examining a given program to determine whether it meets the criteria for a meaningful evaluation. Even when it fails to do so, it still contributes to the improvement of the program. This article describes the principles of the evaluability assessment and provides an example of its application. It concludes by presenting some lessons learned from the evaluation of Quebec's Quit Smoking Centres.