CES members Ardyn Nordstrom and Grace Shen-Tu have recently published, with Aditya Maheshwari and Rheann Quenneville, an article entitled Combining Text Mining and Manual Thematic Analysis to Understand Participant Experiences With Surveys in the open-source International Journal of Qualitative Methods. Here is a short version of the article abstract.
The authors posit that large-scale surveys constitute a critically important source of information in social research and evaluation. But they are successful only when participants engage meaningfully in them. A superior form of engagement is responding fully to open-ended survey questions which can capture nuanced responses, but can be challenging to analyze at scale.
The study combined computational text mining and manual thematic analysis to examine over 15,000 open-ended survey responses concerning participants’ experiences with longitudinal public health surveys administered by Alberta’s Tomorrow Project. The text mining analysis consists of both sentiment analysis and topic modelling to identify broad patterns in participant experiences. This is combined with a thematic analysis of a purposive sample of 852 of these responses to validate the computational methods and identify themes not captured by the text mining analysis.
It was found that the conclusions from the analyses were largely consistent across the computational text mining and manual thematic analyses. However, the computational approaches overgeneralize or miscategorize certain responses, highlighting the benefits of in- corporating both methods to analyse large volumes of qualitative survey responses.