Ecological Momentary Assessment of Mental Health Problems Among University Students: Data Quality Evaluation Study

Abstract

Background: The use of ecological momentary assessment (EMA) designs has been on the rise in mental health epidemiology. However, there is a lack of knowledge of the determinants of participation in and compliance with EMA studies, reliability of measures, and underreporting of methodological details and data quality indicators. Objective: This study aims to evaluate the quality of EMA data in a large sample of university students by estimating participation rate and mean compliance, identifying predictors of individual-level participation and compliance, evaluating between- and within-person reliability of measures of negative and positive affect, and identifying potential careless responding. Methods: A total of 1259 university students were invited to participate in a 15-day EMA study on mental health problems. Logistic and Poisson regressions were used to investigate the associations between sociodemographic factors, lifetime adverse experiences, stressful events in the previous 12 months, and mental disorder screens and EMA participation and compliance. Multilevel reliability and intraclass correlation coefficients were obtained for positive and negative affect measures. Careless responders were identified based on low compliance or individual reliability coefficients. Results: Of those invited, 62.1% (782/1259) participated in the EMA study, with a mean compliance of 76.9% (SD 27.7%). Participation was higher among female individuals (odds ratio [OR] 1.41, 95% CI 1.06-1.87) and lower among those aged ≥30 years (OR 0.20, 95% CI 0.08-0.43 vs those aged 18-21 years) and those who had experienced the death of a friend or family member in the previous 12 months (OR 0.73, 95% CI 0.57-0.94) or had a suicide attempt in the previous 12 months (OR 0.26, 95% CI 0.10-0.64). Compliance was particularly low among those exposed to sexual abuse before the age of 18 years (exponential of β=0.87) or to sexual assault or rape in the previous year (exponential of β=0.80) and among those with 12-month positive alcohol use disorder screens (exponential of β=0.89). Between-person reliability of negative and positive affect was strong (RkRn>0.97), whereas within-person reliability was fair to moderate (Rcn>0.43). Of all answered assessments, 0.86% (291/33,626) were flagged as careless responses because the response time per item was <1 second or the participants gave the same response to all items. Of the participants, 17.5% (137/782) could be considered careless responders due to low compliance (<25/56, 45%) or very low to null individual reliability (raw Cronbach α<0.11) for either negative or positive affect. Conclusions: Data quality assessments should be carried out in EMA studies in a standardized manner to provide robust conclusions to advance the field. Future EMA research should implement strategies to mitigate nonresponse bias as well as conduct sensitivity analyses to assess possible exclusion of careless responders.