Objective: In obstructive sleep apnea patients on continuous positive airway pressure (CPAP) treatment there is growing evidence for a significant impact of the type of mask on the residual apnea-hypopnea index (rAHI). Here, we propose a method for automatically classifying the impact of mask changes on rAHI. Methods: From a CPAP telemonitoring database of 3,581 patients, an interrupted time series design was applied to rAHI time series at a patient level to compare the observed rAHI after a mask-change with what would have occurred without the mask-change. rAHI time series before mask changes were modelled using different approaches. Mask changes were classified as: no effect, harmful, beneficial. The best model was chosen based on goodness-of-fit metrics and comparison with blinded classification by an experienced respiratory physician. Results: Bayesian structural time series with synthetic controls was the best approach in terms of agreement with the physician.s classification, with an accuracy of 0.79. Changes from nasal to facial mask were more often harmful than beneficial: 13.4% vs 7.6% (p-value < 0.05), with a clinically relevant increase in average rAHI greater than 8 events/hour in 4.6% of cases. Changes from facial to nasal mask were less often harmful: 6.0% vs 11.4% (p-value < 0.05). Conclusion: We propose an end-to-end method to automatically classify the impact of mask changes over fourteen days after a switchover. Significance: The proposed automated analysis of the impact of changes in health device settings or accessories presents a novel tool to include in remote monitoring platforms for raising alerts after harmful interventions.