Encounter Appropriateness Score for You Model: Development and Pilot Implementation of a Predictive Model to Identify Visits Appropriate for Telehealth in Rheumatology

Objective: We aimed to develop a decision-making tool to predict telehealth appropriateness for future rheumatology visits and expand telehealth care access. Methods: The model was developed using the Encounter Appropriateness Score for You (EASY) and electronic health record data at a single academic rheumatology practice from January 1, 2021, to December 31, 2021. The EASY model is a logistic regression model that includes encounter characteristics, patient sociodemographic and clinical characteristics, and provider characteristics. The goal of pilot implementation was to determine if model recommendations align with provider preferences and influence telehealth scheduling. Four providers were presented with future encounters that the model identified as candidates for a change in encounter modality (true changes), along with an equal number of artificial (false) recommendations. Providers and patients could accept or reject proposed changes. Results: The model performs well, with an area under the curve from 0.831 to 0.855 in 21,679 encounters across multiple validation sets. Covariates that contributed most to model performance were provider preference for and frequency of telehealth encounters. Other significant contributors included encounter characteristics (current scheduled encounter modality) and patient factors (age, Routine Assessment of Patient Index Data 3 scores, diagnoses, and medications). The pilot included 201 encounters. Providers were more likely to agree with true versus artificial recommendations (Cohen's κ = 0.45, P < 0.001), and the model increased the number of appropriate telehealth visits. Conclusion: The EASY model accurately identifies future visits that are appropriate for telehealth. This tool can support shared decision-making between patients and providers in deciding the most appropriate follow-up encounter modality.