A reinforcement learning approach for individualizing erythropoietin dosages in hemodialysis patients

This paper presents a reinforcement learning (RL) approach for anemia management in patients undergoing chronic renal failure. Erythropoietin (EPO) is the treatment of choice for this kind of anemia but it is an expensive drug and with some dangerous side-effects that should be considered especially for patients who do not respond to the treatment. Therefore, an individualized treatment appears to be necessary. RL is a suitable approach to tackle this problem. Moreover, resulting policies are similar to medical protocols, and hence, they can easily be transferred to daily practice. A cohort of 64 patients are included in the study. An implementation of the Q-learning algorithm based on a state-aggregation table and another implementation using the multi-layer perceptron as a function approximator (Q-MLP) are compared with the protocols followed in the Nephrology Unit. The policy obtained by the Q-MLP approach outperforms the hospital policy in terms of the ratio of patients that are within the targeted range of hemoglobin (11.5-12.5 g/dl) at the end of the analyzed period, since an increase of 25% is observed. It ensures an improvement in patients' quality-of-life and considerable economic savings for the health care system due to both the expensiveness of EPO treatment and the costs incurred by the health care system in order to alleviate problems related to EPO over-dosing. It should be pointed out that the approach presented here is completely general, and therefore, it can be applied to any problem of drug dosage optimization. © 2009 Elsevier Ltd. All rights reserved.