Detection of anomalous program behaviors based on hidden Markov models

To improve detection accuracy, a new intrusion detection method with high efficiency was presented. The method is based on hidden Markov model (HMM) to profile normal program behaviors using traces of system calls generated during the normal execution of processes. At the stage of anomaly detection, a testing trace of system calls is divided into short system call sequences by moving along the trace with a sliding window. The output probability of a short system call sequence embedded in the testing trace is calculated based on the normal model. If the output probability of a short system call sequence exceeds a preset threshold, the short system call sequence is identified as a mismatch. If the ratio of the number of mismatch system call sequences to the number of all sequences embedded in the trace exceeds another preset threshold, the trace is then considered as an intrusion. Experimental results show that the proposed method improves the detection accuracy by at most 590% compared to both Forrest's and Lee's methods.