Securing virtual machines from anomalies using program-behavior analysis in cloud environment

Cloud Computing is the key technology of today's cyber world which provides online provisioning of resources on demand and pay per use basis. Malware attacks such as virus, worm and rootkits etc. are some threats to virtual machines (VMs) in cloud environment. In this paper, we present a system call analysis approach to detect malware attacks which maliciously affect the legitimate programs running in Virtual Machines (VMs) and modify their behavior. Our approach is named as 'Malicious System Call Sequence Detection (MSCSD)' which is based on analysis of short sequence of system calls (n-grams). MSCSD employs an efficient feature representation method for system call patterns to improve the accuracy of attack detection and reduce the cost of storage with reduced false positives. MSCSD applies Machine Learning (Decision Tree C 4.5) over the collected n-gram patterns for learning the behavior of monitored programs and detecting malicious system call patterns in future. We have analyzed the performance of some other classifiers and compared our work with the existing work for securing virtual machine in cloud. A prototype implementation of the approach is carried out over UNM dataset and results seem to be promising.