Smartphone devices are prevalent today with an estimate of around 2 billion shipments in 2015. Off these, the Android platform constitutes over 1 billion devices. Android also happens to be the most vulnerable platform among smartphone devices. In this paper we perform a Random Forest Classification on Android feature dataset to measure the reliability of behaviour analysis. We perform comparison between performance of the algorithm as its parameters are changed. We conduct a 10-fold cross validation and also perform test with a separate training set and validation set. We compare to see if use of SMOTE algorithm to generate instances of the under sampled class helps in either cross validation or separate training - validation set tests. We also provide a description of the framework that can be used to actively monitor Android devices by running a service on the device. According to our evaluation, a 10 fold cross validation gives a 96.40 percent correct result using the SMOTE algorithm but drops to 81.64 percent using a validation set test.