Traffic anomaly detection is one of the most important technologies that should be considered in network security and administration. In this paper, we propose a traffic anomaly detection mechanism that includes traffic monitoring and traffic analysis. We develop an analytical system called WISE-Mon that inspects the traffic behavior by monitoring and analyzing the traffic. We establish a criterion for detecting abnormal traffic by analyzing training set of traffic and applying Fisher linear discriminant method. By using the properties of distributions such as chi-square distribution and normal distribution to the training set, we derive a hyperplane which enables to detect abnormal traffic. Since the trend of traffic can be changed as time passes, the hyperplane has to be updated periodically to reflect the changes. Accordingly, we consider the self-learning algorithm which reflects the trend of traffic and so enables to increase accuracy of detection. The proposed mechanism is reliable for traffic anomaly detection and compatible to real-time detection. For the numerical results, we use a traffic set collected from campus network. It shows that the proposed mechanism is reliable and accurate for detecting the abnormal traffic. Furthermore, it is observed that the proposed mechanism can categorize a set of abnormal traffic into various malicious traffic subsets.