Anomaly detection in dynamic graphs is an emerging data mining research topic. However, applying anomaly detection to real-world industry problems such as insider threats, and banking fraud, is full of challenges. The multi-million dollar question is: What is a high-quality anomaly? In this paper we address the importance of reducing false positives and associating them with anomaly triage. After a review of recent graph-based anomaly detection research, we propose a novel triaging definition for anomalies in dynamic graphs with three categories: node level, community level, and evolutionary path level. With this extensive triaging system, we create an integrated framework that detects anomalies in large dynamic graphs with a reduced rate of false positives. We benchmark the performance of our proposed framework on both synthetic and real-world datasets such as data from Facebook Newsgroups. Our experiments demonstrate the effectiveness and consistency of our framework in detecting dynamic anomalies.