Rumors can spread fast on Online Social Networks (OSN) which may cause serious public issues. Thus the detection of rumors on OSN has become a hot research topic in recent years. While most of the previous work proposed some supervised methods like classification to deal with the rumor detection problem, we view the rumors as outliers among the recent Weibos posted by a user and propose a novel outlier detection method to detect them. An improved PCA method is proposed to deal with both the categorical and numerical features used for detection and to preserve the most significant information that we are interested in. Then a distance-based outlier detection method is applied to detect the potential rumors. Experimental results show that our method can achieve a better F1 and time performance compared to previous work.