The horizon in marine scenes provides an important prior feature for unmanned surface vehicles (USV) based research and applications. However, most of existing research in horizon detection usually consider specific or simple scenarios. In this paper, we propose a novel approach to detect the horizon in maritime images with various situations by applying the algorithm of random sample consensus (RANSAC) hierarchically. First, a rough horizon line is estimated with RANSAC in the gradient map of downsized image. Thus, a region of interest (ROI) is defined by the neighborhood of the estimated horizon. Then a proper amount of patches are sampled from the edge map of the original image in the ROI, and a straight line is fitted in each patch using RANSAC. Finally, patches with lower rate of outliers for their fitted lines are selected and aggregated to compute the final horizon via RANSAC. Experimental results on our own dataset with diverse scenarios demonstrate that the proposed approach is more robust and more accurate than the traditional methods.