Adaptively tracking tram railway in video-based complex scene is difficult because of road curving and environment changing. In this paper, we introduce an adaptive railway recognition method by analyzing gray distribution features of railway region. This method firstly segments track regions using multiple thresholds which can be dynamically optimized based on the change of local accumulation histogram with the change of scenes. Then, on the basis of binary image, combined with connectivity and skeleton extraction, track feature points are automatically extracted from the position of the track starting point. A suitable curve model is chosen to construct the railway equation. The proposed method is able to achieve accurate recognition of railway in different scenes.