This paper presents an infrared single‐pixel video imaging to surveille sea surface. Based on the temporal redundancy of the surveillance video, a two‐step scheme, including low‐scale detection and high‐scale detection, is proposed. For each frame, low‐scale detection performs low‐resolution single‐pixel imaging to obtain a “preview” image of the scene, where the moving target can be located. These targets are further refined in the high‐scale detection where the high‐resolution single‐pixel imagings focusing on these targets are used. The frame is reconstructed by merging these two‐level images. The simulated experiments show that for a video with 128 × 128 pixels and 150 frames, the sampling rate of our scheme is about 17.8%, and the reconstructed video presents a good visual quality.