The scene map building is the key of robot navigation. This paper proposes the principle of projection of virtual height line (PVHL) and its application in the scene map building. At first partition the scene scope uniformly into grids. Then introduce virtual height lines (VHLs) which do not exist in the scene. Each VHL is projected into left image and right image, and two projection lines are generated. Each projection line has a point of intersection with the scene, which is the only visible point and has the maximum similarity metric. The height of the point is that of the corresponding scene. Finding the height of the scene is converted to prove whether the point of intersection has the maximum similarity metric or not. Apply the principle of PVHL to build the 2D and 3D scene map. The experiments to build 2D map and 3D map are performed and the results show the proposed principle is feasible and effective.