In video coding, the coding efficiency of intra frames (I frames) is limited since they are coded independently. Besides, in videos with repeated scenes, high correlations exist between video contents far apart, which can be used to improve the coding efficiency. In this article, a library based video coding scheme is proposed. In the proposed scheme, a scene library containing the common scene frames (CSFs) of a video is firstly built, and then original I frames are coded as inter frames by referencing CSFs in the library. With the aid of the library, the correlations between video contents far part can be well exploited. Also by only referencing the library, the coding bits of traditional I frames are reduced while the functionalities of original I frames are still preserved. Experimental results show that the coding efficiency of videos with repeated scenes is significantly improved by the proposed scheme.