This letter proposes an efficient wavelet-based Fine Granularity Scalable (FGS) coding scheme, where the base layer is encoded with a newly designed wavelet-based coder, and the enhancement layer is encoded with Progressive Fine Granularity Scalable (PFGS) coding. This algorithm involves multi-frame motion compensation, rate-distortion optimizing strategy with Lagrangian cost function and context-based adaptive arithmetic coding. In order to improve efficiency of the enhancement layer coding, an improved motion estimation scheme that uses both information from the base layer and the enhancement layer is also proposed in this letter. The wavelet-based coder significantly improves the coding efficiency of the base layer compared with MPEG-4 ASP (Advanced Simple Profile) and H.26L TML9. The PFGS coding is a significant improvement over MPEG-4 FGS coding at the enhancement layer. Experiments show that single layer coding efficiency gain of the proposed scheme is about 2.0–3.0dB and 0.3–1.0dB higher than that of MPEG-4 ASP and H.26L TML9, respectively. The overall coding efficiency gain of the proposed scheme is about 4.0–5.0dB higher than that of MPEG-4 FGS.