In-loop filters are widely employed in video coding standards to improve coding efficiency by reducing quantization noise. However, only local spatial correlation of video frames is exploited in existing in-loop filters, which limits the improvement of video coding efficiency. In this paper, a higher order transform domain filter (HOTDF) is proposed to further reduce quantization noise. The filter fully exploits non-local spatial correlation by grouping similar patches and sparsely representing them in a higher order transform domain. Then quantization noise can be well removed by preserving large transform coefficients conveying majority of original signals and discarding the remaining smaller ones mainly caused by quantization. Experimental results show that compared with HM16.0, the proposed algorithm achieves 1.9%, 3.0% and 2.5% BD-rate reduction for luminance component on average for all intra, low delay P and random access configurations, respectively.