Inverse halftoning is a key technology to yield a continuous tone image from a halftone image. The main application is to make some image enhancement or further compression of halftones more feasible. Many former approaches have been proposed in the literature. Among these, a recent method proposed by Chung and Wu using edge-based lookup table achieves good image quality, where the edge feature is adopted to refine the trained look-up table (LUT). However, it has three deficiencies, including 1) the edge features are limited in some predefined cases, which cannot full represent every potential possibility, 2) the lookup table grows exponentially when extreme grayscales are attempted to be recorded, and 3) the trained lookup table cannot fully include all the cases, which leaves some halftone patterns in practice have no associated output grayscale. Chang et al.'s method employed one trained filter to compensate the halftone patterns that are not recorded in LUT. However, one filter cannot fully characterize the full textures in an image. In this study, the halftone patterns are classified according to its variance and then used to train the corresponding filter sets, which are then employed to provide higher prediction accuracy by inner product with the corresponding halftone patterns. As documented in the experimental results, the proposed inverse halftoning provides excellent performance in image quality and memory consumption than former approaches.