Classification rules are the interest of most data miners to summarize the discrimination ability of classes present in data. A classification rule is an assertion, which discriminates the concepts of one class from other classes. The most classification rules mining algorithm aims to providing a single solution where multiple solutions exist. Moreover, it does not guarantee the optimal solution and user has not any control over the classification error rate. In this paper, we addressed these problems inherent in mostly used classification algorithms. A solution has been proposed to solve these problems and it has been tested with experimental data.