It is very important to construct the training set and determine the sample number in the regression problem. In this paper, a new idea of constructing the training set is elaborated. The key point of this idea is to choose the hyper-parameters before determining the training set. More importantly, a heuristic approach is proposed to select samples of support vector machine (SVM). Using these methods, the relationship between generalization error and the number of training samples on a given confidence level is computed. The empirical results on benchmark data (Boston Housing) and engineering data indicate that the proposed approach can give a reference to construct the proper training set. Moreover, the proposed approach has practical significance for other parametric learning machine.