Estimating the number of endmembers and their corresponding spectral signatures in the hyperspectral imagery is crucial for successful application of the spectral un-mixing process. This paper proposes a novel two-stage process for estimating the number of endmembers. In the first stage of this process, a subset of convex pixels is located using principal components analysis (PCA) and a sequence of projective 3D convex hull. In the second stage, an endmember is identified from the subset of convex pixels by applying orthogonal subspace projection (OSP) and the subset of convex pixels is clustered using a spectral angle mapper (SAM), iteratively. The proposed method was carried out with both synthetic and real images for estimating the number of endmembers. The results demonstrated that the proposed method can be used to estimate a more reasonable and precise number of endmembers than the eigenvalue-based methods and that it can also be used to simultaneously extract an endmember and the anomaly spectra.