We describe a new method to relieve common assumptions/ restrictions in head tracking by using a model-based approach. This improves local feature matching which only considers the pattern around the extracted feature excluding the object shape, so that misalignment can occur. In this paper, to overcome constraints on motion we consider region- and distance-based feature refinement methods to validate the local features used when tracking the ellipsoidal object. We also present a direct mapping method to reconstruct 3D feature positions for tracking. The utility of the new method has been demonstrated for face pose estimation using the Boston face database.