Tracking the pose of an object is a fundamental operation in computer vision. Yet, achieving this task for arbitrary objects without requiring a priori knowledge remains a major stumbling block. This paper introduces a method for tracking the pose of a moving object without requiring its 3D model or textured surfaces. In the first step, a sequence of images-poses pairs is obtained and PCA coefficients are derived from the image sequence. Then, a piecewise linear observation mapping is build between the poses and the PCA coefficients. The mapping is then used in the observation model of a Kalman filter that tracks the pose of the object.