For a complete calibration of optical surface topography measuring instruments, which encompass their ability to measure slope and curvature, a determination of their optical transfer function is required. Errors induced by non-linearity of the scales of the instrument can affect their shift invariant properties, which in turn affect their transfer function. The non-linearity can be caused by distortion produced by the quality of the optical setup. A method to develop a correction model that combines a simple model of optical distortion with error separation techniques is discussed. Experimental tests of the method are presented and measurement uncertainties are investigated.