In this study, a visible-light based fast iris ellipse fitting based gaze tracking scheme is developed for wearable eye trackers. First, after image enhancement pre-processing of eye images, the two-level binarization identifies the iris contour, and the candidate points for ellipse fittings are selected from the binaried iris profile. Next, by fast Random Sample Consensus (RANSAC) ellipse fitting, the iris centers are predicted, and finally the gaze tracking acts effectively when the calibration is done. In our experiments, the averaged estimation errors of iris centers are smaller than 7 pixels. At training mode, the indoor averaged horizontal and vertical accuracies of gaze tracking are 1.80 and 1.59 degrees, respectively. At testing mode, the indoor averaged horizontal and vertical accuracies of gaze tracking are 1.83 and 1.96 degrees, respectively. Besides, the outdoor averaged accuracies of gaze tracking are less than 2.5 degrees. The proposed main function executes up to 83 frames/sec by a personal computer at 3.4GHz working frequency.