To improve the hand gesture recognition accuracy based on Hu moment features, a new recognition algorithm was developed based on the fingertip structure detection. Firstly, the geometric features, the areas of skin region and the image, were designed to segment the skin region from the background in space of hue, saturation and value of brightness. Secondly, the cam point inspections were carried out and the fingertip was detected after the contour approximation. After that, the 7-dimensional feature vector was built. Finally, the distance marching criterion was utilized to recognize the hand gesture. Compared with the Hu moment feature recognition, the developed recognition algorithm improved the recognition accuracy by 2.7%.