This paper proposes a new method to detect mouth in color image. Instead of segmenting images based on lip color (like other methods often do), we segment images based on skin. After that, we add some specific techniques to approach the mouth in color image efficiently to determine mouth candidates quickly and then classify mouth candidates by neural network. Therefore, the runtime of method is improved remarkably and the result is good (about 85.5%). Moreover, this method allows robust mouth detection under different light conditions and with any particular make-up.