As a special task beyond general object detection, pedestrian detection has attracted much attention in recent years. Despite the significant improvements, detecting pedestrians is still a challenging task, especially for small-size pedestrians. In this paper, we present a multi-scale feature fusion convolutional neural network (MFF-CNN) for pedestrian detection. The MFF-CNN is benchmarked on three challenging pedestrian datasets: Caltech, INRIA and ETH datasets. The experimental results show that our proposed approach achieves highly competitive results and has a great advantage in small-size pedestrian detection.