This paper addresses the problem of efficient pedestrian detection using features that are extracted by convolving feature channels with a very small number of filters. The method uses as feature channels low level features such as LUV colour and HOG, and trains a boosted decision forest on top of the learned features. The feature selection is guided by a greedy search or by an exhaustive search on a few number of scaled versions of simple horizontal, vertical and uniform filters. Extensive results on the challenging Caltech dataset show that with only 3 filters we obtain state-of-the-art results achieving a 18.3% miss rate (MR). Using optical flow as an additional input, we further improve the results and obtain a 15.5% MR.