Tracking-by-detection based on online learning has shown superior performance in visual tracking of unknown objects. However, most existing approaches use a fixed-size box to represent objects and can merely show the unoccluded area of the object. To overcome the limitations, we propose a novel tracking-by-detection approach based on local patches. We extend ferns forest to visual tracking and optimize online learning with the reliability of the predicted object. Moreover, a re-sampling technique is used to obtain the scale of the object and locate its unoccluded area. To show the benefits of our approach, we run our algorithm on various challenging sequences, and compare it with the state-of-the-art methods. The experiment results show that our algorithm receives an accurate tracking and a good robustness in tracking rigid and non-rigid objects.