Due to the development of hand-held Plenoptic cameras, the light filed camera has become more and more popular in recent years. The light field camera can capture the angular and depth information and is able to construct a stereoscopic image. To obtain a high quality reconstructed image, the disparity should be estimated accurately. However, for the sub-images taking from the light field camera, especially for those of the microscope case, it usually happens that there is no feature point within a patch and the disparity is hard to be determined accurately. In this paper, to improve the efficiency and the accuracy of disparity estimation for light field cameras, we propose a multilayer matching architecture. Moreover, a gradient map is applied to assign the weight for disparity matching. In addition, we apply the segmentation method to the rendered final image and refine the disparity estimation result. Simulations show that a more accurate depth estimation result can be achieved when using the proposed architecture.