Lenstra-Lenstra-Lovász (LLL) is an effective lattice reduction algorithm in multi-input-multi-output (MIMO) system. Its modified version Fix Lenstra-Lenstra-Lovász (F-LLL) algorithm uses fix method to substitute round method in original LLL algorithm in processing the parameter aiming to largely reduce the algorithm complexity. However, with the number of antennas and iteration times increasing, performance of F-LLL gradually becomes unsatisfactory. It will cause much performance loss especially in large MIMO system. In this paper, a novel hybrid algorithm which inserts a pre-judging procedure and combines both fix method and round method is proposed. The pre-judging procedure firstly searches all the size reduction pairs to calculate all the possible values. Then analyze these values by making histogram. Fitting curve of this histogram will be presented as Gaussian distribution that mean value and standard deviation will be determined. The iteration criterion is settled equals to standard deviation. Hybrid F-LLL algorithm makes a better trade-off between algorithm complexity and performance. Also it can solve the problem of large performance loss in F-LLL.