In this paper, a novel algorithm based on logistic regression digital backpropagation (LR‐DBP) is proposed to mitigate the effect of dispersion and nonlinearity in a long‐distance optical transmission system. The LR‐DBP can evaluate nonlinear effects by estimating the cross‐entropy error and adjusting the structure of DBP. The experiment is setup to demonstrate an 80 Gbit/s 16QAM signal transmitted across 1000 km single‐mode fiber. The results show that LR‐DBP achieves a 1.8 dB launch power improvement over the dispersion compensation algorithm and a 0.6 dB launch power gain with DBP. The proposed algorithm could obtain with 19.2% computational complexity decrease and 19.6% central processing unit (CPU) running time decrease by calculating the average numbers of real multiplications and CPU running times.