We consider the problem of bit error rate (BER) degradation because of the power gain imbalance between horizontal (H)‐polarization and vertical (V)‐polarization components in an orthogonal dual‐polarization transmission system. To alleviate the aforementioned BER degradation problem, we propose a non‐orthogonal polarization‐domain rotation scheme where the axes of H‐polarization and V‐polarization components are rotated with different angles at the transmitter and de‐rotated at the receiver. In addition, in order to assess the effectiveness of the polarization‐domain rotation scheme, we derive the closed‐form BER expression under a practical dual‐polarized channel model, which is represented by cross‐polarization ratio and co‐polarization ratio (CPR). We also derive the approximated BER expressions for the two asymptotic values of CPR: balanced CPR and infinite CPR. With the derived BER expressions, we find the optimal rotation angles that jointly minimize the BER. According to the numerical results, it is shown that about 3dB Eb/N0 gain is obtained at the BER of 10−4 and the CPR of 10dB by the polarization‐domain rotation scheme with optimal rotation angles compared with the conventional orthogonal dual‐polarization transmission. Copyright © 2015 John Wiley & Sons, Ltd.