The frustrated spin-1/2 J1a–J1b–J2 antiferromagnet with anisotropy on the two-dimensional square lattice was investigated, where the parameters J1a and J1b represent the nearest neighbor exchanges and along the x and y directions, respectively. J2 represents the next-nearest neighbor exchange. The anisotropy includes the spatial and exchange anisotropies. Using the double-time Green’s function method, the effects of the interplay of exchanges and anisotropy on the possible phase transition of the Néel state and stripe state were discussed. Our results indicated that, in the case of anisotropic parameter 0 ≤ η ≤ 1, the Néel and stripe states can exist and have the same critical temperature as long as J2 = J1b/2. Under such parameters, a first-order phase transformation between the Néel and stripe states can occur below the critical point. For J2 ≠ J1b/2, our results indicate that the Néel and stripe states can also exist, while their critical temperatures differ. When J2 > J1b/2, a first-order phase transformation between the two states may also occur. However, for J2 < J1b/2, the Néel state is always more stable than the stripe state.