We investigate the layer undulations that appear in smectic A liquid crystals when a magnetic field is applied in the direction parallel to the smectic layers. In an earlier work (García-Cervera and Joo in J Comput Theor Nanosci 7:795–801, 2010) the authors characterized the critical field using the Landau–de Gennes model for smectic A liquid crystals. In this paper, we obtain an asymptotic expression of the unstable modes using Γ-convergence theory, and a sharp estimate of the critical field. Under the assumption that the layers are fixed at the boundaries, the maximum layer undulation occurs in the middle of the cell and the displacement amplitude decreases near the boundaries. Our estimate of the critical field is consistent with the Helfrich–Hurault theory. When natural boundary conditions are considered, the displacement amplitude does not diminish near the boundary, in sharp contrast with the Dirichlet case, and the critical field is reduced compared to the one calculated in the classical theory. This is consistent with the experiments carried out by Ishikawa and Lavrentovich (Phys Rev E 63:030501(R), 2001). Furthermore, we prove the existence and stability of the solution to the nonlinear system of the Landau–de Gennes model using bifurcation theory. Numerical simulations are used to illustrate the predictions of the analysis.