Based on the linear matrix inequality (LMI) method, this paper is concerned with the robust asymptotic stability of neutral stochastic neural networks with delay. First, we discuss the asymptotic stability in mean square of neutral stochastic neural networks without delay in stochastic perturbation and obtain a sufficient condition. This condition is dependent on the size of the time delay, which is usually less conservative than delay-independent ones. Then, we extend the method to cope with the robust asymptotic stability analysis of neutral stochastic neural networks with uncertainties and delay in stochastic perturbation. By employing a Lyapunov-Krasovskii functional, Jensen's inequality and conducting the stochastic analysis, an LMI approach is developed to derive the stability criteria. The proposed criteria can be checked readily by using some standard numerical packages, and no tuning of parameters is required. Examples are provided to show the effectiveness and applicability of the proposed criteria.