We will present a microscopic model to understand the nonreciprocal optical responses, i.e., the second-harmonic generation through electric and magnetic dipole moments and the magneto-electric effects. Here the twist crystalline field, which reflects the real crystal structure, is found to play an important role in both channels of the second-harmonic generation. Then we can explain the spectra of the second-harmonic generation through electric and magnetic dipoles. Absolute as well as relative magnitudes of the second-order susceptibilities through both channels are also estimated to be reasonable. In terms of the same model, we can also understand the magneto-electric spectra, i.e., the electric- and magnetic-dipole interference in the linear response.