We study an interaction between the solar wind and crustal magnetic fields on the lunar surface using SELENE (Kaguya) data. It has been known that magnetic enhancements are at times detected near the limb external to the lunar wake, which is thus called lunar external magnetic enhancement (LEME), as a result of direct interaction between the solar wind and lunar crustal fields. Although previous observational studies showed that LEMEs in the high solar zenith angle region favor stronger interplanetary magnetic field (IMF) and higher solar wind density, the relation between the IMF and the crustal field orientation has not been taken into account. We show evidence that the relation between the IMF and crustal field orientation is also one of the key factors that control the extent of LEME, focusing on one-day observations at 100km altitude that include data above strong crustal fields around South Pole-Aitken (SPA) basin. Strong LEMEs are detected at 100km altitude around SPA basin under the stronger and northward IMF condition, while they weaken under southward IMF. All LEME's peaks are located in the region where unperturbed crustal fields at 300km altitude are directed northward while they are less related to unperturbed crustal fields at 100km or lower, which suggests that lunar crustal fields are compressed by the solar wind dynamic pressure, and its large scale component parallel to the IMF is essential to the formation of the LEME.