A new algorithm is proposed to retrieve sea surface wind speed and wind direction from marine X-band radar images measured in the nearshore region. To remove the influence of ships, a preprocessing method is proposed to identify them from gray-level radar image sequence. To retrieve wind speed under different sea states, the probability distribution function of gray levels and significant wave height are used to develop a wind speed model. The azimuthal dependence of gray levels is used to retrieve wind direction, with a deviation added to adjust the influence of land. The root mean square error (RMSE) between the wind speeds measured by anemometer and retrieved from marine X-band radar image sequences is 1.37m/s for wind speed of 0–20m/s, and the RMSE of wind direction between them is 26.2° for wind speed of 5–20m/s, which show that the proposed method is reasonable. The influences of sea states and ships to the algorithm are also discussed.