We present a fast and low-cost fractional motion estimation (FME) and its corresponding VLSI architecture for H.264/AVC. The proposed direction-based FME algorithm just searches 8 points and employs four hadamard transforms to reduce computation time and hardware cost. Additionally, we propose two mode reduction methods to skip most search modes to further reduce computation complexity. Our FME can code a macroblock within 631 cycles on average and quite enough to achieve real-time coding for HD1080p video sequences operating at the frequency of 155 MHz.