A thin palladium composite membrane without any modified layer was successfully obtained on a rough porous alumina substrate. Prior to the fabrication of palladium membrane, a poly(vinyl) alcohol (PVA) layer was first coated onto the porous substrate by dip-coating technique to improve its surface roughness and pore size. After deposition of palladium membrane on the PVA modified substrate, the polymer layer can be completely removed from the composite membrane by heat treatment. The microstructure of the palladium composite membrane was characterized in detail using SEM, EDXS and XRD analysis. Permeation measurements were carried out using H2 and N2 at temperatures of 623 K, 673 K, 723 K and 773 K. The results indicated that the hydrogen permeation flux of 0.238 mol m−2 s−1 with H2 separation factor α(H2/N2) of 956 for the as-prepared palladium membrane was obtained at 773 K and 100 kPa. Furthermore, the good membrane stability was proven during the total operation time of 160 h at the temperature range of 623 K–773 K and gas exchange cycles of 30 between hydrogen and nitrogen at 723 K.