A novel type of nano-tapered Ag/PAA (Porous Anodic Alumina) hybrid structure was conveniently fabricated by a multistep anodizing and etching pore-widening method and thermal evaporation of a Ag film. Different Ag film thickness and annealing process were introduced to investigate the Ag/PAA morphology related optical property and SERS performances. It was found that the Ag/PAA hybrid structures had a strong and broad-band absorption in both ultraviolet and visible light regions owing to the geometry trapping of inverted nanotaper structure as well as the localized surface plasmon resonance (LSPR) of Ag nanostructures. The LSPR of Ag nanoparticles was further utilized for surface enhanced Raman scattering (SERS) applications. It was found that the SERS intensity was strongly dependent on the annealing-induced formation of Ag particles and interparticle nanogaps. The annealed Ag (30nm)/PAA nanostructure was demonstrated to be the most active SERS substrates with strong Raman enhancement. This hybrid structure has strong potential towards applications such as photocatalyst, biological detection and solar cells.