In this paper, we study spectrum overlay access to share spectrum between primary users (PUs) and secondary users (SUs). Our goal is to maximize the average throughput by optimal power allocation within finite time duration. To do this, we formulate this optimization problem as a Markov Decision Process with continuous state. An approximate value method with pre-allocation mechanism is proposed, which can effectively protect the PUs by obtaining a continuous closed-form solution rather than a discrete one. Numerical results show that the proposed algorithm exhibits better performance than traditional methods while guaranteeing non-interference to PUs.