In this paper, the work addresses the notion of generating test cases by applying Q-learning algorithm in source code to obtain the optimal solution. The test cases are generated manually using the state diagram and automatically using obtained optimal solution from Q-learning algorithm. Here, the shortest path algorithm is chosen and the optimal solution is obtained for each and every initial states. It mainly focuses on the analysis between the manually generated test cases and automatically generated test cases.