To enhance the generalization ability of average reward reinforcement in continuous state space, this paper proposes an improved algorithm based on fuzzy inference system. In reinforcement learning, agent accesses to knowledge through the interaction with the environment. The reward signal from the environment carries out comprehensive appraisal of the effect of the action, which is operated by the agent. However, when the learning environment is complex and continuous, it will increase the difficulty of learning process and reduce the learning efficiency, even can not accomplish. In response to these disadvantages, we use fuzzy inference system as the approximator to generalize the continuous state space. Duo to this solution, the application scope of reinforcement learning can be effectively expanded. To verify the reasonableness of algorithm, we apply new version to Robocup simulation soccer. Through the analysis of experimental data, the results show that the performance of the improved algorithm is better than the original algorithm.