This paper proposed a framework based on reinforcementlearning for color object segmentation in videosequences. This agent-based method is used to find the appropriatevalues to detect an object. The reinforcement learningagent contains some sample images and their ground truthfrom which to learn. The information obtained from theexploration/exploitation of the solution space is used by theintelligent agent to enhance the quality of the segmented imageover time. As the proposed method uses a dynamic framework, it has a good potential for color object segmentation applicationin video sequences.