Speckle is a granular multiplicative noise that reduces the resolution and contrast of the image there by degrading the diagnostic accuracy of the Ultrasound image. Speckle reduction technique has to be followed, to enhance the quality of ultrasound image. In this paper, an efficient algorithm capable of removing speckle noise using Particle Swarm Optimization (PSO) technique is presented. The algorithm initially uses Modified Hybrid median filter to reduce speckle noise present in the corrupted image, which computes the median of the diagonal elements, and maximum of the horizontal and vertical elements in a moving window and finally the two values are compared with the central pixel and the median value of the three values will be the new pixel value. It uses PSO to return optimized weighting factor of median of diagonal element and maximum of horizontal and vertical element to recover the corrupted image. The variance of the extracted uniform block is the objective function of PSO. The performance of the algorithm is tested and compared with Active filter, Lee filter, Frost filter, and Modified Hybrid median filter. Experimental results shows that the proposed method removes the speckle noise effectively without losing the information in the images.