The fatigue strength of a high V alloyed powder metallurgy tool steel with two different inclusion size levels, tempered at different temperatures, was investigated by a series of high cycle fatigue tests. It was shown that brittle inclusions with large sizes above 30 μm prompted the occurrence of subsurface crack initiation and the reduction in fatigue strength. The fracture toughness and the stress amplitude both exerted a significant influence on the fish-eye size. A larger fish-eye area would form in the sample with a higher fracture toughness subjected to a lower stress amplitude. The stress intensity factor of the inclusion was found to lie above a typical value of the threshold stress intensity factor of 4 MPa·m1/2. The fracture toughness of the sample with a hardness above HRC 56 could be estimated by the mean value of the stress intensity factor of the fish-eye. According to fractographic evaluation, the critical inclusion size can be calculated by linear fracture mechanics.