To determine the accuracy of image-based diagnosis for stage 4 or worse retinopathy of prematurity (ROP) disease.Prospective cohort study.We prospectively obtained data, from 8 major ROP centers, for 1220 eye examinations from 230 infants.An ophthalmologist at each center provided a clinical diagnosis using indirect ophthalmoscopy. Wide-angle retinal images (RetCam; Clarity Medical Systems, Pleasanton, CA) were then obtained, and these were independently read by 2 ROP experts using a web-based system for an image-based diagnosis.Sensitivity and specificity of image-based diagnosis from the ROP experts were calculated using the clinical diagnosis as the reference standard.Of 1220 examinations, 28 (2%) had a clinical diagnosis of stage 4 or worse. Sensitivity and specificity for stage 4 or worse disease were 75% and 99% for expert 1, and 86% and 99% for expert 2. Sensitivity and specificity for the detection of stage 5 disease were 69% and 99% for both experts.There are inconsistencies in the accuracy of image-based diagnosis of stage 4 and stage 5 ROP when compared with the clinical diagnosis.