Traumatic brain injury (TBI) due to falls, car accidents, and warfare affects millions of people annually. Determining personalized therapy and assessment of treatment efficacy can substantially benefit from longitudinal (4D) magnetic resonance imaging (MRI). In this paper, we propose a method for segmenting longitudinal brain MR images with TBI using personalized atlas construction. Longitudinal images with TBI typically present topological changes over time due to the effect of the impact force on tissue, skull, and blood vessels and the recovery process. We address this issue by defining a novel atlas construction scheme that explicitly models the effect of topological changes. Our method automatically estimates the probability of topological changes jointly with the personalized atlas. We demonstrate the effectiveness of this approach on MR images with TBI that also have been segmented by human raters, where our method that integrates 4D information yields improved validation measures compared to temporally independent segmentations.