This study aims to measure the associations of physical activity and one of its components, sport and exercise, with at-risk substance use in a population of young men.Baseline (2010–2012) and follow-up (2012–2013) data of 4748 young Swiss men from the Cohort Study on Substance Use Risk Factors (C-SURF) were used. Cross-sectional and prospective associations between at-risk substance use and both sport and exercise and physical activities were measured using Chi-squared tests and logistic regression models adjusting for covariates.At baseline, logistic regression indicated that sport and exercise is negatively associated with at-risk use of cigarettes and cannabis. A positive association was obtained between physical activity and at-risk alcohol use. At baseline, sport and exercise was negatively associated with at-risk use of cigarettes and cannabis at follow-up. Adjusted for sport and exercise, physical activity was positively associated with at-risk use of cigarettes and cannabis.Sport and exercise is cross-sectionally and longitudinally associated with a low prevalence of at-risk use of cigarettes and cannabis. This protective effect was not observed for physical activity broadly defined. Taking a substance use prevention perspective, the promotion of sport and exercise among young adults should be encouraged.