The study purpose was to examine the strength, direction and shape of the associations of environmental perceptions with recreational walking and leisure-time moderate-to-vigorous physical activity, using pooled data from four study sites (Baltimore [USA], Seattle [USA], Adelaide [Australia] and Ghent [Belgium]). Moreover, site- and gender-specificity of the associations were examined. In total, 6014 adults (20–65 years, 55.7% women) completed the Neighborhood Environmental Walkability Scale and the International Physical Activity Questionnaire. Both a ‘recreational walking-friendliness’ index and a ‘leisure-time activity friendliness’ index had a positive linear association with recreational walking and leisure-time moderate-to-vigorous physical activity, respectively. The associations were significant in all study sites except Ghent. Present findings were clearly site-specific, imposing possible challenges for built environment recommendations. In Belgium, interventions to promote leisure-time activity may need to target promotion of existing opportunities rather than built environment improvements.