This study focuses the inter-relationships among the men, the use, and the intra-specific cassava diversity, under the perspective of this crop evolutionary dynamics. The origin, the use and the current local management of varieties with high and low cyanogenic potential are important questions around cassava domestication. We collected 169 local varieties identified as “sweet” or “bitter” cassava by traditional farmers from Atlantic Forest and Amazon (Medium Negro River Basin), Brazil. Using a population genetics and an ethnobotany approach, the diversity and the genetic structure of cassava were evaluated. We found a total of 115 vernacular names, and in the Atlantic Forest sample the average genetic diversity (H S = 0.654) was higher for the sweet varieties than for bitter ones (0.582). The genetic differentiation coefficient (R ST), used to estimate the diversity among groups, was 0.057 (P < 0.001), indicating that the divergence between the two groups is low. We obtained a low correlation between the morphological and genetic distances, and the congruence was high when the ethnoclassification and the genetic structure were considered. We discuss the adaptive advantages of the sweet varieties use, the current socio-economic changes in bitter varieties use, and the ecological history of these variety groups.