The remarkable molecular diversity of breast cancers affects therapeutic options and clinical outcome for patients. Understanding the cell of origin and the factors that impact stem cell differentiation and proliferation are crucial aspects for the development of effective breast cancer prevention, diagnostic, and therapeutic strategies. Evidence suggests that molecular subtypes, such as luminal and basal breast cancers, may originate from cells in different stages of epithelial differentiation. Here we review studies that suggest basal breast cancer may have a stem or progenitor cell origin. This work has been augmented by the availability of BRCA1 mutant tissues and mouse models of basal breast cancer. We also discuss environmental factors that have been shown to affect luminal breast cancer initiation and progression. Further studies are needed to obtain comprehensive answers to many of these questions, yet the results underscore the importance of breast stem cells in tumorigenesis.