Fitness trade‐offs across episodes of selection and environments influence life‐history evolution and adaptive population divergence. Documenting these trade‐offs remains challenging as selection can vary in magnitude and direction through time and space. Here, we evaluate fitness trade‐offs at the levels of the whole organism and the quantitative trait locus (QTL) in a multiyear field study of Boechera stricta (Brassicaceae), a genetically tractable mustard native to the Rocky Mountains. Reciprocal local adaptation was pronounced for viability, but not for reproductive components of fitness. Instead, local genomes had a fecundity advantage only in the high latitude garden. By estimating realized selection coefficients from individual‐level data on viability and reproductive success and permuting the data to infer significance, we examined the genetic basis of fitness trade‐offs. This analytical approach (Conditional Neutrality‐Antagonistic Pleiotropy, CNAP) identified genetic trade‐offs at a flowering phenology QTL (costs of adaptation) and revealed genetic trade‐offs across fitness components (costs of reproduction). These patterns would not have emerged from traditional ANOVA‐based QTL mapping. Our analytical framework can be applied to other systems to investigate fitness trade‐offs. This task is becoming increasingly important as climate change may alter fitness landscapes, potentially disrupting fitness trade‐offs that took many generations to evolve.