We characterize the lives of 1.2 million inventors in the United States by linking patent records to tax data. Tracking these inventors from birth through their careers, we establish three empirical results that shed light on the key factors that determine who becomes an inventor. First, rates of innovation vary substantially by parent income, race, and gender. Differences in ability account for relatively little of these gaps and inventors from under-represented groups do not have higher quality patents on average, contrary to existing models of selection into innovation. Second, exposure to innovation during childhood plays a critical role in determining children’s propensities to innovate. Growing up in an area or in a family with a high innovation rate in a particular technology class leads to a higher probability of patenting in exactly that technology class. Third, the private returns to innovation are highly skewed and are typically earned many years after career choices are made. Using a simple model that matches these facts, we show that providing children from under-represented backgrounds greater exposure to innovation have more potential to increase innovation rates than increasing the private returns to innovation.