This paper shows that much of the decline in labor force participation of U.S. prime age men comes from “in-and-outs” – whom I define as men that temporarily leave the labor force. Individuals moving in and out of the labor force have been an understudied margin of labor supply and account for roughly one third of the decline in participation between 1977 and 2015. Most in-and-outs take an occasional short break in between jobs, but are otherwise continuously attached to the labor force. In-and-outs are also qualitatively different from long-term non-participants. Half of the rise of in-and-outs has come from married or cohabiting men and I show that this rise can be explained by a wealth effect from their partners' growing earnings, using variation in the growth of female wages across demographic groups. In addition, I find that changes in household structure can account much of the rest of the rise of in-and-outs. To capture both effects in a unified framework, I construct and estimate a dynamic model of labor supply and household formation. The model estimates imply that labor supply factors are responsible for nearly the entire rise of in-and-outs, while changes in labor demand have contributed little.