The dynamical glass transition is typically taken to be the temperature at which a glassy liquid is no longer able to equilibrate on experimental timescales. Consequently, the physical properties of these systems just above or below the dynamical glass transition, such as viscosity, can change by many orders of magnitude over long periods of time following external perturbation. During this progress toward equilibrium, glassy systems exhibit a history dependence that has complicated their study. In previous work, we bridged the gap between structure and dynamics in glassy liquids above their dynamical glass transition temperatures by introducing a scalar field called ``softness,'' a quantity obtained using machine-learning methods. Softness is designed to capture the hidden patterns in relative particle positions that correlate strongly with dynamical rearrangements of particle positions. Here we show that the out-of-equilibrium behavior of a model glass-forming system can be understood in terms of softness. To do this we first demonstrate that the evolution of behavior following a temperature quench is a primarily structural phenomenon: The structure changes considerably, but the relationship between structure and dynamics remains invariant. We then show that the relaxation time can be robustly computed from structure as quantified by softness, with the same relation holding both in equilibrium and as the system ages. Together, these results show that the history dependence of the relaxation time in glasses requires knowledge only of the softness in addition to the usual state variables.