PRIMUS: An observationally motivated model to connect the evolution of the AGN and galaxy populations out to z~1