Panchromatic Observations of SN 2011dh Point to a Compact Progenitor Star