The Granular Origins of Aggregate Fluctuations