Joint Frequency-and Image-Space Learning for Fourier Imaging