A Hybrid Deep Learning Approach to Cosmological Constraints from Galaxy Redshift Surveys