Robust Inference for Misspecified Models Conditional on Covariates