Cross-validated Area Under the ROC curve for Stata users: cvauroc

Presentation Date: 

Wednesday, October 24, 2018

Location: 

Pompeu Fabra University, Barcelona, Spain (Spanish Stata Users Meeting, 2018)

Presentation Slides: 

Receiver operating characteristic (ROC) analysis is used for comparing predictive models, both in model selection and model evaluation. This method is often applied in clinical medicine and social science to assess the trade-off between model sensitivity and specificity. After fitting a binary logistic regression model with a set of independent variables, the predictive performance of this set of variables - as assessed by the area under the curve (AUC) from a ROC curve - must be estimated for a sample (the 'test' sample) that is independent of the sample used to predict the dependent variable (the 'training' sample). An important aspect of predictive modelling (regardless of model type) is the ability of a model to generalize to new cases. Evaluating the predictive performance (AUC) of a set of independent variables using all cases from the original analysis sample tends to result in an overly optimistic estimate of predictive performance. K-fold cross-validation can be used to generate a more realistic estimate of predictive performance. To assess this ability in situations in which the number of observations is not very large, cross-validation and bootstrap strategies are useful. cvAUROC is a user written Stata command that implements k-fold cross-validation for the AUC for a binary outcome after fitting a logistic regression model and provides the cross-validated fitted probabilities for the dependent variable or outcome, contained in a new variable named _fit. Different options and examples for the use of cvAUROC can be downloaded at https://github.com/migariane/cvAUROC and can be directly installed in Stata using ssc install cvAUROC.