The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence from a Three-Year National Study

Citation:

Yoon Soo Park, Ajit K Sachdeva, Kathy Liscum, Adnan Alseidi, Melissa Gesbeck, Patrice Gabler Blair, Edgardo Salcedo, Maura Sullivan, and Georges Bordage. 2021. “The American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA): Validity Evidence from a Three-Year National Study.” Ann Surg.

Abstract:

OBJECTIVE: To gather validity evidence supporting the use and interpretation of scores from the American College of Surgeons Entering Resident Readiness Assessment (ACS ERRA) Program. SUMMARY AND BACKGROUND DATA: ACS ERRA is an online formative assessment program developed to assess entering surgery residents' ability to make critical clinical decisions, and includes 12 clinical areas and 20 topics identified by a national panel of surgeon educators and residency program directors. METHODS: Data from three national testing administrations of ACS ERRA (2018-20) were used to gather validity evidence regarding content, response process, internal structure (reliability), relations to other variables, and consequences. RESULTS: Over the three administrations, 1,975 surgery residents participated from 125 distinct residency programs. Overall scores (Mean = 64% [SD = 7%]) remained consistent across the three years (p = .670). There were no significant differences among resident characteristics (gender, age, IMG status). The mean case discrimination index was 0.54 [SD = .15]. Kappa inter-rater reliability for scoring was 0.87; the overall test score reliability (G-coefficient) was 0.86 (Φ-coefficient = 0.83). Residents who completed residency readiness programs had higher ACS ERRA scores (66% vs. 63%, Cohen's d = 0.23, p < .001). On average, 15% of decisions made (21/140 per test) involved potentially harmful actions. Variability in scores from graduating medical schools (7%) carried over twice as much weight than from matched residency programs (3%). CONCLUSIONS: ACS ERRA scores provide valuable information to entering surgery residents and surgery program directors to aid in development of individual and group learning plans.