Our objective was to determine the ability of the internal medicine In-Training Examination (ITE) to predict pass or fail outcomes on the American Board of Internal Medicine (ABIM) certifying examination and to develop and externally validated predictive model and a simple equation that can be used by residency directors to provide probability feedback for their residency programs. We collected a study sample of 155 internal medicine residents from the three Virginia internal medicine programs and a validation sample of 64 internal medicine residents from a residency program outside Virginia. Scores from both samples were collected across three class cohorts. The Kolmogorov-Smirnov z test indicated no statistically significant difference between the distribution of scores for the two samples (z=1.284, p=.074). Results of the logistic model yielded a statistically significant prediction of ABIM pass or fail performance from ITE scores (Wald=35.49, SE=0.036, df=1, p<.005) and overall correct classifications for the study sample and validation sample at 79% and 75%, respectively. The ITE is a useful tool in assessing the likelihood of a resident’s passing or failing the ABIM certifying examination but is less predictive for residents who received ITE scores between 49 and 66.