(1108) Physician vs. Partometer: a machine learning model predicts mode of delivery as well as physicians

Objective: Performance of predictive machine learning (ML) models has rarely been assessed against clinician intuition. The Partometer is a validated ML model which uses data from the electronic health record to generate ongoing predictions of a patient’s likelihood of vaginal delivery (VD). The aim of this study is to compare calibration and accuracy of the Partometer in predicting VD compared to the physician.

Study Design: This was a prospective study of all patients presenting for a planned VD. At 4 hours, we conducted two concurrent assessments: 1) to the patient’s physician “on a scale of 1-10 (ordinal; 10 = highest), what is your patient’s likelihood of a vaginal delivery?”; 2) we queried the Partometer on the same continuous scale.

Patient information was chart abstracted; physician characteristics were obtained from public sources. We assessed calibration by fitting a linear mixed model for difference in Brier scores with random intercept to account for the predictions made by the same physician. For accuracy, we used >6 as predicting a VD (i.e., if the prediction was >6 and a VD occurred, this was a true positive). We stratified by patient characteristics. Subgroup analysis was performed among patients who were nulliparous, term, singleton vertex (NTSV).

Results: There were 203 predictions from 46 clinicians. The Partometer was equally well calibrated (Partometer 0.11 +/- 0.18, physician 0.12 +/- 0.22) and accurate (85.7% [80.1-90.2] vs. 80.8% [74.7-86.0]) as physicians in predicting VD. The findings persisted after adjustment for patient and clinician characteristics. On demographic-adjusted analysis, both the Partometer and physicians were less well calibrated for patients who were non-Hispanic Black, with obesity, or nulliparous. For the NTSV subgroup, predictions on subjects age ≥35 were less well calibrated by both Partometer and physicians. Differences in calibration were found in the NTSV subgroup among the clinicians < 40 years old and/or < 10 years of practice.

Conclusion: ML model to predict mode of delivery was as well calibrated and accurate as physicians’ own predictions of their patients.