(822) Evaluating an Artificial Intelligence Platform Answering Common Pregnancy Questions: Satisfactory but Inconsistent.

Friday, January 31, 2025

10:30 AM – 12:30 PM

Submitting Author and Presenting Author(s)

Leonardo F. Antelo, MD, N/A

Resident Physician
Mount Sinai West Obstetrics and Gynecology
New York, New York, United States

Coauthor(s)

LB

Lois Brustman, MD

Icahn School of Medicine at Mount Sinai West
New York, New York, United States
ZA

Zainab Al-Ibraheemi, MD

Icahn School of Medicine at Mount Sinai West
New York, New York, United States
GA

Graham Ashmead, MD

Mount Sinai West
New York, New York, United States

Objective: The availability of artificial intelligence platforms is rapidly increasing. We assessed the platform ChatGPT 3.0 and its ability to answer questions about common pregnancy conditions. This study analyzed the appropriateness of responses, the consistency of responses over multiple iterations, and the presence of misinformation within the responses.

Study Design: Seven questions were generated for each of the following pregnancy conditions to yield a total of 28 questions: Gestational Diabetes, Preeclampsia, Fetal Growth Restriction, and Oligohydramnios. Questions were entered into ChatGPT 3.0 in an identical manner three separate times, generating 84 total responses. Since ChatGPT forms a unique response even when exposed to the same prompt, each set of responses was reviewed by one of three board-certified MFM physicians. The accuracy of responses to each specific question was rated 1) Satisfactory, 2) Incomplete, or 3) Inadequate. ChatGPT responses contain information beyond the scope of the question. This additional information was evaluated for the presence or absence of misinformation by each reviewer.

Results: The responses of 13 out of 28 questions received consistent Satisfactory ratings over the three separate iterations (47%). 26 out of 28 questions had at least one response that was satisfactory (93%). When assessing the total 84 responses for misinformation, 22 out of 84 contained misinformation (27%). 51 out of 84 were rated as both Satisfactory and did not contain misinformation (61%). Only 7 out of the 28 questions generated consistent answers by the reviewers, receiving both satisfactory ratings and also not containing misinformation (25%).

Conclusion: Our results suggest that Chat GPT 3.0 is able to generate satisfactory responses to common pregnancy questions. However, we found it to be inconsistent over multiple iterations of the same question. This platform should be used with caution, and the presence of misinformation should be noted when counseling patients on the use of AI.