Skip to main content
SMFM 2025 Pregnancy Meeting Main banner
Final Program


Program Book
Icon Legend
Presentation Icons
Additional registration fee required
Additional registration fee required
CME Credit Offered
CME Credit Offered
Abstract Award
Abstract Award
Poster Icons
Additional registration fee required
Additional registration fee required
Faculty have requested this content not be shared on social media
Faculty have requested this content not be shared on social media
CME Credit Offered
CME Credit Offered
Abstract Award
Abstract Award
Back

Poster Session 4

(1202) Assessing the Reliability of ChatGPT for Evaluation of Risk-of-Bias in Randomized Clinical Trials

Friday, January 31, 2025
4:00 PM – 6:00 PM  

    Submitting Author and Presenting Author(s)

  • Takeshi Nagao, MD

    The Jikei University School of Medicine
    Minatoku, Tokyo, Japan

    • Coauthor(s)

  • George R. Saade, MD (he/him/his)

    Professor and Chair, Associate Dean for Women's Health Obstetrics and Gynecology
    Macon & Joan Brock Virginia Health Sciences Eastern Virginia Medical School at Old Dominion University
    Norfolk, Virginia, United States

  • Tetsuya Kawakita, MD, MS

    Associate Professor
    Macon & Joan Brock Virginia Health Sciences Eastern Virginia Medical School at Old Dominion University
    Norfolk, VA, United States

Objective: To examine whether ChatGPT can be useful in the assessment of the risk-of-bias in Randomized Clinical Trials (RCTs) traditionally performed by reviewers performing systematic reviews.

Study Design: The study analyzed systematic reviews and meta-analyses published in the American Journal of Obstetrics and Gynecology from 2021 to 2023, focusing on the risk-of-bias assessments in RCTs using Version 2 of the Cochrane risk-of-bias tool for randomized trials (RoB 2). ChatGPT-4 was trained on RoB 2 guidelines and then tasked with evaluating each RCT across five domains: Randomization process, Deviations from intended interventions, Missing outcome data, Measurement of the outcome, and Selection of the reported result. Each domain, along with the overall bias, was rated as Low risk of bias, Some concerns, or High risk of bias. ChatGPT's evaluations were compared to those of human reviewers, as described in published papers, using Cohen's Kappa to determine agreement. Additionally, for each RCT, ChatGPT conducted three separate evaluations to calculate the intra-ChatGPT agreement rate using Fleiss' Kappa.

Results: The study included 161 RCTs. The Cohen's Kappa scores for agreement between ChatGPT's evaluations and those of the reviewers were as follows: Domain 1 (Randomization) 0.485, Domain 2 (Deviations) 0.489, Domain 3 (Missing) 0.596, Domain 4 (Measurement) 0.446, Domain 5 (Selection) 0.484, and Overall bias 0.262. For the three separate evaluations conducted by ChatGPT on the same RCTs, the Fleiss' Kappa scores were Domain 1 (Randomization) 0.245, Domain 2 (Deviations) 0.359, Domain 3 (Missing) 0.555, Domain 4 (Measurement) 0.493, Domain 5 (Selection) 0.351, and Overall bias 0.760.

Conclusion: ChatGPT's risk-of-bias assessments moderately agreed with human reviewers across five domains but showed only fair agreement in overall bias assessment. Intra-ChatGPT evaluations were inconsistent for certain domains (Randomization, Deviations, Selection), though the overall bias assessment showed substantial agreement.