TL;DR
Get health and wellness essentials delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Physicians reviewing ChatGPT responses to IVF questions rated them lower than American Society for Reproductive Medicine education materials on accuracy, completeness, safety and overall quality, while finding their clarity comparable. The small study also found unsupported claims in half of the chatbot responses and misinformation in 4.2%; it did not assess patient outcomes.
Physicians rated ChatGPT’s answers to common IVF questions below educational materials from the American Society for Reproductive Medicine (ASRM) for accuracy, completeness, safety and overall quality, according to a comparative analysis presented at the society’s annual meeting. The chatbot’s clarity scores were similar to ASRM’s, but reviewers also found unsupported claims and some misinformation in its responses.
Researchers compared ChatGPT responses about the IVF process and associated risks with two ASRM education documents. They used the free GPT-5.6 Luna version, ran each prompt three times, and collected 12 chatbot responses. Four physicians reviewed the responses and documents, rating them on a five-point scale and checking whether they covered key concepts.
ChatGPT’s mean scores were 3.98 for accuracy, 3.40 for completeness, 2.92 for safety and 3.38 for quality. ASRM materials scored 5.00, 4.88, 4.88 and 5.00 on those measures, respectively. For clarity, the scores were close: 4.60 for ChatGPT and 4.63 for ASRM. The checklist covered by the chatbot averaged 94.1%, compared with 100% for the society materials.
The researchers said half of ChatGPT’s answers contained an unsupported claim, while 4.2% contained misinformation; reviewers found none in the ASRM responses. In a separate comparison, the chatbot’s estimates of IVF success for ages 20 through 45 generally tracked CDC and Society for Assisted Reproductive Technology data and showed declining birth rates with age. However, it overestimated live birth rates at age 35 and underestimated them at ages 37 and 40.
Clear Answers Can Still Miss Key Details
The findings matter because people may use chatbots to make sense of complex fertility treatment and its risks. A response that is easy to read can appear dependable even when it leaves out important information or includes an unsupported statement. In this analysis, similar clarity scores did not translate into similar physician ratings for safety, completeness or quality.
The results do not show that every AI answer about IVF is wrong, or establish how chatbot use affects medical decisions. They do point to a practical concern: readability is not a substitute for reliability, particularly when a question involves treatment choices or individualized chances of success. The study’s author encouraged clinicians to ask patients whether they use chatbots, help them request sources, and direct them to trusted educational materials.
Products Worth Considering
KitchenAid Artisan 5-Quart Tilt-Head Stand Mixer, KSM150PS, Almond Cream
Vitamix Ascent X3 Professional-Grade Blender, 3 Programs, Shadow Black
Cuisinart Chef's Classic 11-Piece Stainless Steel Cookware Set
As an affiliate, we earn on qualifying purchases.
How Researchers Compared the Answers
The project had two parts. First, the team compared chatbot responses to common questions about IVF procedures and risks with ASRM’s patient education materials. Physicians scored the materials and responses, assessed checklist adherence and red flags, and measured readability using the Flesch-Kincaid grade-level scale.
ChatGPT answers based on conversational prompts were assessed at about an eighth-grade reading level; responses to generic prompts were closer to a tenth-grade level. ASRM documents were roughly at a twelfth-grade level. The researcher said this suggests the wording of a patient’s prompt can affect readability. The second part compared chatbot estimates of IVF success by age with CDC and SART data; it was a comparison of estimates, not an assessment of outcomes for individual patients.
“The key point here is that fluent, confident wording can coexist with omissions, outdated information, and errors.”
— Nicole Fischer, MD, MPH, reproductive endocrinology and infertility fellow at the University of Pittsburgh
Limits of This Small Comparison
The analysis involved one version of one chatbot and a limited set of prompts and responses. Fischer identified the small sample as a limitation and said newer or specialized systems, including ChatGPT Health, should be tested separately. The findings should not be treated as a rating of every AI model or of future versions.
Some measures were subjective or descriptive, and the study did not track patient-level outcomes. It therefore cannot establish whether patients who consult a chatbot receive worse care, misunderstand their options, or make different treatment decisions. The report also does not establish how often patients encounter these kinds of errors in everyday use.
Testing More Models and Patient Use
Further evaluations could compare more chatbot versions, specialized health tools and a wider range of IVF questions against current professional and public-health sources. Researchers would also need to examine whether the answers change when prompts are repeated or phrased differently, and whether the information patients receive affects their understanding or decisions.
For now, Fischer recommended that clinicians ask patients about chatbot use, encourage them to request cited sources and point them toward trusted resources. The study’s authors’ next suggested step is testing newer and specialized models; the report provides no timeline for that work.
Key Questions
Did ChatGPT perform as well as ASRM materials on IVF questions?
In this analysis, physician ratings for ChatGPT were lower on accuracy, completeness, safety and quality. Its clarity rating was close to that of the ASRM materials.
How often did reviewers find unsupported claims or misinformation?
The researchers reported that half of the chatbot answers contained an unsupported claim and 4.2% contained misinformation. They reported no misinformation in the ASRM responses reviewed.
Were ChatGPT’s IVF success estimates accurate?
Most estimates across ages 20 to 45 were described as in line with CDC and SART data, and the chatbot showed birth rates declining with age. It overestimated live birth rates at age 35 and underestimated them at ages 37 and 40.
Can the findings be applied to every AI chatbot?
No. The researchers tested one version of a single chatbot, and Fischer said newer and specialized models need evaluation. The study also had a small sample and did not measure patient outcomes.
Source: medpagetoday.com
