If you don’t completely trust Doctissimo to diagnose the ailments you suffer from, don’t trust ChatGPT when it comes to diagnosing your child either.
This is a perfect illustration of the risks and progress that remains to be made in the field of AI. A study published in the American medical journal JAMA Pediatrics illustrates how ChatGPT is far from replacing your GP when it comes to finding out what’s wrong with your child’s health.
An incorrect diagnosis in 83% of cases
Questioned about a hundred cases of childhood illness, the chatbot (in its version 4.0) failed in 83% of diagnoses. In 72% of cases, the entire response given by ChatGPT was incorrect and, in the remaining 11%, the chatbot established a partial, but insufficient, diagnosis. This study follows another published in July 2023 which showed that ChatGPT was wrong in 61% of diagnoses for complex medical cases.
ChatGPT’s inability to correctly diagnose childhood illnesses is partly because the robot is unable to extrapolate or analyze from the partial symptoms reported by a child (often less accurate in identifying ailments than a adult patient). “Pediatric diagnoses require consideration of patient age alongside symptoms», Explains the study.
AI is also unable to make connections between certain pre-existing problems. For example, ChatGPT was not able to make a link between autism and vitamin deficiency, even though the latter are more common in people with autism spectrum disorder. In this specific case, the OpenAI robot made a mistake in diagnosing a rare autoimmune disease.
AI is making its mark in medicine
ChatGPT’s problems with medical diagnoses aren’t exactly surprising. The AI was fed by terabytes of more or less reliable data drawn from the Internet. Scientific literature is often less accessible than blogs and other freely available content, so the AI made do with what it had at hand. The robot was clearly not designed to perform this type of task.
However, the use of broad language models in the medical field is on the rise, as shown by certain experiments carried out in the Netherlands or certain programs set up by Silicon Valley billionaires. The solution could be found in the construction of an AI specialized in the diagnosis of medical cases and which would have access to all current scientific literature. But even there, the risk of bias is present. Not sure, therefore, that AI will replace our doctors right away.
Source : Jama Pediatrics via Ars Technica
8