Significant Errors With ChatGPT for Migraine Drug Treatment

SAN DIEGO — ChatGPT, an artificial intelligence (AI) tool that can provide headache information, makes significant errors regarding migraine drug treatments but does better when it comes to nonpharmacologic therapies, an assessment by headache specialists showed.

The findings suggest doctors should counsel patients not to rely solely on ChatGPT for accurate information on migraine treatments, at least not in its current iteration, investigator Robert Bonakdar, MD, director of Pain Management, Scripps Center for Integrative Medicine, San Diego, told Medscape Medical News.

The good news, though, is that more accurate AI alternatives may be on the way, Bonakdar added.

The findings were presented on June 13 at the American Headache Society (AHS) Annual Meeting 2024.

Can I Trust AI?

Clinicians have mixed feelings about AI, including Large Language Models like ChatGPT. They are intrigued by and dabbling with it but are aware of its limitations and outdated information, said Bonakdar, who has treated migraines for over two decades and has experience using ChatGPT.

Although there are other AI tools, ChatGPT is one of the most extensive and is the one headache specialists are most likely to use, said Bonakdar. Studies show the tool is useful in some areas of medicine, like orthopedics, but less so in others, such as neurosurgery, he added.

“The issue is, can I trust AI for migraine?” he asked.

For this study, investigators asked ChatGPT 14 common questions related to treatment options for migraine. The questions focused on the efficacy and safety of conventional treatments (preventive medications, acute treatments, non-oral acute therapies such as patches and sprays, and procedures involving nerve blocks) and nonpharmacologic treatments (diet/supplements, exercise, stimulation devices, physical therapies such as massage, self-care, and behavior therapies).

Investigators then presented the AI responses to four headache specialists: A neurologist, a neurology physician assistant, a United Council for Neurologic Subspecialties certified physician, and a headache researcher.

These specialists rated the ChatGPT answers on a Likert scale from 1 (completely disagree) to 5 (completely agree) in completeness, accuracy, and helpfulness.

Participants gave higher overall scores for nonpharmaceutical treatments. Their highest rating — for behavioral therapies — was 4.75, while their lowest rating — for diet/supplements — was 4.

“ChatGPT did decently well in nonpharmacologic therapy advice, but it wasn’t perfect,” commented Bonakdar.

For pharmacologic treatments, the ratings were lower. The highest was 3.95 (for procedures), and the lowest was 3.5 (for preventative medications).

The AI errors here “were much more pronounced” than for nondrug interventions, said Bonakdar.

Inappropriate Options

The raters noted ChatGPT also recommended some inappropriate treatment options, such as opioids and migraine surgery. 

They also noticed the tool inaccurately claimed Cefaly requires a prescription when external trigeminal nerve stimulation is available over the counter. ChatGPT also failed to note potential health risks for migraine treatments during pregnancy, including feverfew, a plant containing compounds believed to help treat or prevent inflammatory conditions.

The headache specialists also pointed out the AI tool didn’t mention approved medications such as Nurtec and Qulipta, both calcitonin gene-related peptide receptor blockers orinclude important safety information such as side effects of valproic acid, the anticonvulsant Topamax, and beta-blockers.

ChatGPT is “not a bad place to start” to get general migraine-related information, but because this information isn’t personalized, it’s important for patients to discuss treatments with their clinician, said Bonakdar.

“ChatGPT doesn’t know you. It doesn’t know your medications, and it doesn’t know your contraindications or your allergies,” he added.

Commenting for Medscape Medical News, Nina Riggins, MD, PhD, president, Brain Performance Center and Research Institute, and director, Headache Center at The Neuron Clinic, San Diego, said that while the study offered important information, she would have liked more assessment on the diagnostic capabilities of ChatGPT.

Riggins noted the importance of understanding the capabilities and limitations of AI in medicine. “We’re aware of AI ‘hallucinations,’ when it doesn’t provide correct information, but we also understand that the use of technology can be extremely important in medicine, even lifesaving.”

Many inaccuracies related to ChatGPT can be attributed to outdated information, she added. And understanding how this AI tool uses such information could “provide specific pathways for improvement of this technology,” said Riggins.

In future, AI might help triage patients with headache disorders, assist in reading imaging reports, and improve overall access to care, she said. And by decreasing the amount of paperwork doctors deal with, AI could free up doctors to spend more time with their patients, she added.

In the meantime, Riggins agreed clinicians should advise patients not to get advice on efficacy and safety of treatments from ChatGPT.

A More Focused AI Tool

Bonakdar also presented findings from a second study on a more focused AI assistant researchers created with Wellkasa, a company investigating AI technology and evidence-based lifestyle medicine. The group trained the AI model, know an My AI Assistant (Maia), to consistently provide the most accurate possible answers.

“For example, if Maia answered that Cefaly needed a prescription, which is false, she received a low score, and if she answered correctly that it didn’t, she would receive a high score,” said Bonakdar.

The model was trained not only in simple true/false areas but also in more subtle lifestyle areas such as stress management.

Researchers asked 23 patients with migraine, all but one woman, to try out Maia for a week. Patients were to ask Maia questions related to migraine, and they could pose questions in a chat room or by phone.

The study found knowledge and confidence scores on migraine prevention, treatment, and lifestyle improved across 11 of 12 items (91.7%).

“These findings are key because we know education on its own can improve migraine status and medication benefits,” said Bonakdar.

While education can be time-consuming and not always personalized, “with Maia, we found migraine education can be simplified because learning about migraines becomes a very personal learning journey through simple and natural conversations,” he said.

The study also showed statistically significant improvements on 4 of 12 items pertaining to medication management, therapy side effects, lifestyle therapies, and multimodal approaches to migraine management.

Participants with less migraine-related disability were significantly more satisfied with Maia than those with more migraine-related disability (P

In providing feedback, study participants said the tool answers questions they don’t want to bother their doctors with and avoids the need for multiple searches about migraine medications. However, some said they wished Maia could address queries related to comorbidities and track their migraines.

While AI is viewed by some as possibly replacing clinicians, it’s more likely patients will use information they glean from AI to engage their doctors, especially in areas such as lifestyle interventions that may be rather nuanced, said Bonakdar.

Riggins said she was particularly impressed with the useful advice provided by Maia on lifestyle modifications such as sleep and diet.

All AI tools, including ChatGPT and Maia, should eventually make migraine education safer for both patients and doctors, said Riggins.

Bonakdar and Riggins reported no relevant conflicts of interest.