Please use this identifier to cite or link to this item: http://hdl.handle.net/1942/43057
Title: Can large language models pass official high-grade exams of the European Society of Neuroradiology courses? A direct comparison between OpenAI chatGPT 3.5, OpenAI GPT4 and Google Bard
Authors: D'Anna, Gennaro
VAN CAUTER, Sofie 
Thurnher, Majda
Van Goethem, Johan
Haller, Sven
Issue Date: 2024
Publisher: SPRINGER
Source: NEURORADIOLOGY,
Status: Early view
Abstract: We compared different LLMs, notably chatGPT, GPT4, and Google Bard and we tested whether their performance differs in subspeciality domains, in executing examinations from four different courses of the European Society of Neuroradiology (ESNR) notably anatomy/embryology, neuro-oncology, head and neck and pediatrics. Written exams of ESNR were used as input data, related to anatomy/embryology (30 questions), neuro-oncology (50 questions), head and neck (50 questions), and pediatrics (50 questions). All exams together, and each exam separately were introduced to the three LLMs: chatGPT 3.5, GPT4, and Google Bard. Statistical analyses included a group-wise Friedman test followed by a pair-wise Wilcoxon test with multiple comparison corrections. Overall, there was a significant difference between the 3 LLMs (p < 0.0001), with GPT4 having the highest accuracy (70%), followed by chatGPT 3.5 (54%) and Google Bard (36%). The pair-wise comparison showed significant differences between chatGPT vs GPT 4 (p < 0.0001), chatGPT vs Bard (p < 0. 0023), and GPT4 vs Bard (p < 0.0001). Analyses per subspecialty showed the highest difference between the best LLM (GPT4, 70%) versus the worst LLM (Google Bard, 24%) in the head and neck exam, while the difference was least pronounced in neuro-oncology (GPT4, 62% vs Google Bard, 48%). We observed significant differences in the performance of the three different LLMs in the running of official exams organized by ESNR. Overall GPT 4 performed best, and Google Bard performed worst. This difference varied depending on subspeciality and was most pronounced in head and neck subspeciality.
Notes: D'Anna, G (corresponding author), ASST Ovest Milanese, Neuroimaging Unit, Legnano, Milan, Italy.
gennaro.danna@gmail.com
Keywords: AI;LLM;chatGPT;Neuroradiology;GPT4
Document URI: http://hdl.handle.net/1942/43057
ISSN: 0028-3940
e-ISSN: 1432-1920
DOI: 10.1007/s00234-024-03371-6
ISI #: 001216135700001
Rights: The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024
Category: A1
Type: Journal Contribution
Appears in Collections:Research publications

Files in This Item:
File Description SizeFormat 
Can large language models pass official high-grade exams of the European Society .pdf
  Restricted Access
Early view950.95 kBAdobe PDFView/Open    Request a copy
Show full item record

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.