Please use this identifier to cite or link to this item:
http://hdl.handle.net/1942/43057
Title: | Can large language models pass official high-grade exams of the European Society of Neuroradiology courses? A direct comparison between OpenAI chatGPT 3.5, OpenAI GPT4 and Google Bard | Authors: | D'Anna, Gennaro VAN CAUTER, Sofie Thurnher, Majda Van Goethem, Johan Haller, Sven |
Issue Date: | 2024 | Publisher: | SPRINGER | Source: | NEURORADIOLOGY, | Status: | Early view | Abstract: | We compared different LLMs, notably chatGPT, GPT4, and Google Bard and we tested whether their performance differs in subspeciality domains, in executing examinations from four different courses of the European Society of Neuroradiology (ESNR) notably anatomy/embryology, neuro-oncology, head and neck and pediatrics. Written exams of ESNR were used as input data, related to anatomy/embryology (30 questions), neuro-oncology (50 questions), head and neck (50 questions), and pediatrics (50 questions). All exams together, and each exam separately were introduced to the three LLMs: chatGPT 3.5, GPT4, and Google Bard. Statistical analyses included a group-wise Friedman test followed by a pair-wise Wilcoxon test with multiple comparison corrections. Overall, there was a significant difference between the 3 LLMs (p < 0.0001), with GPT4 having the highest accuracy (70%), followed by chatGPT 3.5 (54%) and Google Bard (36%). The pair-wise comparison showed significant differences between chatGPT vs GPT 4 (p < 0.0001), chatGPT vs Bard (p < 0. 0023), and GPT4 vs Bard (p < 0.0001). Analyses per subspecialty showed the highest difference between the best LLM (GPT4, 70%) versus the worst LLM (Google Bard, 24%) in the head and neck exam, while the difference was least pronounced in neuro-oncology (GPT4, 62% vs Google Bard, 48%). We observed significant differences in the performance of the three different LLMs in the running of official exams organized by ESNR. Overall GPT 4 performed best, and Google Bard performed worst. This difference varied depending on subspeciality and was most pronounced in head and neck subspeciality. | Notes: | D'Anna, G (corresponding author), ASST Ovest Milanese, Neuroimaging Unit, Legnano, Milan, Italy. gennaro.danna@gmail.com |
Keywords: | AI;LLM;chatGPT;Neuroradiology;GPT4 | Document URI: | http://hdl.handle.net/1942/43057 | ISSN: | 0028-3940 | e-ISSN: | 1432-1920 | DOI: | 10.1007/s00234-024-03371-6 | ISI #: | 001216135700001 | Rights: | The Author(s), under exclusive licence to Springer-Verlag GmbH Germany, part of Springer Nature 2024 | Category: | A1 | Type: | Journal Contribution |
Appears in Collections: | Research publications |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Can large language models pass official high-grade exams of the European Society .pdf Restricted Access | Early view | 950.95 kB | Adobe PDF | View/Open Request a copy |
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.