Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

Abstract

We evaluated the performance of a large language model called ChatGPT on the United States Medical Licensing Exam (USMLE), which consists of three exams: Step 1, Step 2CK, and Step 3. ChatGPT performed at or near the passing threshold for all three exams without any specialized training or reinforcement. Additionally, ChatGPT demonstrated a high level of concordance and insight in its explanations. These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making.

Keywords

ConcordanceUnited States Medical Licensing ExaminationMedical educationComputer scienceLicensureMedical schoolArtificial intelligenceMedicineInternal medicine

Affiliated Institutions

Related Publications

How Does ChatGPT Perform on the United States Medical Licensing Examination (USMLE)? The Implications of Large Language Models for Medical Education and Knowledge Assessment

Aidan Gilson , Conrad Safranek , Thomas Huang +4 more

Background Chat Generative Pre-trained Transformer (ChatGPT) is a 175-billion-parameter natural language processing model that can generate conversation-style responses to user ...

2023 JMIR Medical Education 1794 citations

Large language models encode clinical knowledge

Karan Singhal , Shekoofeh Azizi , Tao Tu +29 more

Abstract Large language models (LLMs) have demonstrated impressive capabilities, but the bar for clinical applications is high. Attempts to assess the clinical knowledge of mode...

2023 Nature 2248 citations

A Survey on Evaluation of Large Language Models

Yupeng Chang , Xu Wang , Jindong Wang +13 more

Large language models (LLMs) are gaining increasing popularity in both academia and industry, owing to their unprecedented performance in various applications. As LLMs continue ...

2024 ACM Transactions on Intelligent Syste... 1741 citations

En Route to Ethical Recommendations for Gene Transfer Clinical Trials

Nancy M. P. King , Odile Cohen‐Haguenauer

In Geneva, Switzerland, on 2–3 April 2007, Clinigene-NoE (the European Network for the Advancement of Clinical Gene Transfer and Therapy) and Consert (program on Concerted Safet...

2008 Molecular Therapy 35 citations

Publication Info

Year: 2023
Type: article
Volume: 2
Issue: 2
Pages: e0000198-e0000198
Citations: 3039
Access: Closed

External Links

View on DOI.org

Social Impact

Altmetric

Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models

PlumX Metrics

Social media, news, blog, policy document mentions

Citation Metrics

3039

OpenAlex

Cite This

APA Style

                            
                                
                                    Tiffany H. Kung, 
                                
                                    Morgan Cheatham, 
                                
                                    Arielle Medenilla
                                
                                et al.
                            
                            (2023). 
                            Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. 
                            PLOS Digital Health
                            , 2
                            (2)
                            , e0000198-e0000198.
                            https://doi.org/10.1371/journal.pdig.0000198
                        

Identifiers

DOI: 10.1371/journal.pdig.0000198