Medizinische Universität Graz - Research portal

Go directly to the nagivation.
Go directly to the content.

Navigation/Search

Selected Publication:

SHR Neuro Cancer Cardio Lipid Metab Microb

Kugic, A; Schulz, S; Kreuzthaler, M.
Disambiguation of acronyms in clinical narratives with large language models.
J Am Med Inform Assoc. 2024; 31(9):2040-2046 Doi: 10.1093/jamia/ocae157 [OPEN ACCESS]
Web of Science PubMed FullText FullText_MUG

Leading authors Med Uni Graz: Kreuzthaler Markus Eduard; Kugic Amila
Co-authors Med Uni Graz: Schulz Stefan
Altmetrics:
Dimensions Citations:
Plum Analytics:
Scite (citation analytics):
Abstract:: OBJECTIVE: To assess the performance of large language models (LLMs) for zero-shot disambiguation of acronyms in clinical narratives. MATERIALS AND METHODS: Clinical narratives in English, German, and Portuguese were applied for testing the performance of four LLMs: GPT-3.5, GPT-4, Llama-2-7b-chat, and Llama-2-70b-chat. For English, the anonymized Clinical Abbreviation Sense Inventory (CASI, University of Minnesota) was used. For German and Portuguese, at least 500 text spans were processed. The output of LLM models, prompted with contextual information, was analyzed to compare their acronym disambiguation capability, grouped by document-level metadata, the source language, and the LLM. RESULTS: On CASI, GPT-3.5 achieved 0.91 in accuracy. GPT-4 outperformed GPT-3.5 across all datasets, reaching 0.98 in accuracy for CASI, 0.86 and 0.65 for two German datasets, and 0.88 for Portuguese. Llama models only reached 0.73 for CASI and failed severely for German and Portuguese. Across LLMs, performance decreased from English to German and Portuguese processing languages. There was no evidence that additional document-level metadata had a significant effect. CONCLUSION: For English clinical narratives, acronym resolution by GPT-4 can be recommended to improve readability of clinical text by patients and professionals. For German and Portuguese, better models are needed. Llama models, which are particularly interesting for processing sensitive content on premise, cannot yet be recommended for acronym resolution.
Find related publications in this database (using NLM MeSH Indexing): Abbreviations as Topic - administration & dosage; Natural Language Processing - administration & dosage; Humans - administration & dosage; Language - administration & dosage; Narration - administration & dosage; Electronic Health Records - administration & dosage
Find related publications in this database (Keywords): natural language processing; large language models; electronic health records; acronyms