UC San Francisco

Emergency departments nationwide are overcrowded and overtaxed, creating pressure on nurses and physicians to accurately triage patients at intake. The Emergency Severity Index triage process is resource-intensive and clinicians frequently face simultaneous urgent demands, making consistent prioritization difficult.

UCSF researchers evaluated ChatGPT-4 (accessed via UCSF's secure generative AI platform with broad privacy protections) on its ability to extract symptoms from clinical notes and determine urgency. The model was tested against 251,000 de-identified adult ED visit records and benchmarked against physician performance on a 500-pair sub-sample.

The LLM correctly identified which patient in a matched pair had the more serious condition 89% of the time across 10,000 pairs. In a 500-pair sub-sample evaluated by both the AI and a physician, the AI scored 88% accuracy versus 86% for the physician. Researchers note the model is not yet ready for clinical deployment without further validation.

• LLMs can match or slightly exceed physician-level accuracy on ED triage prioritization using only symptom text from clinical notes.
• Using real-world clinical data (251,000 visits) rather than simulated scenarios strengthens the validity of findings, but bias in training data remains an unresolved concern.
• Clinical deployment requires additional validation, bias mitigation, and prospective trials before responsible use.