Introduction: Audits are an integral part of effective modern healthcare. However, the collection of data for such audits is human resource intensive. Artificial intelligence (AI) in the form of large language models (LLM) may be able to assist with these audit processes.
Method: Discharge summaries from a retrospective cohort of stroke admissions from one month at a tertiary hospital were collected. A locally deployed LLM, LLaMA3, was then used to extract a variety of routine stroke audit data from free-text discharge summaries. These data were compared to the previously collected human audit data in the statewide registry. Manual case note review was undertaken in cases of discordance.
Results: Overall, there was a total of 144 data points that were extracted (9 data points for each of the 16 patients). The LLM was correct (or concordant) in 135/144 (93.8%) of individual datapoints. This performance included binary categorical, multiple-option categorical, datetime, and free-text extraction fields.
Conclusions: LLM may be able to assist with the efficient collection of stroke audit data. Such approaches may be pursued in other specialties. Future studies should seek to examine the most effective way to deploy such approaches in conjunction with human auditors and researchers.