EDBERT: Predicting Emergency Department Disposition Using a BERT-Based Architecture

Abstract

Objective: To develop and evaluate EDBERT (Emergency Department Bidirectional Encoder Representations from Transformers), a BERT-based architecture built around a multi-input fusion network that enriches free-text triage notes with the additional context of presenting complaints and patient age to predict Emergency Department (ED) disposition, supporting early decision-making and reduced ED length of stay (LOS). Methods: A retrospective cohort of 570,143 ED presentations to the Royal Melbourne Hospital, Australia, was used. At the core of EDBERT, a fusion network integrates three complementary input sources—triage notes, presenting complaints, and patient age—through learnable weights, with the two free-text inputs encoded by customised eight-layer BERT encoder stacks pre-trained on triage text with Masked Language Modelling (MLM). The resulting architecture is also lightweight, containing 66 million parameters, substantially fewer than the 110 million of BERT-BASE. Eighty percent of the dataset was used for training, and twenty percent for testing. Performance was benchmarked against BERT-Base configurations of varying depth. Results: EDBERT achieved an accuracy of 83.26%, a macro-averaged F1-score of 81.99%, and an area under the receiver operating characteristic curve of 0.91, outperforming all single-input BERT-Base variants, including a depth-matched eight-layer model (82.08%), indicating that the gain derives from the fusion of complementary inputs combined with domain-adaptive pre-training rather than from model capacity. Conclusions: A fusion-based BERT architecture that supplements the triage narrative with presenting complaint and age context predicts ED disposition more accurately than larger single-input BERT models, while requiring substantially less memory and computation, making it practical for deployment in hospitals with limited computing resources.

1. Introduction

Australia faces an ever-growing demand for emergency care and hospital admissions [1]. Emergency Departments (EDs) provide 24 h assistance for people with severe or life-threatening injuries or illnesses, and they are experiencing increased patient demand that is surpassing population growth [2]. In 2023–2024, an estimated 9 million patients presented to EDs in Australia [2]—24,658 patients per day—with approximately 31% subsequently admitted to an inpatient ward [3].

Emergency Department length of stay (LOS) is the duration between a patient’s arrival at the ED and their departure from it [4,5]. Emergency Departments are often overcrowded [6], leading to prolonged ED LOS, which correlates with increased morbidity and mortality [7]. Jones et al. reported a time-associated linear increase in all-cause 30-day mortality for patients who remain in the ED for more than 5 h after arrival, with one additional death occurring for every 82 patients delayed for over 6 to 8 h [8]. The early identification of patients who require admission could improve ED flow, optimise resource allocation, and improve patient outcomes.

The triage process serves as the first point of contact for patients in the ED [9]. Before assessment, demographic information and the reason for presentation are collected and documented in the triage note by a triage nurse [10]. The presenting problem, also referred to as the chief complaint [11], provides a description of the primary concern that prompted the patient to attend the ED. The triage note is a brief clinical record that typically includes the reason for presentation, relevant symptoms, medical history, and physiological observations used to determine clinical urgency. The quality and completeness of triage documentation can influence the efficiency and effectiveness of early ED assessment and care [12]. Nonetheless, ED triage notes are commonly written in a highly compressed style and may contain abbreviations, symbols, incomplete phrases, and locally used shorthand. The following de-identified example from the study dataset illustrates these characteristics:

Presenting problem: Left axillary pain and rectal bleeding.

Triage note:

SP: this PM, intermittent L) axilla pain, sharp localised, also c/o blood in stool few times this week, small, bright. PHx nil. GCS 15, RR 16, SpO₂ 100%, BP 126/72, T 36.3, pain 2/10--declines analgesia, BOC 0, nil COVID sx.

This short narrative includes clinically relevant information such as the timing, location, and characteristics of the pain; the additional report of rectal bleeding; previous medical history; physiological observations; pain severity; and the patient’s decision to decline analgesia. Although such documentation is efficient for rapid clinical recording, its abbreviated and locally variable form creates challenges for consistent interpretation and automated natural language processing (NLP).

With the recent significant advancements in NLP, the adoption of large language models (LLMs) in the medical context has gained considerable attention [13]. These models have shown significant efficacy in downstream medical NLP tasks when trained on large datasets of clinical text [14]. Large language models such as Bidirectional Encoder Representations from Transformers (BERT) [15] and its pre-trained variants, including ClinicalBERT [16] and BioBERT [17], have shown significant potential for medical applications. In 2021, Tahayori et al. reported the first Australian application of BERT to free-text ED triage notes, predicting patient disposition with an accuracy of 83% and an area under the receiver operating characteristic curve (AUROC) of 0.88 [18]. Ferri et al. developed a deep ensemble multitask model incorporating BERT to predict emergency jurisdiction from multimodal emergency medical call incidents, reporting an accuracy of 0.80 [19]. Building on these findings, Chang et al. demonstrated that machine learning models integrating both structured and unstructured triage data could outperform emergency physicians in disposition prediction [20], and Chen et al. introduced a deep neural network integrating physicians’ clinical narratives with vital signs that outperformed traditional logistic regression and early warning scores in terms of F1-score [21]. Boughorbel et al. further explored multimodal outcome prediction in the ED using a perceiver-based language model that combined textual and structured clinical information, highlighting the potential of multi-input representations for ED outcome prediction [22]. More recently, Chen et al. used BlueBERT, a biomedical domain-specific language model [23], and achieved an AUROC of 0.9014, demonstrating that domain-adapted pre-trained models provide considerable gains over general-purpose LLMs [24]. Task-specific architectural adaptations have also demonstrated benefits in other deep learning domains, including computer vision, where multi-scale feature extraction has been used to improve cross-view gait recognition [25].

However, two important gaps remain in this body of work: First, most existing BERT-based disposition models rely on the free-text triage note as a single input, without systematically integrating the readily available structured context that accompanies it at triage, such as the presenting complaint and patient age; indeed, Tahayori et al. anticipated that adding age as a distinct feature would further improve predictive performance [18]. Second, developing, fine-tuning, and maintaining LLMs for healthcare applications is resource-intensive [26]: BERT-BASE contains approximately 110 million parameters [27], demanding substantial computational power that makes it less practical for hospitals with limited computing resources.

In this study, we developed EDBERT (Emergency Department Bidirectional Encoder Representations from Transformers), a lightweight BERT-based framework organised around a multi-input fusion network. Rather than introducing an entirely new Transformer architecture, EDBERT adapts the BERT framework to the ED setting by combining triage notes with complementary information from the presenting complaint and patient age. These three inputs are integrated through learnable weights to form a unified representation for disposition prediction. The Transformer encoder is deliberately designed to be lightweight, with fewer layers, attention heads, and hidden dimensions than conventional BERT configurations, thereby reducing memory requirements and computational cost while retaining predictive performance. The main contributions of this study are therefore as follows: (i) a fusion-based BERT framework that integrates free-text triage notes with presenting complaints and patient age for ED disposition prediction; (ii) a lightweight, domain-tailored encoder architecture designed to reduce memory and computational overhead; and (iii) a comparative evaluation of EDBERT against other BERT-based architectures using real-world ED data.

2. Materials and Methods

2.1. Data

We used a retrospective dataset of electronic patient triage records from the ED at the Royal Melbourne Hospital, Australia. Approximately 5% of presentations had other disposition categories, such as transfer to another hospital, leaving the ED before completion of care, or death in the ED, which were excluded from this study. Following these exclusions, the final analytical dataset comprised 570,143 presentations, of which 211,810 (37.2%) resulted in admission and 358,333 (62.8%) had a “Home” or “discharge” disposition. Each record included the triage note and presenting complaint as free-text fields, and patient age as an integer value.

Eighty percent of the final analytical dataset was used for model training, while the remaining twenty percent was reserved as the test set to evaluate final predictive performance on unseen observations.

2.2. The EDBERT Model

EDBERT is organised around a fusion network that integrates three complementary input sources (triage notes, presenting complaints, and patient age) into a single representation for disposition prediction. The two free-text inputs are encoded by customised, lightweight BERT encoder stacks (66 million parameters in total). Below, we describe the encoder stack and the fusion network that forms the core of the architecture.

2.2.1. Customised BERT Encoder Stack

The Transformer encoder stack in EDBERT (Figure 1) is a modified version of the BERT architecture [15]. While BERT-BASE consists of 12 transformer layers, 12 attention heads, a hidden size of 768, and an intermediate size of 3072, EDBERT uses a more compact architecture comprising eight transformer layers, eight multi-head self-attention heads, a hidden size of 512, and a feedforward intermediate size of 2048. The model vocabulary was tailored to include additional domain-specific tokens and expanded to 40,000 tokens, while the maximum positional embedding length of 512 was retained.

Table 1. Architectural differences between BERT-BASE and EDBERT’s customised BERT encoder stack.

Component	BERT-BASE	EDBERT
Hidden Size (H)	768	512
Number of Transformer Layers (L)	12	8
Number of Multi-Head Self-Attention Heads (A)	12	8
Maximum Position Embeddings	512	512
Intermediate Size (I)	3072	2048

summarises these architectural modifications. The empirical analysis of encoder depth that informed the selection of the 8-layer architecture is provided in the Supplementary Materials (Table S1, Figures S1 and S2).

2.2.2. Fusion Network

The fusion network integrates three distinct input sources—triage notes, presenting complaints, and patient age—each providing complementary information for disposition prediction. Triage notes serve as the primary source because they contain rich, unstructured clinical information, including detailed symptom descriptions, severity indicators, contextual factors, and triage nurse observations. However, presenting complaints and age were also included because they provide concise and standardised information that may not always be stated explicitly or consistently within a triage note. Presenting complaints capture a patient’s primary reason for attending an ED, while age provides an important indicator of baseline clinical risk. The inclusion of presenting complaint and age was informed by both the prior literature and routine ED practice, as these variables provide complementary, routinely available information that may not be fully captured in the triage note. Previous studies have similarly shown that routinely collected triage features, including age, triage category, arrival mode, and vital signs, are among the strongest contributors to predictive performance [28].

Each input is first processed through a dedicated encoding pipeline. The triage notes and presenting complaints are tokenised and encoded separately using two identical customised encoder stacks (Section 2.2.1), pre-trained on domain-specific triage data, to produce contextual embeddings. During training, each embedding passes through a dropout layer to reduce overfitting. The age input is projected into the same embedding space as the BERT hidden layers via a fully connected linear transformation.

To combine the input sources, the model applies learnable scalar weights to each of the three embeddings, and the weighted embeddings are summed to form a single fused representation:

$$\mathrm{Fusion}=w_{\mathrm{text}}·E_{\mathrm{text}}+w_{\mathrm{pc}}·E_{\mathrm{pc}}+w_{\mathrm{age}}·E_{\mathrm{age}}$$

(1)

This fused embedding is then passed through the final classification layer, which predicts the ED disposition (Figure 2).

2.3. Model Training

EDBERT was trained in two stages: In the first stage, the two customised encoder stacks were pre-trained using the Masked Language Modelling (MLM) technique [15]: 15% of the input tokens were randomly selected for masking, and the model was trained to reconstruct their original values from the surrounding context. This enables the model to learn deep, context-sensitive representations and capture the complex semantic and syntactic relationships within triage texts. In the second stage, the full model was trained end-to-end by calculating the loss between the predicted and actual disposition labels using the cross-entropy loss function.

The weights applied to the three input sources were determined through grid-search optimisation using only the training partition, with the held-out 20% test set reserved exclusively for final model evaluation. The resulting weights were $w_{\mathrm{text}}=0.65$ for triage notes, $w_{\mathrm{pc}}=0.20$ for presenting complaints, and $w_{\mathrm{age}}=0.15$ for age. The test set was not used for fusion-weight optimisation, hyperparameter selection, or Masked Language Modelling (MLM) pre-training. MLM pre-training was likewise restricted to the training partition to minimise information leakage. The Adam optimiser [29] was used with a learning rate of $2\times {10}^{-5}$ and a weight decay of 0.01. The batch size was set to 64 for training and 32 for evaluation, and a dropout rate of 0.5 was applied during training to mitigate overfitting.

Table 2. Hyperparameter values used during model training.

Hyperparameter	Value
Learning Rate	$2\times {10}^{-5}$
Weight Decay	0.01
Train Batch Size	64
Test Batch Size	32
Dropout Percentage	0.5

summarises the hyperparameter configuration.

2.4. Evaluation

Predictive performance on the held-out test set was assessed using accuracy, macro-averaged precision, recall, F1-score, the receiver operating characteristic (ROC) curve, and the area under the ROC curve (AUC). EDBERT was benchmarked against BERT-Base configurations with 2 to 12 transformer layers (in increments of two). Two ablation analyses were conducted to justify the final design: (i) the effect of encoder depth on predictive performance, parameter count, and training time; and (ii) the contribution of MLM pre-training. The complete results of these analyses are reported in the Supplementary Materials. Finally, we compared EDBERT’s performance against several pre-trained BERT variants, including BERT-Base [15], MedBERT [30], DistilBERT [31], TinyBERT [32], and ALBERT [33], on the held-out set.

3. Results

The triage notes had a maximum length of 101 words and an average length of 391 characters, including spaces.

EDBERT achieved the highest performance of all evaluated models, with an accuracy of 83.26% and macro-averaged precision, recall, and F1-score of 82.14%, 81.85%, and 81.99%, respectively, outperforming the standard 12-layer BERT-Base model across all metrics (

Table 3. Performance comparison of EDBERT and the standard BERT-Base model, evaluated using accuracy and macro-averaged precision, recall, and F1-score percentages. Values in parentheses represent the corresponding 95% confidence intervals (CIs). Confidence intervals were obtained by parametric bootstrap over the class-conditional counts. BERT-Base was evaluated on $N=\mathrm{114,029}$ and EDBERT on $N=\mathrm{118,957}$.

Model	Accuracy	Precision	Recall	F1-Score
BERT-Base (12 layers)	82.19	81.22	80.16	80.62
	(81.97–82.41)	(80.98–81.46)	(79.91–80.41)	(80.38–80.86)
EDBERT (8 layers)	83.26	82.14	81.85	81.99
	(83.05–83.47)	(81.92–82.36)	(81.63–82.07)	(81.77–82.21)

). Notably, EDBERT also exceeded the depth-matched eight-layer BERT-Base model trained on triage notes alone (83.26% vs. 82.08% accuracy; Supplementary Table S1), indicating that the improvement stems primarily from the fusion of complementary inputs and domain-adaptive pre-training rather than from model capacity. The ROC curve of EDBERT on the test set is shown in Figure 3, with an AUC of 0.91, indicating strong discriminative ability between the two disposition classes.

The supporting analyses that informed the final architecture are summarised below, with the full details provided in the Supplementary Materials. First, varying the encoder depth of BERT-Base from 2 to 12 layers produced only marginal accuracy differences (81.47% to 82.23%; Supplementary Table S1 and Figure S1), while parameter count and training time per epoch increased substantially with depth (Supplementary Figure S2). Deeper models did not yield performance gains, which may be attributed to the low lexical variability of the triage notes, potentially leading to overfitting in larger models. The eight-layer configuration offered the most favourable balance between predictive performance and computational efficiency, and it was therefore adopted for EDBERT. Second, MLM pre-training on triage notes improved EDBERT’s accuracy from 82.18% to 83.26% (Supplementary Table S2 and Figure S4), confirming the value of domain-adaptive pre-training.

Table 4. Performance comparison of different BERT variants, evaluated using accuracy and macro-averaged precision, recall, and F1-score. Values in parentheses represent estimated 95% confidence intervals (CIs).

Model	Accuracy	Precision	Recall	F1-Score
BERT-Base	81.70(81.48–81.92)	80.73(80.51–80.95)	79.68(79.45–79.91)	80.13(79.90–80.36)
MedBERT	82.19(81.97–82.41)	81.22(81.00–81.44)	80.16(79.93–80.39)	80.16(79.93–80.39)
DistilBERT	81.60(81.38–81.82)	79.75(79.52–79.98)	79.58(79.35–79.81)	79.13(78.90–79.36)
TinyBERT	80.90(80.68–81.12)	79.75(79.52–79.98)	78.90(78.67–79.13)	79.13(78.90–79.36)
ALBERT	81.40(81.18–81.62)	79.75(79.52–79.98)	79.39(79.16–79.62)	79.13(78.90–79.36)

shows the performance comparison of pre-trained BERT variants, including BERT-Base, MedBERT, DistilBERT, TinyBERT, and ALBERT. All variants performed within a comparable band of 80.9% to 82.1% accuracy, with macro-averaged precision, recall, and F1-scores clustered between 81% and 82%. MedBERT, having already been pre-trained on the medical domain, achieved the highest accuracy (82.19%), marginally ahead of BERT-Base (81.7%). The compressed architectures—DistilBERT (81.6%), ALBERT (81.4%), and TinyBERT (80.9%)—performed comparably despite having considerably fewer parameters. Importantly, every BERT variant evaluated on triage notes alone remained below the 83.26% accuracy achieved by EDBERT.

4. Discussion

In this study, we designed and evaluated a domain-tailored BERT architecture whose central component is a fusion network that enriches the free-text triage note with the additional context of the presenting complaint and patient age. EDBERT predicts ED disposition with an accuracy of 83.26% and has an AUC of 0.91, while using 66 million parameters—approximately 40% fewer than BERT-BASE. These results compare favourably with previous work. Tahayori et al. reported an accuracy of 83% and an AUROC of 0.88 using a full-size BERT model on triage notes alone [18], and Chen et al. achieved an AUROC of 0.9014 using BlueBERT, a domain-specific model pre-trained on large biomedical corpora [24]. EDBERT achieved comparable or better discriminative performance with a substantially smaller architecture, without relying on external biomedical pre-training corpora.

The improved performance of EDBERT relative to larger, general-purpose BERT configurations can be explained by several factors. First, and most importantly, the fusion network supplements the triage narrative with the complementary context of the presenting complaint and patient age. A depth-matched eight-layer BERT-Base model trained on triage notes alone reached 82.08% accuracy, whereas EDBERT reached 83.26%, indicating that enriching the input representation, rather than adding model capacity, drives the gain. The ablation results suggest this enrichment acts in combination with domain-adaptive pre-training, as the fusion-based model without MLM pre-training reached 82.18% (Supplementary Table S2). The grid-searched fusion weights confirm the dominant contribution of the triage narrative ($w_{\mathrm{text}}=0.65$), consistent with prior evidence that free-text triage narratives carry the greatest predictive signal for disposition [10,20], while the presenting complaint ($w_{\mathrm{pc}}=0.20$) and age ($w_{\mathrm{age}}=0.15$) contribute additional discriminative context that the note alone does not capture. Second, domain-adaptive MLM pre-training on triage text allowed the model to learn the specialised terminology, abbreviations, and contextual patterns characteristic of ED documentation, which a general-purpose model has less opportunity to capture; removing MLM pre-training reduced accuracy by more than one percentage point. Third, triage notes are short and telegraphic, with a maximum length of 101 words in our dataset, and exhibit very low lexical variability (1.01% unique word ratio); the representational capacity of a 12-layer, 110-million-parameter model is therefore excessive and may promote overfitting, whereas EDBERT’s reduced encoder depth is matched to the linguistic complexity of the input.

The lightweight design of EDBERT has direct practical implications. Full-size LLMs are resource-intensive to fine-tune and maintain [26], which is a barrier for hospitals with limited computing infrastructure. By reducing memory usage and training time while preserving predictive performance, EDBERT is more feasible for local training and deployment within existing hospital systems. Early identification of patients likely to require admission could allow bed allocation to begin at the time of presentation, streamline care, reduce the cognitive load on ED clinicians, and ultimately contribute to reduced ED LOS [7,8].

An analysis of misclassified cases identified two recurring patterns, illustrated in

Table 5. Representative EDBERT misclassification examples.

Age	Presenting Complaint	Triage Note	Predicted Disposition (Probability Score)	Actual Disposition
63	Gastrointestinal	Due for ascitic tap on 7/10. Awoke this AM with worsening SOB and severe abdominal distension. Last tap on 20/09 drained 5L. PMHx: liver CA, CKD, HTN, T2DM, IHD.	Admission (98.3%)	Discharge
20	Other Medical	Right hand burned on hot oil; partial-thickness/superficial dermal burn with blistering to fingers. First aid applied. PHx: nil.	Discharge (98.1%)	Admission

.

False positives commonly involved older patients with substantial comorbidity and high-acuity symptom language who were ultimately discharged through planned or established care pathways; for example, 110 false-positive cases contained references to liver disease or cirrhosis. In contrast, false negatives often involved younger patients with short, apparently low-acuity presentations whose need for specialist admission was not evident at triage; 189 false-negative cases involved burns. These findings indicate that final disposition may depend on downstream specialist assessment, investigations, or care pathways not captured in the triage information. This supports future investigation of additional clinical context and uncertainty-aware prediction mechanisms.

More recent research has also emphasised the importance of uncertainty-aware prediction and effective collaboration between clinicians and predictive models. Abdulai et al. proposed a conformal prediction framework that enables a model to abstain when a case is ambiguous, thereby reducing the risk of overconfident misclassification and supporting safer clinical use [34]. Similarly, Barak-Corren et al. directly compared clinician and algorithmic predictions in a prospective observational study. They showed that physicians correctly predicted 57% of admissions with a specificity of 88%, whereas a random forest model using electronic health record (EHR) data from the first hour of the ED visit achieved a sensitivity of 69% and a specificity of 90%. Combining clinician judgement with algorithmic predictions produced the highest positive predictive value of 86% [35]. These findings indicate that predictive performance alone is insufficient for safe deployment in a high-risk clinical environment. Future work should incorporate uncertainty estimation or abstention mechanisms, allowing EDBERT to defer ambiguous cases for clinician review and evaluate the model within a human-in-the-loop framework to determine whether its predictions complement, rather than replace, clinician judgement. Prospective shadow-mode evaluation and subsequent clinician–model collaboration studies would help establish the safety, reliability, and practical value of EDBERT in real-world ED workflows.

Limitations

This study has several limitations. First, it used data from a single site. Because triage documentation style varies considerably between hospitals, the model may need to be re-trained on local data before implementation elsewhere, and a multi-centre study is required to evaluate the generalisability of the proposed architecture. Second, presentations with dispositions other than “Admit” and “Home” (approximately 5% of the cohort) were excluded, so the model’s behaviour on these less common outcomes remains untested.

Third, EDBERT currently produces point predictions without uncertainty estimates or an abstention mechanism. Recent work has emphasised the importance of uncertainty-aware prediction for safe clinical use: Abdulai et al. proposed a conformal prediction framework that enables a model to abstain when a case is ambiguous, reducing the risk of overconfident misclassification [34]. Incorporating such mechanisms would allow EDBERT to defer ambiguous cases for clinician review rather than producing potentially unreliable predictions.

Finally, EDBERT was evaluated retrospectively, without clinician interaction, so determining whether its predictions complement rather than replace clinician judgement remains untested. Barak-Corren et al. found that combining clinician judgement with algorithmic predictions produced the highest positive predictive value (86%) [35]. These findings suggest that predictive performance alone is insufficient for safe deployment in a high-risk clinical environment. Prospective shadow-mode evaluation and subsequent clinician–model collaboration studies within a human-in-the-loop framework would help establish the safety, reliability, and practical value of EDBERT in real-world ED workflows.

5. Conclusions

This study introduces EDBERT, a specialised Transformer-based architecture for ED disposition prediction whose central component is a fusion network that integrates the free-text triage note with the complementary context of the presenting complaint and patient age through learnable weights. Supplementing the triage narrative with this additional context, together with domain-adaptive MLM pre-training, enabled EDBERT to consistently outperform larger single-input BERT baselines. By further tailoring the encoder stacks to the specific characteristics of ED text, including reduced Transformer layers, attention heads, and hidden dimensions, EDBERT delivers this performance with fewer parameters and reduced training time. Overall, EDBERT offers a scalable, high-performing, and resource-efficient solution for predicting patient disposition in emergency settings, supporting timely and informed clinical decision-making. Future work will explore the generalisability of EDBERT across multiple hospital sites, the incorporation of additional patient features, uncertainty-aware prediction, and real-time deployment in ED triage workflows.