5.1. Model Performance Analysis
In the NER task, hyperparameter settings significantly impact the model performance. The learning rate, batch size, and epoch are selected for sensitivity analysis to comprehensively evaluate the model’s robustness and generalizability. This is to explore the fluctuation of model performance under different configurations to provide theoretical support and practical reference for subsequent model optimization.
Learning rate significantly affects how quickly and smoothly the model converges. We therefore tested five values (1 × 10
−5 to 5 × 10
−5) while keeping all other hyperparameters fixed (
Figure 5). The learning rate of 2 × 10
−5 achieved the best performance, with the highest F1 score of 0.8832, and exhibited stable convergence. A larger learning rate may lead to unstable optimization, whereas a smaller learning rate typically results in slower convergence and may require more training steps. Therefore, we use 2 × 10
−5 as the learning rate in subsequent experiments.
Figure 6a indicates that adopting batch size 8 achieves a higher F1 score, reaching 0.8832. When the batch size increases to 64, the F1 score drops considerably. This may be because larger batch sizes reduce the frequency and stochasticity of gradient updates, which can weaken the model’s adaptability to fine-grained entity boundaries in a relatively small domain-specific corpus. As illustrated in
Figure 6b, the F1 improves with more training epochs; however, when epochs exceed 90, the performance of a few batch-size configurations starts to decrease, suggesting the onset of overfitting. This indicates that excessive training may cause the model to further fit domain-specific expressions and annotation patterns in the training data while reducing its generalization ability on unseen samples. Considering both performance and training efficiency, we adopt 90 epochs in this study.
In the NER task, the partitioning strategy of the dataset has a considerable impact on the training effect. This study adopts three ratio settings to evaluate the impact of different portioning schemes, and
Figure 7 summarizes the comparative results. The results show that adopting the 8:1:1 split technique enables the model to achieve the highest F1 score. This ratio provides sufficient training data while reserving appropriate subsets for validation and testing. Both alternative partitioning schemes (7:1.5:1.5 and 6:2:2) result in lower performance than 8:1:1, possibly because they allocate fewer samples for training. Overall, the 8:1:1 data partitioning method performs well for this task and is recommended for subsequent experiments.
5.2. Ablation Experiment and Model Effectiveness Analysis
5.2.1. Comparison of the Adaptability of Various BERT Models
This study compares six mainstream models, including the standard BERT, RoBERTa, MacBERT, ChineseBERT, mBERT, and TinyBERT, to assess the efficacy of BERT and its variants in the NER task of BFPDCs.
Figure 8a depicts the experimental findings for each model. The results indicate that RoBERTa delivers the best performance, achieving an F1 score of 0.8832, which is higher than that of the standard BERT baseline. This result suggests that RoBERTa’s pretraining strategy and representations may be more effective for this task. Furthermore, MacBERT performs better (F1 score = 0.8736), ranking second only to RoBERTa. In comparison, ChineseBERT attains an F1 score of 0.8394, indicating that its performance is relatively lower on this specialized dataset. Although the lightweight TinyBERT model yields a lower F1 score (0.8501), its smaller model size and faster inference potential make it an attractive option for deployment in resource-constrained scenarios.
To further understand how RoBERTa interacts with other modeling components, this study proposes experiments that eliminate specific structures and examine how well the combination of RoBERTa-BiLSTM-CRF, RoBERTa, and BiLSTM-CRF performs. As presented in
Figure 8b, the RoBERTa-BiLSTM-CRF arrangement is the most effective at recognizing entity boundaries and interpreting meanings.
The F1 score for the structure with only RoBERTa is 0.8618, indicating proficient contextual semantic modeling capabilities; nevertheless, it lacks explicit modeling of label dependencies, resulting in a relatively ambiguous recognition entity boundary. Similarly, while BiLSTM-CRF achieves a recall of 0.8719, the absence of pretraining semantic support may result in a precision of 0.8568, yielding an F1 score of 0.8696, thereby highlighting the important role of pretrained models in enhancing entity extraction performance.
In summary, the synergistic combination of RoBERTa and BiLSTM-CRF remarkably improves the ability to model long-distance dependencies and professional terminology structures in BFPDCs texts.
5.2.2. Analysis of the Role of BiLSTM with Parameter Tuning
BiLSTM is introduced as a downstream module to further enhance the sequence modeling capability of the proposed model. Built on top of contextual embeddings, BiLSTM can better capture word order and bidirectional dependency relationships, which is suitable for sequence labeling in building fire protection codes with complex structures. To validate the performance of BiLSTM hyperparameters, this study designs parameter ablation experiments, including num_layers, hidden_size, and dropout, and
Table 6 reports the results.
Experiments show that in a single-layer BiLSTM, when the hidden_size is 256 and the dropout is 0.3, the model achieves superior performance, suggesting that an appropriate model capacity and regularization are helpful in this setting. The two-layer BiLSTM structure achieves the highest F1 score in this group of experiments with a dropout of 0.3 and a hidden_size of 256, indicating that a deeper BiLSTM can be beneficial, but its performance depends on suitable regularization and model size. When the number of layers increases but the dropout is too small or the hidden size is too large, the model exhibits performance fluctuations in some experimental groups, reflecting the sensitivity of the BiLSTM structure to hyperparameters in complex tasks.
Figure 9a shows the model performance on the NER task under different BiLSTM parameter settings. The results indicate that the model performance generally improves as num_layers and hidden_size increase within the tested range. When dropout is applied, the performance is more stable across settings. The best performance is achieved with num_layers = 2, dropout = 0.3, and hidden_size = 256, reaching an F1 score of 0.8832.
Figure 9b further analyzes the contribution of the BiLSTM component via structural ablation. Compared with the baseline model (RoBERTa-CRF), which only includes RoBERTa and CRF, the RoBERTa-BiLSTM-CRF model performs better in Precision (0.8834), Recall (0.8831), and F1 score (0.8832). Although the overall performance of the RoBERTa-CRF model remains strong, it lacks additional order-sensitive sequential modeling in this setting. In particular, when complex contextual linkages exist between entities, their recall is lower, at 0.8738.
Comprehensive results indicate that introducing the BiLSTM layer enhances context modeling capability, particularly for texts containing many professional terms and long sentence dependencies related to BFPDCs. Its dual-channel modeling mechanism of forward and backward semantics can make entity predictions both more accurate and more stable.
5.2.3. Comparison with Traditional NER Baselines
To further evaluate the effectiveness of the proposed model, traditional NER baselines were introduced for comparison, including a rule-based method, HMM, and independent CRF. Unlike pretrained language models such as BERT and RoBERTa, HMM and CRF are traditional statistical sequence labeling models trained directly on the annotated corpus. The rule-based method was constructed using domain dictionaries, trigger expressions, and regular expressions. Specifically, domain dictionaries were developed for common building types, space types, fire protection facilities, building materials, and regulatory expressions, while regular expressions were used to identify numerical requirements and clause references. All baseline models used the same dataset split and evaluation metrics as the proposed model.
As shown in
Table 7, the proposed RoBERTa-BiLSTM-CRF model achieves the best overall performance. The rule-based method achieves relatively high recall because it can identify entities with stable terminology and explicit patterns, but its precision is limited when entity boundaries are long or context-dependent. The HMM baseline obtains comparable but slightly lower overall performance, indicating that character-level emission and transition probabilities are insufficient for modeling complex regulatory semantics. Independent CRF performs better than rule-based and HMM baselines by using handcrafted contextual features and structured sequence decoding, but it still lags behind the proposed model. In contrast, RoBERTa-BiLSTM-CRF benefits from pretrained contextual representations, bidirectional sequence modeling, and CRF-based label transition constraints, enabling more accurate recognition of domain-specific entities in building fire protection design codes.
5.3. Recognition Challenges for Fine-Grained Entities
Although the model demonstrates high performance in the overall NER task, which indicates that it has excellent generalization ability in dealing with most of the entity categories, it still faces certain challenges dealing with fine-grained entity categories, particularly in the context of fuzzy semantic boundaries and functional overlap where the extraction accuracy is degraded.
Table 8 indicates that for entity types “FPR” and “FR,” the F1 scores are 0.8037 and 0.8106, respectively, which are much lower than those for clearer categories such as “SC” (F1 = 0.9811) or “NR” (F1 = 0.9554). This result suggests that when dealing with entity categories that have unclear definitions, the proposed model struggles to understand meaning and pick up important context.
According to the category-level results in
Table 8, BM, FPR, and FR are the most difficult entity categories to recognize, with F1 scores of 0.8025, 0.8037, and 0.8106, respectively. These categories are more challenging because their semantic boundaries are relatively ambiguous: BM entities are often combined with fire-resistance properties and numerical constraints, while FPR and FR frequently share similar normative expressions such as “shall be provided” and “shall meet”. In addition, long clauses containing multiple parallel spaces, facilities, or requirements may lead to omission or boundary errors. These findings suggest that future work should further refine annotation rules for overlapping categories and incorporate domain knowledge to improve fine-grained entity recognition.
Table 9 presents representative examples of entity recognition challenges in building fire protection design codes. These examples show that the proposed model performs well for entities with clear boundaries and stable terminology, but recognition becomes more difficult when multiple technical meanings are combined in one clause or when different entity categories share similar regulatory expressions.
Overall, the examples indicate that most recognition challenges are related to entity boundary ambiguity, category overlap, and long clauses containing multiple parallel entities. These cases further support the need to refine annotation guidelines and incorporate domain knowledge and boundary-aware constraints in future work.
Table 10 presents the performance of the model under different CFPDB documents. The model exhibits relatively consistent performance in the first three codes, with F1 scores higher than 0.9255. However, the performance of the model drops considerably in the code for design of automatic fire alarm system and the technical standard for smoke management systems in buildings. This performance fluctuation may be related to the differences in text style, term density, and sentence complexity, suggesting that cross-code generalization remains challenging under the current setting.
Although the last two codes show lower F1 scores, the overall micro-averaged performance remains high because entity distributions differ across codes.
Table 11 further explains the cross-code performance differences using derived statistical indicators. The automatic fire alarm system code and the smoke management system standard have high FPF proportions, reaching 47.2% and 40.3%, respectively. Their dense facility-oriented terminology, control relationships, and functional descriptions make entity boundary detection and category classification more difficult, which may partly explain their lower F1 scores.
5.4. Comparison with Existing Literature Methods
To further contextualize the performance of the proposed model, several representative NER studies discussed in the related work are summarized in
Table 12. These studies cover technical-domain NER, Chinese construction document NER, construction risk text mining, general Chinese NER, and Chinese medical NER. Since the compared studies are based on different corpora, entity schemas, annotation strategies, and application scenarios, the comparison is intended to provide a performance reference rather than a strictly controlled benchmark comparison.
As shown in
Table 12, the proposed model achieves an F1 score of 88.32% on Chinese building fire protection design codes. This result is lower than the power dispatch NER model reported by Chen et al. [
9], which may be attributed to the relatively stable terminology and clearer entity boundaries in power dispatch texts. However, the proposed model slightly outperforms the CRF-based Chinese construction document NER method [
10] and achieves better F1 performance than the construction supply chain risk NER model [
11], the general Chinese NER model [
14], and the Chinese medical NER model [
16].
These results indicate that the proposed model achieves competitive performance under a specialized regulatory-text scenario. Compared with general-domain or event-oriented texts, Chinese building fire protection design codes contain dense technical terminology, hierarchical clause structures, numerical requirements, and compliance-oriented semantic constraints. These characteristics make entity boundary recognition and category classification more challenging. Therefore, the proposed RoBERTa-BiLSTM-CRF architecture provides an effective solution for extracting structured entities from fire protection design codes and supports downstream automated compliance checking.
Beyond entity recognition performance, the extracted entities can also provide structured knowledge units for downstream decision support in automated compliance checking. For example, building type, space type, fire protection facility, numerical requirement, and standard provision entities can support clause retrieval, rule formalization, BIM-based checking, and evidence tracing, which is consistent with recent decision-support studies showing that structured engineering knowledge can be transformed into practical engineering decision-making tools [
68].
In a practical automated compliance checking workflow, the proposed NER model serves as an upstream information extraction module. The extracted entities can be normalized and mapped to BIM information, where space type entities correspond to BIM Room or Space objects, fire protection facility entities correspond to fire safety components, and numerical requirement entities can be converted into computable rule constraints. Downstream rule-checking modules can then compare BIM-derived information with structured regulatory requirements and generate compliance results with traceable code evidence.
Although this study focuses on Chinese building fire protection design codes, the proposed framework has potential transferability to other building code systems. When applied to international codes such as NFPA, IBC, or Eurocodes, the entity schema, technical terminology, measurement units, regulatory expressions, and clause structures should be adjusted according to the target code system. In addition, annotated samples from the target corpus are required for model fine-tuning, and multilingual applications may further require cross-lingual terminology alignment and semantic adaptation.