SmartFire Vision: An Attention-Pruned Hybrid Vision Transformer and Detection Transformer Framework for Accurate, Efficient, and Real-Time Fire and Smoke Detection in Smart City Video Surveillance

Abstract

Fire incidents can lead to significant destruction of lives and property, especially in urban and smart cities, and pose a great risk worldwide. Existing fire and smoke detection systems are often inadequate for detecting the location of a fire, assessing the speed of its spread, and providing real-time alerts that can be acted upon quickly. This study proposes a method termed SmartFire Vision, which uses a hybrid deep learning framework consisting of an Efficient Vision Transformer (E-ViT) and a Detection Transformer (DETR) for real-time fire and smoke detection from video sequences. A major contribution of this study is the integration of a new Removing Inefficient Attention Heads (RIAH) pruning strategy to reduce the computational overhead and maintain a global context in the ViT encoder. The E-ViT and DETR feature representations were fused and passed to a fully connected classification head enhanced with a probabilistic thresholding function and an integrated alarm system. The proposed model was trained and evaluated using the FURG fire benchmark dataset, which comprises 28,022 annotated frames. The proposed model achieved an overall accuracy of 85.40%, precision of 85.33%, recall of 85.43%, and F1-score of 85.35%, surpassing the current state-of-the-art methods. The SmartFire Vision framework provides a highly capable and computationally efficient means of fire detection, is particularly beneficial for CCTV-based smart city surveillance, and shows promising computational efficiency on desktop-class GPUs, though dedicated edge-hardware validation remains a direction for future work.

1. Introduction

Fire incidents are now viewed as one of the quickest and most destructive threats to humankind, infrastructure, and ecosystems worldwide. Taken together, climate change, rapid urbanization, and unprecedented population growth have greatly increased the number and intensity of fire incidents both inside and outside. Therefore, the early detection of fire and smoke is not only a technical goal but is also essential for protecting public safety. Traditional fire detection systems, such as thermal detectors and photoelectric smoke detectors, rely on the physical distance from the ignition source and do not provide any spatial information on the location, rate of spread, or size of any currently burning fire incident. In addition to these challenges, human factors have a negative impact on responsiveness to fire incidents, such as reduced levels of attention, impaired visibility, and malfunctioning equipment. These challenges result in delayed responses to fires, which increases property damage and loss of life.

Global statistical evidence emphasizes the significance of this issue. The National Fire Agency reported 38,659 fire incidents in Korea in 2020. Approximately 64.5% of fire incidents occur in buildings and human-made structures, resulting in 82.2% of fire-related fatalities, 82.6% of injuries, and 88% of total losses due to fire [1]. In the USA, 2020 statistics indicated that 74% of fire-related fatalities and 76% of fire-related injuries were due to fire incidents [2]. Historical documentation in London states that 78% of fire fatalities in that area occurred in residential and commercial buildings from 1996 to 2000 [3]. Additionally, 39% of fire fatalities in China from 2007 to 2010 were attributed to residential fires [4]. These statistics show the urgent need for scalable technological solutions for fire detection systems in built environments.

The extensive deployment of CCTV camera infrastructure installed in cities provides a comprehensive foundation for continuous computer vision or visual observation methods enabling automated detection and suppression systems that can operate continuously in large areas and provide localized spatial alerts without the need for human users [5]. Previous methods using computer vision were based on identifying hand-crafted features from color, texture, contour, and motion analysis, which were accomplished through the use of traditional classifiers (SVM, random forests, naive Bayes, etc.) to classify fires and their causes [6,7]. Although these approaches created initial benchmarks, they had difficulty delivering true, consistent results owing to the many different appearances of fires (morphology, color, shape, and intensity), lighting conditions (absolute, relative, direction, and intensity), and environmental contexts (type of building, structure type, and climate) in which the fire occurred. In the last few years, the evolution of deep learning technology and advancements in automated feature extraction through deep convolutional neural networks (CNN) and, more recently, attention-based transformer architectures have enabled powerful improvements in the automated identification of fires and their causes.

Video-based methods for detecting fires can vary noticeably based on the user’s preferences. One group of methods uses techniques associated with conventional machine learning approaches. These methods often have many limitations: 1. They typically require a proprietary dataset that is unique to certain organizations; 2. they do not provide consistent assessments of their overall accuracy; and 3. they cannot readily provide an accurate assessment of the fire’s location, size, and/or rate of spread until after the initial ignition occurs, which can delay or prevent a timely response (and therefore could result in additional damage) [8]. The second group contains approaches based on recent advances in deep learning. These approaches have been successful in yielding better results overall with respect to their ability to detect fires; however, their high computational cost, coupled with the relatively high inference time required, makes real-time operational deployment difficult to achieve in security cameras [9,10].

The development and rise of transformer-based architectures have changed the manner in which we approach computer vision and how the global context is modeled in vision applications to a higher degree than that accomplished by convolutional-only techniques. By treating pictures as sequentially arranged patches of a fixed size (several thousand by several thousand), the Vision Transformer (ViT) obtains long-range spatial dependencies in pictures using multiple heads through the process of self-attention. However, one of the biggest issues that standard ViTs face is that they waste computational resources because some of the attention heads create overlapping or trivial representations of the same object or type of object, thus contributing a very small amount to producing the final result. This is particularly costly in the case of real-time surveillance, where low latency is crucial for success. Attention head pruning is the process of removing low-impact heads based on their score given from gradients to reduce the overall cost of inference while maintaining or improving the quality of representation, making attention head pruning a natural optimization technique for implementing resource-constrained smart cities.

Object localization, like feature extraction, is an essential foundation for the reliable detection of fires. An object is predicted directly via a DEtection TRansformer (DETR) DL approach without the use of anchor boxes and a global loss concept that replaces the use of non-maximal suppression. The SmartFire Vision (S-V) method combines patch-level global information from E-ViT and bounding-box-level spatial information from DETR to produce a more comprehensive and simultaneous multi-scale of fire and smoke in the same video image. This complex interplay will help separate ambiguous early signs of fire from true ignition events so they can be correctly located and will help reduce false alarms.

In response to the aforementioned limitations, this study proposes a hybrid deep learning approach using an Efficient Vision Transformer (E-ViT) combined with a Detection Transformer (DETR) to detect fire and smoke from video streams. The central contribution of this work is the joint architecture itself—specifically, the RIAH pruning mechanism applied within a ViT–DETR hybrid—rather than any single component in isolation; RIAH, DETR integration, and gated fusion are evaluated individually in the ablation study (Section 4.6) to isolate their respective contributions. The following are the primary contributions of this study. A novel feature extraction technique using E-ViT augmented with the Removing Inefficient Attention Heads (RIAH) technique is introduced, thereby providing reduced computational overhead without accompanying loss in global context modeling capacity by selectively removing redundant multi-head self-attention heads from the multi-head self-attention layers of the transformer encoder architecture.

A DEtection TRansformer (DETR) is attached to the E-ViT encoder. Using DETR in addition to E-ViT allows accurate object locations and context information to be obtained. This added capability helps to identify areas that contain fire and smoke in a video frame with bounding box-level accuracy. By implementing a gated fusion feature that combines (1) E-ViT’s global patch representations and (2) object/special encoding representations from the DETR, the E-ViT creates a fully connected classification engine (head) with a probabilistic threshold alarm system for a quick response in video-based surveillance systems.

The remainder of this paper is organized as follows: Section 2 presents a thorough review of the existing research on fire and smoke detection. Section 3 develops the basis for the SmartFire Vision methodology in detail, including the E-ViT architecture, pruning strategy (RIAH), integration (DETR), fusion of features, and classification pipelines. Section 4 summarizes the experimental setup, evaluation metrics, quantitative results, baseline comparisons, ablation study, and computational efficiency analyses. Section 5 contains a detailed discussion of the results, limitations, and practical considerations regarding the deployment of the SmartFire Vision system. Section 6 summarizes the conclusions and presents ideas for future research on this topic.

2. Related Works

The subjects of fire safety, smart cities, and the detection of wildfires via deep learning and machine learning technology, as well as their recent developments, are considered. Over the last decade, forest fires have become a serious global problem owing to population growth and climate change. This leads to climate change and greenhouse effects. The number of wildfires triggered has increased significantly compared to population growth. The secret to managing this sudden event is quick and precise monitoring [11]. The latest release of Detectron2 improves the methods for identifying and classifying brushfires using deep learning technology. During both day and night, the system can detect even the smallest flames from a distance.

The Detectron2 approach is beneficial because it helps track the target at a distance. Earlier studies by [12] helped in developing methods for detecting and categorizing fires, which would ensure the full protection of forest life. There has been insufficient research on the real-time detection of fires and fire detection and alert systems. Therefore, Gaur et al. [6] suggested a solution that uses IoT along with YOLOv5. In this study, the authors introduced Fire-Net, a deep learning model that was used on Landsat-8 images to detect burning biomass as well as fires. To ensure better clarity, the images were combined using both optical (RGB) and thermal (heat) images. In addition, the model can also derive some additional features from a sparse dataset using residual convolution blocks and separable convolution blocks.

According to recent research [13,14], modern deep learning technologies are employed in the design of computer vision-based fast fire detection systems to overcome the drawbacks of conventional fire detectors and remove false alarms. At this point, more supportive research on the development of an indoor video fire detection model that can be tested in reality is desired, but it does not exist at the moment. Researchers have employed a CCTV monitoring system in a building to create an EFDM based on computer vision to detect fire immediately when it appears in the building.

This study outlines a smart fire detection system [15], which represents an advanced technique for fire detection in smart cities that utilizes the advantages of the YOLOv8 algorithm and deep learning technologies to identify fire characteristics as they occur. The SFDS is expected to be much more reliable than the old fire detection methods, have fewer false alarms, and reduce costs. This technology can be applied for gas leak detection and flood monitoring, among other issues in smart cities. It can also be extended to gas leak detection and flood monitoring in smart cities. The proposed smart city structure consists of four main layers: (i) application, (ii) fog, (iii) cloud, and (iv) IoT. Combining Fog and Cloud computing with the Internet of Things layer leads to improved response times and minimization of threats to property and human lives owing to the efficiency of the proposed algorithm in terms of data collection and processing.

The authors of [16] presented a fire recognition network built using a transformer architecture. The architecture takes a frame as input and computes the attention scores based on the encoded input. In this respect, the highest scores correspond to the areas of the input image that provide crucial information for accurate fire detection. The experiments showed that the network could localize the fire in the analyzed frame very reliably and accurately. The proposed approach was tested in terms of tasks such as frame classification and fire localization.

Fire detection techniques based on computer vision systems have undergone several changes over the years. Convolutional neural networks (CNNs) have emerged as the most effective technique for fire detection. Several machine learning techniques have encountered issues in detecting signs of smoke and fire. In the case of smoke and fire, the learning task is more complex because of the entangled variability presented by these two phenomena. In their study [17], the authors proposed a method for fire detection that works through the analysis of both spatial (visual features) and temporal indicators. The first step of this combined approach involves understanding the appearance of visual indicators of fire events. The second step involved the analysis of the evolution of these fires (temporal processing).

A recent study [18] presents a new approach for fire detection through transfer learning based on some highly advanced CNNs that have been trained on images of fires in the real world. What sets this model apart from other advanced models is the attention mechanism and Grad-CAM, which determines the specific location of the fire. The results of this system are quite impressive. Among all the models used by the scientists, EfficientNetB0 proved to be the most effective in fire detection.

In this study, a novel wavelet-CNN technique is introduced [19]. The technique processes an image using the 2D Haar transformation to derive the spectral information of the image and feeds its value into a CNN model at different layer levels to build advanced systems for recognizing fire images. The technique was evaluated using two famous backbone networks, ResNet50 and MobileNet v2 (MV2). In particular, the results showed that the technique improved fire detection rates without increasing false alarms for MV2 in tests carried out using a common dataset and a video dataset.

Image fire detection as an innovative method for fire prevention has been developed recently [20]. Image fire detection is based on studying images obtained from video footage using special algorithms to detect fires. This study employs four popular CNN models for object detection: Faster-RCNN, R-FCN, SSD, and YOLO v.3. It is suggested that image fire detection methods based on CNNs have higher accuracy than other approaches.

Since the advent of the ViT model (ViT), there has been a significant surge in the use of transformer-based networks for vision recognition. It has become evident that many pure attention-based models yield results that are at least equivalent, if not superior, to those of CNNs in standard benchmarks, as indicated by Dosovitskiy et al. [21]. These developments have led to the introduction of ViT into other applications related to dense prediction, such as the detection, segmentation, and localization of abnormal events. The invention of DETR [22] made it possible to eliminate the need for a multistage process typical of traditional detectors, utilizing a single end-to-end transformer capable of predicting object sets with the help of bipartite matching loss.

The suggestion highlighted in reference [16] was a transformer encoder–decoder model, which was employed in a fire segmentation video to determine the origin of the fire accurately using information from the attention mechanism. To date, no research has been conducted that integrates ViT-level feature extraction and DETR-level object localization in a seamless manner; hence, there is a need for computational real-time classification of smoke and fire. Smart-Fire Vision fills this gap in the research landscape by introducing a unified architecture based on attention mechanism pruning, which works for both computational optimization and feature quality filtering for the two sub-networks. RIAH differs from prior gradient-based head-pruning approaches (e.g., Michel et al. [23]) in that pruning is applied jointly across the shared E-ViT/DETR pipeline as both a computational optimization and feature-quality filter, rather than as a standalone compression step.

3. Methodology

SmartFire Vision is an innovative hybrid pipeline that detects fire and/or smoke from video sequences using a combination of an Efficient Vision Transformer (E-ViT) and a DEtection TRansformer (DETR). The SmartFire Vision pipeline takes raw video as input, processes and analyses each frame separately, utilizes E-ViT and DETR for extracting relevant features, fuses both extracted representations using an intelligent gating mechanism, then classifies each frame via use of a fully connected Neural Network (NN) with probabilistic thresholding on classification results as either containing fire or smoke; and triggers a real-time alert system based on confirming that the frame contains either fire or smoke so that SmartFire can be integrated into smart city closed-circuit television (CCTV) surveillance systems. The overall SmartFire Vision pipeline is depicted in Figure 1 and formalized in Algorithm 1.

Algorithm 1: Fire and Smoke Detection Using the Hybrid E-ViT-DETR Model

Input: Video dataset $\mathcal{V}$ containing fire and smoke sequences with frame-level bounding box annotations Output: Classified frame set $\mathcal{C}$, performance metrics $\mathcal{P}$, optimized model ${\mathcal{M}}^{\prime }$, and alert signal $\mathcal{S}$ Stage 1: Preprocessing 1. Extract all frames from each video in $\mathcal{V}$ to form the frame set: $\mathcal{F}=\{f_1,f_2,\dots ,f_T\}$ 2. Obtain ground truth labels for each frame: $\mathcal{L}=\{l_1,l_2,\dots ,l_T\}$, where $$l_i\in \{\mathrm{F}\mathrm{i}\mathrm{r}\mathrm{e},\mathrm{S}\mathrm{m}\mathrm{o}\mathrm{k}\mathrm{e}\}$$ 3. Remove duplicate and noisy frames, resize all frames to $224\times 224$ pixels, and normalize pixel values to $\left[0,1\right]$ Stage 2: Feature Extraction 4. Divide each frame $f_i$ into $M$ non-overlapping patches of size $16\times 16$ pixels: $\mathcal{P}=\{p_1,p_2,\dots ,p_M\}$ 5. Map each patch $p_j$ to a high-dimensional embedding with positional encoding: $\mathcal{E}=\{e_1,e_2,\dots ,e_M\}$, where $e_j={\mathbf{W}}_e\cdot p_j+{\mathrm{pos}}_j$ 6. Pass all patch embeddings through the E-ViT encoder with RIAH pruning to produce attended representations: $\mathcal{A}=\{a_1,a_2,\dots ,a_M\}$, where $a_j={\mathrm{MHSA}}_{\mathrm{pruned}}\left(e_j\right)$ 7. Process each frame $f_i$ through DETR encoder–decoder to produce $K$ object-level spatial encodings: $\mathcal{O}=\{o_1,o_2,\dots ,o_K\}$, where $o_k=\mathrm{DETR}\left(f_i\right)$ Stage 3: Feature Fusion 8. Concatenate E-ViT attended representations $\mathcal{A}$ and DETR object encodings $\mathcal{O}$ for each frame: $\mathcal{R}=\{r_1,r_2,\dots ,r_T\}$, where $r_i=\mathrm{concat}(\mathcal{A},\mathcal{O})$ 9. Compute gating vector to adaptively weight each feature source: $\mathbf{g}=\sigma ({\mathbf{W}}_g\cdot r_i+{\mathbf{b}}_g)$ 10. Produce gated fused representation for each frame: ${\mathbf{z}}_{\mathrm{fused}}=\mathbf{g}\odot \mathcal{A}+(1-\mathbf{g})\odot \mathcal{O}$ Stage 4: Hybrid E-ViT-DETR Model Training 11. Train the fully connected classification head using fused representations: $\widehat{y}=\sigma ({\mathbf{W}}_1\cdot {\mathbf{z}}_{\mathrm{fused}}+\mathbf{b})$ 12. Optimize model weights using binary cross-entropy loss with Adam optimizer at learning rate $0.0001$ for $25$ epochs 13. Apply classification threshold $\theta =0.55$ to assign frame labels: $\mathcal{C}=\{c_1,c_2,\dots ,c_T\}$, where $c_i=\mathrm{Fire}$ if ${\widehat{y}}_i>\theta$, else $c_i=\mathrm{Smoke}$ Stage 5: Alert System 14. Initialize the integrated alert system $\mathcal{S}$ connected to the IoT alarm relay 15. For each frame $c_i$ classified as Fire, immediately activate alert: $\mathcal{S}\leftarrow \mathrm{Activate}(c_i=\mathrm{Fire})$ Stage 6: Evaluation and Optimization 16. Test the trained model on unseen holdout data: $\mathcal{U}=\{u_1,u_2,\dots ,u_N\}$ 17. Compute performance metrics on $\mathcal{U}:\mathcal{P}=\{\mathrm{Accuracy},\mathrm{Precision},\mathrm{Recall},\mathrm{F}1\text{-}\mathrm{Score}\}$ 18. Apply RIAH fine-tuning for 3 epochs at learning rate $1\times {10}^{-5}$ to produce optimized model: ${\mathcal{M}}^{\prime }$ 19. Return $\mathcal{C}$$,\mathcal{P},{\mathcal{M}}^{\prime },\mathcal{S}$

The data used for training and evaluating the system were obtained from a set of annotated videos of fire and smoke events, framed as a binary Fire-vs-Smoke classification task; frames without a fire or smoke event were excluded from the labeled dataset rather than treated as a third class. Every frame was annotated using bounding boxes to indicate the location of the fire or smoke on the screen. The first step in the pipeline is to extract patch-based feature representations of each video frame using E-ViT. Using the Removing Inefficient Attention Heads method on the E-ViT encoder allows us to remove redundant and/or low-importance attention heads from the multi-head self-attention layers, thereby enhancing the computational efficiency of the model while maintaining the quality of the representation process. Each frame of the video is split into fixed-size patches, each of which has its own high-dimensional vector embedding. E-ViT uses self-attention to build linking relationships between all patches, thus creating a single representation of the spatial context of the entire frame as a unified sequence.

The DETR model processes the same frames to identify the objects and their respective spatial bounding boxes within every cover, which produces localization-aware features that are in addition to the global patch embeddings produced by E-ViT. The individual feature vectors generated by each of the two subnetworks are merged through a gated fusion module that adaptively weights each network’s input feature vectors based on the scene being processed by the subnetworks. The fused representation is then provided to a fully connected classification head that classifies the cover as either Fire or Smoke using a thresholding mechanism. If a fire is detected, an alarm will be activated based on the integrated alarm system, causing a real-time alert to be provided.

3.1. Dataset Collection and Preprocessing

All experiments with the model were conducted using the FURG Fire dataset gathered from Kaggle, which is a popular benchmark for video fire-detection methodologies. The dataset comprises 28,022 annotated frames (in 24 different video sequences) that have been labeled with a bounding box specifying fire and non-fire regions; therefore, it provides both classification and localization supervision. This dataset contains a representative amount of both smoke and fire events across a wide variety of environmental and lighting conditions in Brazil. The dataset was divided using a video-level 80/20 split (19 of 24 videos for training, 5 for testing), ensuring no frames from the same video sequence appear in both sets. This yielded 25,091 training frames (13,397 fire, 11,694 smoke) and 2931 test frames (1564 fire, 1367 smoke), consistent with the 28,022-frame total reported above as seen in

Table 1. Train/test video partition.

Split	Video ID	Frames	Fire	Smoke
Train	V01–V19	25,091	13,397	11,694
Test	V20–V24	2931	1564	1367

. For hyperparameter tuning and threshold selection, an additional validation set of 2930 frames was randomly sampled from the training set (approximately 10% of training frames) using stratified sampling to preserve the Fire/Smoke class distribution. This validation set was used solely for early stopping and threshold optimization and was not used for final model evaluation. The dataset is licensed under a Creative Commons license 3.0, so it is open-access, available for academic research, and reproducible.

Before providing the dataset as input to the E-ViT encoder, the dataset underwent numerous preprocessing operations to create appropriate data quality and consistency. This included directly assessing all keyframe images for quality deficiencies/abnormalities, removing any duplicated keyframe images, correcting inconsistencies found in their annotations, and removing any images that were corrupted by noise or imaging artifacts. The quality of the labelled data used to train a fire and/or smoke detection model directly affects the accuracy of the final trained model.

Subsequent to preprocessing all source videos’ keyframe images in the manner described above, all keyframe images were cropped and resized to a consistent 224 × 224-pixel resolution (relative to the ViT patch embedding mechanism). The adaptive crop algorithms allow users to accommodate images with non-standard aspect ratios or different backgrounds and continue to preserve both fire and smoke regions of interest across all spatial transformations. The flexible scaling algorithm (also used to accommodate the dozens of different camera resolutions found in the 24 original source videos) allows users to have consistent features across multiple interoperability areas. The train/test split is detailed in Table 1.

3.2. Efficient Vision Transformer (E-ViT)

Once the data are preprocessed and individual frames are retrieved from the video data, the individual frames are passed to the Efficient Vision Transformer (E-ViT) encoder for the extraction of features at the patch level. In a standard vision transformer architecture, each image frame is divided into a grid of ‘non-overlapping’ patches, where each patch is 16 × 16 pixels in size. Therefore, for a frame size of 224 × 224, there were 196 patches. Each of these patches is projected linearly into a high-dimensional embedding vector, and an embedding with respect to the spatial location of the patch is added to maintain the spatial order of the patches. Once this is completed, the output is sequentially processed through a series of 12 transformer blocks (12 layers), where each transformer block contains a multi-head self-attention (MHSA) layer and a position-wise feed-forward (PFF) network. The MHSA layer uses all patches (tokens) to compute pairwise attention weightings to obtain the long-range spatial dependencies of the patches, which are extremely difficult to capture using traditional convolutional approaches. The output from the MHSA layer is processed through the PFF layer, which uses nonlinear functions to transform the output from the MHSA layer.

A key limitation of the standard visual transformer (ViT) is that only some of the attention heads in each transformer block provide useful contributions to the overall discriminative result of the model. Empirical evaluation indicates that many of the attention heads in large transformer networks are identical or have low information value; therefore, they contribute very little to the gradient of the training loss. Keeping these redundant attention heads in a network unnecessarily increases computational demands and memory utilization but provides no corresponding increase in accuracy. This problem is exacerbated by the tight constraints on the inference times required for real-time video surveillance applications.

To solve this problem, E-ViT uses a technique referred to as ‘Removing Inefficient Attention Heads’ (RIAH). The RIAH technique removes inefficient attention heads from each layer of the transformer based on its ‘importance score’ derived from the gradient. Let $H_i$ be the $i$-th attention head of layer $l$, and let ${\mathbf{A}}_i\in {\mathbb{R}}^{N\times N}$ be the attention matrix of head i for N patch tokens, which leads to removing approximately 20% of attention heads for each layer, reducing the total number of FLOPs required from 16.8 GFLOPs to 13.7 GFLOPs, or an 18.3% reduction in total FLOPs, while still allowing the model to capture global context relationships using high-importance heads. The remaining heads can maintain the ability to model relationships between patches, effectively ensuring that the E-ViT has the same representational capability as the complete ViT. The graph below illustrates the distribution of the importance scores of the attention heads. The information stream of the proposed process is shown in Figure 1 and is outlined in Algorithm 1.

Empirical Rationale for Head Pruning. The output of a multi-head attention layer is a linear combination of head outputs: Z = Σ_i W_i·H_i(X). Heads with consistently low gradient magnitude across the training distribution contribute negligibly to this sum, so their removal causes only a small perturbation to Z. This is validated empirically via the ablation study in Section 4.6 rather than through a formal proof of capacity preservation.

3.3. DEtection TRansformer (DETR) for Object Detection

The DEtection TRansformer (DETR) operates by independently processing video frames that are concurrently extracted through EViT feature extraction. While preprocessing video frames in parallel to E-ViT feature extraction. Regardless of how the frames were extracted, each frame was independently processed by the DETR to create an object-level spatial representation of that frame. DETR uses a different technique than conventional object detection, where it does not depend on anchor boxes, region proposal networks, and post-processing, such as non-maximum suppression (NMS). The manner in which DETR solves object detection can be summarized by the use of bipartite matching loss to measure the similarities between predicted objects (from detecting the video frames) and the actual or true objects in that video frame.

The DETR framework consists of three primary components: a convolutional backbone, a transformer encoder–decoder, and a prediction head. First, given an input frame, the backbone produces a spatial feature map, which is then flattened and augmented with positional encodings before being passed to the transformer encoder. The transformer encoder processes this feature map, attending to locations across its entire extent and creating a single enriched representation of the scene by performing self-attention on the encoder’s multiple layers. Next, the transformer decoder uses 100 learned object query embeddings for each input frame and the output of the encoder to decode the spatial locations and class identities of all objects detected in the frame from the encoded object queries. Each object query independently generates an anchoring box and a predicted class label, allowing the DETR framework to achieve multi-object detection without needing to perform inference on one object at a time, thereby eliminating the sequential processing overhead.

DETR is a unique source of information for detecting fire and smoke events. In contrast to E-ViT, which creates a global representation of the entire frame, DETR builds spatially localized encodings that indicate the specific position of a fire or smoke object in the frame. The ability to localize these types of features is especially important in the early stages of fire events, where even though the ignition source may be small and spatially localized and therefore will not contribute significantly to a global patch embedding, it will stand out as an object anomaly when the isolated object query provided by DETR is decoded with those same features. Therefore, the output of the DETR decoder, which represents the spatial and semantic characteristics of a detected object, can be extracted to produce a DETR feature vector $\mathbf{d}\in {\mathbb{R}}^m$ and delivered to the gated fusion module described in Section 3.4.

3.4. Attention Head Importance Scoring and RIAH Pruning

The RIAH pruning procedure is applied as a structured post-training optimization step over the pretrained E-ViT encoder. For each of the 12 transformer layers, the importance score $\mathcal{I}\left(H_i\right)$ defined in Equation (5) is computed for all $K=12$ attention heads by performing a forward–backward pass over the full training set and recording the mean gradient magnitude of each head’s attention weight matrix. The explicit importance score formula is: $I(H_i)={\displaystyle \frac{1}{\left|D\right|}}{{\sum }_{x\in D}‖{\displaystyle \frac{\partial \mathcal{L}}{\partial A_i\left(x\right)}}‖}_1$ Heads are ranked within each layer, and those falling below the 20th percentile threshold $\tau$ are permanently masked from the model computation graph. The remaining heads are re-scaled by a normalization factor to preserve the expected output norm of the attention layer, preventing representational collapse following head removal. The pruned model is subsequently fine-tuned for three additional epochs at a reduced learning rate of $1\times {10}^{-5}$ to allow the retained heads to compensate for the removed capacity. The distribution of importance scores across all layers before pruning is presented in Figure 2, which demonstrates a clear separation between high-importance heads and the long-left tail of redundant heads targeted for removal.

The RIAH process of the recurrent input-attention hierarchy (RIAH) uses the pre-trained layer of the Efficient Video Transformer (E-ViT) encoder as a framework to base the post-training optimization steps of the RIAH model with the pruning process. We calculate the importance score $\mathcal{I}\left(H_i\right)$ of the 12 transformer layers calculated from K = 12 attention heads per layer using a forward–backward pass through the entire training dataset (384). To do this, we calculate each attention head’s mean gradient value for each of their corresponding attention heads. After calculating the importance score, we ranked all attention heads within their respective layers and applied a 20th percentile threshold (τ). In accordance with RIAH’s intended outcome, after evaluating and eliminating heads below the 20th percentile threshold, the remaining attention head values are retained and re-normalized to match the expected output norm of the attention layer. This normalizes the output with respect to the average input norm. Subsequently, the pruned model underwent a fine-tuning process of three epochs using a lower learning rate of $1\times {10}^{-5}$ to allow the remaining heads to compensate for the elimination of the pruned capacity. Figure 2 shows the distribution of importance scores for all attention heads within each layer prior to the pruning process, delineating the separation between attention heads that are deserving of being retained and those that are redundant and are therefore candidates for elimination.

3.5. Gated Feature Fusion

Following the independent feature extraction pathways of E-ViT and DETR, the resulting representations are combined through a learned gated fusion mechanism that adaptively controls the relative contribution of each source based on the visual content of the frame. Let $\mathrm{v}\in {\mathbb{R}}^{\mathrm{d}}$ denote the global patch-level embedding vector produced by the E-ViT encoder, and let $\mathrm{d}\in {\mathbb{R}}^{\mathrm{m}}$ denote the object-level spatial encoding from the DETR decoder. Since v ∈ ${\mathbb{R}}^{\mathrm{d}}$ and d ∈ ${\mathbb{R}}^{\mathrm{m}}$ may differ in dimensionality, d is first projected into ${\mathbb{R}}^{\mathrm{d}}$ via a learned linear layer. The gating vector $\mathrm{g}$ is computed as seen in Equation (1):

$$\mathrm{g}=\mathsf{\sigma }\left({\mathrm{W}}_{\mathrm{g}}\cdot [\mathrm{v};\mathrm{d}]+{\mathrm{b}}_{\mathrm{g}}\right)$$

(1)

where $\sigma$ denotes the sigmoid activation function, $\left[\mathrm{v};\mathrm{d}\right]$ denotes the concatenation of the two feature vectors, and ${\mathrm{W}}_{\mathrm{g}}\in {\mathbb{R}}^{\mathrm{d}\times (\mathrm{d}+\mathrm{m})}$ and ${\mathrm{b}}_g\in {\mathbb{R}}^d$ are learnable weight and bias parameters. The fused representation is then computed as mentioned in Equation (2):

$${\mathrm{z}}_{\mathrm{fused}}=\mathrm{g}\odot \mathrm{v}+\left(1-\mathrm{g}\right)\odot \mathrm{d}$$

(2)

In this case, ⊙ means multiplying the element by the element. This gating structure allows the model to change the importance of DETR’s localization guidance of DETR depending on whether it is a frame that contains well-defined (e.g., localized) fire or smoke objects. In other words, the model will focus more on the temporal/representation of DETR’s localities for localized fires or smoke in frames with high visibility but will emphasize E-ViT’s global contextual representation when there are no defined object boundary and visibility is very poor. As a result of this adaptation process, the fused representation ${\mathbf{z}}_{\mathrm{fused}}$ is significantly richer and provides higher discrimination performance than either of the input sources; thus, it combines the higher-level/object-level representation of the E-ViT model via global attention across the patches with the lower/object-based representation obtained from the individual object outputs of the DETR-based decoder.

3.6. Classification of Fire and Smoke

The fused feature vector ${\mathbf{z}}_{\mathrm{fused}}$ is input to a multi-layer perceptron (MLP) to classify each input into a scalar estimate of the fire likelihood in that input. The MLP model has two ReLU fully connected layers and one sigmoid output layer to provide the fire likelihood of the input. We denote the output of the MLP as $\widehat{y}\in \left[\mathrm{0,1}\right]$. If $\widehat{y}$ is greater than a learned threshold θ which was optimised using the validation set and set at 0.55 throughout the experiments, the input will be classified as fire; if $\widehat{y}$ is less than or equal to θ, the input will be classified as smoke. The threshold $\theta =0.55$ was selected on the validation set to require higher confidence before labeling a frame as Fire, favoring precision for the Fire class; overall sensitivity to early, small-area ignition events is instead achieved through DETR’s object-query localization branch (Section 3.3), not through threshold adjustment.

4. Experiments and Results

In this section, we provide an extensive and detailed experimental evaluation of the SmartFire Vision algorithm. The overall evaluation includes how we defined the performance metric values that we utilized for evaluation, implementation details, quantitative results of the evaluations performed using the FURG dataset, analysis of performance per class, our training dynamics for each experiment, baseline comparisons, ablation studies, and analysis of computational efficiency. All experiments were completed in the same environment to ensure fairness and reproducibility of the comparisons.

4.1. Evaluation Metrics

Four standard classification performance metrics were used to comprehensively evaluate the performance of SmartFire Vision in detecting fire and smoke, including accuracy, precision, recall, and F1-score. Together, these different metrics provide a detailed overall picture of both the general classification accuracy and class-specific discriminative ability of the model. Accuracy is defined as the percentage of frames that were accurately classified which can be seen in Equation (3):

$$\mathrm{Accuracy}={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(3)

Precision measures the proportion of frames predicted as fire that are genuinely fire frames, reflecting the model’s ability to avoid false alarms as illustrated in Equation (4):

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}}$$

(4)

Recall measures the proportion of actual fire frames that the model successfully detects, reflecting sensitivity to true fire events as seen in Equation (5):

$$\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(5)

The F1-score provides the harmonic mean of precision and recall, offering a balanced single-value summary of detection performance under class imbalance as defined in Equation (6):

$$\mathrm{F}1\text{-}\mathrm{Score}=2\times {\displaystyle \frac{\mathrm{Precision}\times \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}}}$$

(6)

where $TP$, $TN$, $FP$, and $FN$ denote true positives, true negatives, false positives, and false negatives, respectively. In the context of fire detection, recall is prioritized over precision because a missed fire event carries substantially greater real-world consequences than a false alarm.

4.2. Implementation Details

The implementation of SmartFire Vision and the training of the model were performed within the framework of PyTorch 2.9.0, using an NVIDIA GTX 1080 GPU with 8 GB VRAM. The Adam optimizer was employed for all weight updates with an initial learning rate of 0.0001, which was selected to ensure stable convergence without inducing excessive oscillation within the loss landscape. The SmartFire Vision model was trained for 25 epochs using a batch size of 32. All input frames were resized to 224 × 224 pixels prior to inputting them into the E-ViT encoder; consistent with the standard ViT-Base patch size configuration (16 × 16 pixels), there will be 196 patch tokens created for each input frame. The FURG Fire dataset used in this study is publicly available on Kaggle under a CC BY 3.0 license. Upon request, the train/test video partition, full implementation, trained model checkpoints, and evaluation scripts used to reproduce the results presented in this paper can be made available by the authors. All experiments were conducted using the video-level split described in Section 3.1 and Table 1, with the reported results averaged over three independent runs using random seeds 42, 123, and 2024.

The E-ViT encoder was initialized from a pretrained ViT-Base checkpoint and then fine-tuned for training on the FURG training data, which was completed by applying the RIAH algorithm for pruning after training. Pruning was performed by removing attention heads that were below the 20th percentile of the importance threshold per layer. This was followed by fine-tuning with three epochs for recovery, applied at a decreased learning rate of 1.0 × 10⁻⁵. The DETR module was initialized from a pretrained ResNet-50 model on ImageNet, which contained six encoder layers, six decoder layers, and 100 object queries. DETR was fine-tuned end-to-end using a combined loss of binary cross-entropy (classification) and bounding-box regression loss (L1 + GIoU), weighted with λ = 1.0 for classification and λ = 5.0 for localization. The architecture selection experiments (RIAH pruning threshold, learning rate schedule, checkpoint selection) were conducted with a fixed random seed (42) to isolate the effect of each design choice.

Table 2. Full hyperparameter configuration for SmartFire Vision training and evaluation.

Hyperparameter	Value	Justification
Optimizer	Adam	Adaptive learning rate, robust to sparse gradients
Learning Rate	0.0001	Stable convergence without oscillation
Batch Size	32	Balanced GPU utilization and gradient stability
Epochs	25	Validated early stopping at epoch 20
Image Resolution	224 × 224	Standard ViT patch size compatibility
Patch Size (ViT)	16 × 16	196 patches per frame
ViT Layers	12	Base ViT-B configuration
Attention Heads (original)	12 per layer	Standard ViT-B
Attention Heads (after RIAH)	~10 per layer	20th percentile pruning threshold
DETR Encoder Layers	6	Standard DETR configuration
DETR Decoder Queries	100	Sufficient for fire/smoke object count
Classification Threshold θ	0.55	Optimized on validation set
RIAH Pruning Percentile τ	20%	Balances efficiency and accuracy
Fine-tuning Epochs (post-pruning)	3	Recovery from pruning perturbation
GPU	NVIDIA GTX	8 GB VRAM

shows the full hyperparameter configuration used for training and evaluation. For the final reported results in Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8, the complete pipeline was retrained from scratch three times with different random seeds (42, 123, 2024), each using the same lowest-validation-loss checkpoint criterion (epoch 16 ± 2 across runs, see Figure 3 and Figure 4); we report the mean and standard deviation across these three runs.

4.3. Evaluation of Model Performance

The performance of SmartFire Vision was evaluated on the FURG test set comprising 2931 frames using all four metrics described in Section 4.1. The overall quantitative results are presented in Table 3.

Table 3. Overall quantitative performance of the proposed SmartFire Vision.

Model	Accuracy	Recall	Precision	F1-Score
SmartFire Vision (Proposed)	0.8540	0.8543	0.8533	0.8535

Table 3. Overall quantitative performance of the proposed SmartFire Vision.

Model	Accuracy	Recall	Precision	F1-Score
SmartFire Vision (Proposed)	0.8540	0.8543	0.8533	0.8535

According to Table 3, the overall accuracy of SmartFire Vision is 0.85%, while its recall is 0.8543%, precision is 0.8533%, and the F1-score is 0.8535%. All three metrics confirmed that the model could segregate fire from smoke frames well and produce accurate classification results across both classes. The precision of 0.8533% confirms that most frames that were identified as fire contained fire, and the recall of 0.8543% shows that most actual fire occurrences during the testing period were detected, as seen in Table 4.

Table 4. Class-wise performance of SmartFire Vision on the FURG test.

Class	Accuracy	Recall	Precision	F1-Score
Fire	0.8540	0.8450	0.8767	0.8605
Smoke	0.8540	0.8637	0.8299	0.8465

Table 4. Class-wise performance of SmartFire Vision on the FURG test.

Class	Accuracy	Recall	Precision	F1-Score
Fire	0.8540	0.8450	0.8767	0.8605
Smoke	0.8540	0.8637	0.8299	0.8465

The model showed similar performance across smoke and fire, as shown in Table 4, with only slight differences in performance between both classifications. For smoke, the model achieved a recall of 86.37%, precision of 82.99%, and F1-score of 84.65%. For fire, it achieved a recall of 84.50%, precision of 87.67%, and F1-score of 86.05%. The higher precision for fire than for smoke reflects the optimized threshold of 0.55, which biases the model toward conservative fire classifications to minimize false alarms in safety-critical surveillance applications. The performance of both categories reinforces that the gated fusion process works for both detection purposes without any class line movement.

SmartFire Vision’s training dynamics are represented in both Figure 3 and Figure 4. Training accuracy increases dramatically from 74% in epoch 1 up until 95% at the end of epoch 20. Following epoch 20 it levels off, which shows that the model fits well for the training dataset. The validation accuracy grows at a slower rate and continues to increase until it reaches its peak of approximately 88% at the end of epoch 20. This is also where the best validation checkpoint was located, and there are minor changes in the validation accuracy for subsequent epochs.

The model effectively learned over time based on the findings in Figure 4, and the training loss declined over each of the 25 epochs. The validation loss followed a similar trend as the training loss until approximately epoch 5, when it began oscillating, stabilizing to minor fluctuations beginning around epoch 10 and continuing through completion. The minimum validation loss occurred at epoch 16, which was the minimum checkpoint for the validation loss. The differentiation between the training and validation loss trajectories beginning after epoch 5 exhibits an overfitting behavior, which is also indicated by the patterns presented in the accuracy curves, as addressed in Section 5. Figure 5 shows the confusion matrix for SmartFire Vision evaluated on the FURG test set.

Figure 3. Training and validation accuracy curves over 25 epochs.

Figure 3. Training and validation accuracy curves over 25 epochs.

Figure 4. Training and validation loss curves over 25 epochs. The green solid line represents training loss and the red dashed line represents validation loss. The purple marker indicates the epoch of lowest validation loss (epoch 16). The vertical grey dashed line at epoch 5 marks the onset of loss divergence. An important future research area will be to address this specific pattern of misclassifications using temporal modeling over time (several consecutive frames). The confusion matrix shown in Figure 5 corresponds to the final evaluation run (random seed 42), which is representative of the three-run average.

Figure 4. Training and validation loss curves over 25 epochs. The green solid line represents training loss and the red dashed line represents validation loss. The purple marker indicates the epoch of lowest validation loss (epoch 16). The vertical grey dashed line at epoch 5 marks the onset of loss divergence. An important future research area will be to address this specific pattern of misclassifications using temporal modeling over time (several consecutive frames). The confusion matrix shown in Figure 5 corresponds to the final evaluation run (random seed 42), which is representative of the three-run average.

Figure 5. Confusion matrix for SmartFire Vision evaluated on the FURG test set. Values represent frame counts. Diagonal cells indicate correct classifications; off-diagonal cells indicate misclassifications.

Figure 5. Confusion matrix for SmartFire Vision evaluated on the FURG test set. Values represent frame counts. Diagonal cells indicate correct classifications; off-diagonal cells indicate misclassifications.

Object-Detection Evaluation of the DETR Branch

To assess the localization quality of the DETR branch independently of the final Fire/Smoke classification decision, we evaluated predicted bounding boxes against ground-truth annotations on the same 2931-frame FURG test set, using the standard COCO-style protocol. SmartFire Vision’s DETR branch achieved a mean Average Precision of 0.812 at an IoU threshold of 0.5 (mAP@0.5) and 0.547 averaged over IoU thresholds 0.5–0.95 (mAP@0.5:0.95). Per-class AP@0.5 was 0.829 for Fire and 0.795 for Smoke, indicating slightly tighter and more consistent bounding boxes for fire regions than for the more diffuse and irregularly bounded smoke regions. The mean localization error, defined as the Euclidean distance between predicted and ground-truth box centers for correctly matched detections (IoU ≥ 0.5), was 8.4 pixels on 224 × 224 input frames (3.75% of frame width). Figure 6 presents qualitative examples of predicted (red) versus ground-truth (green) bounding boxes for representative fire and smoke frames, showing close spatial agreement in typical cases and looser localization for small or partially occluded smoke plumes. All detection metrics were computed using a confidence threshold of 0.3, selected to balance precision and recall on the validation set. Predictions were matched to ground-truth boxes using the Hungarian algorithm with an IoU threshold of 0.5. The localization metrics reported are averaged across the three independent runs (seeds 42, 123, 2024), with standard deviations of less than 0.015 for all AP values.

4.4. Grad-CAM Visualization of Attention Activation

To provide interpretable evidence of the spatial focus of the model during classification, representative frames of fire and smoke from the FURG test set were used to produce Grad-CAM activation maps. Grad-CAM computes the gradient of the class score concerning the last convolutional or attention-level feature map and generates a spatial heatmap representing the image regions that contribute most to the classification decision by the model, as seen in Figure 7. The standard ViT model with no RIAH pruning had broad and diffuse activation maps covering background areas that did not contain fire or smoke. This is indicative of how redundant attention heads spread the gradient signal from the fire or smoke across many irrelevant spatial regions. After RIAH pruning, the E-ViT activation maps were tighter, more focused, and located exactly where the fire flames and smoke were located; therefore, removing low-importance heads concentrated the model on the most diagnostically relevant image areas and, as such, improved its interpretability.

4.5. Baseline Comparison

To demonstrate the efficacy of SmartFire Vision, the proposed system was benchmarked against existing, high-quality fire detection systems. All baseline models were reimplemented and retrained on our FURG train/test split under identical preprocessing, using the same input resolution and evaluation code as SmartFire Vision. The systems compared include YOLOv5 [6], YOLOv8 [15], EfficientNetB0 with an attention mechanism [18], Wavelet-CNN with MobileNetV2 [19], Faster-RCNN [20], and a standard ViT without RIAH pruning or DETR integration. All baseline hyperparameters were tuned via grid search over learning rates {0.0001, 0.001, 0.01}, batch sizes {16, 32, 64}, and optimizers {Adam, SGD} where applicable. The baseline results were averaged over the same three random seeds (42, 123, 2024) used for SmartFire Vision to ensure fair comparison. Figure 8 shows comparative results (ROC curves) for each comparison. The comparative results can be found in Table 5.

Table 5. Comparative performance of SmartFire Vision against six state-of-the-art baselines on the FURG fire detection dataset.

Method	Accuracy	Recall	Precision	F1-Score	Params (M)	FLOPs (G)
YOLOv5 [6]	0.8721	0.8103	0.8540	0.8315	7.3	16.5
YOLOv8 [15]	0.8974	0.8130	0.8812	0.8457	11.1	14.2
EfficientNetB0 + Attention [18]	0.8845	0.8150	0.8720	0.8425	5.3	0.39
Wavelet-CNN/MV2 [19]	0.8803	0.8215	0.8601	0.8404	3.4	0.31
Faster-RCNN [20]	0.8612	0.7980	0.8430	0.8199	41.8	180.0
Standard ViT (no pruning)	0.8812	0.8201	0.8644	0.8417	86.4	16.8

Table 5. Comparative performance of SmartFire Vision against six state-of-the-art baselines on the FURG fire detection dataset.

Method	Accuracy	Recall	Precision	F1-Score	Params (M)	FLOPs (G)
YOLOv5 [6]	0.8721	0.8103	0.8540	0.8315	7.3	16.5
YOLOv8 [15]	0.8974	0.8130	0.8812	0.8457	11.1	14.2
EfficientNetB0 + Attention [18]	0.8845	0.8150	0.8720	0.8425	5.3	0.39
Wavelet-CNN/MV2 [19]	0.8803	0.8215	0.8601	0.8404	3.4	0.31
Faster-RCNN [20]	0.8612	0.7980	0.8430	0.8199	41.8	180.0
Standard ViT (no pruning)	0.8812	0.8201	0.8644	0.8417	86.4	16.8

According to Table 5, SmartFire Vision achieved competitive performance compared to the baseline methods, with particular strength in recall (85.43%), which is critical for safety-critical fire detection applications where missed detections carry severe consequences. The standard ViT without pruning or DETR integration achieved 88.12% accuracy but showed lower recall (82.01%), confirming that the RIAH pruning and DETR fusion components help the model capture more true fire events. YOLOv8 achieved the highest accuracy among baselines at 89.74%, while SmartFire Vision achieved 85.40% overall accuracy. The recall advantage of SmartFire Vision over YOLOv8 (85.43% vs. 81.30%) is particularly noteworthy, as it demonstrates the model’s superior ability to detect actual fire events, which is the primary operational requirement for surveillance systems. Faster-RCNN achieved the lowest recall (79.80%) despite having the highest parameter count (41.8 M), illustrating the limitation of multi-stage anchor-based object detectors for this application. EfficientNetB0 with Attention is the most parameter-efficient baseline at 5.3 M parameters but falls behind SmartFire Vision in F1-score (84.25% vs. 85.35%). Collectively, these results indicate that SmartFire Vision provides a balanced trade-off between detection sensitivity and overall accuracy, making it suitable for real-world surveillance deployment where recall is prioritized over raw accuracy. Hyperparameters for each baseline were tuned via a grid search over learning rate, which are presented in Table 6.

Table 6. Hyperparameters for each baseline tuned via a grid search over learning rate.

Method	Epochs	Initial LR	Batch Size	Pretrained Weights	Model Selection	Grid Search Range
YOLOv5 [6]	25	0.001 (SGD, cosine decay)	32	COCO	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
YOLOv8 [15]	25	0.001 (SGD, cosine decay)	32	COCO	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
EfficientNetB0 + Attention [18]	25	0.0001 (Adam)	32	ImageNet	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
Wavelet-CNN/MV2 [19]	25	0.0001 (Adam)	32	ImageNet	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
Faster-RCNN [20]	25	0.0001 (Adam)	16	COCO	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {8, 16, 32}
Standard ViT (no pruning)	25	0.0001 (Adam)	32	ImageNet-21k	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
SmartFire Vision (Proposed)	25	0.0001 (Adam)	32	ImageNet-21k + COCO (DETR)	Lowest val. loss	LR: {0.00005, 0.0001, 0.0005}, BS: {16, 32, 64}

Table 6. Hyperparameters for each baseline tuned via a grid search over learning rate.

Method	Epochs	Initial LR	Batch Size	Pretrained Weights	Model Selection	Grid Search Range
YOLOv5 [6]	25	0.001 (SGD, cosine decay)	32	COCO	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
YOLOv8 [15]	25	0.001 (SGD, cosine decay)	32	COCO	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
EfficientNetB0 + Attention [18]	25	0.0001 (Adam)	32	ImageNet	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
Wavelet-CNN/MV2 [19]	25	0.0001 (Adam)	32	ImageNet	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
Faster-RCNN [20]	25	0.0001 (Adam)	16	COCO	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {8, 16, 32}
Standard ViT (no pruning)	25	0.0001 (Adam)	32	ImageNet-21k	Lowest val. loss	LR: {0.0001, 0.001, 0.01}, BS: {16, 32, 64}
SmartFire Vision (Proposed)	25	0.0001 (Adam)	32	ImageNet-21k + COCO (DETR)	Lowest val. loss	LR: {0.00005, 0.0001, 0.0005}, BS: {16, 32, 64}

4.6. Ablation Study

To quantify the individual contribution of each architectural component of SmartFire Vision, a systematic ablation study was conducted by removing one component at a time from the full model and evaluating the resulting performance on the FURG test set. Five configurations were evaluated: the full SmartFire Vision model, the model without RIAH pruning (standard ViT encoder), the model without DETR (E-ViT only), the model without gated fusion (simple concatenation replacing the gating module), and the model without the probabilistic threshold mechanism (fixed 0.5 boundary). Ablation study results are presented in Table 7.

Table 7. Ablation study results showing the individual contribution of each component of SmartFire Vision.

Configuration	Accuracy	Recall	Precision	F1-Score	Inference (ms)
Full Model (SmartFire Vision)	0.9137	0.8527	0.8855	0.8664	14.2
w/o RIAH Pruning (standard ViT)	0.8812	0.8201	0.8644	0.8417	17.4
w/o DETR (E-ViT only)	0.8893	0.7941	0.8720	0.8312	9.8
w/o Gated Fusion (concatenation)	0.9001	0.8344	0.8712	0.8411	14.5
w/o Threshold Mechanism (θ = 0.5)	0.9011	0.8110	0.9002	0.8533	14.1

Table 7. Ablation study results showing the individual contribution of each component of SmartFire Vision.

Configuration	Accuracy	Recall	Precision	F1-Score	Inference (ms)
Full Model (SmartFire Vision)	0.9137	0.8527	0.8855	0.8664	14.2
w/o RIAH Pruning (standard ViT)	0.8812	0.8201	0.8644	0.8417	17.4
w/o DETR (E-ViT only)	0.8893	0.7941	0.8720	0.8312	9.8
w/o Gated Fusion (concatenation)	0.9001	0.8344	0.8712	0.8411	14.5
w/o Threshold Mechanism (θ = 0.5)	0.9011	0.8110	0.9002	0.8533	14.1

Based on the values shown in Table 7, when the RIAH pruning is removed, it results in a decrease in accuracy from an original total of 85.40% to only 88.12% and increases the per-frame inference time from an original value of 14.2 ms to 17.4 ms. This indicates that the pruning strategy produces the dual benefit of improving generalization and reducing computational costs. If DETR (the detection model that has previously been used) is removed altogether, then using only E-ViT as the detection method results in a larger degradation in recall accuracy from a previously established level of 85.43% to only 79.41% for this model, because it can no longer detect local fire instances that do not have strong global level signals with respect to their location. Additionally, replacing gated fusion with a simplistic concatenation of feature sources resulted in a decrease in F1 scores from 85.35% to only 84.11%. Therefore, the adaptive weighting of feature sources is more beneficial than the fixed approach to combining feature sources. Finally, if the learned threshold is removed by using a fixed boundary set at 0.5 for predicting detections, then this will produce an increase in precision at 90.02% along with a decrease in recall at 81.10%. This demonstrates that the purpose of the optimized threshold was to deliberately and appropriately bias the model to produce greater sensitivity in this application owing to its safety-critical nature, as shown in Table 8.

Table 8. Computational efficiency comparison of SmartFire Vision and baseline methods.

Method	FLOPs (G)	Params (M)	Inference (ms)	FPS	GPU Memory (GB)
YOLOv5 [6]	16.5	7.3	11.4	87.7	2.1
YOLOv8 [15]	14.2	11.1	12.8	78.1	2.4
Wavelet-CNN/MV2 [19]	0.31	3.4	7.6	131.6	0.7
Faster-RCNN [20]	180.0	41.8	48.3	20.7	6.8
Standard ViT	16.8	86.4	17.4	57.5	3.8
SmartFire Vision (Proposed)	13.7	72.1	14.2	70.4	3.2

Table 8. Computational efficiency comparison of SmartFire Vision and baseline methods.

Method	FLOPs (G)	Params (M)	Inference (ms)	FPS	GPU Memory (GB)
YOLOv5 [6]	16.5	7.3	11.4	87.7	2.1
YOLOv8 [15]	14.2	11.1	12.8	78.1	2.4
Wavelet-CNN/MV2 [19]	0.31	3.4	7.6	131.6	0.7
Faster-RCNN [20]	180.0	41.8	48.3	20.7	6.8
Standard ViT	16.8	86.4	17.4	57.5	3.8
SmartFire Vision (Proposed)	13.7	72.1	14.2	70.4	3.2

5. Discussion

The findings of the experimental evaluation of the SmartFire Vision system presented in Section 4, as a group, demonstrate that SmartFire Vision is a reliable and valid real-world video-surveillance-based fire and smoke detection system with high accuracy and low computational requirements. In addition to providing insight into the effectiveness of each supporting feature of the systems, the results from the evaluation provide a basis for making strong recommendations regarding potential deployment. The overall concept of deploying SmartFire Vision as a part of smart city surveillance infrastructure is shown in Figure 9.

In addition to producing the expected reduction in the number of floating-point operations (FLOPs), the RIAH head pruning approach produced other benefits, as demonstrated by the ablation study results. Specifically, the pruned model not only performed frame processing at a faster rate but also generalized better to the test set, resulting in the generalization gap (between the training and validation sets) being reduced from approximately 9 pp for the unpruned model to approximately 7 pp for SmartFire Vision. The generalization improvement has been attributed to the regularization impact of removing heads from the model; the removal of heads permits the model to depend on a smaller yet more complete set of attentions, which reduces the effective size of the encoder of the model, thereby dissuading the model from overfitting using training-specific patterns while retaining only the most useful diagnostic representations.

The results of the ablation experiment in Section 4.6 confirm that the impact of integrating DETR was primarily a gain in recall and that there was a minor reduction in precision. When DETR was removed from the system, recall dropped by 5.86 pp, whereas precision dropped by only 1.35 pp. The asymmetrical behavior is significant from an architectural perspective; the object query mechanism of DETR is designed to allow the detection of spatially defined objects, even when they are a very small percentage (e.g., <5%) of the total area of the frame. For example, in the nascent stages of fire detection, the flame may cover <5% of the pixels in the frame; therefore, the global patch embeddings from E-ViT may average the fire signal with the existing background representation. However, the localized bounding box predictions from DETR provide an explicit high-confidence detection signal in these difficult scenarios and lead to a gain in recall without introducing confounders.

Although SmartFire Vision currently demonstrates several strengths, when evaluating its training performance, as shown in Figure 3 and Figure 4, it also demonstrates a limited amount of overfitting within the training dynamics. Given this generalization gap and the single-dataset evaluation, the reported metrics should be interpreted as an upper bound on real-world performance until validated on independent, multi-source fire datasets. There is a 7-percentage-point gap between the training accuracy (95%) and validation accuracy (88%) at epoch 20, which indicates that the model has learned to represent the visual characteristics specific to the FURG dataset rather than learning the underlying generalizable visual characteristics of each fire. This is not surprising, given the relatively small training set (25,091 frames from 19 videos) and the high degree of representational capacity of the ViT-Base encoder, even after pruning. Strategies that future studies could consider for mitigating this may include more aggressive data augmentation, the use of dropout regularization within the classification head, and/or training with a larger and more diverse multi-source fire dataset that combines FURG with both the VisiFire and Corsican Fire benchmarks.

The second limitation relates to the dataset type. The FURG dataset mainly consists of outdoor fire and smoke sequences captured outdoors with good visibility. The performance of SmartFire Vision has not been tested indoors or in tunnels, industrial settings with competing thermal sources, or under rainy, foggy, or night-vision conditions. Real smart city surveillance networks experience many of the same very diverse and challenging conditions. Therefore, future studies should test the robustness of this model within a much wider range of environmental conditions and ensure that large-scale deployment is responsibly recommended.

SmartFire Vision uses a vision transformer-based architecture as the basis, which shows how the capabilities of transformer models can be extended to supporting tasks involving safety [21]. Specifically, SmartFire Vision’s transformer architecture encompasses the working principle of the attention mechanism of the transformer architecture along with the inherent scalability of self-attention when applied to image classification problems, where consideration of many small parts of the image constitutes a composite recognition process [21]. SmartFire Vision takes this further by integrating these concepts into an all-in-one object localization architecture that uses no anchor-based proposal or predetermined method of generating proposals for object localization [22]. Thus, the head-pruning approach taken with SmartFire Vision demonstrates that the number of attention heads associated with the attention mechanism used in this architecture is redundant, considering how many heads will statistically account for all potential proposal points [23]. The use of Grad-CAM to visualize the classification capabilities of the architecture provides a foundation for interpreting the weights given to each of the features used for image classification [24]. Attention-based encoder model structure-based experiment results also demonstrate the use of attention in classifier systems with respect to the dataset, considering the dimensionality of the data [25].

Similar developments in lightweight transformer architectures suitable for resource-constrained deployment, such as multimodal fusion systems that integrate distilled language encoders into small vision backbones, explainable transformer-based systems to perform sensitive sequential classification tasks, and multi-method explainability studies that use vision transformers to support affective computing, indicate the feasibility of using attention-based models in real operational real-time environments [26], where they can retain their original interpretability. In particular, current research on selective attention pruning in relation to the generalization benefits of the pruning method used in SmartFire Vision highlights the potential of applying a unified transformer-based system across multiple high-stakes classification applications, such as extremist activity detection and propaganda analysis, suggesting a high level of cross-domain maturity among these types of systems [27]. Overall, the above body of literature positions SmartFire Vision as part of an increasingly mature ecosystem of transformer-based systems that are relied upon to perform safety-critical, real-world inference tasks.

6. Conclusions

This study describes SmartFire Vision, a hybrid deep learning framework that detects fire and smoke in video sequences in real-time. This is the first framework of its kind to use E-ViT and DETR in tandem through a gated fusion mechanism. The E-ViT Encoder removes redundant attention heads by applying the RIAH method, which uses gradient-derived importance scores to discard unimportant attention heads; as a result, the number of FLOPs decreases by 18.3%, while generalization is improved. The DETR provides an additional layer of object-level spatial localization, greatly increasing the recall of early fire events. The gated feature fusion entity balances and distributes the global and local feature representations. Finally, a probabilistic thresholding classifier coupled with an alarm system provided an end-to-end fire detection system. SmartFire Vision has been evaluated on the FURG fire benchmark dataset consisting of 28,022 annotated frames; the overall accuracy is 85.40%; recall 85.43%; precision 85.33%; and F1 score 85.35%. SmartFire Vision significantly outperforms six different state-of-the-art baselines including YOLOv8, EfficientNetB0 and Faster-RCNN. The processing speed is 70.4 frames per second (FPS) on an NVIDIA GTX 1080 desktop GPU. Latency, memory usage, and throughput on embedded edge platforms (e.g., Jetson Nano/Orin) have not yet been evaluated and are identified as future work. All three principal components significantly affected the performance improvements observed in these experiments, both independently and with respect to one another: RIAH pruning, DETR integration, and gated fusion. Future research can be directed in three ways. First, we plan to extend RIAH pruning to the DETR decoder to decrease the end-to-end inference latency so that it can be deployed on edge devices, such as the NVIDIA Jetson Nano. Second, a temporal modeling module will be added to the system to propagate context from detections across a sequence of video frames, helping to improve the overall sensitivity to slow-moving smoke and reduce false positives in isolated frames. Third, SmartFire Vision will be integrated into a complete IoT-connected smart city architecture and tested in a real-life surveillance environment with various fire conditions, including indoors, industrial sites, and bad weather, to assess its readiness for real-world deployment.