Real-Time Analysis and Intervention of Classroom Behavior Using Multi-Modal Fusion and Spatiotemporal Context

Abstract

Analyzing classroom engagement is essential for developing effective smart learning environments. Conventional methods often face challenges in achieving reliable identification of individual students, accurately recognizing their behavioral states, and providing timely support. This paper presents a multimodal sensing and supportive feedback system built upon an end–edge–cloud collaborative architecture. By integrating RFID-based seat association, fingerprint verification, and computer vision-based activity analysis, the system establishes a reliable link between student identity and observed activities. Key computational tasks, including activity recognition, spatiotemporal context matching, and rule-based assessment, are executed locally on edge nodes. This enables low-latency, privacy-conscious feedback delivered via Bluetooth, effectively avoiding delays associated with cloud processing. Experimental results indicate that the proposed system significantly enhances both activity recognition accuracy and identity–behavior association reliability in typical classroom scenarios while substantially reducing the average feedback latency compared to traditional approaches.

1. Introduction

Students’ classroom behavior states serves as a key indicator for evaluating teaching effectiveness. It is also essential for enabling personalized instruction. These states play a significant role in improving classroom teaching quality and learning efficiency. Traditional classroom management relies heavily on teachers’ direct observation and experiential judgment. As a result, it is difficult to achieve continuous and objective monitoring of all students. With the advancement of smart education, automated analysis of classroom behaviors has gained increasing attention. Real-time feedback through information technology has become a key research direction in the digital transformation of education [1].

Research on classroom behavior recognition has been extensively conducted by scholars worldwide. Early studies primarily relied on wearable sensors [2]. Data were collected using devices such as accelerometers and gyroscopes. Although these methods achieved relatively high accuracy in recognizing specific actions, several limitations emerged. These included high equipment costs, student resistance to wearing devices, and difficulties in large-scale deployment. In recent years, computer vision-based methods have become the mainstream research direction [3]. Such approaches enable contactless behavior recognition by analyzing features such as student posture and facial orientation from surveillance videos. However, these methods face significant challenges in real classroom settings. First, environmental factors such as lighting variations, occlusion by front-row students, and small target sizes in the back rows degrade the accuracy of visual recognition models [4] specifically optimized YOLOv8 for small target detection in classroom scenarios, providing an important foundation for improving visual perception accuracy.

Furthermore, a systematic review on classroom behavior recognition also points out that current methods generally suffer from “anonymous recognition” and insufficient identity association, which severely limits their application in personalized instructional interventions [5].

Second, most existing methods produce “anonymous” behavior detection results. They lack reliable techniques to associate detected behaviors with specific student identities, making it difficult to support targeted instructional interventions. Furthermore, many systems adopt a cloud-based processing paradigm. Video data must be uploaded to remote servers for analysis. Consequently, there is a noticeable delay between the occurrence of a behavior and the teacher’s awareness of it. This latency fails to meet the requirements for real-time classroom interaction and timely management.

In summary, existing classroom behavior-monitoring technologies still face two critical challenges. The first is the accurate association between identity and behavior—how to reliably map visual recognition results to individual students. The second is the bottleneck of real-time feedback—how to reduce system response time to enable timely intervention. To address these issues, this paper proposes a real-time classroom behavior analysis method based on multimodal machine learning theory [6]. The proposed method integrates multimodal perception with collaborative edge intelligence. A dual-factor authentication mechanism combining radio frequency identification and biometric features is employed to establish a reliable “identity–seat” binding. An edge computing architecture is adopted to perform behavior recognition, spatiotemporal association, and rule-based judgment locally in the classroom. Low-power wireless communication is utilized to achieve real-time feedback at the hundred-millisecond level. Compared to existing approaches, the proposed method maintains high behavior recognition accuracy while significantly improving both identity association reliability and system responsiveness. This work provides a novel technical pathway toward building efficient and practical smart classroom behavior analysis systems.

2. System Architecture and Model Design

2.1. Overall System Architecture

The proposed system adopts a three-tier collaborative architecture consisting of a terminal layer, an edge layer, and a cloud layer, as illustrated in Figure 1. This architecture is designed to meet the requirements of real-time processing, privacy preservation, and scalability in classroom scenarios.

The terminal layer comprises student terminals and classroom cameras. Each student terminal integrates an RFID reader, a fingerprint sensor, a Bluetooth communication module, and a feedback execution unit. The feedback execution unit consists of a vibration motor and a display screen. Classroom cameras capture video streams at a fixed frame rate of 15 fps and transmit them to the edge layer via wired networks.

The edge layer is deployed on edge computing nodes within the classroom and undertakes the core computational tasks. This edge computing architecture brings computational capabilities close to the data source, thereby overcoming the latency bottlenecks associated with cloud-based processing [7]. Recent surveys on human activity recognition also emphasize that real-time performance and edge deployability are becoming important research trends [8]. In addition, the complementarity between visual and wireless signals has been validated in identity recognition tasks, which aligns with the multimodal design philosophy of the proposed system [9]. The edge layer incorporates a behavior recognition model, a spatiotemporal association module, a credential management module, and a rule engine. The edge layer receives video streams and identity credentials from the terminal layer, performs behavior recognition, identity association, and rule-based judgment, and then generates feedback instructions. These instructions are sent directly to the corresponding student terminals via Bluetooth.

The cloud layer consists of remote servers responsible for persistent data storage, offline analysis, and visual presentation. The cloud layer uses an encrypted database to store behavioral event logs persistently. Data transmitted to the cloud are encrypted using TLS 1.3 to ensure confidentiality and integrity during transmission. For storage, sensitive fields (e.g., student names and IDs) are encrypted at the field level using AES-256, and the entire database is protected by Transparent Data Encryption (TDE) against unauthorized physical access. Access control follows a Role-Based Access Control (RBAC) model; only authorized teachers and researchers can view raw data. All access operations are logged for auditing and security review. Logs are retained according to the school’s data management policy (default one year), after which they are automatically and securely deleted to ensure long-term compliance with student privacy requirements. The edge layer asynchronously uploads records of associated events to the cloud, enabling teachers to conduct instructional analysis and historical reviews.

2.2. Multimodal Identity–Seat Binding Mechanism

Reliable binding between student identity and seat location serves as the foundation for establishing behavior–identity associations. The proposed system employs a dual-factor authentication mechanism combining Radio Frequency Identification (RFID) and fingerprint biometrics. Authentication is performed locally on the terminal to ensure uniqueness, real-time responsiveness, and privacy security of identity binding.

The binding process begins when a student presses the fingerprint sensor after taking a seat. Fingerprint verification utilizes a 1:1 local matching algorithm based on minutiae features [10]. Let the registered template be represented as a feature set:

$$M_{\mathrm{reg}}=\left\{(x_i,y_i,{\theta }_i,t_i)\right\}$$

(1)

where $(x_i,y_i)$ denotes the minutiae coordinates, ${\theta }_i$ represents the orientation angle, and $t_i\in \{\mathrm{ending}\mathrm{point}, \mathrm{bifurcation}\mathrm{point}\}$ indicates the minutiae type.

Let the real-time acquired feature set be $M_{\mathrm{in}}$. The matching score $S_{\mathrm{match}}$ is computed by measuring the similarity between the two feature point sets under spatial and orientation constraints:

$$S_{\mathrm{match}}=\sum _i\sum _j{w}_{ij}·exp\left(-\alpha ·d_{ij}^2\right),d_{ij} = \parallel (x_i,y_i)-(x_j,y_j){\parallel }_2+\beta ·|{\theta }_i-{\theta }_j|$$

(2)

where $\alpha$ and $\beta$ are weighting parameters. If $S_{\mathrm{match}}>\tau$ (system threshold $\tau =0.75$), fingerprint verification is considered successful.

Upon successful verification, the terminal activates the RFID reader to retrieve the unique identifier SeatID from the passive tag embedded beneath the seat. The reliability of RFID localization is enhanced through distance estimation based on the received signal strength indicator (RSSI), combined with filtering using a path loss model [11]. Recent advances in wireless sensing, such as RIS-assisted WiFi sensing [12], demonstrate the potential for enhancing wireless signal perception in complex environments, which is conceptually related to the RFID localization enhancement in this work.

$$\mathrm{RSSI}\left(d\right)=P_{\mathrm{tx}}-P{L}_0-10\gamma ·{log}_{10}(d/d_0)+X_{\sigma }$$

(3)

where $P_{\mathrm{tx}}=20\mathrm{dBm}$ denotes the transmission power, $P{L}_0=40\mathrm{dB}$ represents the path loss at reference distance $d_0=1\mathrm{m}$, $\gamma =2.2$ is the path loss exponent, and $X_{\sigma }\sim \mathcal{N}(0,4)$ accounts for shadow fading. A Kalman filter is applied to smooth the RSSI sequence, achieving a seat recognition accuracy of 99.8%.

By combining the student ID $S_{\mathrm{ID}}$, SeatID, and current timestamp t, encrypted with a pre-shared 256-bit symmetric key K, the terminal then constructs a time-sensitive binding credential C:

$$C={\mathrm{Encrypt}}_{\mathrm{AES}\text{-}256\text{-}\mathrm{GCM}}\left(S_{\mathrm{ID}}\Vert \mathrm{SeatID}\Vert t\Vert H(S_{\mathrm{ID}}\Vert \mathrm{SeatID}\Vert t\Vert K)\right)$$

(4)

where $H(·)$ denotes the SHA-256 hash function, ensuring message integrity. The encrypted credential is broadcast to the edge node via Bluetooth Low Energy 5.0 (BLE 5.0).

The edge node maintains a credential cache, structurally implemented as a key value store $\mathrm{Cache}\left[\mathrm{SeatID}\right]=(S_{\mathrm{ID}},t)$. Each credential is assigned a validity window of $T_{\mathrm{w}}=300\mathrm{s}$. The system periodically invalidates expired credentials where $t_{\mathrm{current}}-t>T_{\mathrm{w}}$, and employs an LRU policy for cache management. This ensures that identity–seat binding states remain synchronized with students’ actual physical positions, providing accurate contextual information for subsequent spatiotemporal association.

During this process, fingerprint templates themselves are not uploaded to the edge or cloud. On the terminal device, registered fingerprint feature templates are encrypted using the same AES-256 algorithm and stored in a local secure storage area. The encryption key is derived from the device’s unique serial number and the student ID via a hash function. All fingerprint verification is performed locally on the terminal; only the matching result (success or failure) is output, without disclosing any raw biometric data. This design ensures the privacy and security of biometric data, adhering to the principle of data minimization in edge computing scenarios, which echoes the cross-modal disentanglement idea in multimodal learning [13].

2.3. Spatiotemporal Association Matching

To establish accurate mapping between anonymous behavior detections from vision-based analysis and identified students, this paper proposes a spatiotemporal association matching algorithm based on coordinate mapping and time window validation. The core of this algorithm lies in constructing a transformation from image pixel coordinates to physical classroom seats, followed by joint validation incorporating the temporal validity of identity binding. The cross-modal feature alignment approach demonstrated in works such as MaskFi [14] and the dual-stream shared embedding space in ViFi-ReID [15] provide theoretical support for the multimodal identity association in this study.

During system deployment, a predefined calibration procedure is performed to establish the corresponding rectangular pixel region $R_k$ in the camera view for each physical seat ${\mathrm{SeatID}}_k$. To account for perspective distortion, an affine transformation is applied to map the physical coordinates $(X_k,Y_k)$ of each seat layout to image coordinates $(u_k,v_k)$, generating a mapping table:

$$M=\{{\mathrm{SeatID}}_k\to R_k\},k=1,2,\dots ,K$$

(5)

where each $R_k$ is defined by its four vertex coordinates.

When the visual detection module outputs a behavior bounding box $B_i=(x_{min},y_{min},$$x_{max},y_{max})$, the Intersection over Union (IoU) between $B_i$ and each calibrated region $R_k$ is computed [16]:

$$\mathrm{IoU}(B_i,R_k)={\displaystyle \frac{|B_i\cap R_k|}{|B_i\cup R_k|}}$$

(6)

IoU is a standard metric for evaluating localization accuracy in object detection.

The seat region with the highest IoU value is selected as the most likely physical location ${\mathrm{SeatID}}_{\mathrm{match}}$ where the behavior occurred. To resolve ambiguity when a bounding box lies near the boundary between two adjacent seats, a threshold ${\mathrm{IoU}}_{\mathrm{th}}=0.3$ is set. A match is considered valid only when $\mathrm{IoU}>{\mathrm{IoU}}_{\mathrm{th}}$.

Once the physical seat of the detected behavior is determined, the algorithm queries the credential cache maintained at the edge node for a valid binding credential matching ${\mathrm{SeatID}}_{\mathrm{match}}$ within the time window $T_{\mathrm{w}}$. A successful match requires that a cache entry $\mathrm{Cache}\left[{\mathrm{SeatID}}_{\mathrm{match}}\right]$ exists and satisfies $t_{\mathrm{current}}-t\le T_{\mathrm{w}}$. If found, a successful association is established, generating an identified behavior event record E:

$$E=\langle S_{\mathrm{ID}},\mathrm{BehaviorType},{\mathrm{SeatID}}_{\mathrm{match}},t\rangle$$

(7)

In actual classroom scenarios, a student leaning forward or sideways may cause their detection box to overlap with the calibrated region of an adjacent seat. To handle such spatial ambiguity, the algorithm adds the following processing on top of the basic IoU matching. First, a match is triggered only when the maximum IoU exceeds the threshold ${\mathrm{IoU}}_{\mathrm{th}}=0.3$; otherwise, the behavior is treated as anonymous. Second, if multiple seat regions have IoU values exceeding the threshold and their differences are less than 0.1, the algorithm preferentially selects the seat that has a valid binding credential within the current time window $T_{\mathrm{w}}$. If multiple candidates are still valid, the one with the highest consistency based on historical association records (frequency of successful seat matches for each student in the last 30 s) is chosen. Finally, a Kalman filter is applied to the IoU sequence over consecutive frames, and a sliding window of 5 frames is used for majority voting; the majority result is taken as the final association, avoiding mismatches caused by single-frame jitter.

This process transforms anonymous visual detection results into identified actionable events, addressing the core challenge of identity association. If no valid credential is found, the detection is treated as an unassociated anonymous event, used only for aggregate statistical purposes without triggering individualized feedback.

2.4. Rule Engine and Feedback Mechanism

Previous studies have demonstrated that timely feedback adapted to individual differences is more effective in promoting learning engagement than fixed-rule strategies [17]. Conventional rule-based feedback approaches fail to accommodate student variability, often leading to alert fatigue or insufficient intervention. The dynamic feedback-driven learning optimization framework proposed by Maier et al. further confirms that personalized feedback based on real-time learning behaviors significantly enhances student course participation and sustained learning motivation [17]. Accordingly, this paper designs an adaptive feedback rule engine based on individual response history.

First, the system maintains a dynamic response rate $R_i$ for each student, representing their compliance level to historical feedback:

$$R_i={\displaystyle \frac{N_{\mathrm{response}}^{\left(i\right)}}{N_{\mathrm{total}}^{\left(i\right)}}}$$

(8)

where $N_{\mathrm{total}}^{\left(i\right)}$ denotes the total number of feedback instances received by student i, and $N_{\mathrm{response}}^{\left(i\right)}$ represents the number of instances where behavioral improvement occurred within an observation window $T_{\mathrm{obs}}=60\mathrm{s}$. The response rate is updated upon each new feedback instance:

$$R_i^{\left(\mathrm{new}\right)}={\displaystyle \frac{N_{\mathrm{response}}^{\left(i\right)}+I_{\mathrm{responded}}}{N_{\mathrm{total}}^{\left(i\right)}+1}}$$

(9)

where $I_{\mathrm{responded}}$ is an indicator function that equals 1 if a response is observed and 0 otherwise.

Based on $R_i$, the interval $T_{\mathrm{interval}}$ until the next feedback is dynamically adjusted:

$$T_{\mathrm{interval}}=T_{\mathrm{base}}·\left[1+\gamma ·(1-R_i)\right]$$

(10)

where $T_{\mathrm{base}}$ denotes the baseline interval (e.g., 3 min), and $\gamma$ is an adjustment coefficient (default value 2.0). This formulation implements a personalized scheduling strategy: higher responsiveness leads to shorter intervals, while lower responsiveness results in longer intervals. When $R_i=1$, $T_{\mathrm{interval}}=T_{\mathrm{base}}=3$ min; when $R_i=0$, $T_{\mathrm{interval}}=T_{\mathrm{base}}\times (1+2)=9$ min. This achieves a balance between timely correction and avoidance of unnecessary disruption.

Simultaneously, feedback intensity is dynamically graded. The feedback intensity level $L\in \{1,2,3\}$ is determined based on behavior confidence score $C_f$ and the number of consecutive non-responses $N_{\mathrm{consecutive}}^{\left(i\right)}$:

• $L=1$ (Mild): Applied when confidence is low ($C_f<0.7$) or for first-time anomalies, triggering a brief vibration (200 ms).
• $L=2$ (Moderate): Applied when confidence is moderate ($C_f\ge 0.7$) or response rate is low ($R_i\le 0.6$), triggering vibration and a 5 s on-screen prompt.
• $L=3$ (Severe): Applied when confidence is high ($C_f\ge 0.9$) and consecutive non-responses occur ($N_{\mathrm{consecutive}}^{\left(i\right)}\ge 2$), triggering strong vibration, an on-screen alert, and event reporting to the teacher interface for manual attention.

Based on the rationale above, a finite state machine (FSM) is adopted to model this closed-loop feedback process [18]. FSM is widely used in behavior recognition and intervention systems for managing complex state transition logic.

• $S_0$ (Normal Monitoring): Continuous behavior monitoring; no anomalies detected.
• $S_1$ (First Feedback): Anomaly detected ($C_f\ge \theta$); first feedback instruction sent according to intensity level L.
• $S_2$ (Response Success): Behavioral improvement detected within $T_{\mathrm{obs}}$; update $R_i$, set $N_{\mathrm{consecutive}}^{\left(i\right)}\leftarrow 0$, and return to $S_0$.
• $S_3$ (Retry Waiting): No improvement detected within $T_{\mathrm{obs}}$; wait for adaptive interval $T_{\mathrm{interval}}$, then increment $N_{\mathrm{consecutive}}^{\left(i\right)}\leftarrow N_{\mathrm{consecutive}}^{\left(i\right)}+1$, and trigger next feedback cycle based on updated parameters (return to $S_1$).
• $S_4$ (Manual Intervention): When $N_{\mathrm{consecutive}}^{\left(i\right)}\ge 3$, the system determines that automated intervention is ineffective, automatically notifies the teacher interface, and enters manual processing. After manual handling, return to $S_0$.

Transitions between states are precisely triggered by the conditions described above, forming a complete closed loop of perception–decision–feedback–evaluation–adaptation. This ensures the system’s adaptivity, robustness, and controllability.

3. Experimental Results and Analysis

The experimental setup strictly simulates real classroom scenarios, with parameters configured according to the multimodal binding and feedback mechanism proposed in this study to ensure reproducibility. The hardware platform consists of the following components: an edge server deployed locally in the classroom, specifically an NVIDIA Jetson Xavier NX (NVIDIA Corporation, Santa Clara, CA, USA) with 8 GB memory; customized student terminals integrating an ESP32-S3 main controller (Espressif Systems, Shanghai, China), an FPM10A fingerprint sensor (Hangzhou Grow Technology Co., Ltd., Hangzhou, China), and an MFRC522 RFID reader (NXP Semiconductors, Eindhoven, The Netherlands); and an Orbbec Gemini 335L binocular depth camera (Orbbec, Shenzhen, China) serving as the perception unit, which simultaneously outputs 1080P RGB video streams and depth information.

On the software side, the system is built upon the PyTorch 1.12.0 framework. The core behavior recognition model adopts a YOLOv8n architecture enhanced with a squeeze-and-excitation (SE) attention module [4], with an input resolution of $640\times 640$. To improve inference efficiency at the edge, the model is ultimately deployed using TensorRT 8.6.1 (NVIDIA Corporation) with FP16 quantization.

The core algorithm parameters directly reflect the design principles of the proposed method. The validity window $T_{\mathrm{w}}$ for identity binding credentials is set to 300 s, achieving an optimal balance between binding success rate and temporal validity. The vision-dominant coefficient $\beta$ in the vision–inertial fusion confidence calculation is configured as 0.6. The observation window $T_{\mathrm{obs}}$ for feedback effectiveness is set to 60 s. The baseline interval $T_{\mathrm{base}}$ for adaptive feedback scheduling is 180 s, with an adjustment coefficient $\gamma =2.0$.

The experimental dataset employed in this study is a self-constructed classroom behavior dataset [4]. It consists of 720 annotated video segments captured in a simulated classroom environment, spanning six behavior categories: “listening”, “hand raising”, “reading”, “writing”, “using mobile phone”, and “sleeping on desk”. The dataset is partitioned into training, validation, and test sets with a ratio of 8:1:1 to ensure objective evaluation. Multimodal datasets that integrate wireless and visual modalities, such as RF-Behavior [19], provide valuable references for the construction of such datasets.

To comprehensively assess system performance, the following quantitative metrics are adopted. Among these, ASR and AFD serve as the core innovative indicators:

• Behavior Recognition Accuracy (BRA): The percentage of correctly classified behavior samples by the model on the test set.
• Identity Association Success Rate (ASR): Calculated as $\mathrm{ASR}={\displaystyle \frac{N_{\mathrm{match}}}{N_{\mathrm{total}}}}\times 100\%$, where $N_{\mathrm{match}}$ denotes the number of detected behaviors successfully associated with the correct student, and $N_{\mathrm{total}}$ represents the total number of detected behaviors.
• Average Feedback Delay (AFD): A key metric for evaluating the closed-loop performance of the proposed system. It is defined as the average time elapsed from the moment a behavior occurs (captured by the camera) to the moment feedback is executed (vibration or display) on the student terminal.
• Intervention Response Rate (IRR): The proportion of interventions that result in behavioral improvement within the observation window $T_{\mathrm{obs}}$, calculated as the number of improved instances divided by the total number of interventions.

3.1. Overall Performance Comparison

To evaluate the effectiveness of the proposed system, we compare it with two representative approaches from the recent literature. Approach A is a full-link intelligent tutoring system that similarly implements a closed loop of perception, analysis, decision making, and feedback [20]. It serves as a baseline to validate the efficiency of our system architecture. Approach B is MD-GCN [21], a behavior recognition model that performs only the recognition task. It is adopted as a baseline to calibrate the upper bound of perception module performance. The comparative results are presented in Figure 2 and Figure 3, with detailed metrics in

Table 1. Performance comparison between the proposed system and baseline Approach A and Approach B.

Method	BRA (%)	ASR (%)	AFD (ms)	IRR (%)
Approach A	86	—	670 ± 150	—
Approach B	91.7	—	—	24
Proposed Method	93.2	98.3	232 ± 45	73.8

.

In comparison with Approach A, which also achieves a full-link closed loop, the proposed system demonstrates superior efficiency. Despite undertaking additional complex tasks including real-time identity binding and spatiotemporal matching, our system achieves an average feedback delay (AFD) of $232\pm 45$ ms, significantly outperforming the $670\pm 150$ ms reported for Approach A. This result validates the high efficiency of the proposed edge computing architecture.

Regarding basic recognition performance, the proposed system achieves an accuracy of 93.2%, surpassing both Approach A (86%) and Approach B (91.7%). This validates the superiority of the perception module. Furthermore, the intervention response rate of 73.8% provides empirical evidence supporting the effectiveness of the proposed identity-based personalized closed-loop feedback mechanism from an application perspective.

More importantly, the proposed system achieves a behavior recognition accuracy of 93.2% and an identity association success rate of 98.3%, representing substantial progress in addressing the challenge of “anonymous recognition”. This contribution is not addressed by either of the baseline systems.

To further demonstrate the advantages of the proposed method over a broader range of existing works,

Table 2. Performance comparison between the proposed method and representative existing works.

Method/System	Reference	BRA (%)	Identity Association	AFD (ms)
Wearable sensor	[2]	≈85	Yes	High
Vision-based	[3]	≈90	No	Cloud dependent
Approach A	[20]	86.0	No	670 ± 150
Approach B	[21]	91.7	No	Not reported
Proposed method	—	93.2	Yes (98.3%)	232 ± 45

extends the comparison across wearable methods, vision-based methods, and recent representative systems in terms of behavior recognition accuracy, identity association capability, and feedback latency.

As shown in Table 2, wearable methods can achieve identity binding, but their recognition accuracy is relatively low (about 85%) and feedback latency is high, making them unsuitable for large-scale classroom applications. Vision-based methods achieve reasonable recognition accuracy (about 90%) but lack identity association capabilities and rely on cloud processing. Recent approaches such as the full-link intelligent tutoring system and MD-GCN have improved accuracy but still do not address identity association, and their feedback latency is much higher than that of the proposed method. In contrast, the proposed method achieves significant advantages across all three key metrics: behavior recognition accuracy of 93.2%, identity association success rate of 98.3%, and feedback latency reduced to $232\pm 45$ ms. It is the only complete solution that simultaneously meets the requirements of high-precision recognition, reliable identity binding, and real-time feedback.

3.2. Analysis of Key Parameter Effects and Robustness

System performance is jointly influenced by two key parameters: the identity binding time window $T_{\mathrm{w}}$ and the behavior–seat Intersection over Union threshold IoU. Figure 4 illustrates the effect of $T_{\mathrm{w}}$ on identity association success rate (ASR) and false positive rate (FPR). The results indicate that ASR increases rapidly with $T_{\mathrm{w}}$ before gradually plateauing, reaching its peak at approximately 300 s. In contrast, FPR exhibits a slow but steady increase as $T_{\mathrm{w}}$ extends. Excessively short windows cause valid bindings to expire prematurely, reducing association success rate. Conversely, excessively long windows accumulate outdated information, increasing the risk of mismatches. Accordingly, $T_{\mathrm{w}}$ is set to 300 s in this study to achieve an optimal balance between these competing factors.

Environmental adaptability is crucial for practical deployment of the proposed system. Figure 5 compares the performance retention rates of the proposed method and the cloud-based baseline under various interference conditions. Under typical classroom disturbances such as simulated network latency, lighting variations, and occlusion by front-row students, the proposed edge-based processing architecture demonstrates significantly stronger robustness. Notably, when 100 ms network latency is introduced, the performance retention rate of the cloud-based solution drops sharply to 76.3%. In contrast, the proposed method retains 99.8% of its original performance, as all critical computations are performed locally. This result strongly validates the advantage of the edge computing architecture in mitigating network uncertainty. The challenge of sensor heterogeneity and modality alignment discussed in the HAR survey [22] further underscores the importance of robust edge processing in real-world deployments.

3.3. Ablation Study

The proposed system mainly consists of three core modules: (1) a visual perception enhancement module that introduces an SE attention mechanism into YOLOv8n to improve behavior recognition accuracy; (2) a dual-factor identity binding module that integrates fingerprint recognition and RFID localization to establish reliable student–seat associations; and (3) a spatiotemporal association matching module that maps visual detection results to identified students through coordinate mapping and time window validation.

To quantitatively evaluate the contribution of each module to system performance, an ablation study was designed by incrementally adding core modules and observing changes in Behavior Recognition Accuracy (BRA) and Identity Association Success Rate (ASR). The experimental settings are as follows:

• Experiment 1: Baseline YOLOv8n model used for behavior recognition only, without the SE attention module, identity binding mechanism, or spatiotemporal association matching.
• Experiment 2: The SE attention module was added to Experiment 1, with the other configurations unchanged, to verify the improvement in BRA from the SE module.
• Experiment 3: A dual-factor identity binding mechanism (fingerprint + RFID) was added to Experiment 2, but without the spatiotemporal association matching algorithm; only simple centroid-based position matching using seat region centers was performed.
• Experiment 4: The complete proposed system, including the SE attention module, dual-factor identity binding mechanism, and spatiotemporal association matching algorithm.

The ablation results are shown in Figure 6.

As seen in Figure 6, comparing Experiment 1 and Experiment 2, the SE attention module increases BRA from 90.6% to 92.3%, an improvement of 1.7 percentage points, verifying its effectiveness in enhancing feature representation under complex classroom environments. Comparing Experiment 3 and Experiment 4, adding spatiotemporal association matching increases ASR from 86.4% to 98.3%, an improvement of 11.9 percentage points, indicating that the IoU-based spatiotemporal matching algorithm with time window validation effectively filters out mismatches and significantly improves identity association reliability. Comparing Experiment 2 and Experiment 3, the introduction of the dual-factor identity binding mechanism does not affect BRA, demonstrating good decoupling between the identity binding module and the visual perception module. Notably, Experiment 4 (complete system) achieves a BRA of 93.2%, a further 0.9 percentage point improvement over Experiment 2 (92.3%). This suggests that although the identity binding and spatiotemporal association modules do not directly participate in behavior classification, they indirectly reduce behavioral misclassification caused by identity confusion by providing more reliable identity context, reflecting the synergistic gain of multimodal fusion.

3.4. Comprehensive Evaluation

Based on the experimental results presented above, the proposed multimodal perception and edge intelligence collaborative approach demonstrates significant advantages across multiple dimensions, including behavior recognition accuracy, identity association success rate, real-time feedback latency, and environmental robustness. The experimental findings validate the effectiveness of the proposed solutions to the two core challenges identified in the Introduction. First, accurate association between identity and behavior is achieved through multimodal binding and spatiotemporal matching algorithms. Second, rapid feedback at the hundred-millisecond level is realized through the edge computing architecture. These results provide solid technical support for constructing truly feasible, reliable, and efficient smart classroom behavior analysis systems.

4. Discussion

The purpose of real-time monitoring in this system is twofold. First, by analyzing individual student behavioral states (e.g., listening, writing, using a mobile phone, sleeping on desk), the system can send tactile or visual feedback to the student’s wristband immediately upon detecting off-task behavior, helping the student adjust their state in a timely manner, thereby effectively improving learning efficiency. At the same time, the automated behavior monitoring and alerting functions replace much of the teacher’s classroom monitoring and verbal reminders, significantly reducing the teacher’s workload in managing classroom discipline during lectures, allowing the teacher to focus more on the instructional content itself. Second, the long-term aggregated anonymized classroom behavior data (e.g., the curve of class-wide attention over time, the frequency of different types of off-task behaviors) can indirectly reflect the impact of different teaching rhythms and methods on student attention. Teachers can use this data for teaching reflection and optimization of instructional design, such as adjusting the time allocation between lecturing and interaction, identifying classroom segments that are prone to student distraction, etc. It should be emphasized that these data are used only for teaching improvement and teacher self-development, and are not used as a basis for teacher performance evaluation.

In terms of technical limitations, the coverage of the system is constrained by the physical characteristics of the Orbbec Gemini 335L camera and the edge computing power. The ideal working distance of the camera is 0.5 m to 6 m; beyond 5 m, joint confidence decreases, and recognition accuracy drops by about 15%. With a horizontal field of view of 90°, a single device can effectively cover about 30–40 students. When the number of students exceeds 40 or the distance is too large, the real-time inference latency of the Jetson Xavier NX edge node in 15 W mode increases, making it difficult to meet the hundred-millisecond feedback requirement. Regarding occlusion handling, the horizontal spacing between adjacent students should be no less than 0.3 m; smaller spacing leads to highly overlapping detection boxes, causing ambiguity in IoU calculation. The proposed method alleviates crowding-induced mismatches through the IoU threshold of 0.3 and the historical seat temporal consistency mechanism described in Section 2.3. However, when severe vertical occlusion occurs between front and back rows (e.g., head occlusion), the extraction of upper-body joint points for students in the back row is still impaired, which is an inherent limitation of single-view solutions. In addition, the system mainly recognizes seated upper-body behaviors; its recognition accuracy for large movements such as standing up or walking around is relatively low. Recent works utilizing WiFi CSI for behavior recognition in occluded environments [23] suggest a promising direction for future integration to address occlusion challenges.

From an engineering deployment cost perspective, the hardware cost for a single classroom (configured for 40 students) is approximately USD 2220 (about RMB 16,000). This mainly includes: one Jetson Xavier NX edge node (approx. USD 450), one Orbbec Gemini 335L depth camera (approx. USD 300), 40 student terminal wristbands (approx. USD 35 each, totaling USD 1400), and 40 RFID passive seat tags (USD 0.5 each, totaling USD 20). Based on this estimate, a typical university with a budget of RMB 1 million could equip about 60 classrooms. The system is suitable for smart classroom retrofitting and classroom teaching quality monitoring scenarios.

In summary, the core positioning of this system is as an assistive tool: it serves both students’ behavioral self-regulation and learning efficiency improvement, and teachers’ instructional optimization and management burden reduction. Future work will focus on multi-camera collaborative deployment to expand coverage, introducing self-supervised learning to improve occlusion robustness, and exploring finer-grained student state recognition methods such as affective computing.

5. Conclusions

This paper addresses two fundamental challenges in classroom behavior monitoring: the difficulty of associating student identities with their behaviors, and the high latency of system feedback. A novel solution is proposed by integrating radio frequency identification, biometric features, and computer vision technologies to establish a reliable identity binding mechanism based on multimodal perception. Furthermore, an edge collaborative computing architecture is introduced to implement a closed-loop processing pipeline encompassing behavior perception, spatiotemporal matching, and real-time decision-making and feedback, all performed locally. This approach fundamentally circumvents the latency bottlenecks caused by network transmission. Experimental validation demonstrates that the proposed methodology effectively enhances the reliability of associating visual recognition results with identified students, while achieving real-time interaction capabilities at the hundred-millisecond level. This study presents a feasible and reliable technical solution for constructing smart classroom behavior analysis systems, offering clear practical value for advancing intelligent management and timely intervention in the teaching process. Future extensions could incorporate privacy-preserving distributed training frameworks such as FedTime-MAGNET [24] to further enhance system scalability, aligning with the emerging trend of real-time and deployable HAR systems [8].