Multi-Target Irregular Behavior Recognition of Chemical Laboratory Personnel Based on Improved DeepSORT Method

Abstract

The lack of safety awareness and the irregular behavior of chemical laboratory personnel are major contributors to laboratory accidents which pose significant risks to both the safety of laboratory environments and the efficiency of laboratory work. These issues can lead to accidents, equipment damage, and jeopardize personnel health. To address this challenge, this study proposes a method for recognizing irregular behavior in laboratory personnel by utilizing an improved DeepSORT algorithm tailored to the specific characteristics of a chemical laboratory setting. The method first extracts skeletal keypoints from laboratory personnel using the Lightweight OpenPose algorithm to locate individuals. The enhanced DeepSORT algorithm tracks human targets and detects the positions of the relevant objects. Finally, an SKPT-LSTM network was employed to integrate tracking data for behavior recognition. This approach was designed to enhance the detection and prevention of unsafe behaviors in chemical laboratories. The experimental results on a self-constructed dataset demonstrate that the proposed method accurately identifies irregular behaviors, thereby contributing to the reduction in safety risks in laboratory environments.

1. Introduction

In recent years, chemical laboratory safety accidents have occurred frequently, resulting in significant loss of life and property. University chemical laboratories are equipped with glassware, chemical reagents, gas cylinders, reactors, and other experimental equipment that poses more safety hazards than corporate factories [1]. Since 2000, China has experienced 113 chemical laboratory accidents resulting in 99 fatalities. Studies have found that human factors are the primary causes of these accidents [2,3], among which irregular behavior is one of the key factors leading to these accidents [4]. Many scholars have proposed strategies to enhance chemical laboratory safety from multiple perspectives [4,5,6], primarily focusing on formulating comprehensive management regulations and enhancing safety awareness among chemical laboratory personnel. However, the implementation of these measures often faces challenges due to their time-consuming and inefficient nature.

With continuous improvements in computer hardware, surveillance equipment, and algorithms, numerous methods for human behavior recognition have emerged. Many scholars have applied machine-learning-based methods to unsafe behavior detection tasks, significantly improving the efficiency of safety management. For example, Ding et al. [7] (2018) developed a method for detecting unsafe behaviors in construction sites and other work activities; Hung et al. [8] (2021) proposed a system for identifying dangerous behaviors of workers using transfer learning methods; and Niu et al. [9] established a classification framework for unsafe driving behaviors of truck drivers. However, there is a relative lack of learning-based methods for identifying unsafe behaviors of chemical laboratory personnel. Currently, human behavior recognition methods can be roughly divided into two categories: manual feature extraction and feature extraction based on deep-learning networks. In general, a method based on behavioral feature design is simpler. This process involves extracting action features from each frame of the video, encoding them into feature vectors, and then using machine-learning algorithms for classification. Bobick et al. [10] (2001) analyzed the differences in contour features between video frames and constructed momentum and motion history maps based on them for behavior recognition. Yang et al. [11] (2014) used data from RGBD cameras to capture 3D depth data of skeleton joints for human movement recognition. Willems et al. [12] (2008) introduced a spatiotemporal interest point detection method capable of detecting the location of spatiotemporal interest points in video 3D data at multiple scales. Wang et al. [13] (2009) proposed a spatiotemporal interest point extraction method based on dense sampling to effectively improve the accuracy of behavioral recognition. Subsequently, Wang et al. [14] (2011) proposed a dense trajectories (DT) extraction method to obtain more detailed video motion information by extracting dense trajectories, including Histograms of Oriented Gradient (HOG), Histograms of Oriented Optical Flow (HOF) and Motion of Boundary History (MBH) features. Finally, Wang et al. [15] (2013) proposed an improved dense trajectory (iDT) method to further enhance recognition accuracy by fusing features such as HOG, HOF, and MBH, as well as optimizing optical flow images, feature normalization, and coding methods. Gao et al. [16] (2015) proposed a multidimensional human action recognition model based on image sets and group sparsity which improved action recognition performance by extracting dense trajectory features, constructing a shared codebook, and utilizing group sparsity on multi-view action datasets. Azher et al. [17] (2017) introduced an adaptive local motion descriptor (ALMD) to analyze static textures and generate persistent codes to describe local textures and finally used random forests to recognize human motion. Weng et al. [18] (2018) proposed a method to achieve effective and compact action representation using Length Variable Edge Trajectories (LV-ET) and Spatiotemporal Motion Skeletons (STMS) to address the challenges posed by camera motion, cluttered backgrounds, and occlusion in unconstrained videos. Khursheed et al. [19] (2019) proposed the MtFR method to fuse multiple types of features and reduce irrelevant features and then used the SVM classifier for behavior recognition. Franco et al. [20] (2020) took full advantage of the potential of the Kinect sensor and designed a method that combined skeleton and RGB data stream analyses to identify robust activities.

Behavior recognition algorithms based on deep learning primarily employ end-to-end detection methods and training models to directly extract features from input videos and identify and classify behaviors. These methods commonly utilize two-stream convolutional neural networks (CNNs), three-dimensional (3D) convolutional neural networks, and long- and short-term memory (LSTM) networks. The two-stream convolutional neural network, introduced by Simonyan et al. [21] (2014), learns features from both temporal and spatial streams, effectively addressing the neglect of temporal stream features in traditional methods. In 2012, Ji et al. [22] proposed a 3D convolutional neural network (C3D) that efficiently fused temporal and spatial feature dimensions for video behavior recognition classification. However, a substantial number of parameters in C3D require significant computational resources and video memory. To mitigate this, Kondratyuk et al. [23] (2021) proposed a three-step approach to improve computational efficiency. Additionally, Wang et al. [24] (2016) introduced the time-domain segmentation network (TSN) in 2016, enhancing the model’s capability to process long-duration videos by recognizing human behaviors through time-domain segmentation and random sampling of videos. Donahue et al. [25] (2015) presented a long recurrent convolutional neural network (LRCN) in 2015, a model that integrates convolutional neural networks with long- and short-term memory networks for enhanced behavior recognition. Compared to methods based on artificial feature design, deep learning-based approaches extract features directly from videos, leading to a significant improvement in recognition accuracy. However, high computational resources and equipment requirements present challenges for real-time detection tasks, particularly in scenarios involving multi-target behavior recognition where recognition performance may be significantly affected.

This study focuses on the characteristics of chemical laboratory environments, such as closed space, complex background, relatively sparse density of experimenters, and high behavioral similarity. It combines deep learning-based methods with traditional manual feature design-based methods to propose a new behavior recognition method based on an improved multi-target tracking algorithm (DeepSORT). This method integrates a deep learning-based approach with a traditional manual feature design. Initially, the lightweight OpenPose algorithm was employed to extract skeletal keypoints from the video data for individual localization. Subsequently, the improved DeepSORT algorithm tracks the human target, whereas the YOLOv5 algorithm detects relevant items during the tracking process. Finally, the SKPT-LSTM network, which is a designed behavior recognizer, fuses the tracked target pose and item location information to identify behavior. A flowchart of this method is shown in Figure 1. The method was tested on a self-built chemical laboratory personnel behavior dataset, and its effectiveness was evaluated using performance indicators. An end-to-end network model was used for the comparison. Finally, a practical scenario application flowchart was designed to test the method in real scenarios. The experimental results show that this method can efficiently and accurately identify irregular behaviors of experimental personnel, significantly contribute to reducing chemical laboratory work accidents, promote safe working conditions, and provide an effective guarantee for the early warning of chemical laboratory accidents.

The unique contributions of this study are as follows:

(1)

For the first time, a behavior recognition method based on skeleton key points was used to identify abnormal behaviors of multiple chemical laboratory personnel.

(2)

We improved the structure of the DeepSORT algorithm so that it could track human targets while identifying their behavior; this is called the improved DeepSORT algorithm.

(3)

We applied the SKPT-LSTM network to fuse the tracked human skeleton keypoint position sequence and related object position sequence for behavior recognition.

(4)

The proposed method effectively improved the recognition accuracy of high-similarity behaviors.

2. Materials and Methods

2.1. Data Collection

“The data collection in this study adheres to ethical standards and has obtained appropriate informed consent.”

According to research on the management of unsafe behaviors in laboratories, irregular behaviors in chemical laboratories are defined as actions that violate standard management and operational protocols. These behaviors negatively impact laboratory personnel, equipment, and the environment and may even threaten safety. Several universities have established stringent safety regulations regarding laboratory personnel. For instance, the Laboratory Safety Manual [26] of the Department of Chemistry and Chemical Biology at Harvard University stipulates that all individuals engaged in experimental research must complete the CCB laboratory safety training. The “Large Instrument Management Regulations” [27] in the rules and regulations of the Modern Life Science Experimental Teaching Center of the National Experimental Teaching Demonstration Center for Life Sciences at Tsinghua University explicitly prohibits snacking in the laboratory. Likewise, the “Laboratory Safety Manual” [28] of Anhui Jianzhu University clearly stipulates that smoking, eating, and sleeping are prohibited within the laboratory. The Laboratory Safety and Environmental Protection Management Regulations of East China University of Science and Technology [29] explicitly state that personal belongings are not permitted to be stored in the laboratory, activities unrelated to the experiment are not allowed in the laboratory, carrying and using mobile phones are prohibited, and smoking and eating are prohibited in the instrument room, storage room, and experimental site. These laboratory rules and regulations provided the foundation for the selection and definition of irregular behaviors in our study.

Based on our research on actual conditions in chemical laboratories, we identified “playing with a mobile phone”, “sleeping”, and “eating” as the most common instances of irregular behavior. These behaviors pose safety risks, such as distraction during experiments and the ingestion of toxic chemicals. Therefore, the timely recognition of these behaviors is crucial for enhancing chemical laboratory safety and management efficiency.

To ensure the applicability of our proposed method to real-world scenarios, we collected data from the microbial chemical laboratory, chemical photocatalytic reaction laboratories, and the chemical engineering simulation operation training laboratory (Figure 2).

In university chemistry laboratories, students generally have lower safety awareness than experienced teachers and accidents are more likely to occur during the experimental stage [30,31]. In recent years, the age distribution of student experimenters has generally been concentrated between 20 and 35 years old. Therefore, to make our experimental data more universal, we selected 12 participants and listed their basic information (age, sex, height, weight, physical health status, and whether they had diseases that might have affected the experimental results) in

Table 1. Summary of participant information.

Participant ID	Gender	Age (Years)	Height (cm)	Weight (kg)	Health Status	Medical Conditions Affecting Experiment Results
Participant 1	Male	22	173	65	Healthy	None
Participant 2	Male	27	180	80	Healthy	None
Participant 3	Male	25	168	70	Healthy	None
Participant 4	Male	34	175	75	Healthy	None
Participant 5	Female	23	165	55	Healthy	None
Participant 6	Female	25	160	50	Healthy	None
Participant 7	Male	19	177	65	Healthy	None
Participant 8	Male	26	185	90	Healthy	None
Participant 9	Female	33	155	60	Healthy	None
Participant 10	Female	26	149	45	Healthy	None
Participant 11	Female	27	173	60	Healthy	None
Participant 12	Male	23	160	55	Healthy	None

. They were then asked to perform three deviant and multiple normal behaviors, as shown in Figure 3. The rear wide-angle camera of the Honor Magic3 Pro smartphone was used for data collection, capturing videos from two different perspectives (“overhead” and “level”) at a resolution of 1920 × 1080 and a frame rate of 30 frames per second.

To facilitate model training, long videos were segmented into 5 s clips, each containing only one behavior per laboratory personnel.

We show the details of data collection in

Table 2. Summary of Behavioral Data Collection.

Participant ID	Sleeping (Desk Rest)	Sleeping (Chair Lean)	Phone Use (Sit)	Phone Use (Stand)	Phone Use (Call)	Eat	Normal (Read)	Normal (Experiment)	Normal (Walk)	Total
Participant 1	25 (33)	24 (31)	27 (34)	29 (18)	23 (21)	25 (20)	27 (31)	26 (27)	18 (19)	224 (234)
Participant 2	22 (37)	23 (32)	25 (32)	22 (20)	23 (20)	23 (20)	21 (26)	18 (28)	17 (13)	194 (228)
Participant 3	22 (38)	25 (33)	24 (32)	24 (20)	12 (17)	20 (18)	28 (29)	25 (25)	16 (22)	196 (234)
Participant 4	23 (40)	22 (22)	30 (35)	30 (15)	30 (17)	21 (15)	32 (32)	36 (33)	15 (31)	239 (240)
Participant 5	24 (36)	25 (26)	26 (37)	34 (13)	29 (17)	21 (18)	22 (34)	23 (30)	24 (19)	228 (230)
Participant 6	24 (36)	22 (26)	29 (33)	26 (16)	20 (28)	15 (17)	26 (29)	25 (34)	18 (13)	205 (232)
Participant 7	24 (32)	23 (30)	27 (34)	28 (18)	22 (21)	25 (20)	27 (30)	25 (27)	18 (19)	219 (231)
Participant 8	21 (37)	22 (31)	24 (32)	22 (19)	22 (20)	22 (19)	20 (26)	18 (28)	16 (13)	187 (225)
Participant 9	22 (37)	24 (33)	24 (33)	23 (19)	12 (17)	20 (17)	28 (29)	25 (24)	15 (21)	193 (230)
Participant 10	23 (40)	21 (22)	29 (35)	30 (14)	30 (17)	21 (15)	31 (32)	35 (33)	15 (31)	235 (239)
Participant 11	24 (36)	25 (25)	26 (37)	33 (13)	28 (16)	21 (17)	22 (34)	22 (30)	23 (18)	224 (226)
Participant 12	23 (36)	22 (26)	29 (33)	27 (16)	20 (28)	15 (18)	26 (29)	24 (34)	17 (12)	203 (232)
Total	277 (438)	278 (337)	320 (407)	328 (201)	271 (239)	249 (214)	310 (361)	302 (353)	212 (231)	2547 (2781)

(mainly including the number of times each experimenter performed the behavior in strict accordance with the collection standards of various behaviors, where the numbers in “()” are the number of video samples collected from the “Eye level” angle, and the numbers outside “()” are the number of video samples collected from the “Overlook” angle). A total of 5328 video samples were divided into training, validation, and test sets at a ratio of 7:1:2.

Explanation of data collection criteria for various behaviors:

“Sleeping” behavior: Defined into two modes, namely sleeping on the laboratory bench and sleeping while lying on one’s side in a chair.

“Eating” behavior: Involves bringing food to the mouth over a period of time, with each dataset containing at least one such action.

“Playing with a mobile phone” behavior encompasses scenarios such as sitting at the laboratory bench while using a mobile phone, walking around the laboratory while making or receiving phone calls, and standing while engaged with a mobile phone.

“Normal” behavior: Refers to laboratory personnel performing regular activities such as conducting experiments, reading, walking, etc., without exhibiting the aforementioned three irregular behaviors.

When constructing the dataset, we found that the behaviors of “Sitting at the laboratory table and playing with mobile phone” and “normal reading and note-taking” have high behavioral similarity; “answering and making phone calls” and “normal walking” and “sitting or standing and playing with mobile phone” have high behavioral similarity; “normal experimental behavior” and “eating” behavior have high behavioral similarity.

2.2. Method Framework

This subsection describes the principles of the Lightweight OpenPose, YOLOv5, SKPT-LSTM, and improved DeepSORT algorithms utilized in this study.

2.2.1. Lightweight OpenPose Algorithm

The behavioral postures of the human body are primarily determined by the relative positions of the human skeletal joints. Human posture estimation algorithms accurately locate skeletal keypoints in complex scenarios. The Lightweight OpenPose [32] algorithm is one of the most widely adopted skeletal keypoint extraction methods. Developed by Intel in response to the computationally intensive nature of the OpenPose [33] algorithm, it has gained widespread application in various fields [34,35,36]. Utilizing a “bottom-up” approach, this algorithm achieves the real-time detection of skeletal information for multiple individuals and exhibits excellent robustness.

The network architecture of the Lightweight OpenPose algorithm is illustrated in Figure 4. It employs MobileNet v1 [37] as the backbone for the feature extraction. Following the feature extraction, the network underwent several refinement modules that were divided into two branches. One branch predicts the confidence map of keypoints and generates a heat map (point confidence map (PCM)) whereas the other predicts the part affinity fields between each keypoint to produce a vector map (part affinity fields (PAF)). Loss calculations and optimization were conducted for both PCM and PAF at each stage, as shown in Equations (1) and (2). Post-processing of the PCM and PAF outputs results in a skeletal keypoint group $\mathbb{P}$ for all human bodies in the input image, as outlined in Equation (3).

$$\left\{\begin{array}{c}{S}_1={\rho }_1\left(F\right)\\ L_1={\phi }_1\left(F\right)\end{array}\right.$$

(1)

$$\left\{\begin{array}{c}{S}_t={\rho }_t(F,S_{t-1},L_{t-1}),\forall t\ge 2\\ L_t={\phi }_t(F,S_{t-1},L_{t-1}),\forall t\ge 2\end{array}\right.$$

(2)

The symbols $S_1$ and $S_t$ represent the confidence maps of the keypoints for the first and t stages, respectively. Similarly, $L_1$ and $L_t$ denote the vector fields for the first and tth stages, respectively. $\rho (·)$ represents the prediction network for point confidence maps (PCM) and $\phi (·)$ represents the prediction network for part affinity fields (PAF).

$$\mathbb{P}=\{{pose}_1,{pose}_2,\cdots {pose}_i\}$$

(3)

$${pose}_i=\{\widehat{B},keypoint,conf\}$$

(4)

$$\widehat{B}=\{top,left,width,height\}$$

(5)

$$keypoint=\left[\left[x_0,y_0,conf,0\right]\cdots [x_j,y_j,conf,j]\right]$$

(6)

In Equation (3), ${pose}_i$ represents the skeletal keypoint group of the ith individual in the current image. In Equation (4), $\widehat{B}$ represents the minimum outer bounding rectangle of all keypoints, that is, the target box, and $conf$ is the confidence level of the current keypoint group. In Equation (5), $top,left$ represents the coordinates of the upper-left corner of the human target box, and $width,height$ represent the width and height of the target box, respectively. In Equation (6), $x_j$ and $y_j$ represent the horizontal and vertical coordinates of the skeletal keypoints, respectively, and j represents the ID pair of skeletal keypoints.

The information of 18 skeletal keypoints extracted by the Lightweight OpenPose algorithm is depicted in Figure 5.

2.2.2. YOLOv5 Object Detection Algorithm

For behaviors with high similarity, relying solely on the position features of skeletal keypoints can make it difficult to distinguish between them, as shown in Figure 6, where the left side depicts the “Reading a book” behavior and the right side depicts the “Playing with a mobile phone” behavior. Therefore, this study adopted the YOLOv5 algorithm as an item detector to detect the positions of relevant items, thereby improving the recognition accuracy for highly similar behaviors.

The YOLOv5 algorithm is a one-stage object detection algorithm [38] which treats the object detection task as a regression problem. It can accurately and rapidly locate objects in images for classification and return their positions to the images. YOLOv5 is suitable for a wide range of scenarios [39,40,41].

The network architecture of YOLOv5, as depicted in Figure 7, accepts images with a resolution of 640 × 640 pixels as the input. After feature extraction using the backbone, the features were passed through the neck layer. In the neck layer, features of different scales were fused, resulting in the generation of three feature maps of sizes 20 × 20, 40 × 40, and 80 × 80. Subsequently, after Non-Maximum Suppression (NMS) processing, the algorithm outputs the category $cls$ and confidence $conf$ of the target objects, as well as the bounding box positions $O{x}_1,O{y}_1,O{x}_2,O{y}_2$ where $O{x}_1$ and $O{y}_1$ represent the horizontal and vertical coordinates of the upper-left corner of the bounding box, respectively, and $O{x}_2$ and $O{y}_2$ represent the horizontal and vertical coordinates of the lower-right corner of the bounding box, respectively.

2.2.3. SKPT-LSTM Behavior Recognizer

Human behavior can be regarded as a time-series classification problem determined by changes in a series of human poses over a certain period of time. Recurrent neural networks (RNNs) [42] are designed to handle data with temporal properties. They possess a “memory” characteristic, allowing them to utilize previous input information to influence the current output. However, traditional RNNs suffer from vanishing or exploding gradient problems, which lead to training failure.

Long Short-Term Memory (LSTM) networks [43] are a variant of RNNs that address this issue by introducing gate units to control the importance of sequence information, enabling the network to effectively learn long-term dependencies and successfully accomplish tasks. The structure of the LSTM unit is shown in Figure 8.

In Figure 8, $F_t$ represents the forget gate, $I_t$ represents the input gate, ${\widehat{C}}_t$ represents the candidate memory, and ${\mathsf{\Omega }}_t$ represents the output gate.

To accomplish the behavior recognition task in this study, we improved the LSTM network by integrating the sequences of skeletal keypoint coordinates and object position coordinates of the target over a certain time range to identify its behavior. This improved network is referred to as the Sequence Key Point Tracking LSTM (SKPT-LSTM) network. The structure of this network is illustrated in Figure 9.

In Figure 9, the SKPT-LSTM network structure diagram, $T$ represents the total number of time steps in a sample, $Fram{e}_t$ represents the input data from the tth time step; $x_0^t,y_0^t$ represent the horizontal and vertical coordinates of the skeletal keypoints representing “nose” at time step t; $O{x}_1^t,O{y}_1^t,O{x}_2^t,O{y}_2^t$ represent the positions of the object detected by the object detector at time step t; $F{c}_1(\cdot )$ is a fully connected layer capable of extracting features from skeletal keypoint coordinates; $F{c}_2(\cdot )$ is a fully connected layer capable of extracting features from object position coordinates; $Fc(\cdot )$ is the feature fusion layer. As shown in Equations (7)–(10).

$$f_{pt}=F{c}_1(x_0^t,y_0^t,x_1^t,y_1^t\dots x_{17}^t,y_{17}^t)$$

(7)

$$f_{bt}=F{c}_2(O{x}_1^t,O{y}_1^t,O{x}_2^t,O{y}_2^t)$$

(8)

$$H_t=LSTM\left(BN\right(f_{pt}+f_{bt}\left)\right)$$

(9)

$$\tilde{y}=S\mathrm{oftmax}(Class\_Fc(BN\left(Fc\right(H_{last\_t}\left)\right)\left)\right)$$

(10)

The SKPT-LSTM network divides the data into two streams before inputting them into the LSTM network: a stream of human-body keypoint coordinates and a stream of item location coordinates. After extracting the features through two fully connected layers (512 neurons per layer), the features of the two streams were summed, fused, and batch-normalized before being inputted into the LSTM network (containing two LSTM units, each with a hidden layer containing 512 neurons). In the last time step of the SKPT-LSTM network, the final hidden layer states of the LSTM network were passed through two fully connected layers containing 256 neurons to extract the features. Finally, a Class_FC layer was processed using a softmax activation function to output the class probabilities of the behaviors. The ReLU activation function was used for all the fully connected layers, except for the last Class_FC layer. The output of the SKPT-LSTM network is the probability that a series of poses from the current input belong to a certain behavior.

• Input of the SKPT-LSTM Network.

The input of the SKPT-LSTM network is represented by (11):

$$X=\left\{\begin{array}{l}({x^{\prime }}_0^1,{y^{\prime }}_0^1),({x^{\prime }}_1^1,{y^{\prime }}_1^1),\dots ({x^{\prime }}_{17}^1,{y^{\prime }}_{17}^1),(O{x^{\prime }}_1^1,O{y^{\prime }}_1^1),(O{x^{\prime }}_2^1,O{y^{\prime }}_2^1)\\ ({x^{\prime }}_0^2,{y^{\prime }}_0^2),({x^{\prime }}_1^2,{y^{\prime }}_1^2),\dots ({x^{\prime }}_{17}^2,{y^{\prime }}_{17}^2),(O{x^{\prime }}_1^2,O{y^{\prime }}_1^2),(O{x^{\prime }}_2^2,O{y^{\prime }}_2^2)\\ ⋮\\ ({x^{\prime }}_0^t,{y^{\prime }}_0^t),({x^{\prime }}_1^t,{y^{\prime }}_1^t),\dots ({x^{\prime }}_{17}^t,{y^{\prime }}_{17}^t),(O{x^{\prime }}_1^t,O{y^{\prime }}_1^t),(O{x^{\prime }}_2^t,O{y^{\prime }}_2^t)\\ ⋮\\ ({x^{\prime }}_0^T,{y^{\prime }}_0^T),({x^{\prime }}_1^T,{y^{\prime }}_1^T),\dots ({x^{\prime }}_{17}^T,{y^{\prime }}_{17}^T),(O{x^{\prime }}_1^T,O{y^{\prime }}_1^T),(O{x^{\prime }}_2^T,O{y^{\prime }}_2^T)\end{array}\right\}$$

(11)

In Equation (11), $\left({x^{\prime }}_{\mathrm{i}\mathrm{d}}^t,{y^{\prime }}_{\mathrm{i}\mathrm{d}}^t\right),$ $\left(O{x^{\prime }}_1^t,O{y^{\prime }}_1^t\right),$ $(O{x^{\prime }}_2^t,O{y^{\prime }}_2^t)$ represent the normalized and standardized coordinates of the skeletal keypoints $id$ at time step $t$ and the detected bounding box positions of the relevant objects, including the upper-left and lower-right corners, respectively. $T$ represents the total number of time steps in a sample. The normalization and standardization equations are represented by Equations (12)–(14), respectively.

$$norm(x)={\displaystyle \frac{x_{orig}}{frame\_w}}$$

(12)

$$norm(y)={\displaystyle \frac{y_{orig}}{frame\_h}}$$

(13)

In the above equation, $x_{orig},y_{orig}$ represent the original unprocessed initial horizontal and vertical coordinate points, respectively, and $frame\_w, frame\_h$ denote the width and height of the current video frame, respectively.

$$x^{\prime }={\displaystyle \frac{norm\left(x\right)-\mu }{\sigma }}$$

(14)

In Equation (14), the $norm\left(x\right)$ represents the normalized coordinate value, where $\mu ,\sigma$ denote the mean and standard deviation of the normalized samples, respectively.

• Loss function of SKPT-LSTM.

The loss function chosen for training the SKPT-LSTM network was the multiclass cross-entropy loss function, as represented by Equation (15).

$$Loss=-\sum _{i=0}^{C-1}{y}_i\mathit{log}\left(p_i\right)=-log\left(p_c\right)$$

(15)

In the above equation, $p=[p_0,p_1,\cdots ,p_{C-1}]$ represents the probability of each behavior category output by the SKPT-LSTM network, $y=[y_0,y_1,\cdots ,y_{C-1}]$ represents the true category to which the training sample belongs; $C$ is the number of categories for the sample, and $i$ represents the current sample.

• Evaluation Metrics of SKPT-LSTM.

The performance of the SKPT-LSTM network was evaluated using the accuracy, precision, recall, and F1 score metrics. Accuracy was calculated as the ratio of the number of correctly detected samples to the total number of samples detected by the model. Precision represents the ratio of correctly predicted positive samples to all samples predicted as positive by the model. Recall, also known as sensitivity, is the ratio of the correctly predicted positive samples to the total number of positive samples. The F1 score, which balances precision and recall, was computed as the harmonic mean of the precision and recall. The formulas for these evaluation metrics are derived from

Table 3. Confusion matrix.

	Prediction	Positive	Negative
Actual
Positive		True Positive(TP)	False Negative(FN)
Negative		False Positive(FP)	True Negative(TN)

and are shown in Equations (16)–(19).

$$Accuracy={\displaystyle \frac{TP+TN}{TP+TN+FP+FN}}$$

(16)

$$P\mathit{recision}={\displaystyle \frac{TP}{TP+FP}}$$

(17)

$$R\mathit{ecall}={\displaystyle \frac{TP}{TP+FN}}$$

(18)

$$F1 score=2\times {\displaystyle \frac{R\mathit{ecall}\times P\mathit{recision}}{R\mathit{ecall}+P\mathit{recision}}}$$

(19)

In Table 3, TP represents the number of positive samples correctly classified by the model, FP represents the number of positive samples incorrectly classified by the model, FN represents the number of negative samples incorrectly classified by the model, and TN represents the number of negative samples correctly classified.

2.2.4. Improved DeepSORT Algorithm

The DeepSORT algorithm [44] is a simple, high-precision, and fast multi-target tracking algorithm. It first utilizes a detector to obtain the detection information of targets in the previous moment and then uses Kalman filtering to predict the position information of the targets in the next moment. Next, by combining the appearance features, motion information, and target information detected by the detector at the next moment, the similarity between the targets at the two moments was calculated, and the Hungarian algorithm was used for matching. Finally, it completes the trajectory update, thereby achieving the target-tracking task. Owing to its excellent performance, the DeepSORT algorithm is widely used for various tracking tasks [45,46,47,48].

In the task of human behavior recognition, the DeepSORT algorithm can track the position of each human target in consecutive video streams frame-by-frame, forming trajectories and assigning each target a unique “ID” to distinguish different targets. However, it cannot recognize the behavior of targets at the current moment. In this study, we improved the DeepSORT algorithm for laboratory personnel behavior recognition. The improved DeepSORT algorithm is shown in Figure 10.

Step 1: Input the video stream frame by frame into the Lightweight OpenPose human posture estimator to obtain all the human skeletal keypoints group $\mathbb{P}$ in the current video frame as in Equation (3) and generate an external rectangular box $\widehat{B}$ (Equation (5)) with all the skeletal keypoints to select the human body. The body frame position information and human skeletal keypoint information are input into the DeepSORT algorithm as detection information (Equation (20)) to generate a new Track$(T_i,i\in \{1,\cdots M\left\}\right)$ (Equation (21)).

Step 2: Kalman filtering predicts the position of the target trajectory at the next moment after transforming the target frame information obtained from the detector (Equation (5)) into the target frame center coordinates, aspect ratio, height, and motion information. This matches the detector’s detection information at the next moment. For a target that fails to match, the decision is made according to the corresponding rules, that is, whether to delete it or match it again in the next frame.

Step 3: For successfully matched target trajectories, the position is updated, and the information of human skeleton points from the matched detection data are stored sequentially in P (Equation (22)). The area where the target was located was then extended outwards by a certain number of pixels (ensuring that all parts of the target were included), cropped, and detected using the YOLO object detector (Equation (24)). The detected relevant object position information is synchronized and stored in $\mathcal{O}$ (Equation (23)), and the process then moves to the next frame of target tracking.

Step 4: When a target is successfully tracked for a certain period (i.e., when the information saved in the target trajectory reaches a certain quantity), the tracked sequential skeleton keypoint and object position information are processed. They were fed into the SKPT-LSTM network to recognize the target behavior and simultaneously update the behavioral state $\mathcal{B}$ of the target (Equation (25)).

$$D_i=\left\{pos{e}_i, features\right\}$$

(20)

$$T_i=\{m,v,ID,state,features,P,\mathcal{O},\mathcal{B}\}$$

(21)

$$P=\left\{pos{e}_0,pos{e}_1,\cdots pos{e}_t\right\}$$

(22)

$$\mathcal{O}=\left\{Obbo{x}_0,Obbo{x}_1,\cdots Obbo{x}_t\right\}$$

(23)

$$Obbo{x}_t=YOLO\left(crop\right(img,to\_bbox(m_t,v_t)+\mathit{margin}\left)\right)$$

(24)

$$\mathcal{B}=SKPT-LSTM\left(preprocess\right(P,O\left)\right)$$

(25)

In Equation (20), $i$ represents the ith detected target, and $features$ denotes the appearance feature vector used during cascade matching.

In Equation (21), $m$ and $v$ represent the mean and variance of the trajectory used by the Kalman Filter (KF) for prediction and update, respectively. $ID,state,features$ denote the target’s identifier (used to distinguish it from other targets), state of the trajectory, and appearance feature vector of the currently tracked target, respectively.

In Equation (22), $P$ stores the poses from time 0 to t, and is updated using the pose from the latest detection at time t.

Equation (23), $\mathcal{O}$ represents the collection of the positions of relevant items detected by the YOLO object detector within the range of human targets from time 0 to t. The calculation formula is shown in (23).

In Equation (24), the function $to\_bbox(m_t,v_t)$ converts the mean and variance of the target trajectory at time t into a bounding box. The function $crop(img,box+\mathrm{margin})$ crops the area near the target with a margin specifying the extent to which the cropping area extends outwards. The $YOLO\left(img\right)$ function was used to detect the positions of the relevant items using the YOLO algorithm. If no detection was detected, $Obbo{x}_t=(\mathrm{0,0},\mathrm{0,0})$. If multiple identical relevant items were found within the range, $Obbo{x}_t=Obbo{x}_{\mathit{max}(conf)}$.

The track successfully matched the detection information and underwent simultaneous updates of P and $\mathcal{O}$ during the Kalman filter-update process.

In Equation (25), $\mathcal{B}$ represents the behavior of the human target. When the target was successfully tracked for a certain period, its behavior was identified using the SKPT-LSTM behavior recognizer.

3. Results

The experimental setup of this study included a CPU: 12th Generation Intel(R) Core (TM) i9-12900H with a clock speed of 2.50 Gigahertz, 16 GB RAM, NVIDIA GeForce RTX 3060 GPU, and Windows 11 operating system. PyTorch2.21 [49] was used as the deep learning framework in this study, and the implementation code was written in Python3.8. NVIDIA CUDA and NVIDIA cuDNN were used for GPU acceleration during the training and inference of the deep learning networks.

3.1. Training and Testing of the Lightweight OpenPose Model

Considering issues such as varying environmental lighting conditions, the distance between experimenters and the camera, and different camera angles, video data in this study were sampled (with one frame sampled every 2 s) to construct the Lightweight OpenPose keypoints training dataset. Subsequently, the dataset was annotated using COCO-Annotator, a web-based image-annotation tool. The Lightweight OpenPose model with a Refinement Stage consisting of three layers and pre-trained on the COCO-Keypoints2017 dataset was trained using the self-constructed dataset. Finally, the trained Lightweight OpenPose model detected the skeletal keypoints of the experimenter, and the detection results are shown in Figure 11.

As shown in

Table 4. Coordinates and confidence scores of key points of the “sleep” behavior.

Keypoints ID	X	Y	Confidence	Behavioral Pictures
1	150	69	0.755
2	133	42	0.872
3	176	56	0.443
4	206	89	0.214
5	163	102	0.694
6	256	115	0.872
7	209	66	0.583
8	74	175	0.722
9	199	188	0.881
10	166	300	0.900
11	117	234	0.622
12	252	267	0.848
13	203	360	0.702
16	206	49	0.729
17	209	85	0.638

, the trained Lightweight OpenPose model effectively predicts the occluded points. Although keypoints 0, 14, and 15 were obscured owing to the serious occlusion of the human body posture, and the confidence level of keypoint 4 was low, the average confidence level of the detected points still reached 0.70, indicating the capability of the trained Lightweight OpenPose model to perform the detection task in this study.

3.2. Training and Evaluation of YOLOv5 Model

This study used the YOLOv5-s model as the object detector. Compared with other versions of the YOLO series, the YOLOv5 model can maintain high accuracy while having a very fast inference speed and is lightweight. Therefore, resource-constrained devices are easy to deploy and suitable for real-time application scenarios.

First, we prepared a dataset for training the item detector. We used the trained Lightweight OpenPose model as a human detector for the keypoint image dataset and cropped a 50-pixel region around the target (to ensure the integrity of the item target). We then used the LabelImg1.8.6 software to label cropped images with cell phone items. Next, we converted the labeled images into the format of the YOLO model training evaluation dataset and trained the YOLOv5-s model for 100 epochs. The training and testing results are shown in Figure 12.

As illustrated in Figure 12, the trained YOLOv5 model demonstrated satisfactory performance, achieving an mAP@0.5 of 97.31% and an mAP@0.5:0.95 of 65.12% for the validation set. Because the YOLO algorithm was exclusively employed for detecting “phone” items in this study, the category loss value consistently remained at zero.

The trained Lightweight OpenPose model combined with the YOLOv5 target detector was integrated into the DeepSORT algorithm to track the pose and identify the target’s items. As shown in Figure 13, the object detector performed well.

3.3. Training and Evaluation of SKPT-LSTM Network

Training of SKPT-LSTM Network

We utilized the DeepSORT algorithm in conjunction with the Lightweight OpenPose model and the YOLOv5 model to process our self-constructed video dataset of chemical laboratory personnel behavior. This process involved extracting the location sequences of keypoints on the skeleton of the experimenter as well as the location sequences of related items for each video sample. Subsequently, we created a training evaluation dataset for the SKPT-LSTM network based on Equation (11).

For the training process, we configured the following parameters: Batch size: 32; Initial learning rate: 0.001; Learning rate schedule: the learning rate was reduced by a factor of 10 every 20 epochs; Total epochs: 100; Optimizer: Adam optimizer; Dropout rate: 0.5; L2 Regularization: Coefficient was set to 0.01.

The loss reduction and accuracy curves for the training process are shown in Figure 14.

Figure 14 shows that the training status of the model was satisfactory. Throughout the training process, both the loss value and accuracy of the model on the test set stabilized within a certain range, indicating that the SKPT-LSTM network that we designed was structurally reasonable.

4. Discussion

4.1. Evaluation of SKPT-LSTM Network

The performance of the trained SKPT-LSTM network was evaluated using a self-built chemical laboratory behavior test dataset.

We used a self-built chemical laboratory behavior test dataset to evaluate the performance of the SKPT-LSTM network. We mixed the validation and test set data to form a test dataset containing 1390 behavior samples, accounting for one-quarter of the entire dataset. Among them, there were 124 samples of “eating” behavior, 179 samples of “sleeping on the laboratory table” behavior, 193 samples of “sleeping on the side of the chair” behavior, 169 samples of “sitting and playing with mobile phones” behavior, 133 samples of “standing and playing with mobile phones” behavior, 125 samples of “making and receiving phone calls” behavior, 189 samples of “normal reading and taking notes” behavior, 169 samples of “normal experiment” behavior, and 109 samples of “normal walking” behavior. Using the method proposed in this paper, we identified the above nine behaviors in the test set, drew a confusion matrix (as shown in Figure 15), and calculated indicators, such as P, R, and F1.

Table 5. Evaluation results of the SKPT-LSTM model for each behavior.

	Metrics	Precision	Recall	F1 Score
Behavior
Sleeping (Chair Leaning)		94.97%	97.93%	0.9643
Sleeping (Desk Resting)		96.07%	95.53%	0.9580
Eating		88.34%	73.39%	0.8017
Phone Use (Sit)		83.33%	88.76%	0.8596
Phone Use (Stand)		85.31%	91.73%	0.8840
Phone Use (Call)		92.77%	61.60%	0.7404
Normal (Read)		87.38%	95.24%	0.9114
Normal (Experiment)		85.88%	89.94%	0.8786
Normal (Move)		75.21%	83.49%	0.7913
Average		87.70%	86.40%	0.8654

presents the results of this study.

In Figure 15, we have simplified the behavior descriptions to enhance clarity and ease of representation. The revised labels are as follows:

“Sleeping (Desk Rest)” is simplified to “sleeping_desk”

“Sleeping (Chair Lean)” is simplified to “sleeping_chair”

“Phone Use (Sit)” is simplified to “play_phone_sit”

“Phone Use (Stand)” is simplified to “play_phone_stand”

“Phone Use (Call)” is simplified to “play_phone_call”

“Normal (Read)” is simplified to “normal_read”

“Normal (Experiment)” is simplified to “normal_experiment”

“Normal (Walk)” is simplified to “normal_move”

These simplifications ensure that the behavior labels are more concise and easily understandable.

By combining the indicator calculation results in Table 5 and the confusion matrix in Figure 15, we analyzed the performance of the SKPT-LSTM model in detail. The model performs best in identifying the two “sleeping” behaviors because in this type of behavior, the sequence position fluctuation of the key points of the human skeleton is small, and the behavior characteristics are relatively obvious. The recall rate of the “eating” behavior is low. By analyzing the confusion matrix, we found that the model misclassified a small number of “eating” behaviors as “normal experiment” and “normal reading and note-taking”. This may be because the titration operation in the chemical experiment and the operation of turning pages in the book are also accompanied by the movement of the “wrist” key point, which leads to the similarity of the characteristics of the “eating” behavior with other behaviors, thereby affecting the recall rate. Among the three types of “playing with mobile phones”, the recall rate of “making and receiving phone calls” behavior is the lowest. Specifically, 11 and 18 “making and receiving phone call“ samples were misclassified as “playing with mobile phones while sitting” and “playing with mobile phones while standing”, respectively, and nine samples were misclassified as “normal walking”. This is because the behavior of “making or receiving calls” usually involves the use of mobile phones, and the SKPT-LSTM network identifies mobile phone usage behaviors in different situations by analyzing the positional relationship between the “mobile phone” position and related skeleton key points. However, owing to the influence of the shooting angle, errors occur in the extraction of the positional relationship features, resulting in behavior recognition errors. In addition, in the process of “making or receiving calls”, “mobile phone” objects are often blocked, making it impossible for the object detector to detect the position of the relevant object and then misjudging it as “normal walking” behavior, which also leads to a low recognition accuracy rate for “normal walking” normal walking behavior.

Figure 16 shows the integration of the trained SKPT-LSTM network into the improved deep-SORT algorithm and identifies various behaviors of the laboratory personnel. It accurately identifies the various behaviors of the laboratory personnel, which shows that the proposed method is practical and effective.

4.2. Comparison of SKPT-LSTM Network

In order to verify that the SKPT-LSTM network can effectively extract behavior recognition features from the positional relationship between sequence human skeleton key points and related items, we selected the basic LSTM, basic recurrent neural network (RNN), and basic convolutional neural network (CNN) for comparison. Initially, we trained these three networks separately, using the same training data and parameters. Subsequently, we tested the trained network models on an identical test dataset and assessed the average accuracy, precision, recall, and F1 score for each behavioral category as well as the number of samples processed per second (PS, Processing Speed).

It can be clearly observed from

Table 6. Performance comparison of four networks.

	Metrics	Accuracy	Average Precision	Average Recall	F1 Score	PS
Model
Conv2D		59.14%	63.66%	47.94%	0.5469	119.82
RNN		69.64%	68.95%	62.57%	0.6560	81.79
LSTM		70.22%	67.21%	61.40%	0.6417	78.84
SKPT-LSTM		87.99%	87.70%	86.40%	0.8654	20.96

and Figure 17 that the SKPT-LSTM network designed in this study can effectively fuse the tracked sequence of human skeleton keypoint groups with the location information of related objects and extract features for behavior recognition. In contrast, the three types of basic neural networks used for comparison exhibited poor model performance when processing the same test dataset because of their inability to extract key features effectively. For example, for the “making and receiving phone calls” behavior, the three models often misclassify it as “sitting and playing with mobile phone” or “standing and playing with mobile phone”. However, because the SKPT-LSTM network considers the spatial position relationship characteristics between the key points of the human skeleton and related items, the recognition speed of the algorithm was lower. However, in terms of laboratory safety, the recognition accuracy of the algorithm was more important than that of the algorithm. Therefore, the design of the SKPT-LSTM network proved reasonable and effective.

4.3. The Impact of Different Shooting Angles on the Performance of the Improved DeepSORT Algorithm

When constructing the dataset for identifying irregular behaviors of laboratory personnel, we used two different angles of equipment for data collection, namely the “horizontal angle” and the “overhead angle”. After data cleaning, the dataset also contained 2297 behavior samples from the “horizontal angle” and 2339 behavior samples from the “overhead angle”. To verify the impact of data samples from different shooting angles on model performance, we divided the datasets of these two angles into training, validation, and test sets in a ratio of 7:1:2 and trained and tested the improved DeepSORT algorithm. The evaluation indicators included average accuracy, average precision, average recall and F1-Score.

It can be clearly seen that

Table 7. Network performance table under different angle dataset training.

	Metrics	Accuracy	Average Precision	Average Recall	F1 Score
Angle
Eye level		87.28%	86.42%	83.35%	0.8486
Overlook		84.63%	85.38%	82.24%	0.8378

shows that the model trained with the “Eye level” angle data has better overall performance than the model trained with the “Overlook” angle data. The dataset was analyzed in combination with the experimental results. In the “Overlook” angle dataset, the positional relationship between the key points of the human skeleton will shift due to changes in the shooting angle. Because this study uses a 2D human posture estimation algorithm, it cannot accurately represent the three-dimensional spatial positional relationship between the key points of the skeleton, which increases the difficulty of the model to extract behavioral features from the spatial positional relationship, resulting in a decrease in recognition accuracy. The data at the “Eye level” angle were closer to the human visual angle, and the distance between the key points of the skeleton was less affected by the shooting angle. Therefore, it can more accurately represent the spatial positional relationship between the key points of the human skeleton.

4.4. Improved DeepSORT Performance Comparison

To verify the superior performance of the improved DeepSORT algorithm proposed in this study in identifying irregular laboratory personnel behaviors, we conducted a comparative analysis with several popular end-to-end behavior recognition networks, including C3D and R3D networks.

R3D (ResNet3D) and C3D (convolutional 3D) networks are deep learning networks dedicated to video and three-dimensional convolution operations, respectively. They are widely employed in tasks such as video analysis and action recognition. Both are predicated on 3D convolution operations, which capture spatiotemporal information within the video by extending 2D convolution in the temporal dimension. However, their implementation approaches and characteristics vary.

The C3D network was among the earliest 3D convolution networks proposed for video comprehension, and its principal attributes are simplicity and directness. It conducts convolution in three-dimensional space to capture the temporal information between video frames. The R3D network is an extension of the ResNet network in three-dimensional convolution. The R3D network incorporated a residual structure into the 3D convolution network, thereby enhancing the training stability and performance of the deep model. In contrast to C3D, R3D possesses a superior gradient transfer capacity in deep structures and is suitable for spatiotemporal modeling of longer video sequences.

We trained these models using our own chemical laboratory personnel behavior dataset and evaluated them using the same test dataset. The comparison indicators included accuracy, average precision, average recall, F1 index, number of parameters(M), applicability to multi-target scenarios, algorithm calculation speed in a single-target scenario (SFPS, the number of video frames processed per second by the algorithm in a single target scene), and algorithm calculation speed in a multi-target scenario (MFPS, the number of video frames processed per second by the algorithm in multi-target scenarios). A comparison of the results is presented in

Table 8. Behavior recognition algorithm performance comparison table.

	Metrics	Accuracy	Average Precision	Average Recall	F1 Score	Parameters (M)	Multi-Target Suitability	SFPS	MFPS
Algorithm
C3D		89.96%	89.40%	86.84%	0.881	78.03	NO
R3D		76.67%	76.81%	74.68%	0.7573	33.18	NO
Proposed Method		87.99%	87.70%	86.40%	0.870	24.86	Yes	17.03	14.42

.

As is conspicuously evident in Table 8, the C3D model showcases the optimal performance metrics on the same test dataset; nevertheless, it also has the highest parameter count, attaining 78.03 M, which imposes substantial demands on the deployment hardware. Additionally, both deep learning-based end-to-end behavior recognition algorithms are ill suited for multi-target scenarios, such as those encountered in chemical laboratory environments. In contrast, the improved DeepSORT algorithm proposed in this study not only has a significantly lower parameter count but also delivers robust performance, providing a superior balance between efficiency and effectiveness. Because the improved DeepSORT algorithm not only has the function of target tracking but also considers behavior recognition, whereas C3D and R3D merely focus on the behavior recognition task, Table 8 presents the average calculation speed of the improved DeepSORT algorithm in single-target and multi-target scenarios.

4.5. Application in Actual Scenarios

To further explore the performance of the SKPT-LSTM network for detailed behavior recognition, we made detailed predictions for various behaviors based on dataset collection standards. However, in actual chemical laboratory application scenarios, we mainly focus on whether the experimenters have irregular behaviors, and the specific details of irregular behaviors are not very important (for example, people only care about whether there is “playing with mobile phones” behavior and will not pay special attention to “sitting”, “standing” or “making phone calls”). Therefore, in actual applications, we simplify the behavior categories into four categories: “eating”, “sleeping”, “playing with mobile phones” and “normal” behavior, and calculate the performance indicators of the simplified four categories of behavior recognized by the improved DeepSORT according to Figure 15,

Table 9. Improved DeepSORT algorithm for each behavior recognition index after category simplification.

	Metrics	Accuracy	Precision	Recall	F1 Score
Behavior
Eating		92.45%	88.35%	73.39%	0.8018
Sleeping			97.35%	98.66%	0.98
Playing with phone			94.33%	89.70%	0.9196
Normal			88.10%	95.07%	0.9145
Average			92.03%	89.21%	0.9040

.

Table 9 indicates that the proposed method exhibited a higher recognition effect in the simplified behavior categories where the accuracy of each behavior reached 92.45%.

Testing conducted in real chemical laboratory environments validated the robustness, generalization ability, and real-time multi-target behavior recognition capability of the proposed method. The testing data were captured using the rear wide-angle camera of Honor Magic3 Pro with a video frame rate of 30 frames per second (fps). The capture locations included the three aforementioned chemical laboratories with personnel in a normal experimental state.

Considering the possibility of multiple laboratory personnel in the laboratory, each with varying appearance times, recognition failures may occur if the appearance time of the laboratory personnel is short and the tracked posture actions are insufficient. Therefore, the behavior recognizer only identifies laboratory personnel who appear for more than 3 s. This testing setup enabled a more realistic assessment of the application of the proposed method in practical scenarios.

Based on the flowchart depicted in Figure 18, the method described in this study was applied to real chemical laboratory scenarios as follows. The video stream is continuously fed into the improved DeepSORT algorithm. Every 5 s, the SKPT-LSTM behavior recognizer identifies the behavior of all tracked human targets that satisfy the tracking conditions (i.e., appearing for more than 3 s) and presents the recognition results in real-time. The parameters “count” indicate the current number of processed video frames, “Total” represents the total number of frames in the video stream, and “Frame_num(Video)” is a function used to obtain the total number of frames in the video stream. If the input data are from a real-time monitoring device, then “Total” is set to zero.

The proposed method was applied in a microbial chemical culture laboratory, a chemical photocatalytic reaction laboratory, and a chemical engineering simulation operation training laboratory. Figure 19 shows the single-target behavior detection of laboratory personnel. It can be observed that the proposed method accurately detects the behavior of laboratory personnel, achieving an average frame rate of 17 fps, thus demonstrating its capability for real-time detection.

Given that both the Lightweight OpenPose and DeepSORT algorithms are suitable for multi-target tasks, the performance of the proposed method was evaluated in scenarios involving multiple laboratory personnel to assess its generalization ability in multi-target behavior recognition tasks.

Figure 20 illustrates that when our method is used for multiple-target behavior detection, it is evident that although the algorithm can accurately identify the behaviors of laboratory personnel, the detection speed is relatively slower than that of single-target detection, stabilizing at approximately 14 frames per second (fps). As the number of laboratory personnel increased, the detection speed of the algorithm decreased. Therefore, our method is more suitable for chemical laboratory scenarios with relatively few personnel.

Even in cases where laboratory personnel are partially or fully occluded, the proposed method can continue to track the poses of targets and accurately identify their behavior, as shown in Figure 21.

Because the proposed method adopts offline data processing and model training, if behaviors or poses not included in the self-constructed dataset occur in practical applications, resulting in the inability of the behavior recognizer to identify them, we employ the strategy of adding unrecognized behaviors to the training dataset of the SKPT-LSTM network and then retrain the network.

5. Conclusions

In this study, we employed the lightweight OpenPose multi-target human pose estimation algorithm and YOLOv5 target detection algorithm to enhance the performance of the DeepSORT multi-target tracking algorithm. By integrating these algorithms, DeepSORT can track the sequence of skeletal keypoint locations and associated sequence of item locations for a target. In addition, we designed a network called SKPT-LSTM for integration into the improved DeepSORT algorithm. This network can fuse tracked sequence information to achieve behavioral recognition. This method was tested on a self-built chemical laboratory personnel behavior dataset. The results showed that the improved DeepSORT algorithm achieved an accuracy of 87.99% for the nine behavior recognition tasks defined in this study, with an average precision of 87.70% and an average recall of 86.40%. Furthermore, the SKPT-LSTM network designed in this study was compared and evaluated with three other classical baseline networks, and showed superior performance in terms of accuracy, precision, recall, and F1 score under the same test set conditions. The algorithm was tested and evaluated using the datasets obtained from various shooting angles. The results show that the performance of the proposed method was better than that of the “Overlook” angle at “Eye level” angle. Finally, we compared the performance of the improved DeepSORT algorithm with those of two end-to-end behavior recognition algorithms based on deep learning. The results showed that the improved DeepSORT algorithm is lightweight and suitable for deployment in chemical laboratory scenarios. When applying the method in this study in an actual chemical laboratory scenario, we first simplified the “nine behaviors” into “four types of behaviors”. The simplified algorithm showed higher performance in practical applications, with a recognition accuracy of 92.43%. The proposed method can accurately identify the behaviors of single- and multi-target experimenters with an average speed of 17.03 frames per second (fps17) and 14.42 frames per second (fps14), respectively, further verifying the effectiveness of the proposed method.

Although the method proposed in this study achieved certain results in the task of laboratory personnel behavior recognition, it also has some limitations. First, in the multi-target behavior recognition task, as the number of targets increases, the detection time also increases accordingly, which, to a certain extent, limits the applicability of this method in practical applications. Second, the definition criteria for irregular behaviors in this study are relatively simple, and it is impossible to effectively identify complex behaviors, such as “eating and playing with mobile phones”, and multiple behaviors simultaneously. To address these limitations, in the future, a lighter and more efficient algorithm model will be used to improve the proposed method, and the scale of the laboratory irregular behavior dataset will be expanded so that this method can identify more complex and important irregular behaviors.

In conclusion, the methodology outlined in this study presents an effective framework for accurately identifying irregular behavior among laboratory personnel. The ultimate goal of this method is to deploy it in a chemical laboratory and use it in the laboratory safety management department to reduce the pressure on laboratory safety management by using computer vision algorithms. Simultaneously, this method can effectively encourage experimenters to comply with regulations in chemical laboratories, reduce the occurrence of laboratory accidents, promote the establishment of a safe working environment, and provide strong guarantees for the early warning of laboratory accidents.