Human Action Recognition from Videos Using Motion History Mapping and Orientation Based Three-Dimensional Convolutional Neural Network Approach

Abstract

Human Activity Recognition (HAR) has recently attracted the attention of researchers. Human behavior and human intention are driving the intensification of HAR research rapidly. This paper proposes a novel Motion History Mapping (MHI) and Orientation-based Convolutional Neural Network (CNN) framework for action recognition and classification using Machine Learning. The proposed method extracts oriented rectangular patches over the entire human body to represent the human pose in an action sequence. This distribution is represented by a spatially oriented histogram. The frames were trained with a 3D Convolution Neural Network model, thus saving time and increasing the Classification Correction Rate (CCR). The K-Nearest Neighbor (KNN) algorithm is used for the classification of human actions. The uniqueness of our model lies in the combination of Motion History Mapping approach with an Orientation-based 3D CNN, thereby enhancing precision. The proposed method is demonstrated to be effective using four widely used and challenging datasets. A comparison of the proposed method’s performance with current state-of-the-art methods finds that its Classification Correction Rate is higher than that of the existing methods. Our model’s CCRs are 92.91%, 98.88%, 87.97.% and 87.77% which are remarkably higher than the existing techniques for KTH, Weizmann, UT-Tower and YouTube datasets, respectively. Thus, our model significantly outperforms the existing models in the literature.

1. Introduction

The number of videos uploaded to the internet has increased dramatically over the last few decades. As a consequence, visual content [1] is becoming more prevalent on social media platforms. More than 500 h of video content is uploaded every minute on YouTube. Academic and industrial researchers are developing more proficient techniques for video processing, such as video summarization [2]. Computer vision applications that recognize actions in the video such as video surveillance and anomaly detection are in high demand. A technique for extracting action from video frames is called action segmentation. In various applications, it is used as part of visual effects in movies, to understand scenes in detail, to create virtual backgrounds, and to identify human action automatically in videos without interference from objects.

Nowadays, technology, such as that used for activity motion identification, is more focused and attractive due to advancements in the field of computer vision, as well as in Machine Learning, and finds applications in several areas such as in sports motion analysis, the prediction of pedestrian behavior, fitness tracking systems, human–computer interfaces and many others. Human activity can be determined from three-dimensional images and videos using algorithms developed specifically for this purpose [3,4]. A wide range of applications can be achieved by recognizing human actions. Even after several decades of research and numerous encouraging advancements, identifying human actions remains challenging. The movement of a human is articulated by nature. It is very challenging to extract motions with such high levels of articulation from monocular video sensors. According to studies conducted in the past decade, video-based Human Action Recognition is limited by this difficulty [5].

To detect human activity, texture information such as spatial–temporal gradients [6], spatio-temporal frequency [7,8] and dynamic texture recognition [9] have been considered adequate. Meanwhile, other groups have analyzed motion features by using optical flows directly, such as clustered motion patterns [10,11], motion heat maps [12], crowd prediction [13], spatial saliency with the motion feature [14], particle trajectory, optical flow fields [15], and local motion histogram [16].

Despite significant efforts made by the computer vision and image processing communities, video action recognition remains a challenging problem due to scene complexity, acquisition complexity, and action complexity. One of the primary challenges is to model actions in video sequences, known as action representation. Dynamic models have been proven to be the most effective among many action representations [17], shape models [18], interest-point models [19], and motion models [20].

Computer-based systems have great potential for recognizing human action or posture, but CNN-based techniques have improved model performance. Motion History Mapping (MHI) is an important technique in which the sequence of the silhouette is converted into a set of gray-scale images while preserving the dominant information of motion. Therefore, it has the major advantage of not being much sensitive to silhouette noises such as shadows and missing parts. MHI expresses the motion flow by using the intensity of every pixel in a temporal manner. Moreover, MHI can be implemented using low-powered CPUs. The ordinary and standard Convolutional Neural Network cannot handle rotational invariance. Rotational invariance plays a very crucial role in Human Activity Recognition. The Orientation-based Convolutional Neural Network (CNN) learns features that are invariant to a rotational aspect.

We propose a novel model in this study to achieve improved performance. The key contributions of our work are listed below.

• In this paper, we have proposed a novel Motion History Mapping and Orientation-based Convolutional Neural Network (CNN) framework for Human Activity Recognition and classification using Machine Learning.
• A KNN-based classification technique is introduced to classify human activities.
• The exposure frames were trained with the Orientation-based 3D Convolutional Neural Network (CNN) model, thus increasing the Classification Correction Rate (CCR).
• This model has used four publicly available and challenging datasets containing movements of human activities.
• The proposed technique outperforms existing techniques used on the four publicly available and challenging datasets, in terms of higher precision and CCR. Therefore, it can be very advantageous for several accuracy-centric applications.
• The proposed model is also computationally efficient.
• Our model performs well even on low-resolution datasets such as UT-Tower, unlike the MediaPipe approach, which is generally not preferable for dark, blurry and noisy low-quality video datasets such as UT-Tower.

Paper Organization

Detailed information about existing research on video processing and Human Action Recognition is presented in Section 2. The four steps in the proposed methodology (viz. rectangle extraction, Histograms of Oriented Rectangles (HORs), training and testing frame sequences, and the classification of K-Nearest Neighbors (KNNs)) are explained in Section 3. Section 4 describes the experimental setup, datasets, results analysis, and discussion to demonstrate the validated findings. Section 5 concludes the proposed work and discusses future work.

2. Literature Review

Research in computer vision and pattern recognition has focused on detecting and recognizing human actions in video sequences for decades [21,22]. Video-salient features are usually extended to video using $(2D+t)$ images combined to form a 3D frame. Feature extraction models are categorized into two categories, namely 2D feature extraction and 3D feature extraction.

(i)

2D Feature Extraction Models

Detectors of 2D features select mainly salient structures in the image, while detectors of 3D features use structured measurements. Using 3D Gaussian filtering, saliency criteria were computed for a 3D autocorrelation matrix [23].

The time domain is not treated similarly to space, and few methods can detect salient features based on motion. These surveys provide detailed information on Human Action Recognition [24,25,26]. The authors use a taxonomy of 2D and 3D feature detectors and recognition methods [27,28,29,30].

Deep learning models [31,32] build high-level features from low-level ones based on learning a hierarchy of features from low-level ones. The Machine Learning (ML) technique is composed of two types—supervised and unsupervised—and is used to train the dataset. The resulting systems can engage in recognizing visual objects competitively [33], Human Action Recognition, Natural Language Processing (NLP) [34], human tracking [35], segmentation tasks [36,37] and denoising [38,39].

The authors of [40] proposed a wearable sensor-based HAR and SA-GAN, a novel method for adversarial knowledge transfer. The concept of spatial–temporal interest points (e.g., cuboids) has become very popular in action recognition [41], 3D Hessians, 3D Harris corners, and 3D salient points [42]. They are mostly extensions of traditional 2D methods used for object detection, which follow a traditional approach for object detection. A codebook is created by computing multiple-scale feature descriptors, clustering them, and assigning them to a Bag-of-Words (BoW) representation. Direct models of spatio–temporal relationships between features have also been proposed [43]. The author of [44] presented a Spatio-Temporal Manifold Network (STMN) system that considered the overfitting problem and used deep feature extraction for feature extraction. Graded algorithms could easily be reversed to address this network’s inefficient processing order. Using independent dictionaries and classifiers, a novel framework is presented by the authors [45].

(ii)

3D Feature Extraction Models

An arbitrary view can be identified with the resulting classification algorithm using a dictionary constructed from a real-world 2D image sequence. Three-dimensional images were also enhanced with dense 3D features. The 3D descriptor was introduced by [46] to recognize activity features. Over the past decade, it has been widely accepted that 2D Human Action Recognition is the most popular form; however, 3D Human Action Recognition is still underdeveloped. Three-dimensional Human Action Recognition has been reported by relatively few authors so far. A 3D environment is established by humans in real-time, unlike traditional cameras that create a 2D scene projection. A survey on pose estimation techniques divides them into 2D and 3D approaches based on achieving a 2D or 3D pose representation [47]. Two-dimensional activities in the image plane can only be analyzed visually because they are projections. Due to this, the projection of actions may not include all information about what was done, depending on the viewpoint. It is possible to resolve this shortcoming by constructing three-dimensional representations based on reconstructed 3D data with two or more cameras. The study by [48] stands out as the sole endeavor to estimate 3D poses directly from individual images. It is also possible to predict the pose of 3D objects by predicting the depth of key points in 2D [49]. LSTMs have been used to predict 3D poses based on single images [50]. Transformer-based models [51] can be used for prediction when dealing with complex and long-range dependency on sensor data, as they follow a parallel processing mechanism. They are computationally expensive to deploy for real-time applications. Temporal Graph Attention Networks (TGATs) [52] use attention mechanisms for emphasizing the most relevant temporal and spatial relationships within the graphs, thereby allowing them to learn how the movements change with time. Being computationally expensive, a sparsity of sensor data can further complicate their operation. An Inflated 3D Convolutional Network (I3D) is a model for classification that utilizes 3D convolutions [53] to capture spatio-temporal features from videos by transforming pretrained 2D CNNs, to handle video frames as a 3D volume. The overfitting problem is the major concern in I3D, i.e., the model may overfit the training data by capturing noise and extraneous information, which can result in weak performance when applied to unseen data. Another network, the Slow–Fast Network [54], is used for HAR by leveraging the advantages of slow and fast processing paths for recognizing a wide range of activities. Slow–Fast Networks involve several complex architectures. This complexity makes them harder to deploy on devices with limited hardware.

A comparative analysis of Human Activity Recognition models based on the latest research is presented in

Table 1. A comparative survey of different Human Activity Recognition (HAR) models.

Authors	Dataset Used	Objectives and Proposed Method	Performance
[55]	Weizmann, Ballet and KTH	The author proposed a new method of recognizing human activity in a video sequence using critical poses of the human. silhouettes using Support Vector Machine (SVM)-Neural Network (NN) with Principal Component Analysis (PCA).	Overall accuracy: 96.70%
[56]	Opportunity activity recognition and hand gesture	Deep Convolutional Neural Networks (CNNs) automate feature learning from raw data.	Average Accuracy: 92.11%
[57]	Triaxial gyroscope and accelerometer	An approach to select discriminant features to construct a better generalizable online activity. recognizer was presented for smart phone-based HAR system.	Accuracy: 84.3% Precision: 84.5 and F1 Score: 84.3%
[58]	Weizmann, Ballet and KTH	The AESI, SDP, and modified SDG evidence were combined with motion evidence to. recognize human activity according to a new framework i.e., SVM-KNN with PCA.	Accuracy: 95.83%
[59]	HASC corpus	A Deep Recurrent Neural Network (DRNN) recognizes human activity with high throughput.	Accuracy: 95.42%
[60]	UTKinect Action-3D and CAD-60	Human daily activities can be recognized by tracking skeletal joint movements using a depth sensor.	Accuracy: 97.00%
[61]	Opportunity, Skoda and Actitracker	CNN ensembles have been proposed to resolve the confusion between walking and going upstairs, two highly similar activities. CNN (two variants, CNN-2 and CNN-7) were proposed for recognizing human activities.	Accuracy: 96.11%
[62]	UCI-HAR, PAMAP2 and, WISDM	An end-to-end model using CNN and Gated Recurrent Units (GRUs) is proposed that performs automated feature extraction and activity classification.	Accuracy: 96.20%, 95.27% and 97.21%
[63]	UTKinect Action3D dataset	The joint angle shift method is combined with a Deep Convolutional Neural Network (DCNN) model to classify 3D datasets encompassing spatial–temporal information from RGB-D video image data.	Accuracy: 96.72%
[64]	Conducted real-life experiments of activities.	A muscle–machine interface is constructed using a Bi-Directional Long Short-Term Memory (BiLSTM) network, which decodes multichannel Surface Electromyography (sEMG) signals into flexion and extension angles of the hip and knee joints in the sagittal plane.	Accuracy: 92.8%
[65]	UCF 101 and Kinetics-400 dataset	Firefly Optimization Algorithm is applied to fine-tune the hyper parameters of both the graph-based model and a CNN baseline for comparison.	Accuracy: 88.9%

.

3. Proposed Methodology

In the proposed methodology, the human body is represented by rectangular patches. Our technique is based on the positions and orientation of these rectangular patches. These rectangular patches are changed over time concerning their actions. Our methodology first extracts the rectangular patches from the video frames over time $t$. Each video frame has a spatial feature concerning the orientations.

As the rectangular patches are grouped over orientations, a spatial histogram is formed, which can be evaluated over time to see how its shape changes. The human body’s location in the given videos is first identified as part of this. A bounding box is then drawn around its silhouette obtained through a background-subtraction approach. Bins of $\mathrm{N}\times \mathrm{N}$ equal dimensions are then defined within this bounding box. The ratio between the torso, legs and head is considered when forming these spatial bins. The orientations of the rectangular patches in all spatial bins are used to create a histogram. $H_t$ is used to represent a pose. Firstly, the frames extracted from any video of the datasets as shown in Figure 1. Each video frame is extracted as its feature. In Figure 2, the rectangle histograms are depicted based on spatial orientation.

A feature point of the human activity is marked on each extracted frame using background-subtraction technique, as shown in Figure 2. Human activity is marked in the rectangle box, and the grid circumscribing the silhouette is divided into equal parts. A histogram is then computed for each oriented rectangle within that region. A detailed block diagram of the proposed model is shown in Figure 3. First and foremost, the frames of the video sequence are extracted, as shown in Figure 1. Then, the foreground and background are differentiated and a silhouette is extracted. The rectangle is extracted from the silhouette, as mentioned in the next section. Each rectangle is represented by a histogram based on different orientations, as explained in Section 3.2. A 3D CNN model is built for testing and training. Finally, the classification is performed by the KNN classifier.

3.1. Rectangle Extraction

Rectangular patches are used to describe human poses. Foreground–background discrimination is usually used to accomplish this. It is the easiest to use background subtraction after forming a reliable background model. We propose a technique for locating moving objects using background-subtraction.

A 3D Convolution Neural Network (CNN) is then used to find rectangular regions over the human silhouette based on the different orientations and scales. It covers a search space of $180°$ using 12 tilting angles spaced $15°$ apart. The orientation scales of the rectangular patches are covered at 180°. A rectangular Gaussian filter ${\mathrm{G}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}}$ is used for performing convolutions of the binary silhouette image $\mathrm{I}(\mathrm{x},\mathrm{y})$ to create a rectangular region $\mathrm{R}\left(\mathrm{x},\mathrm{y}\right).$

R(x,y) = G _rect (x,y) × I(x,y)

(1)

A $2D$ Gaussian $\mathrm{G}(\mathrm{x},\mathrm{y})$ patch is applied to ${\mathrm{G}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}}$ to form a zero-padded rectangle.

$$\mathrm{G}\left(\mathrm{x},\mathrm{y}\right)={\displaystyle \frac{1}{2\mathsf{\pi }{\mathsf{\sigma }}^2}}{\mathrm{e}}^{-({\mathrm{x}}^2+{\mathrm{y}}^2)/2{\mathsf{\sigma }}^2}$$

(2)

There is a greater chance that patches of a particular kind will be found in areas with a higher response to this filter. In the rectangle extraction phase, approximately $~1000$ rectangles are extracted per frame. A histogram can be formed using all of these rectangles by representing each limb with a representative rectangle. In rectangular filters, a representative rectangle covers the silhouette completely. Those candidates with higher responses that are further apart are chosen to meet these constraints. Each frame contains only $~10$ rectangles.

3.2. Histograms of Oriented Rectangles (HORs)—Pose Descriptor

The Histogram of Oriented Rectangles (HOR) can be used to define poses after identifying the rectangular patches on the human body. The orientation of rectangular patches is used to calculate a histogram. Each rectangle is represented by a histogram with twelve circular bins based on $15°$ orientations. This circular histogram is evaluated on an $\mathrm{N}\times \mathrm{N}$ grid over the human body to account for spatial information. The best results are obtained with $\mathrm{N}=3$. This grid is created by splitting the silhouette in the $\mathrm{Y}\mathrm{d}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}$ according to the leg length. A right-to-left and top-to-bottom equalization has been applied to Figure 4. As a result, we provide action space for the arms above the head by allowing some space.

The feature vector is formed by concatenating the HORs of each spatial bin. A coarser grid on the human body and coarser granularity in orientation bins are also examined, which provide more compact features but coarser pose details.

3.3. Capturing Local Dynamics

It is impossible to distinguish two actions based on a single pose in action recognition. A shape-based action descriptor is insufficient since temporal dynamics must also be considered. Rather than using single frames to calculate the HOR, snippets of frames can be used. A HORW is a histogram of rectangles over windows of frames, where every $i$th the frame has a histogram of rectangles.

$$\mathrm{H}\mathrm{O}\mathrm{R}\mathrm{W}(\mathrm{i})=\sum _{\mathrm{k}=\mathrm{i}-\mathrm{n}}^{\mathrm{i}}\mathrm{H}\mathrm{O}\mathrm{R}\left(\mathrm{k}\right)$$

(3)

The HOR over this window of frames captures the local dynamics over n frames. According to our experiments, HORW is more helpful in discriminating human actions that are similar in pose domain but differ in speed.

3.4. Train and Test Model

Two-dimensional Convolutional Neural Networks: The convolutional layer of 2D CNNs implements 2D convolution to extract local neighborhood features from the precious layer’s feature maps. A sigmoid function and an additive bias are applied to the outcome. The $i$th layer and $j$th feature map, $u_{ij}^{xy}$, describing the unit at the position $\left(\mathrm{x},\mathrm{y}\right)$, are given by:

$${\mathrm{u}}_{\mathrm{i}\mathrm{j}}^{\mathrm{x}\mathrm{y}}=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({\mathrm{b}}_{\mathrm{i}\mathrm{j}}+\sum _{\mathrm{m}}\sum _{\mathrm{p}=0}^{{\mathrm{P}}_{\mathrm{i}}-1}\sum _{\mathrm{q}=0}^{{\mathrm{Q}}_{\mathrm{i}}-1}{\mathrm{w}}_{\mathrm{i}\mathrm{j}\mathrm{m}}^{\mathrm{p}\mathrm{q}}{\mathrm{v}}_{\left(\mathrm{i}-1\right)\mathrm{m}}^{\left(\mathrm{x}+\mathrm{p}\right)\left(\mathrm{y}+\mathrm{q}\right)}\right)$$

(4)

A hyperbolic tangent function is defined as $\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left(.\right)$, a bias function for this feature map is defined as ${\mathrm{b}}_{\mathrm{i}\mathrm{j}}$, a set of feature maps in the $(i-1)$th layer that corresponds to the current feature map is defined as m, ${\mathrm{w}}_{\mathrm{i}\mathrm{j}\mathrm{m}}^{\mathrm{p}\mathrm{q}}$ is the position $\left(\mathrm{p},\mathrm{q}\right)$ of the kernel linked to the $\mathrm{m}$th feature map, and the height and width of the kernel are described by ${\mathrm{P}}_{\mathrm{i}}$ and ${\mathrm{Q}}_{\mathrm{i}}$, respectively. In subsampling layers, features are grouped over local neighborhoods, reducing their resolution while making them more insensitive to input distortions. An alternate layer of convolution and subsampling can be used to build CNN architectures. It is usually possible to train the parameters of a CNN either supervised or unsupervised—for example, bias ${\mathrm{b}}_{\mathrm{i}\mathrm{j}}$ and kernel weight ${\mathrm{w}}_{\mathrm{i}\mathrm{j}\mathrm{m}}^{\mathrm{p}\mathrm{q}}$ [66,67].

Three-dimensional Convolutional Neural Networks: The CNNs compute features from 2D feature maps based on spatial dimensions. It is appropriate to capture the angle information of numerous contiguous frames when analyzing video action detection and recognition problems. The proposed method includes 3D convolutions and orientation changes for computing features from CNNs during the convolution stages. In 3D convolution, several contiguous frames are stacked to form a cube convolved with a 3D kernel. A convolution layer captures motion information by connecting various contiguous frames in a previous layer. In the $\mathrm{i}$th and $\mathrm{j}$th layers, position $(\mathrm{x},\mathrm{y},\mathsf{\alpha })$ is represented by the equation mentioned below.

$${\mathrm{u}}_{\mathrm{i}\mathrm{j}}^{\mathrm{x}\mathrm{y}}=\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}\left({\mathrm{b}}_{\mathrm{i}\mathrm{j}}+\sum _{\mathrm{m}}\sum _{\mathrm{p}=0}^{{\mathrm{P}}_{\mathrm{i}}-1}\sum _{\mathrm{q}=0}^{{\mathrm{Q}}_{\mathrm{i}}-1}\sum _{\mathrm{r}=0}^{{\mathrm{R}}_{\mathrm{i}}-1}{\mathrm{w}}_{\mathrm{i}\mathrm{j}\mathrm{m}}^{\mathrm{p}\mathrm{q}}{\mathrm{v}}_{\left(\mathrm{i}-1\right)\mathrm{m}}^{\left(\mathrm{x}+\mathrm{p}\right)\left(\mathrm{y}+\mathrm{q}\right)}\right)$$

(5)

where ${\mathrm{R}}_{\mathrm{i}}$ is the size of the 3D kernel along with its orientation change. In the previous layer, ${\mathrm{w}}_{\mathrm{i}\mathrm{j}\mathrm{m}}^{\mathrm{p}\mathrm{q}}$ represents the $\mathrm{m}$th feature map connected to the $(\mathrm{p},\mathrm{q},\mathrm{r})$th kernel. This network consists of M cliques that cooperate to complete the final task. Cliques are subparts of networks stacked over multiple layers. Cliques extract features from video segments associated with subactions within the overall activity. This method uses a gray scale and orientation change input, with at least m video frames each. To mitigate local body deformations and noise in our proposed model, we apply a max-pooling operator to each 3D convolutional layer. Two fully connected layers are developed next to correlate activity labels with the convolution results of different cliques.

Table 2. Detailed architecture of a 3D-CNN designed for HAR.

Layer Name	Stride	Input Dimension	Output Dimension	Filter Size	Parameter Size	Activation Function	Filter/Kernel Size
Conv 3D Layer1	(1,1,1)	(16,64,64,1)	(14,62,62,20)	(3,3,3)	560	ReLu Activation	Convolution layer with 20 filters.
Max Pooling 3D Layer1	(2,2,2)	(14,62,62,20)	(6,30,30,20)		-	-	Max pooling with 3 × 3 Kernel.
Conv 3D Layer2	(1,1,1)	(6,30,30,20)	(4,28,28,20)	(3,3,3)	5420	ReLu Activation	Second convolution layer with 20 filters.
Max Pooling 3D Layer2	(2,2,2)	(4,28,28,20)	(1,13,13,20)		-	-	Max pooling to reduce the spatial dimensions further.
Flatten Layer	-	(1,13,13,20)	(1,3380)	(3,3,3)	-	-	To flatten the output for the fully connected layers.
Fully Connected Layer1		(1,3380)	512	(3,3,3)	172,352	ReLu Activation	Fully connected layer 1 with 512 neurons.
Fully Connected Layer2		512	128		65,664	ReLu Activation	Fully connected layer 2 with 128 neurons.
Output Layer		128	11		1419		This is the final output layer having 11 neurons for 11 classes.

shown below describes the details of the 3D CNN of the proposed model.

As shown in Figure 5, these components are further defined in the following paragraphs.

• Max-pooling Operator

A max-pooling operation follows every 3D convolution. It is possible to determine the invariance of shift and deformation of a system using the proposed method [68,69]. The max-pooling process is used to subsample a set of feature maps, resulting in a similar number of feature maps but with reduced spatial resolution. As an example, if a $2\times 2$ max-pooling operator has been applied to $a_{1\times }{a}_2$ feature map, then the max values in each $2\times 2$ non-overlapping region of a $2\times 2$ feature map should be accumulated to form a new feature map of size $a_1/2\times a_2/2$.

• Full Connection Layer

The proposed model contains two full connection layers. Feature maps from different $M$ model cliques are first concatenated into an extended feature vector. In addition, each vector unit is connected to the neurons in the first layer of full connections. Each output neuron signifies the probability of an action hypothesis, and as a result, each output neuron K represents the number of groups of actions. The softmax function is applied to the output labels to normalize their probability.

$$\sigma \left(z_i\right)={\displaystyle \frac{\exp \left(z_i\right)}{{\sum }_{k=1}^K\exp \left(z_k\right)}},$$

(6)

In this formula, $z_i$ is the value of the $\mathrm{i}$th output neuron calculated by multiplying its values with its weights, and $\mathsf{\sigma }\left({\mathrm{z}}_{\mathrm{i}}\right)$ is its probability. Note that ${\sum }_{k=1}^K\exp \left(z_k\right)=1$. Each video frame contains a grey and depth channel. The grey and depth images are obtained from the raw data. The first layer of the convolutional kernel duplicates the channels. The convolved feature maps are the same size since the convolution results are combined. It is also possible to apply the proposed model to multiple-channel video frames.

3.5. Oriented Rectangle Histograms: Recognizing Activities

To evaluate the proposed pose descriptor’s performance for every video frame in the action classification, we incorporate the four matching methods after estimating pose descriptors for every video frame. To simplify the process, we consider single frames (or an extracted number of frames) as the matching procedure criteria, disregarding the sequence’s dynamics. In the test instance, we label each frame with its database is classified based on its nearest neighbor.

The nearest neighbor classification employs voting techniques throughout the entire sequence. A jogging sequence from the KTH database is classified based on its nearest neighbor, as mentioned in Figure 6.

Histogram distance is used to compute the ${\mathrm{X}}^2$ distance between frames [70]. Frames with histograms ${\mathrm{H}}_{\mathrm{i}}$ are labelled with the class of the frame, with histograms ${\mathrm{H}}_{\mathrm{j}}$ having the smallest distance ${\mathrm{X}}^2$ among them.

$${\mathrm{X}}^2\left({\mathrm{H}}_{\mathrm{i}},{\mathrm{H}}_{\mathrm{j}}\right)={\displaystyle \frac{1}{2}}\sum _{\mathrm{n}}{\displaystyle \frac{{({\mathrm{H}}_{\mathrm{i}}\left(\mathrm{n}\right)-{\mathrm{H}}_{\mathrm{j}}\left(\mathrm{n}\right))}^2}{{\mathrm{H}}_{\mathrm{i}}\left(\mathrm{n}\right)+{\mathrm{H}}_{\mathrm{j}}\left(\mathrm{n}\right)}}$$

(7)

Due to variations in binning the rectangles when the bounding box center of the human silhouette changes, ${\mathrm{X}}^2$ and ${\mathrm{L}}_2$ distance functions are very susceptible to noise. For these, we consider the distances among the bins at the expense of the longer computation time, and we can use the Earth Mover’s Distance [71] or Diffusion Distance [72], which have been shown to be most proficient for histogram assessment in the presence of shifts.

Global Histogramming: A global histogram method [73] is proposed using an approach similar to a Motion Energy Image (MEI). Motion energy mapping is a technique in which the dynamic energy of the movement is visualized and analyzed using a spatial map. A motion energy template provides a high-level representation of the video. All spatial histograms from the oriented rectangles in the sequence are summed up to represent the entire video compactly. A global histogram is formed based on each $\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}$ by summing all the histograms in ${\mathrm{H}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}$ and forming a global histogram based on the sum.

$${\mathrm{H}}_{\mathrm{g}\mathrm{l}\mathrm{a}\mathrm{b}\mathrm{a}\mathrm{l}}\left(\mathrm{d}\right)=\sum _{\mathrm{t}}{\mathrm{H}}_{\mathrm{t}}\left(\mathrm{d}\right)$$

(8)

There is a link between each test instance’s ${\mathrm{H}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}$ and the training instances ${\mathrm{X}}^2$, fomed using distances and a description of the nearby contests.

It is most efficient to use a global histogramming method with long sequences that captures the associated pose information with each pass. A histogram that tracks the accumulated results of a series of actions is more accurate when the sequences are long enough, such as in the case when each action is repeated multiple times. In the case of short sequences and a different number of cycles, the accumulated histogram may not deliver a good picture of the action’s structure.

K-Nearest Neighbor Technique Classification: The K-Nearest Neighbor (KNN) is a simple and easy-to-understand method for learning. Meanwhile, KNN has the advantage of being an effective deployment and implementation technique. Additionally, KNN has the advantage of working accurately with multiclasses, achieving better accuracy. Furthermore, it is a simple technique, which earned it a reputation in Machine Learning (ML). Thus, it is easy to implement, in addition to the advantages mentioned earlier. According to the classification approach, the closest training points for the human pose descriptor are used in the classification process. KNNs learn by analogy by comparing test cases with similar-looking training data [74]. Every row represents an $\mathrm{n}$ attribute (feature) in the $\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}$ space described by the training data. In the first step, KNN searches the dataset with the most similar pattern space to the test dataset. Therefore, this training dataset is considered the K-Nearest Neighbor of the test dataset. Distance metrics are used to compare training and test data. In an $\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{a}\mathrm{l}$ space, a KNN can also use the Euclidian distance to find the distance between two points. The Euclidian distance is calculated using Equation (9).

$$\mathrm{d}\mathrm{i}\mathrm{s}\mathrm{t}\left({\mathrm{X}}_1,{\mathrm{X}}_2\right)=\sqrt{\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{X}}_{1\mathrm{i}}-{\mathrm{X}}_{2\mathrm{i}}\right)}^2}$$

(9)

${\mathrm{X}}_1\mathrm{and}{\mathrm{X}}_2$ are the two points ${\mathrm{X}}_1=\left({\mathrm{X}}_{11},{\mathrm{X}}_{12},\dots ,{\mathrm{X}}_{1\mathrm{n}}\right),$ and ${\mathrm{X}}_2=({\mathrm{X}}_{21},{\mathrm{X}}_{22},\dots ,{\mathrm{X}}_{2\mathrm{n}})$. The two-level classification system of the proposed model is based on this observation. According to the average velocities of the training classifications, each action in action set A = $\left\{a_1\dots a_n\right\}$ is fitted with a Gaussian function. A subsequent probability is calculated for each action $a_i\in A$ over these Gaussians based on the test instance. If the subsequent probability $a_i$ of each action exceeds the threshold, we add it to the probable set of actions for a sequence $t=0.1$. After preprocessing, we estimate only the $a_k\in S$ output for actions in the second level. KNN classifiers that have the maximum response are selected to be used for classification.

Dynamic Time Warping (DTW): Actions are sequences of poses using the proposed representation. Each pose is represented as a distribution of rectangles. A 3D representation of an action is created by concatenating 2D feature vectors per pose. There is a wide variation in the time and speed of action sequences performed by different humans. Dynamic Time Warping (DTW) is the most popular method for handling similarities among time series. A similar approach to compare three-dimensional representations of action sequences is taken by the authors [75]. It compares two different 2D action representations located at the location of the exact rectangle with the help of DTW. We measure fluctuations at the exact rectangle location over time using a specific orientation and size of rectangles. The voxel grid of some body regions does not change throughout an action. In this way, some boxes in the rectangle histogram rarely change data series. To measure changes in rectangles with different sizes and orientations over time, we measure the variance at each rectangle’s location. A time warp is used in this model to align samples of consistent points. Using the specified two time series, $x_1\dots x_n$ and ${\mathrm{y}}_1\dots {\mathrm{y}}_{\mathrm{m}}$, the distance $\mathrm{D}(\mathrm{i},\mathrm{j})$ is calculated, as shown in Equation (10).

$$\mathrm{D}\left(\mathrm{i},\mathrm{j}\right)=\left\{\begin{array}{c}D\left(\mathrm{i},\mathrm{j}-1\right)\\ D\left(\mathrm{i}-1,\mathrm{j}\right)\\ D\left(\mathrm{i}-1,\mathrm{j}-1\right)\end{array}\right\}+\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{j}}\right)$$

(10)

where $\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{j}}\right)$ is the local distance function as per the application. The ${\mathrm{X}}^2$ distance function in the proposed implementation is $\mathrm{d}\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{j}}\right)$, as given in Equation (7). The histograms are shown in Figure 7 with DTW along each dimension. Each 2D series of the histogram react angles in the test video has a DTW distance of $\mathrm{D}\left(\mathrm{X}\left(\mathrm{d}\right),\mathrm{Y}\left(\mathrm{d}\right)\right)$ compared to its corresponding 2D series in the training video. The DTW distance ${\mathrm{D}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}$ between two videos is computed by adding both dimensions’ distances together. Training video instances are labelled with the smallest ${\mathrm{D}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}$ label.

$${\mathrm{D}}_{\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{l}}\left(\mathrm{X},\mathrm{Y}\right)=\sum _{\mathrm{d}=1}^{\mathrm{M}}\mathrm{D}\left(\mathrm{X}\left(\mathrm{d}\right),\mathrm{Y}\left(\mathrm{d}\right)\right)$$

(11)

Using the total number of rectangles in the histograms, we exclude the top k distances to reduce the effects of shifted rectangles. Accordingly, we choose $k=⌊\#\mathrm{n}\mathrm{u}{\mathrm{m}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{l}\mathrm{e}\mathrm{s}}/2⌋$ with $\#\mathrm{n}\mathrm{u}{\mathrm{m}}_{\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{g}\mathrm{l}\mathrm{e}\mathrm{s}}$ as the total rectangles in the spatial grid according to the feature vector size.

4. Experiments and Results Analysis

Four publicly available datasets are used to evaluate the Human Action Recognition framework: KTH [76], Weizmann [75] UT-Tower [77] and YouTube. The UT-Tower and Weizmann datasets contain silhouette sequences, whereas the YouTube and KTH datasets do not. The query segment was taken from a single action frame, and all other frames were used as training segments (except those from the same video sequence). The query segment has been categorized based on its action class. Splitting the datasets is crucial for evaluating the model’s performance. We applied five-fold cross-validation, since it ensures that the model is evaluated on different data subsets, reducing the bias caused by a single random split. Also, K-Fold splitting provides a robust evaluation by splitting the datasets into K datasets and testing the model on each while training on the remaining data. It also reduces the likelihood of overfitting and offers a very reliable estimate of the performance of the model across different subsets of data.

4.1. Datasets Used

KTH Dataset: KTH actions include six human actions: walking, running, jogging, hand waving, clapping, and boxing. It is one of the most commonly used datasets. It is a dataset in which 25 actors perform each action, and each actor’s setting is adjusted systematically per action. ${\mathrm{S}}_1$ Indicates an outdoor setting, ${\mathrm{S}}_2$ indicates an outdoor setting with scale variation, ${\mathrm{S}}_3$ indicates an outdoor setting with different clothing, and ${\mathrm{S}}_4$ indicates an indoor setting, as shown in Figure 8. A total of 2391 sequences are currently stored in the database. Each sequence was captured with a static camera at a 25 fps frame rate over a homogeneous background. The sequences are downsampled to 160 × 120 pixels with an average duration of four seconds. The techniques are tested with these variations by identifying actions regardless of the actors’ background, appearance or scale.

Weizmann Dataset: There are 2513 images of human activity in the Weizmann dataset. Nine actors represent five types of human behavior. An online dataset called the Weizmann dataset originally contained 81 low-resolution videos of nine subjects performing nine different actions (viz. running, jumping in place, bending, waving with one hand, jumping jacks, jumping sideways, jumping forward, walking, and waving with two hands). A 10th action was then added: jumping on one leg. Figure 9 illustrates each of these actions with examples.

UT-Tower Dataset: As part of the Semantic Description of Human Activities (SDHA) competition at ICPR 2010, the UT-Tower action dataset is used. With a pixel resolution of 1080 and a frame rate of 59 frames per second, the dataset contains 108 low-resolution video sequences. During the contest, video sequences are categorized according to nine types of human actions. Foreground masks, bounding boxes, and ground truth action labels accompany the video sequences. There are nine categories of human actions shown: pointing, standing, digging, walking, carrying, running, wave 1, wave 2, and jumping. Each of the nine actions was recorded in 12 video sequences. For each action category, six individuals performed them twice. The grass is used for carrying, running, wave1, wave2, jumping, and pointing, while concrete is used for standing, digging, and walking videos.

YouTube Dataset: The dataset based on YouTube videos is called the YouTube dataset. The 11 action classes in the dataset are bicycle riding, diving, basketball shooting, golf swinging, soccer juggling, horses, trampoline jumping, tennis swinging, walking with a dog, and volleyball spiking. The sample images of the YouTube dataset are presented in Figure 10. There are numerous variations in camera motion, cluttered background, acquisition viewpoint, etc.

4.2. Performance Evaluation Metrics

The proposed model was evaluated using evaluation metrics, such as Classification Correction Rate (CCR) and training loss. True Positives (TPs) and True Negatives (TNs) are recognized correctly, and False Positives (FPs) and False Negatives (FNs) are recognized incorrectly. The model is specifically proposed for accuracy sensitive applications.

Classification Correction Rate (CCR): The Correction Classification Rate (CCR) of a proposed model is calculated by adding up the True-Positive and True-Negative samples and dividing them by the sum of all samples—viz. TP, FP, TN, and FN, as mentioned below in Equation (12). The CCR is the percentage of correctly classified query segments in the dataset.

$$\mathrm{C}\mathrm{C}\mathrm{R}(\%)={\displaystyle \frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N}}}\times 100$$

(12)

Recall (R): The model’s sensitivity is measured with this metric. As a proportion of the total number of actions, the recall represents how many anomalous actions were detected, as mentioned in Equation (13).

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}\left(\mathrm{R}\right)={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}}$$

(13)

Precision (P): In terms of precision, it refers to the number of original actions detected among those detected. Measuring false alarms directly correlates with precision metrics. As false alarms are detected over the correct detection of anomalies, False Positives (FPs) are measured as mentioned in Equation (14).

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\left(\mathrm{P}\right)={\displaystyle \frac{\mathrm{T}\mathrm{P}}{(\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P})}}$$

(14)

The F1 score is an important metric that combines both precision and recall, resulting in a a single score, as mentioned below in Equation (15).

F1 score = (2 × Precision × Recall)/(Precision + Recall)

(15)

A confusion matrix is used to evaluate the performance of the model using TPs, FPs, FNs and TNs, as shown below. The diagonal elements are correctly predicted instances and off-diagonal elements are misclassification values.

TP	FN
FP	TN

Since this model is designed for improving accuracy, all the CCR values are compared.

4.3. Result Analysis and Discussion

This section describes our experimental results and examines the four publicly available human activity datasets for different contexts. A CNN comprises a convolution layer with partial weight sharing, employing a combination of 20 filters and a pool size of three. The reason for taking a pooling size of three is that a larger window, such as 4 × 4 or 5 × 5, would result in too much downsampling, thereby losing the finer details. However, taking a smaller window, such as 2 × 2, would preserve the information spatially, thereby reducing the effectiveness of dimensionality reduction, and it would also make the model less computationally efficient. The size of the filter plays an important role in founding the key features. The reason for using 20 filters in our model is that it allows the model to learn a wider range of features which is crucial for recognizing human activities, thereby leading to greater accuracy. The choice of taking 20 filters is also due to maintaining a balance between the model’s ability to learn and the computational efficiency. A top layer with a softmax is added to the two fully connected layers for generating state posterior probabilities. Human activity is classified using KNN as a label predictor.

A system equipped with a 64-bit processor with DDR4 RAM up to 2400 MHz and 64 GB and an Intel Core i7 was used to conduct this research. Since the work as conducted on i7, the training time for the model is 6 h. However, this training time could be significantly reduced by using GPU. The initial learning rate taken was 0.001. An adaptive learning rate optimizer was used as it adapts the learning rate. The learning rate was exponentially decayed as the model trained to allow for even finer adjustments while reaching the end of the training process. All the CNNs are trained for 50 epochs using cross-entropy loss and optimizing the network. These parameters were selected for every CNN trained to improve its convergence. The proposed method improved CCR when compared to the KTH, Weizmann, UT-Tower and YouTube datasets, as shown in Figure 11. The Weizmann dataset’s CCR is $98.88\%,$ which higher than that of the KTH, UT-Tower and YouTube datasets. The accuracy increased rapidly just after five epochs for all four datasets. The Weizmann and KTH datasets have low training loss and higher classification accuracy for Human Action Recognition.

When the calculated CCR is higher, human activities are classified more correctly. Classifying an entire sequence of frames containing an action is usually more helpful than just one in isolation. First, frames from each video sequence are obtained to classify the entire sequence since they provide time-localized action information. Figure 12 shows the CCR performance analysis based on the KTH dataset’s action. The running and hand-waving actions show a higher rate of CCR (%), but walking and handclapping show a similar performance. In the KTH dataset, boxing has a lower percentage recognition rate of 79% because boxing activity involves swift hand movements. The overall performance of the proposed model is highly acceptable because the lowest recognition rate is 79%, and the highest is 97.4% for the KTH dataset.

For a quantitative comparison, we have also presented a comparative analysis of the KTH dataset, shown in

Table 3. Comparative analysis of proposed model with state-of-the-art models on KTH dataset.

Authors	Proposed Method	NN Classifier	SLA Classifier	Wong et al.	Dollar et al.
CCR (%)	92.91	89.55	90.84	81	81.2

. The proposed model’s performance is compared to NN classifiers [78], SLA classifiers [79] Wong et al. [80] and Dollar et al. The proposed technique shows a higher classification rate as compared to the existing techniques for the KTH dataset. The proposed technique is remarkable because it outperforms all existing techniques with a CCR of $92.91\%.$ Figure 13 illustrates Table 3 graphically.

Figure 14 shows Weizmann dataset’s calculated CCR (%) that correctly classified the action. For bend, jack, sjump, side, walk, wave1 and wave2 human activities, the proposed technique achieved a CCR of 99.6%, 99.5%, 99.5%, 98.9%, 99.0%, and 97.4%. This suggests that jumps, runs, and skips are harder to distinguish from other actions, such as jacks, bends, sjumps, walks, wave1, side and waves. The side, bend, jack, and sjump actions have a higher recognition rate in the Weizmann dataset. Similarly, skip and run show the lowest recognition rate because of the video speed and orientation mismatch.

Compared with some current state-of-the-art techniques, the proposed technique performs better. The proposed technique is compared with the existing techniques of NN, SLA, Gorelick et al. and Ali et al. [81] The proposed technique shows a CCR performance of 98.88%, which is higher than that of the existing techniques. Figure 15 shows a graphical representation of

Table 4. Comparison with state-of-the-art models on Weizmann Dataset.

Authors	Proposed Method	NN Classifier	SLA Classifier	Gorelick et al.	Ali et al.
CCR (%)	98.88	97.05	96.74	97.83	95.75

.

Figure 16 shows the calculated CCR (%) for the UT-Tower dataset, correctly classifying the action. The proposed technique achieved a CCR (%) of 76.30, 96.80, 98.50, 99.97, 99.75, 89.60, 99.71, and 99.57 for pointing, standing, digging, walking, carrying, running, wave1, wave2, and jumping. This indicates that some actions, such as jumping, pointing and running, are more difficult to distinguish from others as they show the lowest recognition rate in the UT-Tower dataset.

The proposed model is quantitatively compared on the UT-Tower dataset with the existing models in the literature, as shown in

Table 5. Comparative analysis with other models on UT-Tower Dataset.

Authors	Proposed Method	NN Classifier	Liu et al.	Ikizler et al.	Le et al.	Wang et al.	Noguchi et al.
CCR (%)	87.97	78.5	71.2	75.2	75.8	84.2	80.4

. The proposed technique’s performance is compared with NN classifiers, Liu et al., Ikizler et al. [82], Le et al. [83], Wang et al. [84], Noguchi et al. [85] and techniques based on the Classification Correction Rate (CCR) (%). The proposed technique’s performance is remarkable because it outperforms all existing techniques, with a CCR of 87.97%. Figure 17 shows a graphical representation of Table 5.

Figure 18 shows the CCR (%) for the YouTube dataset that correctly classified the activity. The proposed technique achieved a CCR (%) of 77.70, 78.89, 94.80, 77.71, 88.55, 79.67, 89.57, 86.96, 95.55, 79.17, and 89,77 for basketball-shooting, biking/cycling, diving, golf swinging, horse riding, soccer juggling, swinging, tennis swinging, trampoline jumping, volleyball spiking, and walking with a dog. The YouTube dataset has complex and various activities that are difficult to recognize, but the proposed model shows more than a 75% performance improvement ratio. The driving, trampoline jumping, and walking with dog activities have a CCR rate of more than 90%.

Table 6. Comparative analysis with other models on YouTube dataset.

Authors	Proposed Method	NN Classifier	Liu et al.	Ikizler et al.	Le et al.	Wang et al.	Noguchi et al.
CCR (%)	87.77	78.5	71.2	75.2	75.8	84.2	80.4

and Figure 19 show a comparison between the proposed and existing state-of-the-art models on the YouTube dataset. Our proposed model outperforms other state-of-the-art techniques like NN classifiers, Liu et al. [78]. ,Ikizler et al. [82], Le et al. [83], Wang et al. [84], and Noguchi et al. [85].

Based on the results on the Weizmann KTH, UT-Tower and YouTube datasets, Figure 20 shows the precision and recall curve for the proposed model. The graphical representation of the precision versus recall represents that the proposed model shows higher performance for the Weizmann dataset and the lowest performance for YouTube dataset. The KTH dataset also shows a higher precision performance versus recall for the proposed approach than the UT-Tower dataset. The combined results are shown in

Table 7. Comparative analysis of our proposed method with state-of-the-art methods.

Authors	Dataset Used for Comparison	Classification Correction Rate (CCR %)
Proposed Method	KTH Dataset	92.91
	Weizmann Dataset	98.88
	UT-Tower Dataset	87.97
	YouTube Dataset	87.77
NN Classifier	KTH Dataset	89.55
	Weizmann Dataset	97.05
	UT-Tower Dataset	78.50
	YouTube Dataset	78.50
SLA Classifier	KTH Dataset	90.84
	Weizmann Dataset	96.74
Wong et al.	KTH Dataset	81.00
Dollar et al.	KTH Dataset	81.20
Gorelick et al.	Weizmann Dataset	97.83
Ali et al.	Weizmann Dataset	95.75
Liu et al.	UT-Tower Dataset	71.20
	YouTube Dataset	71.20
Ikizler et al.	UT-Tower Dataset	75.20
	YouTube Dataset	75.20
Le et al.	UT-Tower Dataset	75.80
	YouTube Dataset	75.80
Wang et al.	UT-Tower Dataset	84.20
	YouTube Dataset	84.20
Noguchi et al.	UT-Tower Dataset	80.40
	YouTube Dataset	80.40

. This clearly shows that our proposed method significantly outperforms all other methods taken for comparison on the abovementioned datasets.

5. Conclusions and Future Work

The increasing number of surveillance cameras installed in public and private locations has made the computer-based automatic and intelligent video analysis of video sequences increasingly important. This paper proposes a novel approach for Human Action Recognition using a Motion History Mapping and Orientation-based CNN. In the proposed work, the human body is signified by rectangular patches, and this technique is based on the positions and orientation of the rectangular patches. These rectangular patches are changed over time concerning their actions. To evaluate our model’s performance, we have conducted extensive experiments on four challenging, widely used and publicly available datasets: KTH, Weizmann, YouTube, and UT-Tower. The obtained experimental results are compared with existing state-of-the-art models. The performance of the proposed model was evaluated in terms of the Classification Correction Rate (CCR). According to the results, our proposed model offers significantly superior and accurate performance when compared to the other models for extracting human actions, i.e., 92.91 (% CCR) for the KTH dataset, 98.88 (% CCR) for the Weizmann dataset, 87.97 (% CCR) for the UT-Tower dataset and 87.77 (% CCR) for the YouTube dataset. The proposed method outperforms the existing methods and is also computationally efficient, thus making our model an important contribution in the field of Human Activity Recognition.

In future work, the proposed model can be combined with the transfer learning approach, where we can pre-train our model on the curated dataset and fine-tune it on a smaller, in-the-wild dataset. This would help the model to leverage the knowledge learnt in the curated dataset and adapt it to new and complex scenarios of the real-world, in-the-wild dataset. This is because the transfer learning approach would allow the model to handle a variety of noise patterns. Also, the proposed work can be extended by using a multi-view fusion approach. This technique can significantly reduce the memory and computational requirements while still capturing wide spatial information. The limitation of our model is that the 3D CNNs lack an attention mechanism. The attention-based deep learning technique allows the network to emphasize important features while suppressing irrelevant ones. Attention-based deep learning and transformer-based mechanisms could extend this work and further enhance performance.