MPE-HRNet^L: A Lightweight High-Resolution Network for Multispecies Animal Pose Estimation

Abstract

Animal pose estimation is crucial for animal health assessment, species protection, and behavior analysis. It is an inevitable and unstoppable trend to apply deep learning to animal pose estimation. In many practical application scenarios, pose estimation models must be deployed on edge devices with limited resource. Therefore, it is essential to strike a balance between model complexity and accuracy. To address this issue, we propose a lightweight network model, i.e., MPE-HRNet${\phantom{.}}^L$, by improving Lite-HRNet. The improvements are threefold. Firstly, we improve Spatial Pyramid Pooling-Fast and apply it and the improved version to different branches. Secondly, we construct a feature extraction module based on a mixed pooling module and a dual spatial and channel attention mechanism, and take the feature extraction module as the basic module of MPE-HRNet${\phantom{.}}^L$. Thirdly, we introduce a feature enhancement stage to enhance important features. The experimental results on the AP-10K dataset and the Animal Pose dataset verify the effectiveness and efficiency of MPE-HRNet${\phantom{.}}^L$.

1. Introduction

Animal pose estimation aims to identify and locate the key points of animal bodies. It plays a crucial role in assessing animal health [1], protecting endangered species [2], and analyzing animal behavior [3]. With the advancement of deep learning technology, it has been an inevitable trend to apply the technology to animal pose estimation. Deep-learning-based solutions can be categorized into bottom-up solutions and top-down solutions [4]. The bottom-up solutions detect key points and then cluster the key points to generate specific poses [5,6]. Although these solutions are relatively fast, they cannot guarantee accuracy in complex scenes. Different from the bottom-up solutions, the top-down solutions [7,8,9] identify individual animal objects, and then estimate the poses of these objects, yielding higher accuracy [5,6].

A lot of effort has been devoted to the top-down solutions [10,11]. For example, some researchers utilize large network models, such as Hourglass [12] and HRNet [8], to achieve higher accuracy at the cost of high complexity [13,14,15,16,17,18,19]. In many practical application scenarios, pose estimation models must be deployed on edge devices. Edge devices are always equipped with limited resource. They are unable to leverage the advantages of large models since they cannot provide the resource that large models need. Inspired by ShuffleNet [20], Yu et al. [9] proposed a lightweight version of HRNet, i.e., Lite-HRNet. Lite-HRNet can estimate the poses of human beings with relatively high speed and accuracy. However, it shows an obvious drop in accuracy when estimating the poses of animals of multiple species. That happens because of two reasons. The first reason is that animals of different species possess different anatomical structures, resulting in significant variations in their poses. The second reason is that Lite-HRNet sacrifices accuracy in favor of a light network structure.

In this paper, we propose a lightweight network to estimate the poses of multiple species on edge devices with higher accuracy. We name the lightweight network MPE-HRNet${\phantom{.}}^L$ and construct it by improving Lite-HRNet from three aspects. Firstly, we improve Spatial Pyramid Pooling-Fast (SPPF) and propose SPPF${\phantom{.}}^{+}$. We apply SPPF and SPPF${\phantom{.}}^{+}$ to different branches in order to enrich the receptive fields of feature maps and mitigate the impact of feature loss caused by pooling operations. Secondly, we propose a mixed pooling module (MPM) and a dual spatial and channel attention mechanism (DSCA). Leveraging MPM and DSAM, we construct a mixed feature extraction (MFE) block with two different versions. Thirdly, we design a feature enhancement (FE) stage, and append it as the last stage of MPE-HRNet${\phantom{.}}^L$. The main contributions of this work are as follows.

• We propose SPPF${\phantom{.}}^{+}$. Comparing with SPPF, SPPF${\phantom{.}}^{+}$ has a stronger feature extraction capability and lower computational load.
• We propose MPM and DSCA. Based on MPM and DSCA, we design MFE block which possesses strong feature extraction capabilities.
• We design the FE stage and take it as the final stage of MPE-HRNet${\phantom{.}}^L$. This stage is introduced to emphasize the semantic feature in the output feature map of MPE-HRNet${\phantom{.}}^L$.

The rest of this paper is organized as follows. Section 2 introduces the theoretical basis. Section 3 elaborates MPE-HRNet${\phantom{.}}^L$. After the experimental results are presented in Section 4, the paper is concluded in Section 5.

2. Related Works

This section presents the significant advancements in animal pose estimation and two important techniques of deep learning. Firstly, Section 2.1 summarizes the research status of animal pose estimation. Secondly, Section 2.2 addresses the evolution and application of attention mechanisms. Finally, Section 2.3 discusses the advantage and the development of spatial pyramid pooling (SPP).

2.1. Animal Pose Estimation

Traditional solutions [21] for animal pose estimation involve obtaining behavioral data through manual observation or attaching sensors to animals. These methods incur high labor costs and implementation difficulties, along with challenges related to equipment vulnerability and data collection for rare species. Deep-learning-based pose estimation solutions can effectively alleviate these problems [22]. These solutions can be classified into bottom-up solutions and top-down solutions. Considering accuracy, this paper solely focuses on the top-down solutions.

To improve accuracy, large network models have been introduced to some of the top-down solutions. Graving et al. [17] developed a software toolkit based on Hourglass and DenseNet [23] to estimate keypoint locations with subpixel precision. Zhou et al. [16] improved Hourglasses to model the differences and spatial correlations between mouse body parts. Wang et al. [13] designed a grouped pig pose estimation model which exploited HRNet [24] to predict the heatmap of keypoints of pigs. Fan et al. [14] and Zhao et al. [18] also exploited HRNet. They constructed a pose estimation network for cattle and Jinling ducks, respectively. He et al. [15] combined Vision Transformer (ViT) [25] and HRNet, and constructed a model for birds. The above solutions are only applicable in scenarios involving single-species pose estimation.

In multispecies pose estimation scenarios, Gong et al. [19] applied dynamic convolution and a receptive field block to HRNet, thereby significantly enhancing the feature extraction of quadrupeds. In order to address the challenges in animal data annotation, Liao et al. [26] proposed THANet to transfer human pose estimation to animal pose estimation. Different from the solution of Liao et al. [26], Hu and Liu [27] utilized a language–image contrastive learning model to address the challenges, while Zeng et al. [28] employed a semi-supervised method with MVCRNet to utilize unlabeled data. All these solutions exploit large models. In many practical application scenarios, models are always deployed on edge devices. Because of the limited memory and computing power, large models are not suitable for those devices. We try to make a trade-off between accuracy and complexity, and design a lightweight model to run animal pose estimation on edge devices with relatively high accuracy.

2.2. Attention Mechanism

Attention mechanism [29] allows models to focus on important areas in an image by mimicking the human visual system. It calculates a weight matrix for an input feature map and depends on the weight matrix to highlight the important features and weaken the irrelevant features in the feature map. SENet [30], ECANet [31], FcaNet [32], and CBAM [33] are all attention mechanisms.

SENet is composed of a squeeze module and an excitation module. The squeeze module extracts global spatial information from input feature maps and then compresses these feature maps using global average pooling. The excitation module uses a fully connected layer to capture the relationships among these compressed feature maps and output an attention weight matrix for the input feature maps. Wang et al. [31] believe that the fully connected layer in the excitation module increases the complexity of SENet. They replaced the fully connected layer with a 1D convolution, from which they came up with ECANet. Qin et al. [32] also made improvements to SENet. They replaced the global average pooling with the following operations: grouping the feature maps and then processing the grouped feature maps with a 2D discrete cosine transform function. CBAM is a hybrid attention mechanism that combines the channel features and spatial features of input feature maps to figure out where to focus and what to focus on.

2.3. Spatial Pyramid Pooling

Convolutional neural networks (CNNs) [34] often require input images to meet a special requirement of size to ensure that the fully connected layer can output feature vectors of a specific size. Traditional methods exploit cropping and stretching to generate images of a specific size, but this can result in image distortion and information loss. SPP [35] utilizes pooling operations of different scales to transform images of different sizes into a specific size, thus avoiding stretching or cropping images. Chen et al. [36] replaced the pooling layers in SPP with multiple dilated convolutions with different dilation factors to generate feature maps with larger receptive fields. Jocher [37] improved SPP and proposed SPPF. SPPF exploits multiple concatenated pooling layers of the same scale to process input feature maps, and fuses the pooling results in parallel to improve speed.

3. Proposed Algorithm

In this section, we illustrate the proposed MPE-HRNet${\phantom{.}}^L$. We outline the structure of MPE-HRNet${\phantom{.}}^L$ in Section 3.1, and then elaborate the improved version of SPPF in Section 3.2. After representing the details of MFE block in Section 3.3, we describe the feature enhancement stage in Section 3.4.

3.1. Architecture of MPE-HRNet${\phantom{.}}^L$

To construct a lightweight network for accurately estimating the poses of multiple animal species, we improved Lite-HRNet and proposed MPE-HRNet${\phantom{.}}^L$. The improvements include optimizing branches with SPPF, SPPF${\phantom{.}}^{+}$ and MFE block, and introducing a feature enhancement (FE) stage. The branch optimization is designed to enhance the feature extraction and representation capabilities of MPE-HRNet${\phantom{.}}^L$. The introduction of the FE stage aims to enhance the important features in the final outputs of MPE-HRNet${\phantom{.}}^L$. Figure 1 and Table 1 show the architecture and the details of MPE-HRNet${\phantom{.}}^L$, respectively.

Figure 1. Overall architecture of MPE-HRNet${\phantom{.}}^L$. MPE-HRNet${\phantom{.}}^L$ includes five stages. MFE block and SPPF${\phantom{.}}^{+}$/SPPF are applied to stages 2 to 4. FE stage is appended as the final stage. The yellow cuboids describe the feature maps processed by our improvements.

Table 1. Structure details of MPE-HRNet${\phantom{.}}^L$.

Stages	Output Size	Operations	Resolution Branch	Output_Channels	Repeat	Num_Modules
stem	64 × 64	conv2d	2 ×	32	1	1
		shuffle block	4 ×	32	1	1
2	64 × 64	SPPF${\phantom{.}}^{+}$	4 × 8 ×	40, 80	1	1
		MFE block	4 × 8 ×	40, 80	2	2
		fusion block	4 × 8 ×	40, 80	1
3	64 × 64	SPPF${\phantom{.}}^{+}$	4 × 8 × 16 ×	40, 80, 160	1	1
		MFE block	4 × 8 × 16 ×	40, 80, 160	2	2
		fusion block	4 × 8 × 16 ×	40, 80, 160	1
4	64 × 64	SPPF	4 × 8 × 16 × 32 ×	40, 80, 160	1	1
		MFE block	4 × 8 × 16 × 32 ×	40, 80, 160	2	2
		fusion block	4 × 8 × 16 × 32 ×	40, 80, 160	1
FE	64 × 64	upsampling	4 ×	80	1	1
		shuffle block	4 ×	80	1
		DConv	4 ×	40	1
		ECA	4 ×	40	1

As Figure 1 and Table 1 show, stage stem only creates a new branch with a conv2d block and a shuffle block. Both stage 2 and stage 3 create a new branch, and apply SPPF${\phantom{.}}^{+}$ to the branches created by the previous stage of each of them. They stack multiple MFE blocks to extract features from all the branches except the ones created by themselves. Similar to stage 2 and stage 3, stage 4 also applies stacked MFE blocks to branches. The difference is that stage 4 does not create any new branches, and it applies SPPF instead of SPPF${\phantom{.}}^{+}$ to the branch created by its previous stage. Stage FE is quite different from the former four stages. It is introduced to enhance the features output by stage 4.

3.2. SPPF⁺

Lite-HRNet utilizes downsampling to create new branches. However, downsampling can only generate feature maps with limited receptive fields. It could probably also lose some important features. SPPF can generate feature maps with rich receptive fields by extracting multiple features of varying scales and fusing them together. However, SPPF has three obvious limitations. Firstly, 1 × 1 convolution struggles to extract complex features. Secondly, pooling operations also lose some features. Finally, concat operation increases computational complexity. To address these issues, we improve SPPF and propose SPPF${\phantom{.}}^{+}$.

In order to enhance the ability to extract features, SPPF${\phantom{.}}^{+}$ replaces the 1 × 1 convolution with a depthwise separable convolution. To ensure that the feature maps generated by pooling operations contain rich features, SPPF${\phantom{.}}^{+}$ inserts fusion operations among the sequential pooling operations. Concretely, it fuses the results of the initial two pooling operations, and then processes the fused results with the last pooling operation. To reduce the computational complexity caused by the concatenation operation, SPPF${\phantom{.}}^{+}$ refrains from concatenating the feature maps produced by the pooling operations with the one yielded by the depthwise separable convolution. Instead, it fuses the results of the last pooling operation with the fused results of the first two pooling operations, and then concatenates the fused results with the results of the depthwise separable convolution. SPPF${\phantom{.}}^{+}$ employs ADD operations to achieve fusion.

Figure 2 illustrates the architecture of SPPF${\phantom{.}}^{+}$. $spp{f}^{+}({\mathbf{X}}_s)$ denotes the outputs of SPPF${\phantom{.}}^{+}$ with the given input ${\mathbf{X}}_s$, which includes multiple channels. It is computed according to Equation (1). In the equation, $con{v}_{1\times 1}(·)$ represents 1 × 1 convolution, $concat(·)$ denotes the concatenation operation, $con{v}_{DC}(·)$ describes depthwise separable convolution, and $mma{p}_p(·)$ represents a series of max pooling operations. $mma{p}_p(·)$ is calculated according to Equations (2)–(5).

$$spp{f}^{+}({\mathbf{X}}_s)=con{v}_{1\times 1}(concat(con{v}_{DC}({\mathbf{X}}_s),mma{p}_p({\mathbf{X}}_s))$$

(1)

$$mma{x}_p({\mathbf{X}}_s)=sma{x}_p({\mathbf{X}}_s)+dma{x}_p({\mathbf{X}}_s)+tma{x}_p({\mathbf{X}}_s)$$

(2)

$$sma{x}_p({\mathbf{X}}_s)=ma{x}_p(con{v}_{DC}({\mathbf{X}}_s))$$

(3)

$$dma{x}_p({\mathbf{X}}_s)=ma{x}_p(sma{x}_p({\mathbf{X}}_s))$$

(4)

$$tma{x}_p({\mathbf{X}}_s)=ma{x}_p(sma{x}_p({\mathbf{X}}_s)+dma{x}_p({\mathbf{X}}_s))$$

(5)

Figure 2. Structures of SPPF${\phantom{.}}^{+}$. © and ⊕ represent concat operation and add operation, respectively.

3.3. Mixed Feature Extraction Block

Lite-HRNet exploits the conditional channel weight block (CCW) to extract features. The CCW includes two weight calculation functions, ${\mathcal{H}}_s$ and ${\mathcal{F}}_s$. ${\mathcal{H}}_s$ employs average pooling to generate low-resolution feature maps, which may loss some important details. ${\mathcal{F}}_s$ neglects spatial information when computing weight matrices, which is crucial for localizing keypoints. To address these issues, we propose WGM along with DSCA, and construct the MFE block based on them, which serves as the foundational module for feature extraction.

3.3.1. Weight Generation Module

To alleviate the feature loss caused by average pooling operation in ${\mathcal{H}}_s$, we propose a mixed pooling module (MPM) and construct WGM by replacing the pooling operation in ${\mathcal{H}}_s$ with MPM. Figure 3 shows the structure of WGM.

Figure 3. Structure of WGM. The top part shows the structure of WGM, and the bottom part describes the structure of MPM. Matrix transformation represents a series of operations, e.g., reshaping, unsqueezing, and permuting, on the dimensions or sizes of three−dimensional matrices. $H^T$ and $W^T$ denote the height and width of a transformed matrix, and N is calculated by dividing $H^T{W}^T$ by $HW$.

In the figure, the area surrounded by dashed lines illustrates the details of MPM. As the figure shows, MPM consists of a context-aware branch, an average pooled branch, and a maximum pooled branch. The three branches are designed to capture spatial contextual features, highlight local prominent features, and preserve global features, respectively.

The context-aware branch uses a point-wise convolution to extract spatial contextual features, followed by a $softmax$ activation to generate a weight matrix. Subsequently, the weight matrix and the input feature maps undergo matrix transformations and multiplication to map the spatial contextual features into the input feature maps.

Taking ${\mathbf{X}}_m$ as the input of MPM, ${\mathbf{X}}_m$ can be described as (${\mathbf{X}}_{m_0},{\mathbf{X}}_{m_1},\cdots ,{\mathbf{X}}_{m_{c-1}}$), where c denotes the total number of channels in ${\mathbf{X}}^m$. The output of MPM is denoted as $mpm({\mathbf{X}}_m)$. It is calculated according to Equation (6). In the equation, $mbrch({\mathbf{X}}_m,i)$ denotes the results of the i-th branch in MPM. It is calculated by Equation (7). In the equation, $gap(·)$ and $gmp(·)$ separately describe global average pooling and global max pooling; $\delta (·)$ denotes the $softmax$ function, and ${(·)}^{\mathit{T}}$ represents matrix transformation.

$$mpm({\mathbf{X}}_m)=\sum _{i=0}^{2}mbrch({\mathbf{X}}_m,i)\times {\theta }_i$$

(6)

$$mbrch({\mathbf{X}}_m,i)=\left\{\begin{array}{cc}{({(\delta ({(con{v}_{1\times 1}({\mathbf{X}}_m))}^{\mathit{T}}))}^{\mathit{T}}\times {\mathbf{X}}_m^{\mathit{T}})}^{\mathit{T}},\hfill & i=0\hfill \\ gap({\mathbf{X}}_m),\hfill & i=1\hfill \\ gmp({\mathbf{X}}_m),\hfill & i=2\hfill \end{array}\right.$$

(7)

Figure 3 also shows the structure of WGM. According to the figure, WGM first performs mixed pooling on input feature maps and then concatenates the pooled results. After that, it processes the concatenated results using two sequential 1 × 1 convolutions, followed by a sigmoid function to the outputs of the convolutions to generate weight matrices.

Taking ${\mathbf{X}}_w$ as the input of WGM, ${\mathbf{X}}_w$ can be described as (${\mathbf{X}}_{w_0},{\mathbf{X}}_{w_1},\cdots ,{\mathbf{X}}_{w_{b-1}}$), where b denotes the total number of the existing branches in the corresponding stage. Moreover, $\forall {\mathbf{X}}_{w_i}({\mathbf{X}}_{w_i}\in {\mathbf{X}}_w)$, ${\mathbf{X}}_{w_i}$ comprises multiple channel feature maps. For the given input ${\mathbf{X}}_w$, the result of WGM is denoted as $wgm({\mathbf{X}}_w)$. $wgm({\mathbf{X}}_w)$ is calculated by Equation (8). In the equation, $ups(·)$ denotes the upsampling operation, and $\sigma (·)$ represents the sigmoid function. $\forall {\mathbf{X}}_{w_i}({\mathbf{X}}_{w_i}\in {\mathbf{X}}_w)$, WGM calculates a weight matrix for each channel feature map in ${\mathbf{X}}_{w_i}$.

$$wgm({\mathbf{X}}_w)=ups(\sigma (co{n}_{1\times 1}(co{n}_{1\times 1}(concat({\mathbf{X}}_w)))))$$

(8)

3.3.2. Dual Spatial and Channel Attention Module

Each stage includes both low-resolution branches and high-resolution branches with channel and spatial features. Channel features are richer than spatial features in low-resolution feature maps, while both channel features and spatial features are rich in high-resolution feature maps. ${\mathcal{F}}_s$ calculates weights based solely on channel features. It does not capitalize on spatial features which are vital for keypoint localization. To leverage both channel and spatial features for weight calculation, we introduce a dual spatial attention module (DSA) and a hybrid attention mechanism, i.e., DSCA. DSCA is constructed by combining DSA with a channel attention module. Figure 4 shows the structure of DSCA.

Figure 4. Structure of DSCA. It comprises a channel attention module and a DSA module. The area surrounded by red dashed lines describes the structure of DSA. ⨀ represents element-wise multiplication.

DSA utilizes four branches to extract spatial features. The first branch processes each input feature map with max and average pooling, then performs a weighted sum. The second, third, and fourth branches separately use a 1 × 1 convolution, a 3 × 3 convolution, and a 5 × 5 convolution to process each feature map and generate spatial features of various receptive fields. The first and second branches extract global spatial features, capturing the overall structure and layout relationships in each input feature map to obtain higher-level, more abstract semantic information. The third and fourth branches focus on extracting local spatial features, enhancing the attention to different regions within an input image.

After extracting spatial features, DSA concatenates the output of each branch and employs a 7 × 7 convolution to process the concatenated results. It generates a spatial attention weight matrix by applying sigmoid activation function to the results of the 7 × 7 convolution. Given the input ${\mathbf{X}}_d$ with c channels, the result of DSA is calculated according to Equation (9). In the equation, $\sigma$ denotes sigmoid function, $con{v}_{7\times 7}$ describes the $7\times 7$ convolution, $conca{t}_{i=0}^3(·)$ represents the operation to concatenate the results of the four branches, and $dbrch({\mathbf{X}}_d,i)$ depicts the results of the i-th branch. $dbrch({\mathbf{X}}_d,i)$ is calculated according to Equation (10) in which $mp(·)$ and $ap(·)$ describe max pooling and average pooling, respectively.

$$dsa({\mathbf{X}}_d)=\sigma (con{v}_{7\times 7}(conca{t}_{i=0}^3(dbrch({\mathbf{X}}_d,i))))$$

(9)

$$dbrch({\mathbf{X}}_d,i)=\left\{\begin{array}{cc}0.5(mp({\mathbf{X}}_d)+ap({\mathbf{X}}_d)),\hfill & i=0\hfill \\ con{v}_{1\times 1}({\mathbf{X}}_d),\hfill & i=1\hfill \\ con{v}_{3\times 3}({\mathbf{X}}_d),\hfill & i=2\hfill \\ con{v}_{5\times 5}({\mathbf{X}}_d),\hfill & i=3\hfill \end{array}\right.$$

(10)

$dsca(·)$ denotes the outputs of DSCA. It can be calculated according to Equation (11). In this equation, $ca(·)$ represents the output of a channel attention module, and ⊙ denotes element-wise multiplication. To simultaneously utilize both channel and spatial features for weight calculation, we utilize DSCA to replace ${\mathcal{F}}_s$ to compute weights.

$$dsca({\mathbf{X}}_d)=ca({\mathbf{X}}_d)\odot dsa(ca({\mathbf{X}}_d)\odot {\mathbf{X}}_d)$$

(11)

3.3.3. Mixed Feature Extraction Block

We construct MFE block by substituting ${\mathcal{H}}_s$ with WGM and ${\mathcal{F}}_s$ with DSCA, respectively. MFE has two architectures, namely, MFE-H and MFE-L, which utilize WGM and DSCA in different manners. MFE-H utilizes both WGM and DSCA, while MFE-L only uses WGM. Figure 5 compares the structures of MFE block with CCW block.

Figure 5. Structures of MFE. Both MFE-H and MFE-L exploit WGM to generate weighted feature maps as the input of the 3 × 3 DWConv. MFE-H also utilizes the proposed DSCA to generate the weighted feature maps involving in the concat operation, while MFE-L still utilizes ${\mathcal{F}}_s$ to generated weighted feature maps.

High-resolution branches are rich in both channel and spatial features, while low- resolution branches are rich only in channel features. Thus, we apply MFE-H to the two highest-resolution branches and MFE-L to the others.

3.4. Feature Enhancement Stage

Lite-HRNet uses bilinear interpolation in stage 4 to perform upsampling and feature fusion step by step from the 1/32 branch, until generating the output feature maps of the 1/4 branch, which is used for generating heatmaps and locating keypoints. However, bilinear interpolation only considers the subpixel neighborhood and generates feature maps within small receptive fields, which is incapable of capturing the rich semantic information needed for pose estimation. Therefore, we introduce the FE stage to capture more semantic information.

Figure 6 illustrates the structure of the FE stage. The FE stage utilizes content-aware reassembly of features (CARAFE) [38] to upsample input feature maps from 1/8 to 1/4 resolution. Then, it uses the efficient channel attention module (ECA) to process the generated feature maps, and fuses the processed results with the input feature maps at 1/4 resolution. Finally, it processes the fused results sequentially with shuffle block, depthwise separable convolutions, and ECA to create feature maps for heatmap generation and keypoint localization.

Figure 6. Structure of FE stage. The cuboids represent feature maps. © denotes concat operation.

4. Experiments

In this section, we conduct experiments on two public benchmark datasets to evaluate MPE-HRNet${\phantom{.}}^L$. Firstly, we represent the details of datasets, training, and evaluation metrics in Section 4.1. Secondly, we analyze the ablation experiments in Section 4.2 and comparative experiments in Section 4.3, respectively. Finally, we discuss the limitation of HRNet${\phantom{.}}^L$ in Section 4.4.

4.1. Experimental Settings

Dataset. We use two datasets, i.e., AP-10K [39] and Animal Pose [40], to evaluate our solution. AP-10K dataset is the a large-scale dataset for animal pose estimation. It includes 10,015 images collected from 23 families and 54 species of animals. The AP-10K dataset contains a total of 13,028 instances, each of which is labeled with 17 key points to describe poses. Table 2 shows the details of those keypoints. In our experiments, the AP-10K dataset is divided into train set, validation set, and test set. The three datasets include 7010 images, 1002 images, and 2003 images, respectively. The Animal Pose dataset contains 4608 images collected from dog, cat, cow, horse, and sheep species. It includes 6117 instances annotated with 20 keypoints. The Animal Pose dataset is also divided into train set, validation set, and test set. Each of these sets include 2798 images, 810 images, and 1000 images, respectively.

Table 2. Definitions of keypoints.

Keypoint	Definition	Keypoint	Definition
0	Left Eye	9	Right Elbow
1	Right Eye	10	Right Front Paw
2	Nose	11	Left Hip
3	Neck	12	Left Knee
4	Root of Tail	13	Left Back Paw
5	Left Shoulder	14	Right Hip
6	Left Elbow	15	Right Knee
7	Left Front Paw	16	Right Back Paw
8	Right Shoulder

Training. All the training is carried out on the same server which is equipped with 5 NVIDIA GeForce RTX 3090 GPUs sourced from Santa Clara, CA, USA, and 376 GB of Kingston memory sourced from Fountain Valley, CA, USA. Our solution and all the comparative solutions are implemented in Python 3.8.19, and executed on PyTorch 2.5.0 framework. Multiple data augmentations are adopted in the training, which include random horizontal flip, random rotation from −80 to 80 degrees, random scaling by a factor of 0.5–1.5, and random body part augmentation generating animals with only the upper or lower part of a body.

To ensure the fairness of training processes, the training parameters, batch_size and the number of epochs, are separately set to 64 and 260, while all the other training parameters are configured with their default values. All parameters are updated by Adam optimizer combined with warm-up learning strategy. Learning rate is configured to $5\times {10}^{-4}$ at the first 170 epochs and reduced to $5\times {10}^{-5}$ and $5\times {10}^{-6}$ at the 170th and 200th epochs, respectively.

Evaluation metrics. Object keypoint similarity (OKS) is a widely accepted criterion for pose estimation. It is calculated according to Equation (12). In the equation, $d_i$ describes the Euclidean distance between the model prediction and the ground truth of keypoint i, $v_i$ denotes the visibility flag of the ground truth of the key point, s represents the scale of the animal object, and $k_i$ describes the constant to control the fall-off of the keypoint.

$$OKS={\displaystyle \frac{{\sum }_i{e}^{\frac{-d_i^2}{2s^2{\hslash }_i^2}}\delta ({\nu }_i>0)}{{\sum }_i\delta ({\nu }_i>0)}}$$

(12)

We use the average precision (AP), $A{P}^{50}$ (AP at OKS = 0.5), $A{P}^{75}$ (AP at OKS = 0.75), $A{P}^M$ (AP for medium objects: 322 < area < 962), $A{P}^L$ (AP for large objects: area > 962), and average recall (AR) to evaluate the accuracy of models. AP is regarded as the main metric for accuracy. In addition, we use Params (M) to evaluate complexity, and use FLOPs (G) and inference time (ms) to evaluate speed.

4.2. Ablation Experiments

To analyze the effectiveness of the improvements in this work, we carry out a series of ablation experiments.

4.2.1. Selection of SPPF⁺ and SPPF

SPPF⁺ fuses the feature maps generated by different pooling operations by add operation. Add operation adds feature maps related to the same channel. It can merge multiple groups of feature maps generated by different pooling operations into one group, thus reducing parameters for subsequent operations. However, it also results in feature loss. Fortunately, the feature loss could be alleviated if the feature maps involving the same add operation are much similar to each other.

For the feature maps generated by the stacked pooling operations in SPPF${\phantom{.}}^{+}$, the similarity between the feature maps related to the same channel increases with the increase in their resolutions. Therefore, SPPF${\phantom{.}}^{+}$ is more suitable for the branches of high resolutions. SPPF does not suffer from the issue. MPE-HRNet${\phantom{.}}^L$ contains multiple branches of different resolutions. In order to improve accuracy through the proper use of SPPF${\phantom{.}}^{+}$ and SPPF in these branches, we carry out the experiments in which SPPF/SPPF${\phantom{.}}^{+}$ are added at the beginning of different branches.

Table 3 records the experiment results of Lite-HRNet and its variants only involving SPPF, SPPF${\phantom{.}}^{+}$, or both of them. +SPPF${\phantom{.}}^{1/8,1/16,1/32}$ denotes the variant with SPPF in the 1/8, 1/16, and 1/32 branches. +SPPF${\phantom{.}}^{+1/8}$ and SPPF${\phantom{.}}^{1/16,1/32}$ describes the variant version with SPPF${\phantom{.}}^{+}$ in the 1/8 branch and SPPF in the 1/16 and 1/32 branches. +SPPF${\phantom{.}}^{+1/8,1/16}$ and SPPF${\phantom{.}}^{1/32}$ represent the variant version with +SPPF${\phantom{.}}^{+}$ in the 1/8 and 1/16 branches and SPPF in the 1/32 branch. +SPPF${\phantom{.}}^{+1/8,1/16,1/32}$ indicates the variant version with SPPF${\phantom{.}}^{+}$ in the 1/8, 1/16, and 1/32 branches. From the experimental results, it can be observed that inserting SPPF${\phantom{.}}^{+}$ in the 1/8 and 1/16 branches and SPPF in the 1/32 branch yields the best results. In the following experiments, we configure SPPF and SPPF${\phantom{.}}^{+}$ according to the best results.

Table 3. Effectiveness study of SPPF${\phantom{.}}^{+}$ and SPPF.

Models	AP	AR	Params (M)	FLOPs (G)
Lite-HRNet	59.8	64.9	1.13	0.35
+SPPF${\phantom{.}}^{\frac{1}{8},\frac{1}{16},\frac{1}{32}}$	61.2	65.9	1.60	0.63
+SPPF${\phantom{.}}^{+\frac{1}{8}}$ & SPPF${\phantom{.}}^{\frac{1}{16},\frac{1}{32}}$	61.3	66.0	1.58	0.63
+SPPF${\phantom{.}}^{+\frac{1}{8},\frac{1}{16}}$ & SPPF${\phantom{.}}^{\frac{1}{32}}$	62.0	66.3	1.56	0.62
+SPPF${\phantom{.}}^{+\frac{1}{8},\frac{1}{16},\frac{1}{32}}$	61.2	66.0	1.46	0.62

4.2.2. Effectiveness Study of All the Improvements

We design five different models to evaluate the effectiveness of each improvement and describe the corresponding experiment results in Table 4. All these results are generated on the AP-10K dataset. In the table, Model0 denotes the original version of Lite-HRNet, while Model1 to Model4 describe different variants of the original version. Model1 and Model2 separately denote the variant with MFE-L in all the branches and the variant with MFE-L in 1/16 and 1/32 branches and MFE-H in 1/4 and 1/8 branches. Model3 represents the variant with MFE-L in 1/16 and 1/32 branches, MFE-H in 1/4 and 1/8 branches, and FE stage.

Table 4. Effectiveness study of all the improvements.

Models	MFE-Lall	MFE-L${\phantom{.}}^{1/16,1/32}$ & MFE-H${\phantom{.}}^{1/4,1/8}$	FE Stage	SPPF+ & SPPF	AP	AR	Params (M)	FLOPs (G)
model0					59.8	64.9	1.13	0.35
model1	✓				60.1	65.0	1.13	0.36
model2		✓			60.4	65.2	1.17	0.44
model3		✓	✓		61.0	65.6	1.25	0.58
model4		✓	✓	✓	62.0	66.3	1.56	0.62

SPPF${\phantom{.}}^{+}$ and SPPF means inserting SPPF${\phantom{.}}^{+}$ and SPPF to the 1/8 and 1/16 branches, and the 1/32 branch, respectively.

According to Table 4, Model1 and Model2 achieved improvements in AP, AR, parameters, and FLOPs(G) separately compared to Model0 and Model1, indicating that the introduction of MFE-L and MFE-H helped to improve the performance of Lite-HRNet. Model3 also achieved performance improvements in the above four evaluation metrics compared with Model2, which verified the effectiveness of the collaboration of MFE-L, MFE-H, and FE stage in improving the performance of Lite-HRNet.

Model4 describes the variant with MFE-L, MFE-H, the new stage, and $SPP{F}^{+}$/SPPF. It executes with a higher performance compared with Model3, which verified the effectiveness of all the improvements proposed in this work.

4.3. Performance Comparison

4.3.1. Comparison on AP-10K Dataset

We carry out experiments to evaluate the detection accuracy and speed of MPE-HRNet${\phantom{.}}^L$ by comparing it with nine important pose estimation solutions, including CSPNeXt-t [41], CSPNeXt-s, ShuffleNetV1 [20], ShuffleNetV2 [42], MobileNetV2 [43], MobileNetV3 [44], Dite-HRNet [45], Lite-HRNet, and DARK [46]. DARK utilizes Lite-HRNet to generate heatmaps but performs post-processing in a different way to the other solutions. All the solutions are lightweight solutions.

Table 5 shows the results of these solutions on the AP-10K validation set. According to the table, MPE-HRNet${\phantom{.}}^L$ obtained the highest AP, $A{P}^{50}$, $A{P}^{75}$, $A{P}^M$, $A{P}^L$, and AR. Compared with ShuffleNetV2, ShuffleNetV1, MobileNetV2, MobileNetV3, CSPNeXt-t, Dite-HRNet, Lite-HRNet, DARK, and CSPNeXt-s, it improved AP in animal pose estimation by 8.8%, 8.5%, 6.4%, 5.5%, 5.3%, 2.3%, 2.2%, 2.0%, and 1.5%, respectively. It also achieved improvements when any of the other metrics was used. It is obvious that MPE-HRNet${\phantom{.}}^L$ outperformed all the comparative solutions in accuracy.

Table 5. Results on the AP-10K validation set.

Methods	AP	AP50	AP75	APM	APL	AR	Inference Time (ms)	Params (M)	FLOPs (G)
CSPNeXt-t	56.7	87.1	58.2	39.2	57.0	61.5	9.74	6.02	1.91
CSPNeXt-s	60.5	88.7	63.2	47.3	60.7	64.8	11.23	8.58	2.36
DARK	60.0	89.1	63.2	46.2	60.3	64.7	27.27	1.13	0.35
Dite-HRNet	59.7	89.1	61.9	48.0	60.0	64.8	33.61	1.19	0.33
Lite-HRNet	59.8	88.9	63.2	51.5	60.1	64.9	27.39	1.13	0.35
ShuffleNetV1	53.5	84.9	54.0	40.9	53.8	58.9	13.61	6.94	1.80
ShuffleNetV2	53.2	83.9	53.0	36.3	53.6	58.7	12.10	7.55	1.83
MobileNetV2	55.6	86.8	57.5	41.7	55.9	61.2	12.27	9.57	2.12
MobileNetV3	56.5	87.3	58.9	48.8	56.8	61.3	13.93	5.24	1.73
MPE-HRNet${\phantom{.}}^L$	62.0	90.1	65.6	53.9	62.3	66.3	29.01	1.56	0.62

We aimed to strike a balance between accuracy and complexity, striving to carefully manage the complexity of MPE-HRNet${\phantom{.}}^L$ during its design process. In all the solutions, DARK and Lite-HRNet have the least Params, and Dite-HRNet has the lowest FLOPs. Comparing with these two solutions, MPE-HRNet${\phantom{.}}^L$ does not increase Params and FLOPs too much while obtaining an improvement in accuracy. Concretely, it has only 0.43 M more parameters than DARK and Lite-HRNet, and 0.29 G more FLOPs than Dite-HRNet, respectively. Comparing with ShuffleNet, MobileNet, and CSPNeXt, MPE-HRNet${\phantom{.}}^L$ has the least Params and the lowest FLOPs(G). The inference time of MPE-HRNet${\phantom{.}}^L$ is 1.74 ms longer than that of DARK and 1.62 ms longer than Lite-HRNet, indicating a slight increase with negligible impact.

Table 6 shows the results on the AP-10K test set. According to the table, MPE-HRNet${\phantom{.}}^L$ still achieved the highest AP in all the comparison solutions. Comparing with ShuffleNetV2, ShuffleNetV1, MobileNetV2, MobileNetV3, CSPNeXt-t, Dite-HRNet, Lite-HRNet, DARK, and CSPNeXt-s, it improved AP by 8.4%, 7.9%, 4.9%, 4.9%, 4.6%, 1.4%, 1.3%, 1.4%, and 1.1%, respectively. MPE-HRNet${\phantom{.}}^L$ also exhibited the highest $A{P}^{50}$, $A{P}^{75}$, $A{P}^M$, $A{P}^L$, and AR in all the other comparison solutions except DARK. Although DARK surpasses our method in the $A{P}^M$ metric, it falls behind MPE-HRNet${\phantom{.}}^L$ in all the other metrics. Considering all the metrics, MPE-HRNet${\phantom{.}}^L$ outperforms all comparison solutions on the test set.

Table 6. Results on the AP-10K test set.

Methods	AP	AP${\phantom{.}}^{50}$	AP${\phantom{.}}^{75}$	AP${\phantom{.}}^{\mathit{M}}$	AP${\phantom{.}}^{\mathit{L}}$	AR	Inference Time (ms)	Params (M)	FLOPs (G)
CSPNeXt-t	55.4	86.2	58.4	46.0	55.9	60.9	10.29	6.02	1.91
CSPNeXt-s	58.9	87.7	62.5	49.2	59.5	63.8	11.84	8.58	2.36
DARK	58.6	87.2	61.0	52.1	59.1	64.0	30.02	1.13	0.35
Dite-HRNet	58.6	87.2	62.1	51.3	59.1	64.2	36.17	1.19	0.33
Lite-HRNet	58.7	87.4	61.9	48.3	59.4	64.0	28.14	1.13	0.35
ShuffleNetV1	52.1	83.6	52.5	42.0	52.7	57.9	13.24	6.94	1.80
ShuffleNetV2	51.6	83.4	51.0	45.0	52.2	57.6	12.21	7.55	1.83
MobileNetV2	55.1	85.8	56.1	41.6	55.7	60.7	12.17	9.57	2.12
MobileNetV3	55.1	86.1	57.3	47.6	55.6	60.5	13.56	5.24	1.73
MPE-HRNet${\phantom{.}}^L$	60.0	88.7	62.9	51.4	60.5	65.0	28.03	1.56	0.62

We also compared MPE-HRNet${\phantom{.}}^L$ with ResNet-101, HRNet-W32, and RFB-HRNet. The latter three solutions are based on large models. According to the results in Table 7, these three solutions exhibit a higher accuracy than MPE-HRNet${\phantom{.}}^L$. However, they also have many more Params than MPE-HRNet${\phantom{.}}^L$. In many practical application scenarios, pose estimation models are required to be deployed on edge devices. Large models struggle to perform optimally on edge devices due to the inability of these devices to provide the resources required by the models.

Table 7. Comparison with solutions based on large models.

Methods	AP	AP${\phantom{.}}^{50}$	AR	Params (M)	FLOPs (G)
ResNet-101	68.1	93.8	77.4	52.99	12.13
HRNet-W32	72.2	94.2	77.0	28.54	10.25
RFB-HRNet	75.0	95.8	78.1	63.97	22.61
Ours	62.0	90.1	66.3	1.56	0.62

The results of RFB-HRNet come from the work of Gong et al. [19].

4.3.2. Comparison on Animal Pose Dataset

We also conducted experiments on the Animal Pose dataset. In these experiments, we compared MPE-HRNet${\phantom{.}}^L$ with ShuffleNetV2, MobileNetV2, and CSPNeXt-s. The corresponding results are recorded in Table 8. Similar to the results on the AP-10K dataset, MPE-HRNet${\phantom{.}}^L$ also achieved the highest AP on the Animal Pose dataset, demonstrating the effectiveness of MPE-HRNet${\phantom{.}}^L$ in improving accuracy.

Table 8. Results on the Animal Pose dataset.

Methods	AP	AP${\phantom{.}}^{50}$	AP${\phantom{.}}^{75}$	AP${\phantom{.}}^{\mathit{M}}$	AP${\phantom{.}}^{\mathit{L}}$	AR	Inference Time (ms)	Params (M)	FLOPs (G)
CSPNeXt-s	61.1	90.3	68.4	61.2	61.3	65.9	8.67	8.58	2.36
DARK	63.3	90.5	70.7	65.0	63.2	68.5	27.47	1.13	0.35
Dite-HRNet	62.0	90.1	67.0	63.3	62.0	67.3	30.10	1.19	0.33
Lite-HRNet	62.2	89.4	68.3	64.4	62.1	67.5	26.47	1.13	0.35
ShuffleNetV2	56.5	86.4	61.8	59.5	56.0	61.9	12.55	7.55	1.83
MobileNetV2	58.7	88.1	64.6	63.1	58.3	64.0	12.07	9.57	2.12
MPE-HRNet${\phantom{.}}^L$	64.9	90.5	72.3	65.9	64.9	69.8	28.17	1.56	0.62

4.4. Limitation Discussion

While MPE-HRNet${\phantom{.}}^L$ obtains accuracy improvements, it still suffers from two obvious limitations. The first limitation is that MPE-HRNet${\phantom{.}}^L$ does not perform well in complex occlusion scenes. The second limitation is that the accuracy of MPE-HRNet${\phantom{.}}^L$ needs to be improved.

Complex occlusion limitation. We chose twelve images with different scenes and depicted the corresponding results of MPE-HRNet${\phantom{.}}^L$ in Figure 7. These images were shot at different distances and angles. They include various animals belonging to different species. Some of them even include shadow and object occlusions. According to Figure 7b,e,h, MPE-HRNet${\phantom{.}}^L$ performed well in the images including multiple objects even if those objects are different sizes. It also performed well in the scenario where shadow exists, according to Figure 7i. MPE-HRNet${\phantom{.}}^L$ can still work well in the scenario where some parts of bodies are occluded lightly, as Figure 7j,k show. However, it does not work well in the scenario where complex occlusion exists, as Figure 7f shows.

Figure 7. Example qualitative results of the AP-10K.

Accuracy limitation. According to Section 4.3.1 and Section 4.3.2, MPE-HRNet${\phantom{.}}^L$ obtains accuracy improvements compared with lightweight models such as Dark, Lite-HRNet, and so on. However, MPE-HRNet${\phantom{.}}^L$ exhibits a relatively low accuracy when compared with the solutions based on large models. In addition, MPE-HRNet${\phantom{.}}^L$ improves accuracy by 1.4%, 1.3%, and 1.4% compared with Dark, Lite-HRNet, and Dite-HRNet, respectively. However, it increases parameters by 38% (0.43 M), 38% (0.43 M), and 31% (0.37 M) compared with the three solutions, respectively. Compared to the improvement in accuracy, the increase in parameters is more pronounced. Nonetheless, the number of parameters of MPE-HRNet${\phantom{.}}^L$ is only 1.56 M. MPE-HRNet${\phantom{.}}^L$ is still a lightweight model. However, MPE-HRNet${\phantom{.}}^L$ does not exhibit any advantage in accuracy compared to large models. It is necessary to improve MPE-HRNet${\phantom{.}}^L$ to obtain higher accuracy.

5. Conclusions

In this work, we focused on the problem of estimating the poses of multiple species with high accuracy in real scenarios. To deal with this problem, we constructed a lightweight network by improving Lite-HRNet. Concretely, we proposed SPPF${\phantom{.}}^{+}$ and applied it as well as SPPF to all branches. We proposed MPM and DSCA, and constructed MFE block by applying them together. We also designed a new stage to enhance the features in the feature map output by MPE-HRNet${\phantom{.}}^L$. The experimental results on the AP-10K dataset and the Animal Pose dataset verified the effectiveness of our work. In the future, we will focus on the problem of animal pose estimation involving complex occlusions.