Dog Activity Recognition Using Convolutional Neural Network ^†

Abstract

We classified common dog activities, such as sitting, standing, and lying down, which are crucial for monitoring the well-being of pets. To create a new model, we used convolutional neural networks (CNNs) on a Raspberry Pi platform and the InceptionV3 model, optimized on a dataset of Siberian Husky photos. The accuracy was 88% on a test set of 50 samples. In the developed model, TensorFlow Keras was used, while the OpenCV library was also used for system interaction with the Raspberry Pi and its Camera module. The model was effective for the image classification of dog behaviors in various environmental circumstances. The model substantially contributes to the development of pet welfare monitoring systems and improves the care for beloved animal companions.

1. Introduction

Dogs are incredibly popular due to their loyalty and affectionate nature. They have been companions to humans for a long time and are often considered part of the family [1]. The number of studies on dog behavior has significantly increased over the past ten to twenty years [2]. As we interact with dogs, we naturally interpret their actions as part of our communication with them. Understanding their behaviors forms the basis of how owners treat their pets. Misinterpreting their actions and having incorrect expectations of them can lead to a fear of dogs and cause inappropriate bite injuries, particularly in children [3].

Monitoring pets can be performed in various ways. One common method is using an electronic collar, which tracks the pet’s location. However, this method has limitations as it only works within a certain range and does not provide information on the pet’s activity or health. The other option is to use a dog monitoring system placed in your home and yard. These devices send notifications when something happens to pets.

Recently, convolutional neural networks (CNNs) have been frequently used to study human behavior. CNNs are also used for pet care. CNNs are a common architecture in deep learning, particularly for image-based detection systems [4]. The main goal of CNNs is to reduce the number of parameters compared to traditional neural networks. This prevents overfitting and improves algorithm speed. In ref. [5], pet activities were detected using accelerometer data, a random forest classifier (RF), a CNN, and a hybrid CNN [5]. The study involved six different dogs performing various activities. Wang et al. utilized CNN models including Alex Krizhevsky’s network (AlexNet) and long short-term memory (LSTM) to detect and classify human behaviors [6]. One of the easiest ways to implement CNN models is to use transfer learning. Transfer learning is used to carry the weights and features of a pre-trained model to a new model to avoid having to create everything from scratch. Then, the time needed to develop and train the model can be shortened [7,8]. These pre-trained models include residual networks (ResNet)-50, visual geometry group (VGG)-16, and InceptionV3. The TensorFlow and Keras libraries are frequently used for training these models [9]. Many researchers implement CNN models with Raspberry Pi, its cameral module, and computer vision. The Raspberry Pi microcontroller is used for image processing due to its flexibility and compatibility [10]. CNN models are widely used in computer vision applications for image processing [11]. The most utilized computer vision library is OpenCV, which is implemented using Python 3 for image processing [12].

Wearable sensors are used in monitoring dog behaviors, while CNN-based image recognition is used for human behavior. In monitoring dog activities, the InceptionV3 algorithm has not been used to recognize the activities of dogs. Therefore, we introduced image processing and a CNN to classify dog activities. To develop a system to detect dog activities, we developed a device to capture an image of a dog and classify its activity at the time of capture. We implemented InceptionV3 as the CNN model for training the dataset and classifying and recognizing images. Finally, a confusion matrix was established to determine the accuracy of the image recognition system.

The developed model in this study benefits animal welfare by understanding dogs’ behaviors and providing information. The research of this study serves as a basis for future studies about understanding dog behavior using image processing.

2. Methodology

In this study, three dog activities of sitting, lying, and standing were monitored. The system consisted of Raspberry Pi IV, Raspberry Pi Camera, and LCD. The model’s architecture was InceptionV3, which focuses on detection, recognition, and adversarial training. A confusion matrix was used to assess the model performance. The dog used in this study was a Siberian husky. A total of 150 images per activity of the dog were collected. The framework of the study is presented in Figure 1.

Raspberry Pi 4 was used as the microcontroller for processing input images, with the Raspberry Pi Camera integrated to capture images of dog activities. Captured images were preprocessed to fit the model’s data requirements. The dog images were resized to 299 × 299 pixels. Then, the pixel values were normalized between 0 and 1 to enhance model performance, stability, and consistency for prediction. The CNN model implemented with transfer learning analyzed the input images and extracted their features for processing. Upon completion, the model output the recognized dog activity based on the features. The model ensured accurate and reliable detection of various dog activities for effective monitoring and analysis.

We designed the device for monitoring using Raspberry Pi parts, a power source to supply 5.1 V and 3 A, and a USB-C [13]. An SD card with a 32 GB capacity was used to run the Raspberry Pi OS and the image classification program. The CNN model trained on InceptionV3 was programmed into the Raspberry Pi for image classification and processing. The Raspberry Pi camera module was used to capture images of the dog with its 8-megapixel camera [14]. The status of the dog’s activity was displayed on the connected 3.5-inch LCD (Figure 2).

Figure 3 illustrates the system operation flowchart, beginning with image data input captured with a Raspberry Pi camera. When the system receives an image of the dog, it preprocessed the image, including resizing and normalization. Next, the image was recognized by InceptionV3, followed by classification of the dog’s activity. To implement the image classification, the OpenCV library was used. The processed output was displayed on the LCD. After displaying the classified image, another image was captured to recognize and classify images.

InceptionV3 is an inception model that combines convolutional filters of various sizes to create a new filter [15]. This design reduces computational complexity by reducing the number of parameters that need to be trained. The InceptionV3 model consists of convolutional, pooling, and fully connected layers, among others. It uses convolutional filters of different sizes (1 × 1, 3 × 3, and 5 × 5) to gather features at various scales [16]. Larger filters capture more complex features, while 1 × 1 filters reduce the dimensionality of the input data.

In this study, InceptionV3 was implemented through transfer learning by loading the pre-trained model from the TensorFlow Keras library. This model was trained on the ImageNet dataset to learn a variety of visual attributes [17]. In model training, the pre-trained layers were frozen to preserve their learned representations and determine their weights. Transfer learning kept the generalized information from the original dataset while including custom layers for classification. The model was modified to a new task to classify the images of Huskies.

Different dog sizes, shapes, and postures were captured by the model for differentiating between different activities. Minor variations in the dog’s body position and orientation that distinguish each activity were examined using the extracted features. A common characteristic of dog standing is that their front and back legs are straight. Its vertical body shape and higher head positions were compared with those of its standing and lying positions to identify the dog’s posture.

For the experiment, an LCD monitor and Raspberry Pi camera were connected to the Raspberry Pi 4. The devices were positioned facing the dog to classify its activity. The camera module was set to have a fixed-focus lens with a focal length of 6 mm and an aperture of f/2 and was positioned 3.5 feet from the dog. The recognition results were displayed on the LCD monitor. The model program in Raspberry Pi OS was stored on an SD card integrated into Raspberry Pi 4 (Figure 4).

A total of 150 images per activity were collected for model training, with 80% allocated for training and 20% for validation. The dataset consisted of a combination of internet-based pictures showing specific dog activities and real-life images. The model was uploaded to the microprocessor to classify the dog’s activity. For testing, 50 images were captured and classified to determine the model’s accuracy.

3. Results and Discussion

Figure 5 displays the graphical user interface (GUI) of the system, which was created using the Tkinter Python library. The GUI included a live camera feed and featured two buttons, “Capture Photo” and “Capture Another.” When the “Capture Photo” button was pressed, the system captured the current image from the camera. The “Capture Another” button was used to reset the system and capture and classify a new image. The intuitive design of the GUI ensures ease of use and accessibility with limited technical expertise. By incorporating real-time feedback, users can precisely control the moment of image capture, enhancing the accuracy of the classification process. The simplicity of the two main buttons minimizes confusion and streamlines capturing and classifying images. This setup is beneficial for applications requiring quick and repeated image analysis for efficient and uninterrupted operation.

When an image was captured and classified, it was identified to depict its positions. The image recognition system analyzed the visual features of the image to determine posture. The image recognition system successfully processed the image and classified postures and activities. The classification accuracy demonstrated the system’s capability to distinguish various postures (Figure 6).

Table 1. Confusion matrix.

N = 50		Predicted
		Standing	Sitting	Lying
Actual	Standing	15	1	0
	Sitting	0	17	0
	Lying	2	3	12

shows the confusion matrix of system performance. From the 50 testing samples, the model predicted 15 out of 16 accurately. The model’s ability to identify standing posture was validated by a high true positive rate. For sitting, the accuracy was 100%, while for lying, the accuracy was 71%. The model had difficulty in distinguishing lying postures. The accuracy was calculated using Equation (1).

$$Accuracy = {\displaystyle \frac{{\sum }_{n=1}^3{A}_{nn}}{{\sum }_{\begin{array}{c}i=1\\ j=1\end{array}}^{3}Aij}}$$

(1)

Equation (1) indicates the relationship between expected and actual values using a confusion matrix. The model correctly predicted 44 out of 50 samples. The overall accuracy was 88% across all postures.

4. Conclusions and Recommendations

We developed a system to capture and classify the images of dog activities. The system integrated InceptionV3 for classifying dog activities. The accuracy was determined using a confusion matrix. The overall accuracy was 88%. To improve the accuracy of the model, images of larger dogs are needed to better capture the dog’s activities, particularly considering that shorter-legged dogs might confuse the system for activity detection. It is necessary to augment the training dataset to include a variety of dog activities. It is also necessary to diversify the dataset with various dog breeds to enhance accuracy. Alternative models need to be constructed for a more accurate classification of dog activities.