Computer-Vision-Based Statue Detection with Gaussian Smoothing Filter and EfficientDet

Abstract

Smart tourism is a developing industry, and numerous nations are planning to establish smart cities in which technology is employed to make life easier and link nearly everything. Many researchers have created object detectors; however, there is a demand for lightweight versions that can fit into smartphones and other edge devices. The goal of this research is to demonstrate the notion of employing a mobile application that can detect statues efficiently on mobile applications, and also improve the performance of the models by employing the Gaussian Smoothing Filter (GSF). In this study, three object detection models, EfficientDet—D0, EfficientDet—D2 and EfficientDet—D4, were trained on original and smoothened images; moreover, their performance was compared to find a model efficient detection score that is easy to run on a mobile phone. EfficientDet—D4, trained on smoothened images, achieves a Mean Average Precision (mAP) of 0.811, an mAP-50 of 1 and an mAP-75 of 0.90.

1. Introduction

Smart cities are based on the notion of using various forms of technology to address people’s daily needs. It provides residents with a great deal of convenience and value. Tourism has an important function in cities, as well as social places, entertainment, and shopping complexes [1]. Travel blogs, tour guides and maps have been the traditional methods used by tourists when they are on tour. In today’s world, those methods are described as outdated. Object detection is a fundamental study area in computer vision and artificial intelligence. Its goal is to find the target of interest in an image, establish the category appropriately, and provide the bounding box of each target. It is a requirement for more advanced computer vision tasks, including target tracking, object detection, pattern recognition, and semantic scene interpretation [2].

The tourist industry is gradually incorporating intelligent tourism applications, and research into these applications has recently begun [3]. Using visual sensors and artificial intelligence, computer vision and machine learning may assist give trustworthy intelligent autonomous solutions to smart tourism in several ways [4,5,6]. Features and image processing are very important in detection. Filtering is a very important task in image processing; it removes noise, enhances contours and improves texture in images. The Gaussian Smoothing filter has proven to be effective in many areas; the filtering enhances the model’s performance in classification [7,8,9]. The Viola–Jones detector’s implementation strategy is similar to that of the standard object detection method. It largely uses artificial design and innovative feature extraction algorithms for recognition and detection [10,11], as well as Support Vector Machines, Decision Trees [12], and other classifiers. The image is frequently preprocessed before detection to improve image quality [13]. To forecast the object, sliding window processing is frequently used for the image during the detection phase. The best detection performance is attained at that moment.

The sliding window approach, on the other hand, lays heavy demands on the computer’s computational capacity because it traverses all potential locations and size ratios. Furthermore, the capacity to express hand-crafted features is limited, adding to a poor overall detection impact. The Region Convolutional Neural Network (RCNN) algorithm [14] was proposed in 2014, and Convolutional Neural Networks(CNN) was used to extract features. CNN is a machine learning method that has pushed the development of object identification tasks to the deep learning level. The gradient descent approach may be used by deep learning to automatically optimize model parameters [15,16,17]. Various object detection tasks have achieved significant progress.

The motivation of the study is to develop a smart tourism application that can serve as a means of information in modern tourism. Object detection models are affected by noise in images. This motivated us to improve the model’s performance by applying the Gaussian Smoothing Filter to enhance the images. The main contributions of this study are as follows:

• Applying the Gaussian Smoothing filter to improve the performance of the trained models.
• A lightweight object detection model that can be deployed to mobile phones and edge devices was proposed to detect statues efficiently.
• A mobile application with the trained model as a backend was developed to detect the statue and give information about the statue.

2. Related Works

In this section, we highlight the main existing alternatives for the targeted problem. We classify these alternatives into single versus two-stage detection approaches.

2.1. Two-Stage Detection

Two steps of the two-stage detection architecture are region proposal and object detection. On the image to be examined, a series of region proposal boxes are first presented. Object detection is then carried out. The RCNN detection system, presented by [14], first generates a region proposal box on the picture using selective search [18]. The CNN is then used to extract features, after which the SVM classifier and bounding-box regression are trained, and the outcome is predicted. Although utilizing CNN for feature extraction improves the detection impact significantly, it comes with several downsides, including a lengthy training process. Fast RCNN has been updated to address the aforementioned concerns [19], and faster RCNN [20] algorithms have been developed. The end-to-end detection procedure is completed faster with RCNN. First, the RPN algorithm was proposed as a replacement for the selective search for regional ideas, resulting in a considerable reduction in the amount of time spent on regional recommendations. Furthermore, by eliminating repeating feature calculations, shared features save time; detection accuracy on the VOC07 dataset [21] is 73.2%, and 42.7% on the COCO dataset. The study referenced in [22] concluded that this not only improves classification risk but also better integrates feature extraction and classifier function, which is critical for pedestrian classification at various sizes. Several studies were carried out on the two-stage detectors and models such as RFCN [23] and the RCNN family [14,19,20,24].

2.2. One Stage Detection

In one-stage detection, the region proposal stage is eliminated; unlike in two-stage detection, bounding boxes were assigned at objects’ centres randomly on feature maps, and then a single network was used in processing the image. This concept improves detection speed in calculating the bounding boxes and the probability of each zone [25,26,27]. However, there is a degradation in accuracy due to the limited number of objects, and the models have issues with smaller objects. Speed in detection is very important, especially in autonomous vehicles [28]. YOLOv2 and YOLOv3 were proposed by [29,30], respectively, as a result of the aforementioned concerns. They were fine-tuned to address the aforementioned difficulties, resulting in a single-stage detector that improved detection accuracy while also striking a reasonable balance between speed and precision.

The researchers [31] have presented a framework for SSD one-stage detection [31], which is different from YOLO. In this technique, SSD, multilayer mapping in the convolutional layer technique, detects multiscale objects via producing feature layers of various sizes; this improves the detection of smaller objects. The RetinaNet detector was proposed by [32] in 2017, which has a new loss function that improves detection by paying attention to classification and imbalance in a data set. The study referenced in [33] proposed ALFnet [33], which detection speed is similar to SDD and detection accuracy is similar to Faster RCNN. To improve the previous loss function, [34] presented DIOU Loss and CIOU Loss in 2019. It considers the overlap area, centre point distance, and aspect ratio, as opposed to the prior object box regression loss. The bounding box with distance loss has a quicker convergence speed and greater convergence accuracy, improving the object detection framework’s detection accuracy. The study referenced in [35] suggested YOLOv4. Backbone partly employs the CSPNet structure [36], which takes advantage of the benefits of numerous detection frameworks. Adding the SPP structure [37] and the PAN structure [38] to the neck section allows for feature fusion. The benefits of clustering [39] are also employed to calculate the projected frame size. The PAN structure combines the information collected from multiple layers, and it can enable the network to integrate the features of different scales. Finally, on the coco dataset, YOLOv4 achieves 65FPS detection speed, reaching the optimum balance of current detection frame speed and accuracy.

The YOLO series approaches [27,29,30,35], namely, deep-learning-based SSD, were proposed in recent years. The YOLO techniques may be utilized for a variety of object detection tasks. Because just a small number of objects are anticipated in each anchor, missing detection is common in congested pedestrian scenes, and the algorithm’s performance suffers as a result. However, because of the fast detection speed of such algorithms, pedestrian detection technology may be used in the field of intelligent driving. The SSD [31] technique is presented for generic object detection, and it can help with the multiscale detection problem in pedestrian detection.

In the broad object detection field, the RetinaNet [32] detector proposes a novel loss function that can increase detection accuracy. The ALFnet [40] algorithm is mostly used to identify pedestrians. It may be extended to generic object detection to some extent due to the effective enhancement of the task of pedestrian detection. The border regression problem in object identification is investigated by the CIOU Loss [34] method, which significantly increases the detection effect of diverse objects.

3. Statue Detection

In this section, we use statue detection as a proof of concept for the proposed approach.

3.1. Dataset and Data Preprocessing

A video of thirty various sculptures surrounding the Near East University was collected for this study; the aim was to capture each statue from a different angle. In total, 300 images were generated from each video. Annotations were carried out on the images using the Roboflow online tool. Images were enhanced by turning them horizontally and vertically, cropping and randomly changing the brightness. The data augmentation will improve the model’s robustness and reduce overfitting [41,42,43,44,45,46,47]. The dataset was split into 80%, 10% and 10% training, validation and testing, respectively.

Table 1. Dataset split information.

Training	Validation	Testing
6696	774	774

gives details about the data.

3.2. Gaussian Filtering (Smoothing)

Gaussian filters are a class of linear smoothing filters with weights chosen according to the form of the Gaussian function [48]. Equation (1) is a very good filter for eliminating noise taken from the normal distribution of the Gaussian smoothing filter.

$$G\left(x\right)=e^{-\frac{x^2}{2{\mathsf{\sigma }}^2}}$$

(1)

Equation (1) determines the width of the Gaussian. The two-dimensional discrete Gaussian zero mean function (2) is used as a smoothing filter for image processing.

$$g\left[i,j\right]=e^{-\frac{\left(i^2+j^2\right)}{2{\mathsf{\sigma }}^2}}$$

(2)

3.3. EfficientDet

EfficientDet [49] is a neural network for object detection. It is a TensorFlow object recognition API that supports many model families, including CenterNet. [50], MobileNet [51], ResNet, and Fast R-CNN. The EfficientDent outperform the pre-existing models employed in [32,52]; they are also lightweight models that can be deployed to edge devices and mobile applications.

3.4. Model Training

In this study, three EffiientDet architecture models were employed to detect statues. The training was carried out in two stages: firstly, the training was conducted on original augmented images, and secondly, training was conducted on augmented Gaussian Smoothened filtered images. Instead of using CSV annotation, as it is the traditional API, we changed the training process to use PASCAL VOC (XML) annotation to reduce the XML-to-CSV conversion stage. Furthermore, a post-training quantization was performed to reduce the model’s size in order to improve CPU and hardware accelerator latency. The post-processing slightly reduces the accuracy of the model.

The three EfficientDet models employed, D0, D2 and D4, were trained for 10 epochs using a batch size of 8. Version D2 was regarded to have a compromise between accuracy and detection speed, making it suitable for deployment to a mobile application. D4 is heavyweight; it has better accuracy but less processing speed on the mobile application. After the models had been trained and exported, the android studio was used to create a mobile app in which the exported model could be used for statue detection. Figure 1 Shows the complete training and deployment process of this study. Google Research created Colaboratory (Colab), which was used for training. Colab is a Linux machine with a user interface based on the Jupyter notebook service that requires no configuration. It offers free access to reasonable computer capabilities, such as the Graphical Processing Unit (GPU) [33]. As a result, popular libraries such as Keras, TensorFlow, PyTorch, and OpenCV are utilized to create deep learning applications. This virtual computer has a 2 core CPU and 12 GB of RAM, which may be expanded to 25 GB for free if necessary. When using the laptop, the GPU will be assigned at random. Nvidia Tesla K80s, T4s, P4s, and P100s are among the GPUs that may be accessed [33].

4. Results and Discussion

In this study, three object detection models, EfficientDet—D0, EfficientDet—D2 and EfficientDet—D4, were employed to detect statues; the performance of two of the models was compared to find the best performing model. The D0 achieves an mAP of 0.348, mAP-50 of 0.652 and mAP-75 of 0.3. The D2 achieves an mAP of 0.63, mAP-50 of 0.894 and mAP-75 of 0.81, whereas the D4 achieves an mAP of 0.751, an mAP-50 of 1 and an mAP-75 of 0.837. The D4 achieves a higher mAP with over 10%, an mAP-50 with more than 20% and an mAP-75 with over 3%. For the models trained with smoothened images, the D0 achieves an mAP of 0.352, mAP-50 of 0.595 and mAp-75 of 0.41. The D2 trained on smoothened images achieves an mAP of 0.694, mAP-50 of 0.966 and mAP-75 of 0.867. The D4 trained with smoothening images achieves an mAP of 0.811, mAP-50 of 1 and mAP-75 of 0.90.

The models trained with smoothened images show improved performance compared with the models trained on the original images. The performance of the models is presented in

Table 2. Performance of the EfficientDet D0, D2 and D4 employed in the study.

Models	mAP	mAP-50	mAP-75
EfficientDet—D0	0.348	0.562	0.4
EfficientDet—D2	0.63	0.894	0.81
EfficientDet—D4	0.751	1	0.837
EfficientDet—D0 + Smoothing	0.352	0.595	0.41
EfficientDet—D2 + Smoothing	0.694	0.966	0.867
EfficientDet—D4 + Smoothing	0.811	1	0.9

. On the COCO datasets, the performance of the EffiecientDet family models was compared and presented in

Table 3. Performance of the EfficientDet D0, D2 and D4 on the COCO dataset.

Model	mAP	mAP-50	mAP-75
EfficientDet—D0	33.8	52.2	35.8
EfficientDet—D2	43	62.3	46.2
EfficientDet—D4	49.9	69	53.4

. The EfficientDet—D4 achieves an mAP of 49.9, mAP-50 of 69 and mAP-75 of 53.4. The performance of the proposed model and the performance of the model on the COCO dataset are presented graphically in Figure 2 and Figure 3, respectively.

To implement the mobile-app-based detection system, the trained model which serves as the backend of the system was incorporated into the mobile app. The mobile is set to detect five objects at a time with a minimum detection threshold of 40%. If the statue detection is not 40%, the model will not present the detected object; once the object is detected, the statue and its information will be presented in another screen called the statue description screen. The detection and information screen can be seen in Figure 4.

5. Conclusions

One area of rapidly growing research is smart education. Several object detection models have been developed to efficiently detect objects with high accuracy. In this study, EfficientDet—D0, EfficientDet—D2 and EfficientDet—D4 were proposed to detect statutes for outdoor tourism and the trained model was used to develop a light mobile application. The EfficientDet—D4 trained on smoothened images achieves a mean average precision (mAP) of 0.811, mAP-50 of 1 and mAP-75 of 0.90. The model’s performance demonstrates that it is capable of carrying out the desired job. The drawbacks of object detection models are datasets and high computational resources. More networks will be examined in a future study to improve detection performance and speed the detection. The study’s drawbacks thus far are the need for extra training in the event of more statues.