Development of a Low Cost and Path-free Autonomous Patrol System Based on Stereo Vision System and Checking Flags

: Nowadays, security guard patrol services are becoming roboticized. However, high construction prices and complex systems make patrol robots di ﬃ cult to be popularized. In this research, a simpliﬁed autonomous patrolling robot is proposed, which is fabricated by upgrading a wheeling household robot with stereo vision system (SVS), radio frequency identiﬁcation (RFID) module, and laptop. The robot has four functions: independent patrolling without path planning, checking, intruder detection, and wireless backup. At ﬁrst, depth information of the environment is analyzed through SVS to ﬁnd a passable path for independent patrolling. Moreover, the checkpoints made with RFID tag and color pattern are placed in appropriate positions within a guard area. While a color pattern is detected by the SVS, the patrolling robot is guided to approach the pattern and check its RFID tag. For more, the human identiﬁcation function of SVS is used to detect an intruder. While a skeleton information of the human is analyzed by SVS, the intruder detection function is triggered, then the robot follows the intruder and record the images of the intruder. The recorded images are transmitted to a server through Wi-Fi to realize the remote backup, and users can query the recorded images from the network. Finally, an experiment is made to test the functions of the autonomous patrolling robot successfully.


Introduction
Home security has always been an issue for everyone. However, hiring security guards involves high costs, and also issues of safety about acquaintances. On the other hand, sweeping robots have gradually become commonplace in every home environment. If there is a simple and low-cost method that can increase the patrol function on traditional sweeping robots, it will effectively solve the problem of home security. The patrol system is highly dependent on the function of simultaneous localization and mapping (SLAM), and it also needs rapid SLAM execution efficiency. Therefore, much research has been devoted to improving the working efficiency of SLAM [1][2][3][4][5][6]. Lee, C.W. et al. proposed a variable size rectangle region to replace the traditional grid method of SLAM, and the proposed method reduces the data storage capacity of built maps [1]. Dwijotomo, A. et al. use the adaptive multistage distance scheduler (AMDS) method to improve the working speed of SLAM with large map size. [2]. Li, X. et al. proposed a monocular vision fast 3D SLAM system by using a fusion of localization, mapping, and parses scenes [3,4]. In addition, Valiente, D. et al. also used an omnidirectional monocular sensor to improve the stability of SLAM [5]. Muñoz-Bañón, M.Á. et al. proposed a novel navigation framework based on the robot operation system (ROS), which can separate a mission into several sub-issues to solve them to improve the SLAM working efficiency [6]. Moreover, numerous researchers have The remainder of this paper is organized as follows. In Section 2, we introduce the system architecture. In Section 3, the algorithm used to direct all patrolling functions is explained. Experiment results and a brief discussion are presented in Section 4. Conclusions are made in Section 5.

System Architecture
The three basic components of the system include a patrol robot, server, and client, as shown in Figure 1. The system architecture of each component is outlined in the following.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 3 of 16 The remainder of this paper is organized as follows. In Section 2, we introduce the system architecture. In Section 3, the algorithm used to direct all patrolling functions is explained. Experiment results and a brief discussion are presented in Section 4. Conclusions are made in Section 5.

System Architecture
The three basic components of the system include a patrol robot, server, and client, as shown in Figure 1. The system architecture of each component is outlined in the following.

Patrol Robot
The patrol robot comprises of a processor, wheeled robot, and stereo vision system (SVS). An ASUS K401D notebook running Windows 7 was used as a processor. USB-connected ASUS Xtion pro live was used as an SVS to provide color images and depth information. Owing to that the depth information provided by SVS is based on its active infrared sensor, and the sensor is very sensitive to sun ray, the proposed system in this research just can be used indoors. The wheeled robot was built by integrating a KOBUKI robot, a product that is designed for programmable control without default functions, and an RFID system using an Arduino Uno R3 controller. Depth information of the environment captured by the SVS is analyzed by the processor to find a suitable path. The direction of movement is transformed into commands controlling the robot. The SVS continues monitoring the environment in order to make adjustments to the path in real-time. The cliff sensor, wheel drop sensor, and bumpers of the KOBUKI are also used to modify the patrol path. The SVS also seeks out inspection points and detects intruders. The robot is programmed to approach the inspection points and follow intruders until they disappear. All of this information is transmitted to a database via Wi-Fi. The structure of the proposed robot is presented in Figure 2.

Patrol Robot
The patrol robot comprises of a processor, wheeled robot, and stereo vision system (SVS). An ASUS K401D notebook running Windows 7 was used as a processor. USB-connected ASUS Xtion pro live was used as an SVS to provide color images and depth information. Owing to that the depth information provided by SVS is based on its active infrared sensor, and the sensor is very sensitive to sun ray, the proposed system in this research just can be used indoors. The wheeled robot was built by integrating a KOBUKI robot, a product that is designed for programmable control without default functions, and an RFID system using an Arduino Uno R3 controller. Depth information of the environment captured by the SVS is analyzed by the processor to find a suitable path. The direction of movement is transformed into commands controlling the robot. The SVS continues monitoring the environment in order to make adjustments to the path in real-time. The cliff sensor, wheel drop sensor, and bumpers of the KOBUKI are also used to modify the patrol path. The SVS also seeks out inspection points and detects intruders. The robot is programmed to approach the inspection points and follow intruders until they disappear. All of this information is transmitted to a database via Wi-Fi. The structure of the proposed robot is presented in Figure 2.

Server
A server is used to provide the Windows communication foundation (WCF) web-service and a database for remote image backup. We developed three web-services: upload, history, and real-time. The upload web-service is used for the transmission of information to the server, including inspection point number, check-in time, time of intruder detection, and images of intruder. Remote users can access historical data by scan time and monitor the robot in real-time. The database was built using Microsoft SQL server 2008, and the web-service was built using Visual Studio C# 2010.

Materials and Methods
The robot first searches for a passable path using depth information from the SVS. While on patrol, the robot follows any detected intruder, takes an image of the individual, and uploads the images in real-time. The robot approaches all detected inspection points and signs in electronically, whereupon the inspection point number and sign-in time are uploaded. When all of the inspection points have been passed, that particular patrolling task is complete. Figure 3 presents a flowchart showing the process of autonomously patrolling a site. In the following section, we introduce the four functions controlled by the proposed algorithm.

Server
A server is used to provide the Windows communication foundation (WCF) web-service and a database for remote image backup. We developed three web-services: upload, history, and real-time. The upload web-service is used for the transmission of information to the server, including inspection point number, check-in time, time of intruder detection, and images of intruder. Remote users can access historical data by scan time and monitor the robot in real-time. The database was built using Microsoft SQL server 2008, and the web-service was built using Visual Studio C# 2010.

Materials and Methods
The robot first searches for a passable path using depth information from the SVS. While on patrol, the robot follows any detected intruder, takes an image of the individual, and uploads the images in real-time. The robot approaches all detected inspection points and signs in electronically, whereupon the inspection point number and sign-in time are uploaded. When all of the inspection points have been passed, that particular patrolling task is complete. Figure 3 presents a flowchart showing the process of autonomously patrolling a site. In the following section, we introduce the four functions controlled by the proposed algorithm.

Server
A server is used to provide the Windows communication foundation (WCF) web-service and a database for remote image backup. We developed three web-services: upload, history, and real-time. The upload web-service is used for the transmission of information to the server, including inspection point number, check-in time, time of intruder detection, and images of intruder. Remote users can access historical data by scan time and monitor the robot in real-time. The database was built using Microsoft SQL server 2008, and the web-service was built using Visual Studio C# 2010.

Materials and Methods
The robot first searches for a passable path using depth information from the SVS. While on patrol, the robot follows any detected intruder, takes an image of the individual, and uploads the images in real-time. The robot approaches all detected inspection points and signs in electronically, whereupon the inspection point number and sign-in time are uploaded. When all of the inspection points have been passed, that particular patrolling task is complete. Figure 3 presents a flowchart showing the process of autonomously patrolling a site. In the following section, we introduce the four functions controlled by the proposed algorithm.

Imaging Space Conversion and Patrol Path Determination
The proposed SVS is mounted on the robot parallel to the ground and separated by 212 mm. The field of view (FOV) is 45 degrees in the vertical and 58 degrees in the horizontal, respectively, as the plan and side views of the robot depicted in Figure 4a. The height and width of the visible windows are as follows: 1000 mm from SVS (828 mm × 1109 mm), 2000 mm from SVS (1657 mm × 2217 mm), and 3000 mm from SVS (2485 mm × 3326 mm), as shown in Figure 4b. The FOV is resolved to 640 × 480 pixels, and each pixel is described with RGB color space and the world coordinates , where x i,j , y i,j , and d i,j are the width, height and depth from the center of SVS, respectively.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 5 of 16 The proposed SVS is mounted on the robot parallel to the ground and separated by 212 mm. The field of view (FOV) is 45 degrees in the vertical and 58 degrees in the horizontal, respectively, as the plan and side views of the robot depicted in Figure 4a. The height and width of the visible windows are as follows: 1000 mm from SVS (828 mm × 1109 mm), 2000 mm from SVS (1657 mm × 2217 mm), and 3000 mm from SVS (2485 mm × 3326 mm), as shown in Figure 4b. The FOV is resolved to 640 × 480 pixels, and each pixel is described with RGB color space and the world coordinates  In order to make the patrol path random, depth information of the image is analyzed for searching possible patrol paths in this research, instead of using existing path planning. In addition, to reduce the computation required for processing information in the identification of a suitable path, a region of interest (ROI) is defined from the following image coordinates (pixel number): (32,290) to (608,340), as shown in Figure 5a. The ROI is chosen based on the height where the SVS is installed and the width of the robot. Figure 5b presents a corresponding color image with ROI. Assume that and are the horizontal and vertical coordinates on the ROI image plane, where 32 ≤ ≤ 608 and 290 ≤ ≤ 340. In order to find a possible path region in ROI, the minimum depth at each column of ROI is selected as ̅ = [ ̅ 1 ̅ 2 ⋯ ̅ ] at first. The size thresholds of a path are defined as follows: depth (1500 mm) and width (3520 mm). In the following, four steps of suitable path identification are described: 1. Initially, compare each ̅ in ̅ with the threshold of depth . The ̅ which are larger than continuously are selected as a sub-section ̅ , which ̅ ∈ ̅ .
3. As long as the calculated width exceeds width threshold , then this route (the region of selected section ̅ ) is adopted as a suitable path. The angle from the center of the image plane (the head of the robot) to the center of the path is then calculated to control the rotation of In order to make the patrol path random, depth information of the image is analyzed for searching possible patrol paths in this research, instead of using existing path planning. In addition, to reduce the computation required for processing information in the identification of a suitable path, a region of interest (ROI) is defined from the following image coordinates (pixel number): (32,290) to (608,340), as shown in Figure 5a. The ROI is chosen based on the height where the SVS is installed and the width of the robot. Figure 5b presents a corresponding color image with ROI. Assume that i and j are the horizontal and vertical coordinates on the ROI image plane, where 32 ≤ i ≤ 608 and 290 ≤ j ≤ 340. In order to find a possible path region in ROI, the minimum depth at each column of ROI is selected as D = d 1 d 2 · · · d n at first. The size thresholds of a path are defined as follows: depth d T (1500 mm) and width w T (3520 mm). In the following, four steps of suitable path identification are described:

1.
Initially, compare each d n in D with the threshold of depth d T . The d n which are larger than d T continuously are selected as a sub-section D k , which D k ∈ D.

2.
Select the largest section D k of D, and calculate the corresponding real width w k of the selected section. Assume the selected section D k = d i,j x k 2 i,j · · · x k m i,j , then the width w k of the section D k can be estimated as follows:

3.
As long as the calculated width w k exceeds width threshold w T , then this route (the region of selected section D k ) is adopted as a suitable path. The angle θ d from the center of the image plane (the head of the robot) to the center of the path is then calculated to control the rotation of the robot. Assume i c is the center of FOV, and i d is the middle of the selected section D k on the horizontal axis of image coordinate, then the angle between the head of the robot and the center of the desired path can be calculated as follows: 4.
If the analysis results fail to provide a viable path, the robot rotates counterclockwise (CCW) in search of another route.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 6 of 16 the robot. Assume is the center of FOV, and is the middle of the selected section ̅ on the horizontal axis of image coordinate, then the angle between the head of the robot and the center of the desired path can be calculated as follows: 4. If the analysis results fail to provide a viable path, the robot rotates counterclockwise (CCW) in search of another route.

Signing in at Inspection Points
The patrolling of the robot does not only follow the path which is determined from the depth image, but also guided to the specified location by detecting inspection points. The inspection points are made by a colored marker and an RFID tag. As shown in Figure 6a, the RFID tag as an inspection point is mounted on walls 12 cm above the ground in accordance with the RFID sensor of the robot. This location was marked using a red circular patch (diameter 4.8 cm) at 21 cm above the ground for the SVS system of robot to identify while the robot moving closer to the inspection point. The identification process is divided into two steps, color and shape recognition. In order to prevent the influence of light condition, color images from the SVS is transferred from RGB (red, green, and blue) to HSV (hue, saturation, and value) color space at first. Assume , , are the values in RGB color space per pixel, [ ] ∈ [0,1], and , are the maximum and minimum one in [ ], then the values in HSV color space per pixel can be calculated from: = .
The transferred image is then binarized by the threshold which is defined as 0°≤ ℎ ≤ 10°, 33.3% ≤ ≤ 56.9%, and 29.4% ≤ ≤ 100% to filter the red color. After the color recognition process, the circle Hough transform (CHT) function of the openCV library is then used for shape recognition. Circles with a radius between 22 to 70 pixels in the binarized images are selected as candidates, and the biggest one is considered as an inspection point, such as depicted in Figure 6b.
While the inspection point is detected, the function of the robot will change from patrolling to signing. Moreover, the algorithm used for signing is designed by controlling the robot to align the

Signing in at Inspection Points
The patrolling of the robot does not only follow the path which is determined from the depth image, but also guided to the specified location by detecting inspection points. The inspection points are made by a colored marker and an RFID tag. As shown in Figure 6a, the RFID tag as an inspection point is mounted on walls 12 cm above the ground in accordance with the RFID sensor of the robot. This location was marked using a red circular patch (diameter 4.8 cm) at 21 cm above the ground for the SVS system of robot to identify while the robot moving closer to the inspection point. The identification process is divided into two steps, color and shape recognition. In order to prevent the influence of light condition, color images from the SVS is transferred from RGB (red, green, and blue) to HSV (hue, saturation, and value) color space at first. Assume p r , p g , p b are the values in RGB color space per pixel, p r p g p b ∈ [0, 1], and p max , p min are the maximum and minimum one in p r p g p b , then the values in HSV color space per pixel can be calculated from: The transferred image is then binarized by the threshold which is defined as 0 • ≤ p h ≤ 10 • , 33.3% ≤ p s ≤ 56.9%, and 29.4% ≤ p v ≤ 100% to filter the red color. After the color recognition process, the circle Hough transform (CHT) function of the openCV library is then used for shape recognition.
Circles with a radius between 22 to 70 pixels in the binarized images are selected as candidates, and the biggest one is considered as an inspection point, such as depicted in Figure 6b.
While the inspection point is detected, the function of the robot will change from patrolling to signing. Moreover, the algorithm used for signing is designed by controlling the robot to align the center of the detected circle. Assume the image coordinate of the detected circle center in horizontal is i cc , and the edges of the center region are i lT and i rT . If i cc is less than i lT , then the robot is turned CCW for 15 degrees. On the other hand, if i cc is larger than i rT , then the robot is turned clockwise (CW) for 15 degrees. The 15 degrees of turning is limited by the minimum movement of the robot. As long as the value of i cc falls between i lT and i rT , then the robot moves toward the circle marker until the RFID tag is detected. This algorithm is described in (6). Based on experiment results, we set the values of i lT and i rT to 310 and 330, respectively.
When the robot is sufficiently close to the RFID sensor to detect the RFID tag, the tag ID and sign-in time are recorded, and the signing task is deemed to have been completed. Based on the left-hand-rule, the robot then rotates CCW until the circle target is outside the viewing window, whereupon it continues its patrolling task.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 7 of 16 is , and the edges of the center region are and . If is less than , then the robot is turned CCW for 15 degrees. On the other hand, if is larger than , then the robot is turned clockwise (CW) for 15 degrees. The 15 degrees of turning is limited by the minimum movement of the robot. As long as the value of falls between and , then the robot moves toward the circle marker until the RFID tag is detected. This algorithm is described in (6). Based on experiment results, we set the values of and to 310 and 330, respectively.
When the robot is sufficiently close to the RFID sensor to detect the RFID tag, the tag ID and sign-in time are recorded, and the signing task is deemed to have been completed. Based on the lefthand-rule, the robot then rotates CCW until the circle target is outside the viewing window, whereupon it continues its patrolling task.

Detection and Tracking of Intruders
When an intruder appears in the FOV of the SVS, the skeleton of the individual is analyzed to identify its feature. The functions of the openNI library is used for skeleton detection, and the identified feature is composed of joints and limb links, as shown in Figure 7a. Given the height of the intruder is about 170 cm, then the proposed distance between the intruder and the robot (or the SVS) for detection is 2500 mm to 3000 mm. In this range, the full body of the intruder can be viewed in the field of view (FOV) of SVS for skeleton analysis without being too close or too far. In order to track the intruder, the center between left and right hip nodes is used to estimate the orientation and the distance between the robot and the intruder. Assume that the image coordinate in horizontal and depth of the left and right hip nodes are ( ℎ ℎ ) and ( ℎ ℎ ), respectively. Thus, the tracking target of the intruder and its distance are defined as ℎ = ( ℎ + ℎ ) 2 ⁄ and ℎ = ( ℎ + ℎ ) 2 ⁄ . The detection results are presented in Figure 7b. In addition, while an intruder is detected, the function of the robot is changed from patrolling to tracking until the intruder disappears from the FOV of SVS.

Detection and Tracking of Intruders
When an intruder appears in the FOV of the SVS, the skeleton of the individual is analyzed to identify its feature. The functions of the openNI library is used for skeleton detection, and the identified feature is composed of joints and limb links, as shown in Figure 7a. Given the height of the intruder is about 170 cm, then the proposed distance between the intruder and the robot (or the SVS) for detection is 2500 mm to 3000 mm. In this range, the full body of the intruder can be viewed in the field of view (FOV) of SVS for skeleton analysis without being too close or too far. In order to track the intruder, the center between left and right hip nodes is used to estimate the orientation and the distance between the robot and the intruder. Assume that the image coordinate in horizontal and depth of the left and right hip nodes are (i lh i rh ) and (d lh d rh ), respectively. Thus, the tracking target of the intruder and its distance are defined as i ch = (i lh + i rh )/2 and d ch = (d lh + d rh )/2. The detection results are presented in Figure 7b. In addition, while an intruder is detected, the function of the robot is changed from patrolling to tracking until the intruder disappears from the FOV of SVS. On the other hand, the tracking algorithm is similar to the inspection point approach algorithm, except for the near and far threshold values of tracking length, and . At first, the robot is turned until ≤ ℎ ≤ . As long as the value of ℎ exceeds the far threshold value of tracking length , then the robot moves forward; i.e., until the value of ℎ is less than . Conversely, the robot moves backward until ℎ is larger than . If the value of ℎ falls between and , then the robot remains in the same position. The code of the tracking algorithm is shown in algorithm 1. Moreover, to view the intruder fully, the values of and are set to 2500 mm and 3000 mm, respectively. The robot also records images of intruders while tracking them, and uploads the images to a server via the upload web-service. When the target moves beyond the viewing window, the robot ceases its tracking task and returns to its patrol task.

Wireless Backup and Query Service
Three web-service functions (upload, history and real-time information service) were installed on the server for the remote backup of monitoring data and the submission of queries. The upload service is used for the transmission of data from the robot to the server, whereas the other two services enable remote users to submit queries pertaining to historical or real-time monitoring data held on the server. The robot backs up information when signing in at inspection points and when an intruder is detected. The processor first connects to a wireless access point using a Wi-Fi module. The human- On the other hand, the tracking algorithm is similar to the inspection point approach algorithm, except for the near and far threshold values of tracking length, d kn and d k f . At first, the robot is turned until i lT ≤ i h ≤ i rT . As long as the value of d ch exceeds the far threshold value of tracking length d k f , then the robot moves forward; i.e., until the value of d ch is less than d k f . Conversely, the robot moves backward until d ch is larger than d kn . If the value of d ch falls between d kn and d k f , then the robot remains in the same position. The code of the tracking algorithm is shown in algorithm 1. Moreover, to view the intruder fully, the values of d kn and d k f are set to 2500 mm and 3000 mm, respectively. The robot also records images of intruders while tracking them, and uploads the images to a server via the upload web-service. When the target moves beyond the viewing window, the robot ceases its tracking task and returns to its patrol task. The robot is turned CCW for 15

Wireless Backup and Query Service
Three web-service functions (upload, history and real-time information service) were installed on the server for the remote backup of monitoring data and the submission of queries. The upload service is used for the transmission of data from the robot to the server, whereas the other two services enable remote users to submit queries pertaining to historical or real-time monitoring data held on the server. The robot backs up information when signing in at inspection points and when an intruder is detected.
The processor first connects to a wireless access point using a Wi-Fi module. The human-machine interface (HMI) contacts the upload web service to transmit the tag number and sign-in time to the server. After receiving the data, the server records the information within a table in the SQL database. When an image of an intruder is captured by the robot, the image data is encoded to base64 format and then transmitted (along with the time of detection) to the upload service. The base 64 data is then decoded back into an image by the server, whereupon it is saved in PNG format within a specific folder. The image file is named using the time of detection to facilitate subsequent queries. A client program was also developed to assist in data queries. This program provides real-time monitoring data as well as historical data by accessing the real-time information service or history service, respectively. The real-time information service searches for the latest data within the database, whereas the history service searches for data pertaining to the specific period of time defined by the user. The architecture of the data remote backup and querying system is presented in Figure 8.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 9 of 16 machine interface (HMI) contacts the upload web service to transmit the tag number and sign-in time to the server. After receiving the data, the server records the information within a table in the SQL database. When an image of an intruder is captured by the robot, the image data is encoded to base64 format and then transmitted (along with the time of detection) to the upload service. The base 64 data is then decoded back into an image by the server, whereupon it is saved in PNG format within a specific folder. The image file is named using the time of detection to facilitate subsequent queries. A client program was also developed to assist in data queries. This program provides real-time monitoring data as well as historical data by accessing the real-time information service or history service, respectively. The real-time information service searches for the latest data within the database, whereas the history service searches for data pertaining to the specific period of time defined by the user. The architecture of the data remote backup and querying system is presented in Figure 8.

Experiment and Results
An environment based on the proposed functions was created to verify the efficacy of the proposed algorithm. The environment was a 9100 mm x 5100 mm square space with several pieces of office furniture located in the center, and inspection points were set in each corner of the room, as shown in Figure 9. At the beginning of the experiment, the robot was placed along a wall. Patrolling began when a user pressed the start button on the robot's HMI. The sequence of the patrol task is outlined in the following.

Experiment and Results
An environment based on the proposed functions was created to verify the efficacy of the proposed algorithm. The environment was a 9100 mm x 5100 mm square space with several pieces of office furniture located in the center, and inspection points were set in each corner of the room, as shown in Figure 9. At the beginning of the experiment, the robot was placed along a wall. Patrolling began when a user pressed the start button on the robot's HMI. The sequence of the patrol task is outlined in the following.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 9 of 16 machine interface (HMI) contacts the upload web service to transmit the tag number and sign-in time to the server. After receiving the data, the server records the information within a table in the SQL database. When an image of an intruder is captured by the robot, the image data is encoded to base64 format and then transmitted (along with the time of detection) to the upload service. The base 64 data is then decoded back into an image by the server, whereupon it is saved in PNG format within a specific folder. The image file is named using the time of detection to facilitate subsequent queries. A client program was also developed to assist in data queries. This program provides real-time monitoring data as well as historical data by accessing the real-time information service or history service, respectively. The real-time information service searches for the latest data within the database, whereas the history service searches for data pertaining to the specific period of time defined by the user. The architecture of the data remote backup and querying system is presented in Figure 8.

Experiment and Results
An environment based on the proposed functions was created to verify the efficacy of the proposed algorithm. The environment was a 9100 mm x 5100 mm square space with several pieces of office furniture located in the center, and inspection points were set in each corner of the room, as shown in Figure 9. At the beginning of the experiment, the robot was placed along a wall. Patrolling began when a user pressed the start button on the robot's HMI. The sequence of the patrol task is outlined in the following.  Figure 9. Environment used in patrol experiment.

1.
From the start point until 21 s, the robot searched for a passable path and followed the resulting path.

2.
At 31 s, the 1st inspection point marker was detected by the robot, to which the robot approached.

3.
At 44 s, the robot reached the 1st inspection point. The RFID tag number was recorded by the robot, thereby completing the 1st inspection point sign-in. 4.
From 44 s to 54 s, the robot rotated CCW in search of its next route to follow.

5.
From 54 s to 74 s, the robot moved from the 1st to the 2nd inspection point. 6.
At 84 s, the robot located the 2nd inspection point. The center of the marker image was not located in the target region of the image plane (between 310 and 330 pixels); therefore, the robot rotated until the image marker was near the center of the image plane (at 94 s), as shown in the figure. 7.
At 94 s, the robot proceeded forward to sign-in at this inspection point at 101 s, whereupon it rotated and began patrolling from the 2nd inspection point to the 3rd inspection point (109 s to the 139 s). 8.
The 3rd and 4th inspection points were identified at 159 s and the 203 s, respectively. After checking-in at the final (4th) inspection point, the robot continued patrolling for approximately 30 seconds before halting.
A color image and the corresponding depth information from the SVS of the above procedure are recorded throughout the experiment, as shown in Figures 10 and 11, and Video S1 (Supplementary Materials).   The depth information ̅ of the ROI was also recorded to analyze the trajectory of the robot while on patrol. In order to fix the distortion caused by size-changing of FOV in different distance from SVS, the recorded data is transformed by Equation (2) and presented in the form of polar space, as shown in Figure 12. The FOV is depicted with a blue dashed line, and the threshold of depth is shown with a red dashed line. In addition, the color scale is used to present the values of each ̅ in ̅ . The analysis results are listed in the following: 1. At 11 s and 129 s, the center of the section ̅ was near the origin of the polar space, which means that the robot was moving toward the center of the patrol path. 2. Between 21 s and 31 s, the robot was approaching the 1st inspection point, as indicated by the decrease in recorded depth. 3. While the robot was signing in at the 1st inspection point (at 44 s), the distance between the wall and the robot was less than the minimum distance required by the SVS for detection. Thus, no depth information appears in the polar space during this time. Similarly, the depth values disappeared at 94 s, 159 s, and 203 s, corresponding to the sign-in process at the 2nd, 3rd, and 4th inspection points. 4. After signing in, the robot rotated CCW to search for a suitable path, whereupon the depth values increased considerably from the left side to the right side from 44 s to 64 s. This is similar to the records from 94 s to 119 s, 159 s to 173 s, and 193 s to 216 s.
As shown in Figure 13, the robot followed the proposed algorithm in identifying patrol paths and signing in at inspection points. The depth information D of the ROI was also recorded to analyze the trajectory of the robot while on patrol. In order to fix the distortion caused by size-changing of FOV in different distance from SVS, the recorded data is transformed by Equation (2) and presented in the form of polar space, as shown in Figure 12. The FOV is depicted with a blue dashed line, and the threshold of depth d T is shown with a red dashed line. In addition, the color scale is used to present the values of each d n in D. The analysis results are listed in the following:

1.
At 11 s and 129 s, the center of the section D k was near the origin of the polar space, which means that the robot was moving toward the center of the patrol path.

2.
Between 21 s and 31 s, the robot was approaching the 1st inspection point, as indicated by the decrease in recorded depth.

3.
While the robot was signing in at the 1st inspection point (at 44 s), the distance between the wall and the robot was less than the minimum distance required by the SVS for detection. Thus, no depth information appears in the polar space during this time. Similarly, the depth values disappeared at 94 s, 159 s, and 203 s, corresponding to the sign-in process at the 2nd, 3rd, and 4th inspection points.

4.
After signing in, the robot rotated CCW to search for a suitable path, whereupon the depth values increased considerably from the left side to the right side from 44 s to 64 s. This is similar to the records from 94 s to 119 s, 159 s to 173 s, and 193 s to 216 s.
As shown in Figure 13, the robot followed the proposed algorithm in identifying patrol paths and signing in at inspection points. There were eight experiments made for patrolling; four of them were patrolling with inspection points placed, and the others did not. The robot started patrolling from the same location in each experiment. The patrol time comparison of each experiment is shown in Table 1. The average time of patrolling with/without inspection points was about 206/205.75 sec, respectively. Though the average time of patrolling with the inspection point was larger than the other one, however, the minimum elapsed time of the former can be reduced to only 194 sec. On the other hand, the maximum elapsed time of the latter was 260 sec, which was larger than any elapsed time of the experiment with inspection points placed. This is because the proposed system runs the sing-in procedure while inspection points are placed in the patrol environment, and the sing-in process took about 5 to 8 sec per inspection point. In addition, without placing inspection points, the robot can only used the first method and the built-in obstacle avoidance function to find the patrol path. While the robot was moving to corners of a room and could not find a way out, this caused time-wasting for backing to patrolling.  There were eight experiments made for patrolling; four of them were patrolling with inspection points placed, and the others did not. The robot started patrolling from the same location in each experiment. The patrol time comparison of each experiment is shown in Table 1. The average time of patrolling with/without inspection points was about 206/205.75 sec, respectively. Though the average time of patrolling with the inspection point was larger than the other one, however, the minimum elapsed time of the former can be reduced to only 194 sec. On the other hand, the maximum elapsed time of the latter was 260 sec, which was larger than any elapsed time of the experiment with inspection points placed. This is because the proposed system runs the sing-in procedure while inspection points are placed in the patrol environment, and the sing-in process took about 5 to 8 sec per inspection point. In addition, without placing inspection points, the robot can only used the first method and the built-in obstacle avoidance function to find the patrol path. While the robot was moving to corners of a room and could not find a way out, this caused time-wasting for backing to patrolling. Several other experiments were conducted to verify the efficacy of the intruder detection function. Figure 13 presents recorded images of the intruder in one of the experiments. While the robot was executing its patrol task, an intruder suddenly ran into the FOV of the SVS (at 10 s), whereupon the SVS detected the skeleton of the intruder (at 13 s). The robot simultaneously ceased its patrol task and initiated the intruder tracking task. From 19 s to 43 s, the intruder was pacing around, during which time the robot rotated to keep the intruder within the center region of the image plane. As the intruder was walking backward (from 43 s to 49 s), the robot followed in order to maintain the depth value of tracking target d h between the near-threshold value d kn and far threshold value d k f . In this experiment, the robot proved highly effective in using the proposed algorithm to detect, identify, and track the intruder. Finally, the recorded images can be queried from a database by the graphic user interface (GUI) built in the user client.
Appl. Sci. 2020, 10, x FOR PEER REVIEW 13 of 16 Several other experiments were conducted to verify the efficacy of the intruder detection function. Figure 13 presents recorded images of the intruder in one of the experiments. While the robot was executing its patrol task, an intruder suddenly ran into the FOV of the SVS (at 10 s), whereupon the SVS detected the skeleton of the intruder (at 13 s). The robot simultaneously ceased its patrol task and initiated the intruder tracking task. From 19 s to 43 s, the intruder was pacing around, during which time the robot rotated to keep the intruder within the center region of the image plane. As the intruder was walking backward (from 43 s to 49 s), the robot followed in order to maintain the depth value of tracking target ℎ between the near-threshold value and far threshold value . In this experiment, the robot proved highly effective in using the proposed algorithm to detect, identify, and track the intruder. Finally, the recorded images can be queried from a database by the graphic user interface (GUI) built in the user client.

Conclusions
This paper outlines the architecture of a novel robot designed to patrol in the workspace. The robot has four functions: independent patrolling, signing in at inspection points, the detection and tracking of intruders, and remote backup of data to facilitate real-time monitoring and user queries. The robot receives from an SVS depth information pertaining to the environment for use in the identification of a suitable path to follow. Inspection point markers encountered along this path cause the robot to veer off the path to check-in. Upon arrival at the inspection point, the number of the RFID tag at that point is recorded. After signing in, the robot rotates CCW and continues its patrol task. In the event that an intruder appears within the FOV of the SVS, the skeleton of the intruder is identified according to the function of the openNI library. The intruder is subsequently tracked and imaged until the intruder moves beyond the viewing window. All data recorded during this encounter is uploaded to the server, to enable monitoring in real-time and facilitate subsequent queries of historical data. Several experiments verify the efficacy of the proposed algorithm.
For comparison to other guard solutions, e.g., security cameras, as mentioned in the introduction, sweeping robots have gradually become commonplace in every home or workspace environment. If there is a simple and low-cost method that can increase the patrol function on traditional sweeping robots, it will effectively solve the problem of home security. In addition, the prices of the retrofit devices (SVS and RFID sensors, etc.) on sweeping robots of the proposed system are lower than N.T.D. 5000, but the prices of a security camera is generally higher than that.

Conclusions
This paper outlines the architecture of a novel robot designed to patrol in the workspace. The robot has four functions: independent patrolling, signing in at inspection points, the detection and tracking of intruders, and remote backup of data to facilitate real-time monitoring and user queries. The robot receives from an SVS depth information pertaining to the environment for use in the identification of a suitable path to follow. Inspection point markers encountered along this path cause the robot to veer off the path to check-in. Upon arrival at the inspection point, the number of the RFID tag at that point is recorded. After signing in, the robot rotates CCW and continues its patrol task. In the event that an intruder appears within the FOV of the SVS, the skeleton of the intruder is identified according to the function of the openNI library. The intruder is subsequently tracked and imaged until the intruder moves beyond the viewing window. All data recorded during this encounter is uploaded to the server, to enable monitoring in real-time and facilitate subsequent queries of historical data. Several experiments verify the efficacy of the proposed algorithm.
For comparison to other guard solutions, e.g., security cameras, as mentioned in the introduction, sweeping robots have gradually become commonplace in every home or workspace environment. If there is a simple and low-cost method that can increase the patrol function on traditional sweeping robots, it will effectively solve the problem of home security. In addition, the prices of the retrofit devices (SVS and RFID sensors, etc.) on sweeping robots of the proposed system are lower than N.T.D. 5000, but the prices of a security camera is generally higher than that.
Secondary, mobility and randomness are the two key points of the patrol system. For the proposed system, there are two functions can achieve these objectives, one is path-free autonomous patrol, and the other is intruder following. The path-free autonomous patrol function creates a random patrol path which can avoid intruders to find possible flaws for planned paths in advance, and the intruder following function can avoid blind spots caused by furniture, and then record more image information of intruders. On the other hand, a security camera is usually installed at a fixed location, and its rotation degree is limited. Therefore, more cameras need to be installed to cover all monitoring ranges, and lack of mobility.
Moreover, information security and privacy are issues that everyone values in recent years. For the proposed system, the user cannot get the image information directly from the patrol robot. The recorded images are saved in a server at first. Users need to login the server to get access to watch the records. Generally, a server has a higher level of information security protection. On the other hand, a security camera has a convenience that users can monitor remotely and directly. However, now a day, "security camera has been hijacked or hacked" is heard from time to time, and these events will cause users to concern about the security of the patrol by security camera.
Summarily, the proposed system is not convenient for users to view patrol results, however, it has a lower price, better path randomness, better mobility, and more information security and privacy. The comparison results are shown in Table 2. Table 2. The comparison between the proposed system and other solutions.

Proposed System
Other Solutions (e.g., Security Camera) Price V X Path Randomness V X Mobility V X Convenient X V Security and Privacy V X Supplementary Materials: The following are available online at http://www.mdpi.com/2076-3417/10/3/974/s1, Video S1: The recorded color images and the depth information from the SVS during the patrol task.

Nomenclature
x i, j Distance from the center of SVS on the x-axis (width) at pixel (i, j) y i,j Distance from the center of SVS on the y-axis (height) at pixel (i, j) d i, j Distance from the center of SVS on the z-axis (depth) at pixel (i, j) d T Threshold of depth used to determine a possible path.