Remote Control of a Mobile Robot for Indoor Patrol

Abstract

This study applies smartphone, Bluetooth, and Wi-Fi wireless network to control a wheeled mobile robot (WMR) remotely. The first part of this study demonstrates that the WMR can be controlled manually by a smartphone. The smartphone can remotely control the WMR for forward, backward, left-turn, and right-turn operations. The second part of this article presents object tracking. The WMR can follow a moving object through the use of image processing for object tracking and distance detection. In the third part, infrared sensor and fuzzy system algorithms are integrated into the control scheme. Through wall-following and obstacle-avoidance control, the WMR can successfully perform indoor patrol.

1. Introduction

Over the past few years, different types of wheeled mobile robots (WMR) have been proposed. The WMR’s advantages include high mobility, high load, and easy control. Previous WMR studies have chiefly focused on developing effective performance and helping people work in various environments. Many robots have been used in different applications such as indoor services, space exploration, military undertakings, entertainment, healthcare services, etc. In recent years, intelligent control has been applied to WMR fora variety of tasks [1,2,3,4,5,6,7,8,9,10,11,12,13]. One of the most common intelligent controllers is the fuzzy controller [9,10,11,12,13] because the design is relatively simple and flexible. This paper presents three control schemes. The first one is the smartphone remote control, which uses the Zigbee wireless transmission to the computer-controlled robot. Forward, backward, left-turn, and right-turn movements can be performed via the smartphone. The second control scheme utilizes the EYECAM camera and image processing to detect moving objects and calculate the center distance, and then applies fuzzy controller to track the moving object. The moving target can be clearly seen on the screen of the smartphone. In the third scheme, the WMR integrates infrared sensors to the fuzzy control system and distance detection. A camera is put on the WMR for surveillance. The surroundings of the robot can be seen on the phone. Home patrol and monitoring can therefore be accomplished.

With advanced technology and industrial development, path following, navigation, target tracking, multi-vehicle coordination, path planning and other applications of WMR have been widely discussed. Many of the applications use intelligent systems in controller design such as fuzzy system and neural network. Fuzzy system is quite similar to human thinking. In the real world, the fuzzy logic is full of greater fault tolerance, has better promotion, and is better suited for a nonlinear system. When the WMR is meant to work in a variety of environments, the choice of the controller becomes a very important issue. In the paper of Juang et al. [10], the automobile used a visual sensor to track a moving object. The authors utilized a simplified type 2 fuzzy system to a WMR for target-following control. Zhan et al. [12] used a rapid path-planning and fuzzy logic control to make the WMR trace expected paths. Seder et al. [14] proposed a method that is based on the integration of a focused D* search algorithm and dynamic window local obstacle-avoidance algorithm with some adaptations that provide efficient avoidance of moving obstacles. In the paper of Chung et al. [15], a position-control algorithm was proposed with two separate feedback loops: a velocity feedback loop and a position feedback loop. The proposed control algorithm is designed to compensate for both internal and external errors. This algorithm makes it possible for the WMR to accurately follow the designed trajectory. Other studies include the use of image processing and a cerebellar model articulation controller for path-following control [1], in addition to the laser sensor-positioning algorithm in controller design for unknown environment map building [4], automobile ultrasonic sensors [6], a localization system for path planning [7], and distance between the object and the camera [9].

In this study, remote control by smartphone in mini vehicle applications [16,17] is proposed. In our previous work [17], the mobile robot can only be turned on via Wi-Fi, and performed preset path-following control. In this study, more functions are added to the mobile robot. In addition to the turn-on control, dynamic object tracking and obstacle-avoidance controls are added. The robot can also be controlled by a personal computer (PC) or a smartphone remotely, and the moving path can be changed by the operator at the remote site. Images captured by the network camera placed on the robot can be transmitted to the remote PC or smartphone. Intelligent control, wireless communications, and sensor fusion are successfully integrated in this study. The outline of this paper is as follows. In the second section, the discussion of image processing technology for filtering noise on floor, capturing the targets from the image frame, as well as computing the distance and the angle via the camera are given. Section 3 shows the WMR dynamic tracking and indoor patrol schemes; fuzzy steering control rules are also described. The experiments’ settings are explained in section 4 and a description of the architecture of the hardware system, which includes sensors, Zigbee, smartphone, camera, and block diagrams of the communications scheme. Section 5 shows five experimental results of the proposed system. Conclusions are then given in the last section.

2. Image Processing

In most image processing methods, the light source is very important to the final result. In this study, the image in the red/green/blue (RGB) color space is transformed to the hue/saturation/value (HSV) color space which can significantly reduce the impact of lightness. The HSV model defines a color space in terms of three constituent components; hue (H), saturation (S), and value (V). H is an angle from 0 degrees to 360 degrees. Saturation indicates the range of grey in the color space. It ranges from 0 to 100%. Sometimes the value is calculated from 0 to 1. Value is the brightness of the color and varies with color saturation. It ranges from 0 to 100%. The ranges of the hue and saturation color space can be set to appropriated values so that the image processing is performed as a filter and is utilized to discard useless information and preserve useful information. The center of an object can be detected by the use of the distance calculation method of the camera. With these methods, the robot can track a moving object. The RGB color space values are transferred to HSV color space values by the following equations [17,18,19]:

$$\begin{array}{l}H={\cos }^{-1}\frac{(R-G)+(R-B)}{\sqrt{{(R-B)}^2+(R-B)(G-B)}},B\le G\\ H=2\pi -{\cos }^{-1}\frac{(R-G)+(R-B)}{\sqrt{{(R-B)}^2+(R-B)(G-B)}},B>G\end{array}$$

(1)

$$S=\frac{\max (R,G,B)-\min (R,G,B)}{\max (R+B+G)}$$

(2)

$$V=\frac{\max (R,G,B)}{255}$$

(3)

After HSV transformation, the image is then transformed to a binary image so that the original image can be easily identified from the foreground and background. The binary image is applied to change the image color from 0 to 255, where 0 represents black and 255 represents white. A designation of 255 is the foreground representing the captured dynamic obstacle, whereas 0 is the background representing the image to be filtered out. In the binary image process, an appropriate threshold first needs to be set up. If the pixels’ values of a captured image are larger than the threshold value, then they are set to 255. If the pixels’ values are smaller than the threshold value, they are set to 0. The threshold will affect the captured image in object tracking control. Figure 1 shows an example of the original RGB image transformed into the HSV and binary images.

Distance between the robot and the moving target can be obtained by a geometry method [20]. Figure 2 and Figure 3 show the top view and side view of the EYECAM camera (Draganfly Innovations Inc., Saskatoon, SK, Canada). Three prerequisite angles can be obtained by Equations (4)–(6), where the angles are shown in Figure 2 and Figure 3.

$$\alpha ={\tan }^{-1}(\frac{E}{b})$$

(4)

$$\theta ={\tan }^{-1}(\frac{E}{L_y})$$

(5)

$$\beta ={\tan }^{-1}(\frac{L_x/2}{b+L_y})$$

(6)

E denotes the height from the camera to the floor, b represents the distance from the projection point of the camera on the floor to the closest viewpoint, and Ly is the largest range that the camera can see, as shown in Figure 3. Lx is the largest width that the camera can see; it is the length from the left to the right of the view, as shown in Figure 2. Coordinates of the target can be obtained as follows:

$$y=\frac{E}{\tan [(\alpha -\theta )\frac{v}{w}]}$$

(7)

$$x=\tan [\beta (\frac{2u}{H})-1]\cdot y$$

(8)

$x$ is the distance of the target away from the center line of the camera, as shown in Figure 2. $y$ is the distance of the target away from the WMR, as shown in Figure 3.

In Equations (7) and (8), u and v are the horizontal pixel and the vertical pixel of the center coordinates of the captured target, respectively. The H and W represent the pixel lengths of an image provided by the camera. In this paper, E is 15 cm, Lx is 160 cm, Ly is 150 cm, b is 10 cm, H is 320, and W is 480. We can obtain the center coordinates of the target after the image processing. The distance estimation is then used to calculate the distance between the WMR and the target. The dynamic tracking task can be performed via this information and the position of the target on an image frame. Figure 4 shows the relative coordinate of the target with respect to the WMR.

3. Fuzzy Control

In the first part of this study, we explored how the mobile robot is designed to track a moving object in complex environments. Figure 5 shows the dynamic tracking control scheme with fuzzy controller. The procedure of dynamic tracking is shown in Figure 6. There are two inputs of the proposed fuzzy steering controller, which are the horizontal position of the target in a frame of captured image and the distance between the moving target and the WMR. The output of the system is the turning angular velocity of the wheeled mobile robot. Figure 7 shows membership functions X of the horizontal position of the target in a frame of captured image; its fuzzy sets are L, MI and R, which represent left, intermediate, and right, respectively. Figure 8 shows membership functions of the vertical position Y of the target in a frame of captured image. Its fuzzy sets are N, M and F, which represent near, medium and far, respectively. The fuzzy sets of the left wheel speed (the output variable is D) are LS, LMS, LMM, LMF, LM, LFS, LFM, LFF, and LF, which represent a turn that is very slow, greatly slow, medium slow, minimally slow, medium, minimally fast, medium fast, fast and very fast, respectively. Membership functions of the left wheel speed are shown in Figure 9, where the value of wheel speed is the digital input code of the wheel’s motor. Definition of the right wheel speed is similar.

Table 1. Fuzzy rules of the left wheel speed for the fuzzy steering controller of dynamic tracking.

	D	Y	N	M	F
X
L			LMS	LMF	LS
MI			LMM	LM	LFF
R			LFM	LFS	LF

illustrates the fuzzy rules of the left wheel speed for the fuzzy steering controller. Fuzzy rules of the right wheel speed can be obtained by a similar design process.

In this second part of the study, a fuzzy control scheme is designed to drive the WMR in an indoor patrol situation. The WMR is controlled by a sensor signal which provides distance information to the robot such that the robot can be driven and moved along a wall. In wall following, a safety distance between the robot and the wall is predefined. The robot moves along the wall and keeps a constant distance from the wall. Installed on the robot are two infrared (IR) sensors, which are located at the right and left sides of the robot. Each IR sensor points 45 degrees away from the heading angle, and are marked as SR_L and SR_R, as shown in Figure 10. The measured distance from the SR_R is dR and the distance measured by the SR_L is dL. Control sequence of indoor patrol is shown in Figure 11.

In fuzzy steering controller, the inputs are two IR detecting signals and the output of the system is the turning angular velocity of the robot. Fuzzy sets of the left IR sensor are LF, LM and LC, which represent fast, medium, and slow, respectively. For the fuzzy sets of the left wheel, speeds are LS, LMS, LMM, LMF, LM, LFS, LFM, LFF, and LF, which represent turning very slow, greatly slow, medium slow, minimally slow, medium, minimally fast, medium fast, fast and very fast, respectively. The fuzzy rules for the indoor patrol controller of the left wheel speed are shown in

Table 2. Fuzzy rules for the indoor patrol controller of left wheel speed.

	D	Y	RF	RM	RC
X
LF			LMS	LMF	LC
LM			LMM	LM	LFF
LC			LFM	LFS	LF

.

4. Experimental Settings

The WMR used in this study is called Boe-Bot Idrobot (Parallax Inc., Rocklin, CA, USA) as shown in Figure 12. In addition to Bluetooth control, the robot can also be controlled by the smartphone (HTC Corporation, Taoyuan, Taiwan) via Zigbee wireless module (Texas Instruments Inc., Dallas, TX, USA). Zigbee is used to transmit command signals between the webcam (Kinyo Inc., Hsinchu, Taiwan), the robot, and the PC (ASUSTeK Computer Inc., Taipei, Taiwan). Wi-Fi communication is applied to the connection of smartphone and PC in indoor patrol. Due to the size of memory space, the Boe-Bot Idrobot can only process and store 2K Bytes. The IR sensors (Parallax Inc., Rocklin, CA, USA) and Zigbee occupied by personal identification number is another problem requiring further study. The transmitting frequency of the robot’s Zigbee and camera (Kinyo Inc., Hsinchu, Taiwan) are 24 GHz. They are assigned to com port 5 (COM5) and COM3 on the PC. The image is transmitted at 50 ms per one frame. Zigbee receives information at 20 ms for one unit of data. Data is then processed by the PC. The dynamic objects are solved by the net wrapper to the open source computer vision library (Emgu.CV) of dynamic link library (DLL) files. Figure 13 shows the buttons of operation: stop, up, down, left, right, track and patrol. We use this operating interface to receive robot information, send control command, and process the feedback values. Image processing and fuzzy control are coded by C# language.

Figure 14 Shows the structure of the experiment settings, where (A), (B), and (C) are described in Figure 15. Figure 15a is the use of radio frequency (RF) wireless from camera to PC network. Figure 15b shows the communication from WMR to PC. First, the IR’s value needs to be obtained, then the value of transcoding using Zigbee wireless network is transferred to the PC. COM5 receives exclusive information. The data is then decoded. Figure 15c shows the control flow chart of the command that is sent to the WMR. Information is processed by fuzzy controller with the use of COM5, and the corresponding decoded speed is then sent to the WMR by the Zigbee wireless network. Figure 16 shows the block diagram of the remote control.

5. Experimental Results

The first experiment is tested in a terrain as shown in Figure 17. Figure 18 shows the PC-based control operation of the wheeled robot for lower right-turn control application. The robot is driven by a simple proportional control scheme.

Figure 19 shows the testing environment of the second test based on smartphone control operation. Figure 20 is the right-turn operation. Manual operation control sequences of the smartphone and the WMR are shown in Figure 21 and Figure 22. After pressing the command arrow, as shown in Figure 20, the command signal will be transmitted to the WMR. The WMR will then perform the required action.

The first and second experiments described above are manually operated. The following three experiments are autonomous control. The flowchart of the automatic dynamic tracking control sequence is shown in Figure 6, whereas the flowchart of the automatic indoor patrol control sequence is shown in Figure 11. In dynamic tracking, Figure 23 shows the tracking operation. Figure 24 shows the images on the PC screen that are captured by the robot’s camera.

Indoor patrol and remote surveillance are also tested. The WMR is controlled by a two-input two-output fuzzy controller. Figure 25 is the testing field. Figure 26 shows different positions of the wall-following patrol. Figure 27 shows the images captured on the PC screen.

Remote surveillance is shown in Figure 28, Figure 29, Figure 30 and Figure 31. Figure 28 is the testing field of the remote surveillance. Figure 29 shows different remote control operations. Figure 30 is the remote site. Figure 31 shows the captured images on the smartphone.

6. Conclusions

This study presents integration of image processing techniques, fuzzy theory, wireless communications, and smartphone to a wheeled mobile robot (WMR) for real-time moving object recognition, tracking and remote surveillance. The WMR uses a webcam to capture its surroundings. The WMR calculates the relative position of the target object through image processing and distance computation algorithms. Fuzzy system is applied to robot control. In this study, three cases of experiments are given. In the first case, a PC and a smartphone are utilized to control directly the WMR’s forward, backward, left-turn, and right-turn movements. The second case focuses on target tracking control. The WMR can track a specific target by the HSV algorithm and fuzzy controller. The target can be clearly seen on the smartphone via the webcam on the WMR. In the third case, the WMR is applied to surveillance usage. The WMR can be controlled remotely by a smartphone via wireless communications. The WMR uses infrared sensor and fuzzy controller for obstacle avoidance and wall-following control. The WMR can perform indoor patrol and monitor its surroundings. The home site conditions can be clearly seen on the smartphone. Experiments show that the proposed control design and system integration of the wheeled mobile robot works well for indoor patrol.