An Efficient Algorithm for Cleaning Robots Using Vision Sensors ^†

Abstract

In recent years, cleaning robots like Roomba have gained popularity. These cleaning robots have limited battery power, and therefore, efficient cleaning is important. Efforts are being undertaken to improve the efficiency of cleaning robots. Most of the previous works have used on-robot cameras, developed dirt detection sensors which are mounted on the cleaning robot, or built a map of the environment to clean periodically. However, a critical limitation of all the previous works is that robots cannot know if the floor is clean or not unless they actually visit that place. Hence, timely information is not available if the room needs to be cleaned. To overcome such limitations, we propose a novel approach that uses external cameras, which can communicate with the robots. The external cameras are fixed in the room and detect if the floor is untidy or not through image processing. The external camera detects if the floor is untidy, along with the exact areas, and coordinates of the portions of the floor that must be cleaned. This information is communicated to the cleaning robot through a wireless network. Thus, cleaning robots have access to a ‘bird’s-eye view’ of the environment for efficient cleaning. In this paper, we demonstrate the dirt detection using external camera and communication with robot in actual scenarios.

1. Introduction

Public places like hospitals and industries are required to maintain standards of hygiene and cleanliness. Traditionally, the cleaning task has been performed by people. However, due to various factors like shortage of workers, unavailability of 24-h service, or health concerns related to working with toxic chemicals used for cleaning, autonomous robots have been seen as alternatives. In recent years, cleaning robots like Roomba [1] have gained popularity. These cleaning robots have limited battery power, and therefore, efficient cleaning is important. Efforts are being undertaken to improve the efficiency of cleaning robots.

The most rudimentary type of cleaning robot is the one with bump sensors and encoders, which simply keeps cleaning the room while the battery has charge. Other approaches use dirt sensors attached to the robot to clean only the untidy portions of the floor. Researchers have also proposed to attach cameras on the robot to detect dirt and clean. However, a critical limitation of all the previous works is that a robot cannot know if the floor is clean or not unless it actually visit that place. Hence, timely information is not available on whether the room needs to be cleaned or not, which is a major limitation in achieving efficiency.

To overcome such limitations, we propose a novel approach that uses external cameras, which can communicate with the robots. The external cameras are fixed in the room and detect if the floor is untidy or not through image processing. The external camera detects if the floor is untidy, along with the exact areas, and coordinates of the portions of the floor, which must be cleaned. This information is communicated to the cleaning robot through a wireless network. Unlike previous works that use on-board robot sensors, the novel contribution of the proposed work lies in using external cameras and intelligent communication between the camera node and robot. In the proposed work, we demonstrate results of dirt detection using an external camera and communication with the robot. Our future work will comprise navigating the robot to the coordinates received from the external camera.

The proposed method enables cleaning robots to have an access to a ‘bird’s-eye view’ of the environment for efficient cleaning. We demonstrate how normal web-cameras can be used for dirt detection. The proposed cleaning algorithm is targeted for homes, factories, hospitals, airports, universities, and other public places. The scope of our current work is limited to indoor environments; however, an extension to external environments is straightforward. In this paper, we demonstrate the current algorithm through actual sensors in real-world scenarios.

2. Related Works

Dirt detection and cleaning using robots is an active area of research. Dirt detection using visual sensors is proposed in [2,3]. Similarly, mud and dirt separation has been proposed in [4]. Some researchers have focused on particular environments in which cleaning robots will be used. This includes dirt detection and cleaning in office environments [2], wall cleaning robot [5], cleaning robot for greenhouse roofs [6], and cleaning robot for swimming pools [7]. In terms of algorithms for dirt detection, histogram and saliency detection based methods have been proposed [8,9]. Other researchers have focused on developing algorithms for cleaning large area with multiple robots [10,11], system design and obstacle avoidance of vacuum cleaning robot [12], algorithms for complete coverage path planning for cleaning task using multiple robots [13], and algorithms for cleaning robot’s cognitive abilities [14]. Commercial cleaning robots are also available and widely used [1,15,16]. Many researchers have focused on outdoor mud and dirt detection on roads for unmanned ground robots and vehicles [17].

A map is a prerequisite for path planning. A map is generated by using any of the SLAM (Simultaneous Localization and Mapping) algorithms available in the literature (see [18,19]). Given a map and goal location, a robot can plan a path from its current position. The literature is full of global and local planning of mobile robots. The most widely used algorithms for global planning are: A* [20], and Dijkstra‘s algorithm [21], among many others.

Networking and communication between robots and sensors is common in many research projects, which include inter-robot communication [22,23] and path sharing [24]. With respect to the proposed research, communication between external camera and robot has been proposed in [25] to guide the path of the robot.

3. Dirt Detection and Robot Notification Algorithm

Figure 1 shows the flowchart of the dirt detection and robot notification algorithm. It is assumed that a camera is setup on the ceiling of the room to monitor the dirt on the floor.

The algorithm starts by reading the current frame ($I_t$) from the camera. If the background image ($BG$) is unset, the frame is set as the background frame. This frame is the one that does not contain dirt and is the ideal state of the room’s floor. This frame could also be setup manually by user’s instruction.

Once the background image is set, the subsequent frames are first blurred or smoothed to reduce noise. Smoothing is done by applying a filter, in which an output pixel’s value (i.e., $g(i,j)$) is determined as a weighted sum of input pixel values (i.e., $I(i+k,j+l)$):

$$g(i,j)=\sum _k\sum _{l}I(i+k,j+l)h(k,l),$$

(1)

where $h(k,l)$ is called the kernel, which has the coefficients of the filter.

The next step is to take the absolute difference ($I_{\mathrm{diff}}$) of the current frame and background image, $I_{\mathrm{diff}}=|I_t-BG|$. Threshold operation is then applied on the smoothed image to generate the threshold image ($I_{\mathrm{thresh}}$),

$$I_{\mathrm{thresh}}=\left\{\begin{array}{cc}255,\hfill & \mathrm{if}{I}_{\mathrm{diff}}\ge \mathrm{Thresh}\hfill \\ 0,\hfill & \mathrm{otherwise}.\hfill \end{array}\right.$$

(2)

We then find contours in the threshold image and check for any blobs, which represent the dirt on the floor. If blobs are found, we calculate the total area of the blobs. If the total area is greater than the threshold area (${\delta }_{\mathrm{thresh}}$), the algorithm calculates the coordinates of the blobs. These coordinates are then transferred to the robot for cleaning.

4. Experiment and Results

The experiment setup is shown in Figure 2. Figure 2a shows the Logicool HD 1080 camera setup in the corner of the experiment room at a height of 255cm on a pole. The camera was connected to a laptop computer for image processing. Figure 2b shows the dimensions (395 × 190 cm) of the experiment area.

Figure 3 shows the results of the experiments. Figure 3a shows the background image, which is the image without dirt. This image is manually set by the user. Since this image contains parts of the room that have furniture and boxes, which could be moved, we set the region of interest by masking the image. This is shown in Figure 3b. Figure 3c shows the image with dirt. For dirt, we used many pieces of paper which were all 3 × 3 cm in size. Figure 3d shows the difference between the background image (Figure 3b) and the current frame (Figure 3c). Threshold operation is then applied on this image and the blobs are detected as shown in Figure 3e. The algorithm calculates the total area of the blob and the cleaning area, which is shown in Figure 3f. The coordinates of the bounding box in Figure 3f are transferred to the robot with an instruction to clean.

The transfer of coordinates was tested between the camera computer and robot computer, which were on the same network. The camera computer was set to IP address 192.168.0.11 and robot computer was on the IP address 192.168.0.15. The transferred data was: $<x:135,y:171,w:379,h:273>$, where $x,y,w$ and h represent the x-coordinate, y-coordinate, width, and height of the dust area, respectively. In the proposed work, we confirmed receiving the data on the cleaning robot’s computer. Actual navigation to the dirty area is the next phase of the project and will be developed in the future.

5. Conclusions

In this paper we proposed an algorithm to improve the efficiency of cleaning robots by using external cameras. Unlike previous research, which uses an on-robot camera for dirt detection, the external camera mounted on the ceiling provides a bird’s-eye view of the environment and detects dirt. We proposed an algorithm to detect dirt and calculate its total area and coordinates. This information is transferred to the cleaning robot. The advantage of the proposed algorithm is that the cleaning robot can remotely know the coordinates of the dirty areas to clean.

In the proposed work, we developed and experimented with dirt detection using an external camera and notification to the robot. In the next phase of the project, we will develop the shortest path algorithm for the robot and navigate the cleaning robot to the coordinates of the dirty areas received from the external camera.