Lab Scale Model Experiment of Smart Hopper System to Remove Blockages Using Machine Vision and Collaborative Robot

: In this study, we propose a smart hopper system that automatically unblocks obstructions caused by rocks dropped into hoppers at mining sites. The proposed system captures RGB (red green blue) and D (depth) images of the upper surfaces of hopper models using an RGB-D camera and transmits them to a computer. Then, a virtual hopper system is used to identify rocks via machine vision-based image processing techniques, and an appropriate motion is simulated in a robot arm. Based on the simulation, the robot arm moves to the location of the rock in the real world and removes it from the actual hopper. The recognition accuracy of the proposed model is evaluated in terms of the quantity and location of rocks. The results conﬁrm that rocks are accurately recognized at all positions in the hopper by the proposed system.


Introduction
The fourth industrial revolution in manufacturing innovation, which started with Industry 4.0 in Germany and the United States of America, was developed based on the concept of "an intelligent society in which everything is connected." Now, this concept is spreading to all industrial fields. In particular, the core technologies of the fourth industrial revolution-artificial intelligence, Internet of Things (IoT) [1,2], cloud computing [3], big data analysis [4,5], smart/wearable devices [6], virtual/augmented/mixed reality [7], 3D printing [8], drones [9], autonomous driving [10], and robotics-are proliferating other fields.
ICTs (Information Communication Technology) generated from the fourth industry are currently being utilized in industries such as manufacturing [11], electricity [12], aviation, transportation [13], urban planning [14], medical care [15], and agriculture. In particular, the introduction of ICT is expected to have a significant effect on the mining sector, which is responsible for supplying high-tech raw materials, e.g., rare metals, to the fourth industry. This is because the mining sector has already initiated a new change of industrial sites via the development of innovative technologies such as smart mining. As a result, as in the case of smart factories in the manufacturing sector, new technologies such as ICT is expected to be indispensable to the optimal operation of smart mines in the mining sector.
In the mining sector, ICTs are being used in various ways. For example, it was used in the form of building a spatial database for a mine site, entering attribute information to visualize it, or performing a pre-simulation by changing attribute information [16][17][18][19]. Recently, high-resolution databases of mining sites have been constructed using drones [20][21][22][23] and unmanned ground vehicles [24][25][26][27][28][29]. Further, small-scale mining sites using a low-power Bluetooth beacon and a smartphone equipped with short-range communication technology are being developed [30][31][32].
In the mining field, research using various ICTs is being conducted, and commercial products are also being developed. Jung et al. [33] simulated the truck-loader transport system of an open-pit mine on AnyLogic software and visualized the results intuitively. Haile Gold Mine in South Carolina in the USA measured and predicted the density of the slurry supplied and discharged to the cyclone accurately [34]. A simulation was conducted in the Ban Houayxai Gold-Silver Mine to estimate the expected output with respect to changes in the blasting design factor and predict the metal recovery rate and prospective improvement of the crushing efficiency [35]. Baek et al. [36] learned big data on the truck movement system at the mine site, optimized it through various machine learning techniques, and predicted productivity indicators. Previous studies have been conducted to analyze and visualize data in virtual space, such as building a database, realtime monitoring, and simulation using ICTs. However, few studies have been conducted to control the physical space using the analysis results in the virtual space. In this study, the image processing-based vision system is used to recognize objects of rocks placed in hopper equipment, one of the mineral processing facilities used in mining sites.
The hopper is a funnel-shaped device that screens dropped rocks and serves as an entrance into the crusher. At a mining site, the hopper equipment is usually installed on the upper part of the crusher that crushes the rocks and provides an avenue through which the rocks input into the gyratory crusher pass before being crushed [37]. Figure 1a illustrates the process by which the transported rocks are dispensed from the truck into the hopper. Appl. Sci. 2022, 12, x FOR PEER REVIEW 2 of 19 low-power Bluetooth beacon and a smartphone equipped with short-range communication technology are being developed [30][31][32].
In the mining field, research using various ICTs is being conducted, and commercial products are also being developed. Jung et al. [33] simulated the truck-loader transport system of an open-pit mine on AnyLogic software and visualized the results intuitively. Haile Gold Mine in South Carolina in the USA measured and predicted the density of the slurry supplied and discharged to the cyclone accurately [34]. A simulation was conducted in the Ban Houayxai Gold-Silver Mine to estimate the expected output with respect to changes in the blasting design factor and predict the metal recovery rate and prospective improvement of the crushing efficiency [35]. Baek et al. [36] learned big data on the truck movement system at the mine site, optimized it through various machine learning techniques, and predicted productivity indicators. Previous studies have been conducted to analyze and visualize data in virtual space, such as building a database, real-time monitoring, and simulation using ICTs. However, few studies have been conducted to control the physical space using the analysis results in the virtual space. In this study, the image processing-based vision system is used to recognize objects of rocks placed in hopper equipment, one of the mineral processing facilities used in mining sites.
The hopper is a funnel-shaped device that screens dropped rocks and serves as an entrance into the crusher. At a mining site, the hopper equipment is usually installed on the upper part of the crusher that crushes the rocks and provides an avenue through which the rocks input into the gyratory crusher pass before being crushed [37]. Figure 1a illustrates the process by which the transported rocks are dispensed from the truck into the hopper. In general, the wide entrance of the hopper facilitates the sequential input of rocks dispensed from the truck into the gyratory crusher. However, when a large number of rocks or a large rock is suddenly input into the hopper, a clogging phenomenon called "jamming" could occur (Figure 1b). In hoppers implemented in existing mining sites, manual intervention either by hand or with mechanical help, e.g., a crane, is necessary to remove the blockage. This requires work to be halted while the blockage is removed. Moreover, a direct manual approach toward the entrance of a crusher can be dangerous. These issues can be resolved by utilizing image recognition and robot technology in the hopper to resolve the clogging phenomena automatically, thereby improving the productivity and stability of mining work.
This study aims to propose a smart hopper system at a mining site using an RGB-D (Red Green Blue-Depth) camera, machine vision technology, and a collaborative robot. The proposed architecture captures real-world RGB and depth images using the RGB-D In general, the wide entrance of the hopper facilitates the sequential input of rocks dispensed from the truck into the gyratory crusher. However, when a large number of rocks or a large rock is suddenly input into the hopper, a clogging phenomenon called "jamming" could occur (Figure 1b). In hoppers implemented in existing mining sites, manual intervention either by hand or with mechanical help, e.g., a crane, is necessary to remove the blockage. This requires work to be halted while the blockage is removed. Moreover, a direct manual approach toward the entrance of a crusher can be dangerous. These issues can be resolved by utilizing image recognition and robot technology in the hopper to resolve the clogging phenomena automatically, thereby improving the productivity and stability of mining work.
This study aims to propose a smart hopper system at a mining site using an RGB-D (Red Green Blue-Depth) camera, machine vision technology, and a collaborative robot. The proposed architecture captures real-world RGB and depth images using the RGB-D camera, detects blockages caused by rocks jammed in the hopper's feeder using machine vision image-processing techniques, and transmits the positions of the classified objects to the collaborative robot, enabling the robot arm to remove the blockages automatically. The performance of the proposed smart hopper system was evaluated by constructing an indoor model laboratory and using a miniature model to perform experiments in various scenarios. The results confirm that all components of the smart hopper system operated correctly in all the scenarios considered.

Design of Smart Hopper System
In this study, a smart hopper system was developed to solve jamming caused by rocks dispensed into a hopper at a mining site. The proposed system identifies and removes unfiltered rocks by integrating technologies such as an RGB-D camera, machine vision technology, and a robotic arm. Figure 2 depicts the overall structure of the proposed smart hopper system. First, the upper part of the hopper was photographed in real-time using the RGB-D camera, and the images were transmitted to a program on a notebook personal computer. Then, the rocks were classified using an image-processing technique in a virtual hopper system implemented in the virtual world, and the appropriate motion was simulated in the robotic arm. Based on the simulation, the robotic arm was operated in the real world to remove the rocks causing the blockage. camera, detects blockages caused by rocks jammed in the hopper's feeder using machine vision image-processing techniques, and transmits the positions of the classified objects to the collaborative robot, enabling the robot arm to remove the blockages automatically. The performance of the proposed smart hopper system was evaluated by constructing an indoor model laboratory and using a miniature model to perform experiments in various scenarios. The results confirm that all components of the smart hopper system operated correctly in all the scenarios considered.

Design of Smart Hopper System
In this study, a smart hopper system was developed to solve jamming caused by rocks dispensed into a hopper at a mining site. The proposed system identifies and removes unfiltered rocks by integrating technologies such as an RGB-D camera, machine vision technology, and a robotic arm. Figure 2 depicts the overall structure of the proposed smart hopper system. First, the upper part of the hopper was photographed in real-time using the RGB-D camera, and the images were transmitted to a program on a notebook personal computer. Then, the rocks were classified using an image-processing technique in a virtual hopper system implemented in the virtual world, and the appropriate motion was simulated in the robotic arm. Based on the simulation, the robotic arm was operated in the real world to remove the rocks causing the blockage.  Figure 3 depicts the operational flowchart of the smart hopper system including data acquisition, processing, virtual simulation, and movement of the robotic arm. First, an RGB image and a depth image were captured using the color sensor and infrared (IR) sensor in the RGB-D camera. Then, the RGB images were converted into grayscale and binary images using machine vision technology and were processed. Only the shapes of rocks were extracted from the entire image. By comparing this with the depth image processed in binary form, the heights of the rocks were estimated. A rock was ignored if it had fallen below the hopper; however, its coordinates were calculated if it was in the hopper. To move the robot arm to the calculated rock coordinates, they were converted to sixaxis joint coordinates for the robot arm. Based on this information, the robot arm was moved along the planned path and the targeted rocks were removed from the hopper. Table 1 lists the detailed specifications of the sensors and controllers used in this study.  Figure 3 depicts the operational flowchart of the smart hopper system including data acquisition, processing, virtual simulation, and movement of the robotic arm. First, an RGB image and a depth image were captured using the color sensor and infrared (IR) sensor in the RGB-D camera. Then, the RGB images were converted into grayscale and binary images using machine vision technology and were processed. Only the shapes of rocks were extracted from the entire image. By comparing this with the depth image processed in binary form, the heights of the rocks were estimated. A rock was ignored if it had fallen below the hopper; however, its coordinates were calculated if it was in the hopper. To move the robot arm to the calculated rock coordinates, they were converted to six-axis joint coordinates for the robot arm. Based on this information, the robot arm was moved along the planned path and the targeted rocks were removed from the hopper. Table 1 lists the detailed specifications of the sensors and controllers used in this study. Appl. Sci. 2022, 12, Figure 4 depicts the machine vision-based image-processing algorithm for the RGB and depth images captured by the RGB-D camera. First, the RGB image was captured using the color sensor, and the region where the hopper was located was selected to be the Region of Interest (ROI). Then, image characteristics, such as brightness, gamma, and contrast, were adjusted based on the environment of the laboratory. Next, a pre-processing operation was performed to convert RGB images into grayscale images, and then into binary images. A suitable threshold was selected based on the characteristics of the experimental environment, e.g., camera angle, resolution, and illumination, to identify the shape of the rock accurately. Finally, the binarized image was pre-processed using a lowpass filter, which removed all elements other than rocks from the image. Thus, the coordinates of the rocks were obtained.
Similarly, depth images were captured using the IR Sensor and the IR Emitter sensor. The area where the hopper was located was selected to be the ROI, and the distance value of the selected area of each depth image was converted into a distance unit in the real world, which was recorded in a matrix. A threshold value was selected to filter for distance values corresponding to the upper portions of the hopper. After filtering the image using the threshold, it was converted into a binary image. The binary RGB and depth images were combined into one by representing each pixel of the two binary images using   Figure 4 depicts the machine vision-based image-processing algorithm for the RGB and depth images captured by the RGB-D camera. First, the RGB image was captured using the color sensor, and the region where the hopper was located was selected to be the Region of Interest (ROI). Then, image characteristics, such as brightness, gamma, and contrast, were adjusted based on the environment of the laboratory. Next, a pre-processing operation was performed to convert RGB images into grayscale images, and then into binary images. A suitable threshold was selected based on the characteristics of the experimental environment, e.g., camera angle, resolution, and illumination, to identify the shape of the rock accurately. Finally, the binarized image was pre-processed using a low-pass filter, which removed all elements other than rocks from the image. Thus, the coordinates of the rocks were obtained.
Appl. Sci. 2022, 12, 579 5 of 18 0 or 1 and multiplying them as matrices. This produced a binary image where only points corresponding to rocks on top of the hopper had a value of 1. Thus, information about the rocks located in the upper portions of the hopper was retained, while that of rocks below the hopper was ignored. Finally, the center of each rock above the hopper was calculated. The vision processing algorithms of National Instrument's LabVIEW and the Vision Assistant software were used to process RGB and depth images in this study. Figure 5 depicts the results obtained from the successive steps of the processing sequence in the case of an RGB image. The RGB image was captured in real-time (Figure 5a), and only the image of the hopper was extracted via the Image Mask function (Figure 5b). In addition, the brightness, gamma, and contrast of the image were adjusted based on the experimental environment using the Brightness function ( Figure 5c). Finally, the RGB image was converted into a grayscale image emphasizing the green plane using the Color Plane Extraction function (Figure 5c,d). Similarly, depth images were captured using the IR Sensor and the IR Emitter sensor. The area where the hopper was located was selected to be the ROI, and the distance value of the selected area of each depth image was converted into a distance unit in the real world, which was recorded in a matrix. A threshold value was selected to filter for distance values corresponding to the upper portions of the hopper. After filtering the image using the threshold, it was converted into a binary image. The binary RGB and depth images were combined into one by representing each pixel of the two binary images using 0 or 1 and multiplying them as matrices. This produced a binary image where only points corresponding to rocks on top of the hopper had a value of 1. Thus, information about the rocks located in the upper portions of the hopper was retained, while that of rocks below the hopper was ignored. Finally, the center of each rock above the hopper was calculated.
The vision processing algorithms of National Instrument's LabVIEW and the Vision Assistant software were used to process RGB and depth images in this study. Figure 5 depicts the results obtained from the successive steps of the processing sequence in the case of an RGB image. The RGB image was captured in real-time (Figure 5a), and only the image of the hopper was extracted via the Image Mask function (Figure 5b). In addition, the brightness, gamma, and contrast of the image were adjusted based on the experimental environment using the Brightness function ( Figure 5c). Finally, the RGB image was converted into a grayscale image emphasizing the green plane using the Color Plane Extraction function (Figure 5c Figure 6 depicts the results of the successive steps of converting the grayscale image in Figure 5 into a binary image and then processing it. A threshold value that included the color value of the rock and removed surrounding elements was selected and the image was converted into a binary image (Figure 6a). Using a low-pass filter and other techniques, the pre-processing algorithm removed the fine noise from the binary image (Figure 6b), and the morphology algorithm removed all objects in all regions except the rocks (Figure 6c). Finally, the coordinates of the extracted object were obtained via particle analysis ( Figure 6d).   Figure 5 into a binary image and then processing it. A threshold value that included the color value of the rock and removed surrounding elements was selected and the image was converted into a binary image (Figure 6a). Using a low-pass filter and other techniques, the pre-processing algorithm removed the fine noise from the binary image (Figure 6b), and the morphology algorithm removed all objects in all regions except the rocks (Figure 6c). Finally, the coordinates of the extracted object were obtained via particle analysis (Figure 6d).  Figure 6 depicts the results of the successive steps of converting the grayscale image in Figure 5 into a binary image and then processing it. A threshold value that included the color value of the rock and removed surrounding elements was selected and the image was converted into a binary image (Figure 6a). Using a low-pass filter and other techniques, the pre-processing algorithm removed the fine noise from the binary image (Figure 6b), and the morphology algorithm removed all objects in all regions except the rocks (Figure 6c). Finally, the coordinates of the extracted object were obtained via particle analysis ( Figure 6d).  Figure 7 depicts the results obtained from the successive steps of the processing sequence in the case of the depth image obtained from the RGB-D camera. First, the depth image was captured and visualized in real-time. Then, the distance values were converted to distance units in the real world and recorded in the form of a matrix (Figure 7a). The value corresponding to each pixel of the depth image was displayed. Points with distances that corresponded to the upper portion of the hopper were retained, while points with all other distance values (i.e., greater than that of the top of the hopper) were removed (Figure 7b). Finally, using the Morphology algorithm, all points apart from those corresponding to the smart hopper were deleted (Figure 7c). Appl. Sci. 2022, 12, x FOR PEER REVIEW 7 of 19 Figure 6. Example of binary image processing: (a) grayscale image is converted into a binary image, (b) low-pass filter is utilized, (c) small objects are removed, and (d) the object's coordinates are calculated. Figure 7 depicts the results obtained from the successive steps of the processing sequence in the case of the depth image obtained from the RGB-D camera. First, the depth image was captured and visualized in real-time. Then, the distance values were converted to distance units in the real world and recorded in the form of a matrix (Figure 7a). The value corresponding to each pixel of the depth image was displayed. Points with distances that corresponded to the upper portion of the hopper were retained, while points with all other distance values (i.e., greater than that of the top of the hopper) were removed (Figure 7b). Finally, using the Morphology algorithm, all points apart from those corresponding to the smart hopper were deleted (Figure 7c).  Figure 8 illustrates the simplified lattice form of the processed depth image of a smart hopper. The binary image of the top of the hopper obtained from the depth image and the binary image of the rock obtained via RGB image processing were treated as matrices of the same size and multiplied. As each pixel of a binary image can be represented by 1 or 0, multiplication of two such matrices produces a matrix whose (i, j)th entry is 1 if and only if the ith and jth entries of the first and second multiplicand matrices, respectively, are 1. Thus, the multiplication set the values of pixels corresponding to rocks below the hopper to 0 as they did not coincide with the pixels demarcating the hopper, and it set those of rocks above the hopper that coincided with the hopper's area to 1. Thus, rocks that had descended through the hopper inlet and were photographed and identified by the proposed system while they were being pulverized by the gyratory crusher were excluded from the blockage removal process, and the robotic arm did not try to dislodge them.  Figure 8 illustrates the simplified lattice form of the processed depth image of a smart hopper. The binary image of the top of the hopper obtained from the depth image and the binary image of the rock obtained via RGB image processing were treated as matrices of the same size and multiplied. As each pixel of a binary image can be represented by 1 or 0, multiplication of two such matrices produces a matrix whose (i, j)th entry is 1 if and only if the ith and jth entries of the first and second multiplicand matrices, respectively, are 1. Thus, the multiplication set the values of pixels corresponding to rocks below the hopper to 0 as they did not coincide with the pixels demarcating the hopper, and it set those of rocks above the hopper that coincided with the hopper's area to 1. Thus, rocks that had descended through the hopper inlet and were photographed and identified by the proposed system while they were being pulverized by the gyratory crusher were excluded from the blockage removal process, and the robotic arm did not try to dislodge them. Figure 6. Example of binary image processing: (a) grayscale image is converted into a binary image, (b) low-pass filter is utilized, (c) small objects are removed, and (d) the object's coordinates are calculated. Figure 7 depicts the results obtained from the successive steps of the processing sequence in the case of the depth image obtained from the RGB-D camera. First, the depth image was captured and visualized in real-time. Then, the distance values were converted to distance units in the real world and recorded in the form of a matrix (Figure 7a). The value corresponding to each pixel of the depth image was displayed. Points with distances that corresponded to the upper portion of the hopper were retained, while points with all other distance values (i.e., greater than that of the top of the hopper) were removed (Figure 7b). Finally, using the Morphology algorithm, all points apart from those corresponding to the smart hopper were deleted (Figure 7c).  Figure 8 illustrates the simplified lattice form of the processed depth image of a smart hopper. The binary image of the top of the hopper obtained from the depth image and the binary image of the rock obtained via RGB image processing were treated as matrices of the same size and multiplied. As each pixel of a binary image can be represented by 1 or 0, multiplication of two such matrices produces a matrix whose (i, j)th entry is 1 if and only if the ith and jth entries of the first and second multiplicand matrices, respectively, are 1. Thus, the multiplication set the values of pixels corresponding to rocks below the hopper to 0 as they did not coincide with the pixels demarcating the hopper, and it set those of rocks above the hopper that coincided with the hopper's area to 1. Thus, rocks that had descended through the hopper inlet and were photographed and identified by the proposed system while they were being pulverized by the gyratory crusher were excluded from the blockage removal process, and the robotic arm did not try to dislodge them.   Figure 9a,b depict images of the hopper and the rocks expressed in binary form following the completion of image processing. Represented as a matrix, the red pixels take the value 1, while the black ones take the value 0. In Figure 9c, the depth and RGB images are superimposed to determine whether the locations of the detected rocks are above or below the hopper. In the example depicted in the figure, one of the two detected rocks is jammed above the hopper, while the other has already passed through the hopper inlet. These two rocks can be differentiated by multiplying the matrix forms of the two binary images; in the product, only pixels that correspond to the first rock take the value 1, while those of the second rock and all others take the value 0. Figure 9d depicts the product image, in which only the pixels of the rock on the hopper remains highlighted in green and all others are excluded. The coordinates of the center of the rock were defined based on the position of the detected rock after calibrating the RGB and depth images to eliminate differences between the coordinate systems of the color sensor and the IR sensor of the Kinect v1 product used in the study [18]. In addition, when rocks were partially supported on the hopper inlet or clustered near the inlet, the distance value of the center point of the rock was calculated and compared with the distance value corresponding to the top of the hopper, thereby determining whether they were above or below the hopper.  Figure 9 presents the results obtained by following the algorithm to exclude rocks located below the hopper after determining the positions of rocks on the hopper. Figure  9a,b depict images of the hopper and the rocks expressed in binary form following the completion of image processing. Represented as a matrix, the red pixels take the value 1, while the black ones take the value 0. In Figure 9c, the depth and RGB images are superimposed to determine whether the locations of the detected rocks are above or below the hopper. In the example depicted in the figure, one of the two detected rocks is jammed above the hopper, while the other has already passed through the hopper inlet. These two rocks can be differentiated by multiplying the matrix forms of the two binary images; in the product, only pixels that correspond to the first rock take the value 1, while those of the second rock and all others take the value 0. Figure 9d depicts the product image, in which only the pixels of the rock on the hopper remains highlighted in green and all others are excluded. The coordinates of the center of the rock were defined based on the position of the detected rock after calibrating the RGB and depth images to eliminate differences between the coordinate systems of the color sensor and the IR sensor of the Kinect v1 product used in the study [18]. In addition, when rocks were partially supported on the hopper inlet or clustered near the inlet, the distance value of the center point of the rock was calculated and compared with the distance value corresponding to the top of the hopper, thereby determining whether they were above or below the hopper.

Coordinate System Transformation and Movement Path Planning Using Inverse Kinematics of Robot Arm
The Niryo One [38] was used as the robot arm in this study. Separate coordinate systems were assigned to the six joints that moved the robot arm, and because the joints operate in series in a predetermined order, they must be operated in a specific method to move the gripper of the arm to the target point. This requires the definition of relations be-tween the joints. Equation (1) presents a transformation matrix representing the correlation between the ith joint and the i + 1th joint.
Here, a i denotes the link length, α i denotes the link twist angle, d i denotes the link offset, and θ i denotes the joint angle. Further, cθ i and sθ i denote cos θ and sin θ corresponding to the ith joint, and cαi and sαi denote cos α and sin α corresponding to the ith joint [39]. Using Equation (1), the correlation equations relating the first joint to the sixth joint of the robot arm can be deduced, as Equation (2). The D-H parameters of the Niryo One robot used in the study are presented in Bugday and Karali [40].
As described above, the movement of the robot arm to the desired point by manipulating each joint can be described in terms of kinematics. However, in this study, when controlling the robot arm, it was necessary to convert the coordinates of the rock detected by the camera into the coordinate system of the robot arm. These coordinates were obtained by applying inverse kinematics, which calculates the joint angle of the robot.

Integration of the Coordinate Systems of the RGB-D Camera and the Robot Arm
The coordinates of the rock identified using the RGB-D camera represented the pixel positions of the rock in the RGB image, and the robot arm had a coordinate system for each of its six joints. Thus, it was necessary to establish a correspondence between the coordinate systems of the RGB-D camera and the robot arm to move the gripper of the arm to the location of the detected rock. Figure 10a depicts the five correction points on the hopper selected to match the coordinate system of the RGB-D camera and the robot arm. The relationship between the x-and y-coordinates of the robot arm and the camera are depicted graphically in Figure 10b. That is, for the five points in Figure 10a, the coordinates on the camera and the coordinates of the robot arm were acquired, respectively, and are shown as graphs in Figure 10b. To obtain the coordinates of the camera, the x and y pixel positions for coordinate points were obtained from the LabVIEW software. In addition, to obtain the coordinates of the robot, the states of the six motors of the robot arm were converted into x and y coordinates at each coordinate point.
The coordinates of the camera and robot arm corresponding to all five positions used during coordinate correction are listed in Table 2. Of the 5 points required for correction, points 1, 4, and 5 have identical x-coordinates, and points 1, 2, and 3 have identical ycoordinates. In Figure 10b, a linear regression line is generated to capture the correlation between the coordinates of the camera and the robot arm. From the results in Table 2, the linear regression line was constructed using linear formulae such as Equations (3) and (4), and represents the formula that can be used to calculate the x-and y-coordinates of the robot arm with respect to the x-and y-coordinates of the camera. The Z coordinates were calculated through the depth value of the detected mineral location. Since most of the minerals are jammed around the crusher inlet and the crusher inlet is almost flat, we calculated constant z-coordinates, and all of them were calculated with the same z-coordinate. In future field experiments, if a large-scale hopper and a high-accuracy RGB-D camera are used, the z-coordinate can be calculated in the same way as the x and y-coordinates. nate. In future field experiments, if a large-scale hopper and a high-accuracy RGB-D camera are used, the z-coordinate can be calculated in the same way as the x and y-coordinates. Figure 10. The calibration points for correcting the coordinate system between the RGB-D camera and the robot arm (a), and the relationship graph between the camera and the robot's x and y coordinates (b).

Laboratory Experiment
In this study, a model experimental setup was designed to reproduce the hopper architecture in an actual mining site. To this end, a hopper with a gyratory crusher was manufactured on a reduced scale, and the shape and number of jammed rocks were classified and the recognition accuracy was evaluated.
Because the supply port of the 54-67-inch gyratory crusher measures about 4.43 m, a hopper model with a 115 mm supply port was manufactured at a scale ration of approximately 1/38 ( Figure 11). In addition, an RGB-D camera was placed to photograph the supply port of the hopper, and a robot arm was implemented to remove rocks. Random dispensation of rocks into the hopper was simulated by dumping rocks from a remote-controlled truck. In addition, to ensure consistency of environment during the implementation of machine vision-based image processing techniques, a constant intensity of illumination was maintained.
hopper model with a 115 mm supply port was manufactured at a scale ration of approximately 1/38 ( Figure 11). In addition, an RGB-D camera was placed to photograph the supply port of the hopper, and a robot arm was implemented to remove rocks. Random dispensation of rocks into the hopper was simulated by dumping rocks from a remote-controlled truck. In addition, to ensure consistency of environment during the implementation of machine vision-based image processing techniques, a constant intensity of illumination was maintained. The purpose of this study was to evaluate the accuracy of object recognition using the machine vision-based image processing technique in the virtual world and the ability to push rocks into the lower part of the hopper using a robot arm in the real world. To this end, based on the locations of the rocks remaining on the hopper, two types of scenarios were identified (rocks in contact with the hopper, rocks crossing the hopper) and classified into two types (two rocks on the hopper, three rocks on the hopper) based on the quantity of rocks remaining on the hopper. In cases in which rocks were observed both above and below the hopper, the accuracy of removal of rocks that had already passed through the hopper was also evaluated. In addition, the accuracy with which the gripper of the robot arm was moved to the target rock was also evaluated, and each experimental process and data processing method was recorded and analyzed. The upper part of the hopper was captured in real-time using an RGB-D camera, but object detection and location recognition are performed whenever a remotely-controlled truck arrives and abandons minerals and disappears. The purpose of this study was to evaluate the accuracy of object recognition using the machine vision-based image processing technique in the virtual world and the ability to push rocks into the lower part of the hopper using a robot arm in the real world. To this end, based on the locations of the rocks remaining on the hopper, two types of scenarios were identified (rocks in contact with the hopper, rocks crossing the hopper) and classified into two types (two rocks on the hopper, three rocks on the hopper) based on the quantity of rocks remaining on the hopper. In cases in which rocks were observed both above and below the hopper, the accuracy of removal of rocks that had already passed through the hopper was also evaluated. In addition, the accuracy with which the gripper of the robot arm was moved to the target rock was also evaluated, and each experimental process and data processing method was recorded and analyzed. The upper part of the hopper was captured in real-time using an RGB-D camera, but object detection and location recognition are performed whenever a remotely-controlled truck arrives and abandons minerals and disappears. Figure 12 illustrates the experimental results obtained using the proposed method in a case involving one rock under the hopper and another above the hopper. The rock on the hopper was located without any protrusions over the edge of the hopper (Figure 12a). Both rocks were accurately recognized using the RGB image-processing technique, as is evident from Figure 12b). The rock located under the hopper was already being crushed in the gyratory crusher or was about to be crushed; therefore, it had to be excluded from the movement path planning phase for the robot arm. By superimposing the depth images of the two rocks and the hopper, only the rock placed on the red background in Figure 12c) was identified to be the final detection object. It was confirmed that only that rock was removed (Figure 12d). Figure 13 depicts another example involving one rock above the hopper and two below it, where the latter are partially hidden by the column of the crusher or already in the process of being crushed, thus making their shape appear irregular. In the image, the rock above the hopper is placed across the opening between the hopper and crusher column, which visually obstructs the space below the hopper. As the depth image was updated in real-time whenever rocks were dispensed into the hopper, the background of the part blocked by the rocks was displayed in red as in the case of the hopper and its upper surface. This indicates that the rock lies on the same level as the hopper. In this case of the experiment, the robot arm successfully removed only the rock lying across the opening of the hopper and the crusher column and ignored the two rocks under the hopper. This confirms that the machine vision-based image-processing algorithm developed in this study successfully detects rocks above the hopper in various positional situations. Figure 12 illustrates the experimental results obtained using the proposed method in a case involving one rock under the hopper and another above the hopper. The rock on the hopper was located without any protrusions over the edge of the hopper (Figure 12a). Both rocks were accurately recognized using the RGB image-processing technique, as is evident from Figure 12b). The rock located under the hopper was already being crushed in the gyratory crusher or was about to be crushed; therefore, it had to be excluded from the movement path planning phase for the robot arm. By superimposing the depth images of the two rocks and the hopper, only the rock placed on the red background in Figure  12c) was identified to be the final detection object. It was confirmed that only that rock was removed (Figure 12d).  Figure 13 depicts another example involving one rock above the hopper and two below it, where the latter are partially hidden by the column of the crusher or already in the process of being crushed, thus making their shape appear irregular. In the image, the rock above the hopper is placed across the opening between the hopper and crusher column, which visually obstructs the space below the hopper. As the depth image was updated in real-time whenever rocks were dispensed into the hopper, the background of the part blocked by the rocks was displayed in red as in the case of the hopper and its upper surface. This indicates that the rock lies on the same level as the hopper. In this case of the experiment, the robot arm successfully removed only the rock lying across the opening of the hopper and the crusher column and ignored the two rocks under the hopper. This confirms that the machine vision-based image-processing algorithm developed in this study successfully detects rocks above the hopper in various positional situations.    Figure 14 depicts a case involving two rocks placed on the hopper. The upper one is located partially over the bottom of the hopper, while the other is clearly above the hopper. Both rocks were detected using the proposed RGB image processing technique. In addition, it was confirmed that both rocks were removed by the robot arm.  Figure 14 depicts a case involving two rocks placed on the hopper. The upper one is located partially over the bottom of the hopper, while the other is clearly above the hopper. Both rocks were detected using the proposed RGB image processing technique. In addition, it was confirmed that both rocks were removed by the robot arm.   Figure 15 depicts a case with three rocks placed together on a hopper-one on top of the hopper, one at the meeting point of the crusher column and the hopper, and one partially intersecting the crusher and the hopper. It can be seen that all three rocks were detected using RGB image processing. By superimposing the images of the individual rocks with the depth image, it was confirmed that all of the rocks were placed on the hopper. Although irregular noise was generated at some points within the hopper in the depth image, it did not affect the recognition of the positions of the rocks (Figure 15c). Based on the RGB image, all three rocks were sequentially removed from the hopper (Figure 15d-f). Table 3 lists the RGB-D coordinates of the detected objects depicted in Figures 12-15 and the transformed coordinates corresponding to the robot arm. One to three objects were detected in all cases, including those in which rocks were placed across hoppers and crusher columns. It was confirmed that the objects under the hopper were detected by the camera and finally excluded based on the depth image processing technique. The coordinates obtained with respect to the RGB-D camera reflected the pixel positions in the RGB image, and it was confirmed that they were converted to coordinate systems of the robot arm using the linear Equations (3) and (4).

Results
tially intersecting the crusher and the hopper. It can be seen that all three rocks were detected using RGB image processing. By superimposing the images of the individual rocks with the depth image, it was confirmed that all of the rocks were placed on the hopper. Although irregular noise was generated at some points within the hopper in the depth image, it did not affect the recognition of the positions of the rocks (Figure 15c). Based on the RGB image, all three rocks were sequentially removed from the hopper (Figure 15d Table 3 lists the RGB-D coordinates of the detected objects depicted in  and the transformed coordinates corresponding to the robot arm. One to three objects were detected in all cases, including those in which rocks were placed across hoppers and crusher columns. It was confirmed that the objects under the hopper were detected by the camera and finally excluded based on the depth image processing technique. The coordinates obtained with respect to the RGB-D camera reflected the pixel positions in the RGB image, and it was confirmed that they were converted to coordinate systems of the robot arm using the linear Equations (3) and (4).  The size of the supply port of the gyratory crusher and the hopper used in realworld mining sites is approximately 4.43 m in length, while the hopper model used in the experiment in this study had a 115 m supply port approximately, making it a 1/38 scale model of an actual hopper. The size of the rocks dispensed during the experiment were also reduced to a similar scale, and the experiment was carried out in a sequence imitating a practical application. However, to evaluate the performance and usability of the smart hopper system developed in this study, it needs to be applied to actual work sites. To this end, a large collaborative robot arm capable of moving unfiltered rocks from the hopper inlet and a high-resolution RGB-D camera are required to be installed. In addition, the image processing algorithm based on an integrated coordinate system with respect to the camera and the robot should be modified to suit the field conditions. Finally, environmental factors such as the illumination of the site and the color and size of rocks should be accounted for. Such applications may improve stability and productivity at mining sites.

Optimizing the Path of the Robot Arm to Remove Rocks
A limitation in the applicability of this study to real-world mining sites arises from the inhomogeneity of sizes of rocks in actual mining scenarios and the possibility of multiple rocks obstructing the hopper. Thus, the gripper arm cannot be reliably guided to a welldefined center point of the rock causing the blockage following a carefully pre-designed path. In addition, if the jammed rocks cannot be pushed out by the robot arm or if the rocks are larger than the size of the inlet in an actual mining application, the gripper of the robot arm will require to be equipped with a crushing mechanism, e.g., a drill. In addition, in this study, the identification of rocks and calculation of their distances were performed using only one camera placed vertically above the inlet. In actual applications, an additional camera should be installed, enabling immediate rock recognition. Finally, when pushing rocks using the robot arm, factors such as size and number of rocks, the direction of inclination, and the center coordinates of rocks should be analyzed while planning a movement path for the robot arm.

Accuracy Degradation Due to Scanning Distance and Dust Air
In this study, a lab-scale model experiment simulating an actual mine site was performed. In the configuration of the experimental site, the distance between the rock and the camera was closely arranged. However, in an actual mining site, a camera must be placed at a high position to scan a large-scale hopper. As the height of the camera increases, the size of each mineral will appear smaller, which will reduce the object recognition accuracy. Of course, in an actual mining site, as the distance of the camera increases, the size of the mineral increases, so recognition is not completely impossible. However, a high-resolution RGB-D camera that reflects the scale of the site and the area of the hopper should be utilized, and a machine vision system should be developed in detail.
At the actual mining site, a large amount of dust will be generated by mineral crushing. This will limit the camera's field of view and reduce the object recognition accuracy. In this study, the chroma and transparency of RGB images were partially adjusted to confirm whether objects were recognized when the dust air environment was simulated. Figure 16 shows the results of the object recognition accuracy experiment for RGB images simulating dust air by adjusting chroma and transparency. Overall, it was confirmed that the rocks at the top and bottom of the hopper were recognized separately from the background. However, when there was a rock under the hopper, it was confirmed that the shape was partially unclear.
The rock recognition results in an environment without dust air are compared with the recognition results in an environment simulating dust air. As a result of rock recognition in Figure 16 and Table 4, all rocks were recognized in the same way as in an environment without dust air, and since the center coordinates of rocks were also calculated almost the same, the accuracy was evaluated through the area of the part recognized as a rock. When there are three rocks on the hopper (Figure 16b), it can be confirmed that the area of the part recognized as a rock is calculated to be narrower by about 2% compared to the non-dust environment. When there are three rocks on the hopper and one rock below (Figure 16d), it was confirmed that the rocks on the hopper were recognized as narrow by about 2% similar to the previous case. On the other hand, the rock under the hopper was recognized to be about 5% smaller depending on the presence or absence of dust air, showing a relatively large difference. Since the intensity of the light transmitted to the rock under the hopper is relatively weak, it was confirmed that it was relatively greatly affected by the light limitation by dust air.
In this study, the chroma and transparency of RGB images were partially adjusted to confirm whether objects were recognized when the dust air environment was simulated. Figure 16 shows the results of the object recognition accuracy experiment for RGB images simulating dust air by adjusting chroma and transparency. Overall, it was confirmed that the rocks at the top and bottom of the hopper were recognized separately from the background. However, when there was a rock under the hopper, it was confirmed that the shape was partially unclear. The rock recognition results in an environment without dust air are compared with the recognition results in an environment simulating dust air. As a result of rock recognition in Figure 16 and Table 4, all rocks were recognized in the same way as in an environment without dust air, and since the center coordinates of rocks were also calculated almost the same, the accuracy was evaluated through the area of the part recognized as a rock. When there are three rocks on the hopper (Figure 16b), it can be confirmed that the area of the part recognized as a rock is calculated to be narrower by about 2% compared to the non-dust environment. When there are three rocks on the hopper and one rock below (Figure 16d), it was confirmed that the rocks on the hopper were recognized as narrow by about 2% similar to the previous case. On the other hand, the rock under the hopper was recognized to be about 5% smaller depending on the presence or absence of dust air, showing a relatively large difference. Since the intensity of the light transmitted  In an actual mining environment, the concentration of dust air is not constant and may be so large that the shape of the mineral is not visible at all. In addition, it can work together with the lighting intensity and shadows of the scene to further reduce the object recognition accuracy. Therefore, to overcome this limitation in actual mining sites, an additional device to periodically measure the dust air concentration using environmental sensors and then remove it will be needed. In addition, a sophisticated image processing algorithm that reflects various conditions in the field should be applied.

Conclusions
In this study, a smart hopper system was proposed based on a machine vision-based image-processing technique, an RGB-D camera, and a collaborative robot. The proposed system captures RGB and depth images using an RGB-D camera and processes it using a machine vision-based image recognition technique in a real-world hopper to detect rocks inside the hopper and calculate their coordinates. In addition, it simulates the movement of the robot arm based on the detected coordinates, enabling it to be maneuvered to remove blockages in the real world. The recognition accuracy of the smart hopper system was evaluated with respect to the location and quantity of rocks. The results confirmed that the rocks were accurately recognized in all situations.
Equipment such as crushers and hoppers used in existing mining sites may be obstructed when a large number of rocks or large rocks are dispensed into the hopper. These rocks are usually removed manually, which poses a safety risk and reduces productivity by delaying the workflow. The utilization of the smart hopper system and robot arm developed in this study is expected to provide an efficient solution to these problems. In addition, a simulation system has been implemented to recognize and process images in the virtual world. The robot arm was directly controlled based on the results and the proposed system was optimized. This paper is also expected to serve as a useful reference for the implementation of the vision system and robot technology in the mining sector in the future.