Development of Fishcake Gripping and Classification Automation Process Based on Suction Shape Transformation Gripper

Abstract

The surge in demand for automating seafood processing necessitates the development of robotic processes for transportation, packaging, and classification. South Korean companies are actively constructing diverse robots and grippers for fishcake handling, yet small workshops face spatial constraints. To address this, the study focuses on creating a gripper capable of versatile fishcake handling within compact spaces. The gripper, designed for single-robot use, employs three suction cups, adapting its grip based on fishcake shapes. Small fishcakes are gripped at the center with one suction cup, elongated ones with two cups aligned to the slope, and wider ones with three cups. A testbed with the gripper attached to a robot facilitates fishcake gripping, classification, and automation testing. Fishcake recognition and gripping tests revealed challenges based on shape, width, and material. Despite difficulties, a commendable 100% success rate was achieved for the majority of fishcakes, showcasing the gripper’s effectiveness. Identified improvements include reducing the suction cup diameter and increasing pressure for enhanced gripping and classification in confined spaces. The study demonstrates the successful development of a gripper for versatile fishcake handling, particularly beneficial for small workshops. The identified improvements offer pathways to enhance efficiency in fishcake gripping and classification within limited spaces.

1. Introduction

As the demand for seafood increases, so does the demand for the robot automation of seafood processing. Simultaneously, there is a growing need for the development of robotic processes in transportation, packaging, and classification for both raw and processed seafood. Moreover, as the workforce for seafood processing gradually diminishes, the demand for robot automation processes is on the rise. Consequently, there is a need for a method to execute the process solely through robot automation, independent of the number of workers [1]. This study introduces a robotic automation solution designed to facilitate classified packaging tailored to the diverse shapes, sizes, and types of fishcakes. The proposed solution applies automated robotic processes to the packaging stage during fishcake processing and validates its performance through testing. In the packaging process after classifying fishcakes, using a gripper type that can handle gripping various types of fishcakes is necessary. Thus, it is essential to build more robots in workplaces that want to distribute different types of fishcakes. Large companies that produce and supply fishcakes have relatively larger workplaces than small and medium-sized factories, and can build various robots and grippers that can differently handle different types of fishcakes, as shown in Figure 1 [2]. As depicted in Figure 1, large companies employ a single robot and gripper to handle one type of fishcake during the transfer process. Managing one fishcake per robot represents a straightforward and convenient approach, making it particularly suitable for large workplaces. However, given the diverse types of fishcakes available in the market, handling each type necessitates different robots and, consequently, demands expansive workplaces where these robots can be installed to accommodate the unique requirements of each fishcake type.

However, it is challenging to build robots and grippers in a small workspace. To address this issue, this study developed a gripper capable of gripping multiple fishcakes with one robot, even in a small workspace. Its practicality is notable, as it can enhance fishcake productivity by effectively handling various fishcakes without the need for numerous robots. In this study, a process testbed was built by applying a gripper designed for gripping various fishcakes to a single robot, and a fishcake gripping and classification test was carried out using this testbed. The results showed the gripper’s capability to rotate and grip various shapes of the incoming fishcakes, effectively addressing the inefficiency observed when a single robot could only grip one type of fishcake. Furthermore, it is expected that a more efficient fishcake gripping and classification process optimized for a small space can be performed by gripping various shapes of fishcakes with a single robot in a small workspace. This study investigates the process of gripping and classifying various fishcakes using a testbed with a single gripper.

The remainder of this paper is organized as follows. Section 2 describes the shape and function of the gripper design for fishcake gripping. Section 3 presents the automation process, using a testbed with the designed gripper. Section 4 provides the results of tests on fishcake gripping and classification carried out using the testbed.

2. Related Works

2.1. Gripping Object

Grippers typically serve the purpose of picking up and relocating objects, but their ability to handle different types of objects is constrained. Consequently, the selection of a gripper should align with the specific characteristics of the targeted object To handle various objects, it is essential to keep changing the gripper or secure many robots capable of installing various grippers. Consequently, grippers that can handle various objects have been developed. Soft grippers that can change the stretched distance of grippers, akin to those illustrated in [3,4], are under development, particularly for applications in the food industry. However, challenges arose when attempting to manipulate flat-shaped fish cakes adhered to surfaces. This study addresses these challenges by introducing a gripper designed to grasp fish cakes in various forms, including the flat shape, regardless of their orientation, position, or specific geometry.

2.2. Object Detection and Classification

Time Series Classification (TSC) presents a significant and challenging problem in data mining, and ongoing research is actively exploring the application of deep neural networks (DNNs). Specifically, architectures like residual and CNN demonstrate strengths in image-based classification by computing and extracting features while preserving spatial information. In a comprehensive overview of deep learning applications across various time series domains, adhering to the unified taxonomy of DNNs for TSC, the CNN architecture garnered high evaluation scores [5]. However, previous research [6] focused on enhancing fishcake quality sustainability through OpenCV, a common API in computer vision. A challenge arose in defining all irregular shapes of fishcakes. To address these issues, this paper employs a preprocessing process with OpenCV and enhances classification efficiency by utilizing the CNN-based YOLOv5 model [7].

3. Suction Shape Transformation Gripper of the Vacuum Ejector Type

Figure 2 illustrates the shape and configuration of the vacuum ejector-type fishcake gripper developed for the fishcake automation process in this study. Unlike previous studies that focused on grippers for various objects, the gripper developed here specifically includes the capability to grasp flat fishcakes [8,9,10,11,12]. The gripper in this study consists of three suction cups, a spring unit, a linear actuator, and a solenoid valve, as depicted in Figure 2a. Schmalz’s SPB4 20 SI-55 G1/8-IG products with a 20 mm diameter were used for all three suction cups. The spring unit is designed to maintain a constant distance and contact force, ensuring smooth contact between the gripper and the targeted fishcake during gripping. In cases where only a portion of the suction cups is used, the unused ones can be adjusted along the length of the suction cup using a linear actuator, depending on the fishcake’s shape. Each suction cup is employed to grip fishcakes, and the number of suction cups used varies based on the fishcake type. For small fishcakes or those with similar distances from the center point to corners, a single suction cup is utilized for gripping, while elongated fishcakes, such as rod-shaped ones, require two suction cups. Additionally, the gripper is configured to handle fishcakes with a larger area using three suction cups.

During gripping, it is crucial to ensure that the fishcake area does not deviate from the diameter of the suction cup. This ensures that the pressure remains consistent, allowing for stable gripping even at high speeds. The gripping points based on this principle are illustrated in Figure 3. The gripper designed in this study is of an assembly type and can be adapted to robots with different types of flanges by altering the shape of the block assembled to the robot arm’s flange. This adaptability means that it can be seamlessly integrated into various robot systems with minimal modifications, showcasing a user-friendly design concerning maintenance and potential repairs.

4. Fishcake Automation Process

4.1. Automation Process Testbed

Figure 4 depicts the configuration of the testbed designed for various fishcake transfers and classification by incorporating the developed gripper into a parallel robot. This testbed comprises a belt conveyor, a darkroom for the vision process, an inner vision camera, dimmable lighting, photo sensors, a parallel robot, and a bucket. The belt conveyor is a product from Ohsung Unicon, the camera is MG-A500K-22 from CREVIS, Yongin 17099, Republic of Korea, and the parallel robot is IRB360 from ABB. Figure 5 illustrates the process flow. Fishcakes are randomly positioned on the belt conveyor and transported along it. A signal is triggered when a transported fishcake is detected by a photo sensor, prompting the vision camera. Once the camera recognizes the signal, it transmits the identified shape and gripping position of the fishcake to the parallel robot via the PC. The recognized image is sent to the PC in a format similar to the study by Sandeep Jagtap et al. [13]. Based on this information, the parallel robot tracks the fishcake and executes gripping motions.

4.2. Image Recognition for Fishcake Shape and Position

After capturing an image of the fishcake using the camera to determine its shape and position, a circumscribed rectangle is generated on the screen, and the fishcake’s center point is computed. The center point is determined as the centroid of the area based on the weight of the fishcake area, rather than the rectangle’s center. Consequently, for specifying the fishcake’s position, the coordinates of the center point identified through image recognition are transmitted to the robot. These camera-recognized coordinates must be converted to correspond to the belt conveyor coordinates, enabling the robot to effectively grip the fishcakes on the belt conveyor. The vision camera screen is depicted in Figure 6a, where the rectangle’s area (b), representing the belt conveyor area, and the actual positions of the four vertices are identical. In the vision camera, the top-left vertex is the origin, with the x-value increasing towards the right. Here, $l_v$ represents the horizontal length of the camera screen and $P_v$ denotes a random point signifying the fishcake’s position on the camera screen. On the belt conveyor, the top-right vertex is the origin, and the y-value increases towards the left. $l_c$ is the horizontal length of the belt conveyor and $P_r$ corresponds to the end of the robot’s virtual tool, positioned identically to the real $P_v$. It is noteworthy that the +x direction on the vision camera screen corresponds to the −y direction on the robot. Consequently, the length of $P_v$ is equal to ($l_c-P_r$) and can be expressed as Equation (1) through the proportional representation of each corresponding dimension. This calibration ensures alignment between the gripper and vision camera coordinates. In the event of future changes in the camera’s position, gripper coordinate recalibration can be conducted based on the camera screen and the conveyor space.

$$P_r=l_c-l_c/l_v\times P_v$$

(1)

Regarding the fishcake shape, multiple datasets of various fishcake shapes are created in advance, and this dataset is learned to identify the shape of the photographed fishcake. To effectively recognize the shape of a fishcake, it is imperative to have a comprehensive dataset for learning. While attempting to identify a fishcake similar to the dataset, even in the absence of specific fishcake data, can yield results, it may lead to inaccuracies. Utilizing diverse datasets specifically curated for fishcakes enhances the accuracy of the recognition process. The dataset employed comprises a total of 1231 images. These images were extracted from a 10 s video recorded on a moving conveyor, segmented into frame intervals. The dataset meticulously considers the location and direction of each type of fishcake, providing a robust foundation for accurate shape recognition.

The YOLO Model was used for training, similar to the image recognition performed by Salaki Reynaldo Joshua et al. [14]. This model can be used to distinguish the fishcake shape. Initially, YOLOv7was under consideration; however, the processing average speed of the C#-based development program ranged between 500 ms and 600 ms. Despite claims in YOLOv7′s paper that it is 6.3% more accurate and 1.5 times faster than YOLOv5s in terms of Mean Average Precision (mAP) and processing speed, practical performance tests revealed that the YOLOv5s model achieved an average processing speed of approximately 80 ms. Surprisingly, YOLOv5 outperformed YOLOv7 by being 6.25 times faster in processing speed. Given the necessity for high-speed processing as actual items move on the conveyor, the decision was made to adopt and develop the YOLOv5′s S model.

Accordingly, the shapes and types of fishcakes are classified by image recognition and the length of each side of the circumscribed rectangle.

4.3. Gripper Suction Shape Transformation According to Fishcake Shape

As mentioned in Section 2, the suction shape of the gripper developed in this study was designed to be varied based on the fishcake shape. The fishcake can be gripped based on the slope by adjusting the tool’s center point to align with the central position of the chosen suction cup, contingent on the fishcake’s shape, and rotating accordingly.

The center point of the suction cup was set as the tool center point (TCP) when using one suction cup, and the center point of the gripper was set to the TCP when using three suction cups. The center point of the two suction cups was set to the TCP when using two suction cups. In this case, the slope of the two points is calculated by receiving the slope of the fishcake as recognized by the camera, and then the fishcake is gripped using it.

Figure 7 shows the rear view of the gripper. The center point of the fishcake varies by the fishcake shape, and the position of the TCP varies by the number of suction cups used. The default center point for the parallel robot used in this study is the center position of the three suction cups. To change the gripping position based on the fishcake shape, the TCP position must be changed. This can be set by applying an offset corresponding to the distance from the center point of the three points in Figure 8.

5. Gripping and Classification of Fishcakes

5.1. Fishcake Gripping and Classification Test

Gripping and classification tests for the fishcake process were conducted using the mentioned automation algorithm. In this examination, the belt conveyor speed, transporting the fishcakes, was set to 500 rpm, while the gripping speed of the parallel robot was set to 800 mm/s. Three types of fishcakes were utilized in the test, characterized by: small or similar distances from the center point to the corners, elongated shapes, and fishcakes with a large area. Figure 9 illustrates the fishcake models employed in this study. The fishcakes categorized as the first class include cuboid, heart, sphere, circle, ellipse, and triangle. The second class comprises long, thicker, and cylindrical fishcakes, while the third class consists of flat square fishcakes.

For fishcakes in the first class, a single suction cup is used for gripping, as depicted in Figure 10a. Fishcakes in the second class are gripped using two suction cups, as shown in Figure 10b. The third class of fishcakes is gripped using all three suction cups, as illustrated in Figure 10c. Subsequently, the fishcakes from each class are segregated into respective buckets. Fishcakes from class 1, gripped with one suction cup, are placed in the first bucket, as seen in Figure 11a. Fishcakes from class 2, gripped with two suction cups, are sorted into the second bucket, shown in Figure 11b. Fishcakes from class 3, gripped with three suction cups, are classified into the third bucket, depicted in Figure 11c.

5.2. Analysis of Test Results

The image recognition and gripping tests for the fishcakes in Figure 8, as mentioned earlier, were conducted 10 times for each shape. The results of these tests are summarized in

Table 1. Success rates of image recognition.

	Cuboid	Heart	Sphere	Circle	Ellipse	Triangle	Long	Thicker	Cylinder	Flat Square
1	1	-	1	1	1	1	2	2	2	3
2	1	1	1	1	1	1	2	2	2	3
3	1	1	1	1	1	1	2	2	2	3
4	1	1	1	1	2	1	2	2	2	3
5	1	1	1	1	1	1	2	2	2	3
6	1	1	1	1	1	1	2	2	2	3
7	1	-	1	1	2	1	2	2	2	3
8	1	1	1	1	1	1	2	2	2	3
9	1	1	1	1	1	1	2	2	2	3
10	1	1	1	1	1	1	2	2	2	3
[%]	100	80	100	100	80	100	100	100	100	100

and

Table 2. Success rates of pick and place.

	Cuboid	Heart	Sphere	Circle	Ellipse	Triangle	Long	Thicker	Cylinder	Flat Square
1	O	O	O	O	O	O	X	X	O	O
	O	O	O	O	O	O	X	X	O	O
2	O	O	O	O	O	O	△	O	X	O
	O	O	O	O	O	O	O	O	X	O
3	O	O	O	O	O	O	X	O	O	O
	O	O	O	O	O	O	X	O	O	O
4	O	O	O	O	X	O	O	O	X	O
	O	O	O	O	X	O	O	O	X	O
5	O	O	O	O	O	O	O	O	O	O
	O	O	O	O	O	O	X	O	X	O
6	O	O	O	O	O	O	X	O	X	O
	O	O	O	O	O	O	X	O	X	O
7	O	O	O	O	O(2)	O	X	X	O	O
	O	O	O	O	O(2)	O	O	X	O	O
8	O	O	O	O	O	O	X	O	O	O
	O	O	O	O	O	O	X	O	O	O
9	O	O	O	O	O	O	O	O	O	O
	O	O	O	O	O	O	X	O	O	O
10	O	O	O	O	X	O	O	O	X	O
	O	O	O	O	X	O	O	O	X	O
[%]	100	100	100	100	80	100	45	80	60	100
	100	100	100	100	80	100	40	80	50	100

. Table 1 provides an overview of the image recognition success rates for fishcakes. The items marked 1, 2, and 3 indicate that they were classified into classes 1, 2, and 3, respectively. ‘-’ indicates that image recognition failed, and they were not classified into any class. The items that failed to be recognized were indicated in a dark color and the items recognized as another class are indicated in grey. Cuboid, sphere, circle, and triangle were consistently recognized as the first class. However, heart and ellipse were identified as the first class with an 80% probability. Heart encountered recognition failure twice, but when not recognized, it was classified as class 1, posing no issues with gripping. Ellipse was identified as class 2 twice due to its elongated shape. Items categorized as long, thicker, and cylindrical were all correctly recognized as class 2, while flat squares were uniformly identified as class 3. In this scenario, all issues appear to stem from insufficient datasets. The input fishcake may be erroneously categorized into different groups based on factors such as lighting, shape, illuminance, etc., or recognition results may not be generated. It is assessed that augmenting the dataset by incorporating more diverse examples could address this problem effectively.

Table 2 provides a summary of the gripping and classification success rates for fishcakes. Instances where gripping and classification failed were denoted with a dark color and an ‘X’. When the gripping was successful but somewhat unclear due to issues such as fishcake width and material, they were marked with a grey color and ‘△’. A 100% success rate in gripping and classification was achieved for all fishcakes except for the ellipse.

As mentioned earlier, recognition failure occurred for the heart; however, there was no issue with gripping and classifying it since it was predetermined to be classified in class 1. In the case of the ellipse, gripping and classification failed twice each. Despite being classified as class 2, gripping succeeded using two suction cups, aligning with the ellipse’s shape.

Gripping class 2 fishcakes proved challenging due to their elongated nature, where even the slightest deviation in the center point could alter their shape. For long, thicker, and cylinder fishcakes, gripping was susceptible to minor position errors. Nevertheless, gripping thicker fishcakes was relatively easy due to their thick and long shape. In contrast, long fishcakes, with their thin profile, and cylinder fishcakes, featuring surface irregularities, experienced a notable decrease in the gripping success rate.

The thinnest fishcake among the long fishcakes has a thickness of approximately 15 mm, while the gripper’s diameter in this test is 20 mm. To reliably grip thin or uneven fishcakes in the future, it may be necessary to consider replacing it with a smaller diameter suction cup. During this transition, it is crucial to select a suction cup capable of securely holding the heaviest fishcakes tested, weighing 86 g. Suction cups with a diameter that is too small may exhibit weak suction force. Additionally, reducing the diameter increases the pressure applied, but excessive pressure can lead to the suction cup collapsing. Hence, it is essential to identify an appropriate pressure level to ensure effective gripping without compromising the suction cup’s functionality.

Class 3, flat square, had a 100% gripping and classification success rate due to its large surface area and stable gripping with three suction cups.

6. Conclusions

In this study, a versatile gripper capable of handling various fishcake shapes was developed, and a testbed for gripping fishcakes with a robot equipped with a gripper was fabricated and used to test gripping. In the fishcake shape recognition process, the fishcake shapes used for testing were recognized at a belt conveyor speed of 500 rpm. Recognition errors occurred only with a few heart or ellipse-shaped fishcakes, which are not simply round. The fishcake gripping and classification process, conducted at a robot speed of 800 mm/s, revealed a high gripping rate for fishcakes with similar distances from the center point to the corners, often reaching 100%. However, the gripping rate for thin fishcakes or those with pronounced irregularities was comparatively lower. While most fishcakes correctly moved to the gripping position based on their center point and shape of the fishcake, it was difficult to grip some fishcakes depending on the thickness and material of the fishcake.

This problem can be improved by improving the recognition program, reducing the diameter of the suction cup of the gripper, and increasing the pressure of the suction cup.

Furthermore, as a future study, modularizing and automating the packaging process post-fishcake classification could potentially streamline several workers’ tasks into a single robotic workstation. This approach has the potential to address challenges related to limited workspace and workforce reduction.

In the current study, the fishcake transfer speed is set at 500 rpm, and the robot gripping speed is 800 mm/s. In practical applications, it may be necessary to operate at significantly higher speeds. Evaluating the feasibility of securely holding fishcakes at increased conveyor and robot gripping speeds is crucial. If challenges arise in maintaining stability during faster operations, design adjustments should be considered to ensure a more secure grip.

The method proposed in this study is expected to enable efficient gripping and classification of various fishcakes even in small spaces.