Deep CNN-Based Materials Location and Recognition for Industrial Multi-Crane Visual Sorting System in 5G Network

Abstract

Intelligent manufacturing is a challenging and compelling topic in Industry 4.0. Many computer vision (CV)-based applications have attracted widespread interest from researchers and industries around the world. However, it is difficult to integrate visual recognition algorithms with industrial control systems. The low-level devices are controlled by traditional programmable logic controllers (PLCs) that cannot realize data communication due to different industrial control protocols. In this article, we develop a multi-crane visual sorting system with cloud PLCs in a 5G environment, in which deep convolutional neural network (CNN)-based character recognition and dynamic scheduling are designed for materials in intelligent manufacturing. First, an YOLOv5-based algorithm is applied to locate the position of objects on the conveyor belt. Then, we propose a Chinese character recognition network (CCRNet) to significantly recognize each object from the original image. The position, type, and timestamp of each object are sent to cloud PLCs that are virtualized in the cloud to replace the function of traditional PLCs in the terminal. After that, we propose a dynamic scheduling method to sort the materials in minimum time. Finally, we establish a real experimental platform of a multi-crane visual sorting system to verify the performance of the proposed methods.

1. Introduction

Intelligent manufacturing methods [1,2,3] have presently attracted increasing attention from both academic and industry domains, in which many state-of-the-art technologies improve dramatic industrial revolution, including the Internet of Things [4], artificial intelligence (AI) [5], computer vision (CV) [6], and other technologies. The visual sorting system [7,8] has an enormous application range with the development of CV and industrial cameras, especially, multi-crane visual sorting systems in iron mining, steel metallurgy, coal mining, and other fields. There are three critical modules that we focus on in multi-crane visual sorting systems, which are the location module, the recognition module, and the control module. The location module aims to locate the position and get the world coordinates of the materials. The recognition module recognizes the type of material on the conveyor belt. The control module combines cloud PLC to sort and put the materials in the specified position within the acceptable time. Many visual sorting technologies were researched in the industry.

The first task for the visual recognition module is to detect the materials that are captured by the camera. In CV, many deep CNN-based algorithms were widely used in object detection [9]. There are two main categories, including one-stage detection frameworks [10,11] that is a single CNN network to directly predict class probabilities and bounding box offsets from original images and two-stage detection frameworks [12,13] that first generate object proposals from original images and then extract features of them to predict object label and bounding box offsets for proposal refinement. Recently, these algorithms are also used in the visual sorting system. Shen et al. [14] applied YOLOv4 which was one algorithm of one-stage detection frameworks to locate and place the garbage in the correct recyclable trash. Song et al. [15] proposed a single-stage grasp detection framework based on region proposal network architecture in the robotic grasp system, the complexity of which was lower than two-stage architecture. Wang et al. [7] used Faster R-CNN to automatically detect and classify the wheel hubs, and then send them to the production lines for high efficiency. However, deep learning-based methods are utilized in robotic grasp systems or wheel hubs detection but are not suitable for the multi-crane sorting system. Compared with two-stage algorithms, one-stage performs better in model complexity and computing resources. Considering the requirement of high accuracy and timeliness, a fast detection algorithm based-YOLOv5 [16] is applied in a a multi-crane visual sorting system for further enhancement. We get pixel coordinates from the detection algorithm. The control module needs to know the world coordinates of materials for the sorting task. Then, the camera calibration method [17] is used to transform the materials from the pixel coordinate system to the world coordinate system.

After detecting materials, the classes are necessary to be recognized preparing for sorting different materials, such as different levels of product quality. Due to the limited experiments, we use Chinese character objects instead of industrial materials. In CV, several techniques for recognizing characters have been reported in many fields [18,19]. Albahli et al. [20] presented an effective and efficient hand-written digit recognition system that designed a customized faster regional convolutional neural network to localize and classify the digits into 10 classes. Cao et al. [21] designed a zero-shot handwritten Chinese character recognition framework based on CNN with a hierarchical decomposition embedding method to achieve competitive performance. Xie et al. [22] presented a vehicle license plate recognition system, in which a novel combined feature extraction model was designed for license plate detection, and a backpropagation neural network was used to recognize the characters of plates. Caldeira et al. [23] proposed an optical character recognition system to recognize characters printed on steel coils in the industry, in which the alignment and segmentation methods were used for filtering non-character components, and a CNN-based classification network was used for recognizing the characters. Gang et al. [24] applied EfficientNet based on CNN to recognize characters of components mounted on the printed circuit board for decreasing defect detection. Most of methods are used to recognize English letters and digits in many areas. However, it is more difficult to recognize Chinese characters due to their complex and diverse features. The ResNet-based network [25] performs significantly in many applications and has extensive research in computer vision. In this article, we plan to explore a ResNet-based network to recognize Chinese characters for high recognition accuracy. The position, types, and timestamp of Chinese chesses will be transmitted to cloud PLC that controls local PLC to conduct the multi-crane for completing material sorting.

The control module plays a significant role in the industry, which aims to use PLCs [26] that sense the inputs, execute the program, and write the outputs. Traditional PLCs are connected to the equipment in the terminal, which cannot realize the decoupling of a traditional PLC and equipment. In addition, the data cannot be interconnected between different equipment due to different industrial control protocols. Many approaches were developed to break the traditional pyramid structure of the industrial internet. Park et al. (Ref. [27]) presented the architecture of a PLC programming environment that employed a virtual plant model consisting of virtual devices to support the specification of discrete event models. A hardware-in-the-loop simulation approach was proposed in the production system, which was directly used for the real plant with minor adaptations [28]. Goldschmidt et al. [29] introduced cloud-based software PLCs to improve the performance of scalability and multi-tenancy, in which the equipment was controlled by the soft PLCs in the cloud that dynamically scaled and assigned workloads. Kalle et al. [30] showed a novel virtual-PLC approach to avoid significant perturbation of the remote attack in industrial control systems. Zhu et al. [31] presented a cloud PLC platform in human–machine interaction for real-time remote monitoring. However, the mentioned PLCs cannot be flexibly deployed in the cloud-terminal-edge. Many emerging technologies cannot be integrated into the industrial structure, such as artificial intelligence, big data, and 5G. In order to meet the requirement of flexibility and scalability in intelligent manufacturing, it is necessary to explore an integrated method to break the data island and improve the coordination between devices. The function of PLCs can be virtualized in the cloud, which not only decreases the cost of PLCs but also makes full use of computing resources to realize data intercommunication.

Motived by the above methods, we developed a novel deep CNN-based multi-crane visual sorting system in intelligent manufacturing, which enables the accurate sorting of materials in real time. The system uses cloud PLC to control the collaborative work of a multi-crane in a 5G network for high reliability and low latency. The main contributions can be summarized as follows:

• We design a novel multi-crane visual sorting system that applies deep CNN algorithm for the material location and recognition, and propose a dynamic scheduling method of materials with cloud PLC in 5G network;
• We apply the YOLOv5 algorithm to locate the position of materials in the industry and use a camera calibration method to realize coordinate conversion. Additionally, we explore a Chinese character recognition network (CCRNet) to significantly recognize the class of each object from the original image;
• We propose a dynamic scheduling method of materials, in which the multi-crane is controlled by local PLCs that receive the commands from cloud PLC and sorts the chesses in specified position with minimum time;
• We establish an experimental platform of the multi-crane visual sorting system using cloud PLC in a 5G network for centralized control and low-latency. Furthermore, we collect the dataset and train the deep CNN-based model. The whole performance of the multi-crane visual sorting system is demonstrated by abundant experiments.

We arrange the remainder of this paper as follows. In Section 2, we introduce the architecture of the multi-crane visual sorting system. Section 3 concretely describes YOLOv5 algorithm for object location, CCRNet for Chinese character recognition, and a novel dynamic scheduling method for the controlling process of the materials. Section 4 shows the established experimental platform and the results. Section 5 obtains the conclusion and future work of this article.

2. Materials and System

The architecture of the multi-crane visual sorting system with cloud PLC in 5G network is shown in Figure 1, which consists of three layers, including the device layer, the transport layer, and the computing layer. The data of all devices are transmitted by a 5G network, and the commands from the cloud PLC also are sent to control the devices by the 5G network.

There are three types of devices in the device layer, including a multi-crane, a conveyor belt, and the cameras. The movement of the multi-crane and the conveyor belt are controlled by local PLCs that receive the commands from the cloud PLC. In order to locate and recognize the materials on the conveyor belt in real-time, the images or the video are transmitted by a 5G network to the cloud, in which an AI server is deployed that integrates deep learning methods to process data.

In the transport layer, a 5G system consists of an access network, a bearer network, and a core network. We deploy two 5G base stations as the access network. The bearer network is as a bridge to transmit the signal between the access network and the core network. In order to meet the requirement of high reliability and low latency about reliable low latency communication (URLLC) [32] as one typical application of 5G, some function of the core network sink to the edge. We can deploy cloud PLC in the edge or in the cloud.

In the computing layer, we design the cloud PLC to coordinatively control remote devices and cooperate with the high-performance server to support many applications that need a large amount of computing, like CV applications and precise location.

For the multi-crane visual sorting system, the first step is to locate the position and obtain the world coordinates of the materials. Second, the class of all materials are recognized from the original image. The process results are transmitted to the cloud PLC by the 5G network. Third, the cloud PLC sends commands to the local PLC that controls the multi-crane according to the dynamic scheduling method. In this article, we focus on three critical technologies that are the location module, the recognition module, and the control module, the details of which are introduced as follows.

3. Methodology

3.1. The Introduction of CNN

CNN has obtained achievements in many applications, such as person re-identification, action recognition, object detection, and image segmentation. CNN is one of the most important neural networks in deep learning, which performs dramatically in feature representation [33]. It is always the backbone of deep CNN. An example of CNN structure is presented in Figure 2, which consists of convolution layers, pooling layers, and fully connection layers. In convolution layers, small convolutional kernels are utilized to convolve the original image and intermediate feature maps to learn edge features, color features, texture features, and other features. These kernels are updated adaptively for many epochs training by back propagation. The pooling layers still use a small kernel following convolution layers to reduce the dimension of feature maps, which also decreases the computing, avoiding overfitting. There are two common types: max pooling and average pooling. After convolution and pooling layers, the feature maps are flattened to one-dimensional vectors through the global average pooling with (1, 1) kernel. In full connection layers, all neurons in the adjacent two layers are connected using an activation function like Relu. In the output layer, cross-entropy or softmax functions are always used to predict the class of object. Deep CNN is composed of many convolution layers, pooling layers, or many blocks of the variants.

3.2. The Detection Module of Materials

In the multi-crane visual sorting system, we apply the YOLOv5 algorithm to locate the materials on the conveyor belt, the architecture of which is shown in Figure 3, which consists of backbone, neck, and prediction. The backbone aims to learn significant features of the original images. In the neck, a path aggregation network is applied to generate feature pyramids and achieve more useful features from the low and high layers. The prediction module outputs the coordinates, classification, and confidence score of the predicted bounding box, which are $(x_1,y_1,x_2,y_2, C, P)$. ($x_1,y_1)$ denotes the lower-left corner coordinates and ($x_2,y_2)$ represents the upper-right coordinates of the bounding box. C is the classification of the bounding box. P is the confidence score that reflects how accurate the bounding box is predicted. These five vectors can be utilized to calculate the loss to optimize the network. The total loss of YOLOv5 contains regression loss, classification loss, and confidence loss.

Regression Loss. We use the diagonal coordinates $(x_1, y_1, x_2, y_2)$ of the predicted and truth values for the bounding box regression. The generalized intersection over union (GIoU) method is used as the regression loss to let the predicted coordinate close to the truth coordinate. Compared with IoU, GIOU can solve the problem that the loss value is 0 when the predicted bounding box and the truth bounding box do not overlap. The diagram of IoU and GIoU are shown in Figure 4, in which A is the predicted bounding box and B is the truth bounding box. We first calculate IoU, that is the ratio of intersection area and union area between A and B. Then, we find the smallest box C that includes A and B. The difference between area C and the union area is obtained. GIoU is the ratio of the difference and area C. The regression loss is given by:

$$L_{reg }=1-GIoU=1+\frac{A^C-A^U}{A^C}-IoU$$

(1)

where $A^C$ is the area of C that is the smallest box that includes A and B. $A^U$ is the area of the union between the truth box A and the prediction box B.

Classification Loss. Binary Cross-Entropy with Logits Loss is used to recognize and optimize the classification of the materials. The classification loss is summarized as:

$$L_{cls}=\frac{1}{N}\sum _{i=1}^N\left[{\widehat{C}}_{i}ln(sigmoid(C_i\right))+\left(1-{\widehat{C}}_i\right)ln\left(1-sigmoid(C_i\right))]$$

(2)

where $N$ is the value of the min-batch, $C_i$ is the predicted class, and ${\widehat{C}}_i$ is the truth class. $sigmoid$ is the sigmoid function that is [0, 1].

Confidence Loss. Binary Cross-Entropy with Logits Loss is used to optimize the confidence of the bounding box. The confidence loss is summarized as:

$$L_{con}=\frac{1}{N}\sum _{i=1}^N\left[{\widehat{P}}_{i}ln(sigmoid(P_i\right))+\left(1-{\widehat{P}}_i\right)ln\left(1-sigmoid(P_i\right))]$$

(3)

where $N$ is the value of the min-batch, $P_i$ is the prediction of the confidence, and ${\widehat{P}}_i$ is the truth of the confidence. The value of ${\widehat{P}}_i$ is [0,1].

The total loss of YOLOv5 is:

$$L_{total}={\lambda }_1{L}_{reg}+{\lambda }_2{L}_{cls}+{\lambda }_3{L}_{con}$$

(4)

where ${\lambda }_1$, ${\lambda }_2$, and ${\lambda }_3$ are weights of the three losses.

We only obtain the pixel coordinates of objects that cannot be used by cloud PLC. Camera calibration [17] is used to describe the collection between the pixel coordinate system and the world coordinate. We define the pixel coordinate system (u, v) and the world coordinate system ($X_w$, $Y_w$, $Z_w$). $P_w$ presents a real P point. The relation between the pixel coordinate system and the world coordinate system can be obtained as:

$$Z_c\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{cc}\begin{array}{cc}{m}_{11}& m_{12}\\ m_{21}& m_{22}\\ m_{31}& m_{32}\end{array}& \begin{array}{cc}{m}_{13}& m_{14}\\ m_{23}& m_{24}\\ m_{33}& m_{34}\end{array}\end{array}\right]\left[\begin{array}{c}\begin{array}{c}{X}_w\\ Y_w\end{array}\\ \begin{array}{c}{Z}_w\\ 1\end{array}\end{array}\right]=M{P}_w$$

(5)

where $Z_c$ is the scale factor. M is the projection matrix of the camera.

For performance evaluation, we use precision, recall, and mean Average Precision (mAP) as the standard methods for performance [34]. After the location module, the world coordinates of the materials are achieved, which are transmitted to cloud PLC.

3.3. The Recognition Module of Chinese Characters

For the sorting task, we need to recognize the class of each object. In the experiments, we use Chinese characters instead of the real materials in the industry due to the limited condition. A CCRNet is designed to recognize the characters, as shown in Figure 5. It uses ResNet [25] as the backbone network and compares them in the experiments. The backbone is used to extract the features-of-interest region. After backbone, we use the fully connected layer to output the class of Chinese character. For example, the input is the image of a Chinese character and the output is the class of “Jing”. We collect the dataset to train and test these models.

The structure of ResNet18, ResNet34, and ResNet50 is introduced in

Table 1. The structure of ResNet18, ResNet34, and ResNet50 [25].

Layer Name	ResNet18	ResNet34	ResNet50
Conv1	(7, 7), 64, stride 2
Conv2_x	(3, 3), max pooling, stride 2
	$\left\{\begin{array}{c}\left(3,3\right),64\\ \left(3,3\right),64\end{array}\right\}\times 2$	$\left\{\begin{array}{c}\left(3,3\right),64\\ \left(3,3\right),64\end{array}\right\}\times 3$	$\left\{\begin{array}{c}\left(1,1\right),64\\ \left(3,3\right),64\\ \left(1,1\right),256\end{array}\right\}\times 3$
Conv3_x	$\left\{\begin{array}{c}\left(3,3\right),128\\ \left(3,3\right),128\end{array}\right\}\times 2$	$\left\{\begin{array}{c}\left(3,3\right),128\\ \left(3,3\right),128\end{array}\right\}\times 4$	$\left\{\begin{array}{c}\left(1,1\right),128\\ \left(3,3\right),128\\ \left(1,1\right),512\end{array}\right\}\times 4$
Conv4_x	$\left\{\begin{array}{c}\left(3,3\right),256\\ \left(3,3\right),256\end{array}\right\}\times 2$	$\left\{\begin{array}{c}\left(3,3\right),256\\ \left(3,3\right),256\end{array}\right\}\times 6$	$\left\{\begin{array}{c}\left(1,1\right),256\\ \left(3,3\right),256\\ \left(1,1\right),1024\end{array}\right\}\times 6$
Conv5_x	$\left\{\begin{array}{c}\left(3,3\right),512\\ \left(3,3\right),512\end{array}\right\}\times 2$	$\left\{\begin{array}{c}\left(3,3\right),512\\ \left(3,3\right),512\end{array}\right\}\times 3$	$\left\{\begin{array}{c}\left(1,1\right),512\\ \left(3,3\right),512\\ \left(1,1\right),2048\end{array}\right\}\times 3$
	Average pooling
	Fully connected layer

. All of them mainly consist of Conv1, Conv2_x, Conv3_x, Conv4_x, Conv5_x, and the fully connected layer. The difference is the number of convolutional layers that are described in Table 1. In this article, the number of the output is 18 classes. We compare the performance of these models about Chinese character recognition in the experiments.

We use softmax cross-entropy loss [35] to optimize CCRNet. The loss function is summarized as:

$$L_{scls}=-\frac{1}{N}\sum _{i=1}^N-log(\frac{e^{Z_i}}{{\sum }_{j=1}^C{e}^{Z_j}})$$

(6)

where N is the batch size, and $Z_i$ is the prediction output of the CCRNet.

3.4. The Control Module Using Dynamic Scheduling Method

The information of all materials in the visual field is transmitted to the cloud PLC. In the initial state, all cranes are waiting for demands. We design a dynamic scheduling method on cloud PLC to control cranes in minimum time. We make a multi-crane list $Cra=\left[i,\left(x_i,y_i,z_i\right)\right], i\in \left[0,1,2\dots ,K\right]$. i is the i^th crane, and ($x_i,y_i,z_i$) is the initial position of the i^th crane. We define a multi-object list $Obj=\left[j,\left(x_j,y_j,z_j,x_j^{\prime },y_j^{\prime },z_j^{\prime }\right)\right], j\in \left[0,1, 2\dots ,J\right]$. j is the j^th object. $\left(x_j,y_j,z_j\right)$ is the position captured of the j^th object by location module. $(x_j^{\prime },y_j^{\prime },z_j^{\prime })$ is the target position of the j^th object to be transported. The scheduling strategy is described as Algorithm 1:

Algorithm 1. Scheduling strategy.

input: All cranes $Cra=\left[i,\left(x_i,y_i,z_i\right)\right], i\in \left[0,1\dots ,K\right].$ All materials Obj$=\left[j,\left(x_j,y_j,z_j,x_j^{\prime },y_j^{\prime },z_j^{\prime }\right)\right],j\in \left[0,1, 2\dots ,J\right]$.

output: The relation between the crane and the object for sorting

Initialize multi-crane state

for each crane i $\in Cra$ do    Record the initial time $t_{i0}$ and the initial position $x_{i0}$ of each crane

for each object $j\in Object$ do

Calculate the arrival time of the ith crane $t_{i1}=t_{i0}+t_{x1}$$, t_{x1}$ is the running time from the initial position to the object position on the x-axis. Record the crane position $x_{i1}$ at $t_{i1}$.

Calculate the completion time of shipment $t_{i2}=t_{i1}+t_{x2}$$, t_{x2}$ is the time of shipment. Record the crane position $x_{i2}$ at $t_{i2}$.

Calculate the time of delivery to the target place $t_{i3}=t_{i2}+t_{x3}$$, t_{x3}$ is the running time from the object position to the target position. Record the crane position $x_{i3}$ at $t_{i3}$. Calculate $\left(t_{y1}, t_{z1},t_{y3}, t_{z3}\right)$. The process is the same with x-axis.

Calculate the whole time of the ith crane that transports the jth object:

$T_j=\max \left(t_{x1}, t_{y1}, t_{z1}\right)+t_{x2}+\max \left(t_{x3}, t_{y3}, t_{z3}\right)$

Until all materials are calculated. We get a list T=$\left[T_0, T_1,\dots ,T_J\right]$. Sequence the element in T from small to large. We can get the object’s ID that the ith crane needs to transport.

Until all cranes are assigned with materials to be transported.

4. Results

4.1. The Experimental Results of Detection Module Based on YOLOv5

We designed a comparative experiment involving YOLOv5 and deployed them on PyTorch using NVIDIA 3090 graphics processing units (GPUs). To improve the robustness of the system, we used data augmentation methods on the original images, including scaling, color space adjustments, and mosaic augmentation. We set the number of epochs for 100 and the batch size to 8. Adam optimizer [36] was utilized for learning the dramatic representations. The initial learning rate was 0.001. We collected 790 images that consisted of chesses and cubes. In more detail, 568 images was selected as the training set, 79 images belong to the testing set, and the other 142 images belong to the validating set.

After training and testing, we obtain the precision, recall rate, and mAP on the training set, testing set, and validating set, which are presented based on the YOLOv5 algorithm in

Table 2. The results of YOLOv5 for object detection on the training set, testing set, and validating set.

Model	Dataset	Number	Precision	Recall Rate	mAP
YOLOv5	Training set	568	99.60%	99.46%	99.44%
	Testing set	79	98.49%	98.81%	98.85%
	Validating set	142	96.55%	95.40%	97.80%

. We achieve the precision, recall rate, and mAP over 99% on the training set. Moreover, the mAP of YOLOv5 on the testing set is up to 98.85%. The precision of the validating set is also up to 96.55%. This further proves the significant performance of YOLOv5 in the multi-crane visual sorting system. From the overall performance, we choose YOLOv5 as the location algorithm and used the process results of YOLOv5 to continue the camera calibration and material sorting experiments.

4.2. The Results of CCRNet on Chinese Characters

We collected the dataset for Chinese character recognition. There were 5333 images of 18 classes, in which there were 14 Chinese characters and 4 English letters. 4799 images were selected as the set, and 534 images belonged to the testing set. We added ResNet18, ResNet34, and ResNet50 as the backbone of CCRNet. Epoch was set to 100 and the batch size was 16. Adam optimizer was also used to optimize the network and the initial learning rate was 0.0001. The loss of ResNet-based models is presented in Figure 6a on the training set. We can see that the convergence rate of ResNet18 was faster than ResNet34 and ResNet50. The performance of ResNet50 was better than ResNet34. Meanwhile, the accuracy of ResNet-based models is presented in Figure 6b on the training set. With the increasing of epoch, the accuracy of these three models is becoming better. After 60 epochs, the trend is becoming more and more flat. These three models performed significantly well in character recognition.

As seen in

Table 3. The accuracy of ResNet for character recognition on the training set and testing set.

Model	Dataset	Number	Accuracy
CCRNet_ResNet18	Training set	4799	99.50%
	Testing set	534	99.25%
CCRNet_ResNet34	Training set	4799	99.49%
	Testing set	534	99.43%
CCRNet_ResNet50	Training set	4799	99.42%
	Testing set	534	99.25%

, we trained the CCRNet model based on ResNet18, ResNet34, and ResNet50 for the testing set. The accuracy was over 99%. On the training set, the accuracy of CCRNet_ResNet18 exceeded 0.01% and 0.08% over CCRNet_ResNet34 and CCRNet_ResNet50. On the testing set, the accuracy of CCRNet_ResNet34 exceeded 0.18% over CCRNet_ResNet18 and CCRNet_ResNet50. The average accuracy of three models was over 99.2%, which demonstrates that ResNet-based models performed significantly well on Chinese character recognition.

In addition, we compared the accuracy of Chinese characters and English letters, and the results can be seen in

Table 4. The accuracy of CCRNet on Chinese characters and English letters.

Model	Chinese Character Accuracy	English Letter Accuracy
CCRNet_ResNet18	98.99%	100%
CCRNet_ResNet34	99.25%	100%
CCRNet_ResNet50	98.99%	100%

. We can see that the accuracy of English letters is 100% and better than Chinese characters. CCRNet_ResNet34 achieves 99.25% of accuracy and is 0.26% higher than CCRNet_ResNet18 and CCRNet_ResNet50.

In Figure 7, the accuracy distribution of each character on CCRNet_ResNet18, CCRNet_ResNet34, and CCRNet_ResNet50 is presented, in which the value on the diagonal represents the number of each character. We can find that four characters are recognized to the error class on CCRNet_ResNet18 and CCRNet_ResNet50. Only three characters are recognized to the error class on ResNet34. Considering the performance and complexity, we used CCRNet_ResNet18 as the final backbone of CCRNet.

The detection and recognition of Chinese characters is shown in Figure 8, in which all objects are precisely detected, and the confidence score is close to 1. In each image, there are different shapes of objects with different classes. All characters are fully recognized and located. From the experimental results, we demonstrated the significance of YOLOv5 and CCRNet on character detection and recognition.

4.3. The Performance of the Visual Sorting System Using Dynamic Scheduling Method

After visual recognition and camera calibration, we obtained the world coordinates, class and timestamp of the materials that are sent to the cloud PLC. We designed a novel dynamic scheduling method in the cloud PLC server that controled two cranes for the sorting task. Two L-PLCs in the terminal picked the materials and put them in the designed place according to commands from the cloud PLCs in 5G environment.

According to the scheduling strategy, the consuming time of the crane is presented in

Table 5. The consuming time of the crane for the sorting task with scheduling strategy.

Time	Prediction Time (s)	Real Time (s)	$\left|\mathit{\Delta }\mathit{t}\right|$
1	[(1, 2.972), (2, 3.282), (3, 3.301), (4, 3.616)]	2.756	0.216
2	[(1, 2.804), (2, 3.179), (3, 3.237), (4, 3.648)]	3.016	0.212
3	[(1, 3.146), (2, 3.160), (3, 3.179), (4, 3.398)]	3.056	0.090
4	[(1, 2.958), (2, 3.037), (3, 3.055), (4, 3.370)]	3.064	0.106
5	[(1, 2.838), (2, 3.179), (3, 3.237), (4, 3.648)]	3.036	0.198

, which includes the running time of the crane from the initial position to the object position, the time of shipment, and the running time of the crane from the object position to the target position. For example, we predict the time for four objects with the crane, like [(1, 2.972 s), (2, 3.282 s), (3, 3.301 s), (4, 3.616 s)]. According to scheduling method, the cloud PLC conducts the crane to sort the first object with minimum time. For the first test, the predicted minimum time is 2.972 s and the real time was 2.759 s. The time difference is 0.216 s between them. For the third test, the time difference was 0.09 s between the prediction and real time. The average time of five times was 0.164 s, which demonstrates the performance of the scheduling method and meets the requirement of the sorting task in real-time.

The experimental results of the multi-crane visual sorting system are shown in Figure 9. The objects on the conveyor belt are correctly sorted into the designated place in real-time. The proposed method can be applied to many applications in the industry, including separation of different materials and material grading. In the future, we will design a dynamic placement method in a cloud PLC, which can place the objects in the specified position according to certain rules.

5. Conclusions and Future Work

The intelligent visual sorting system plays important role in intelligent manufacturing, in which artificial intelligence and computer vision technologies improve the intelligent and unmanned development of the industry. We establish a deep CNN-based multi-crane visual sorting system with cloud PLCs in a 5G environment. A YOLOv5-based algorithm and a Chinese character recognition network were developed to significantly locate and recognize materials from the original image. mAP of YOLOv5 on validating set is up to 98.85%, and the confidence score is close to 1, which significantly shows the performance of the location module. The accuracy of Chinese character recognition is also up to 99.43% on testing set, which fully met the requirement of different material sorting in the industry. Moreover, cloud PLCs are investigated to be flexibly deployed in the terminal and in the cloud, which allows low-level devices to cooperate with each other. Therefore, the control scheduling method is designed in cloud PLCs that cooperates with AI algorithms to control the multi-crane in minimum time. The average time of sorting is about 3 s, which contains the whole process of the crane from starting to sorting completion. The real experimental platform of the multi-crane visual sorting system verifies the performance of the proposed methods.

Although the experimental results perform dramatically, there is much room to improve the multi-crane sorting system. In the industry, there are massive data streams that are transmitted by a 5G network. We will design 5G and TSN bridge as a transmission layer to highly guarantee deterministic communication. In addition, we will integrate cloud PLCs and an AI platform on the same server in the cloud for stable performance. We will develop some scheduling method to control the low-level devices for low energy and high efficiency. Furthermore, we will continue to improve the accuracy of AI algorithms that can be used in real industrial applications.