Automatic Ladle Tracking with Object Detection and OCR in Steel Melting Shops ^†

Abstract

A ladle tracking system in steel production plants is essential for optimizing the ladle transportation between different processing units. The currently used technologies for ladle tracking, including Radio Frequency Identification (RFID) systems, are not effective due to their high maintenance costs and poor performance in harsh conditions, leaving a significant gap in developing an automated ladle tracking system. This paper proposes two innovative solutions to address these problems: a computer-vision-based ladle tracking system and an integrated approach of preprocessing techniques with optical character recognition (OCR) algorithms. The first method utilizes a YOLOv8 framework for detecting the two classes from the input images, such as the ladles and their unique numbers. This method achieved a precision of 0.983 and a recall of 0.998 in detecting the classes. The second method involves several preprocessing steps prior to the application of OCR. This is suitable for challenging environments, where the clarity of the images may be compromised. EasyOCR with enhanced preprocessing was able to extract the ladle number with a confidence score of 0.9948. The results demonstrate that vision-based automated ladle tracking is feasible in steel plants, improving operational efficiency, ensuring safety, and minimizing human intervention.

1. Introduction

The steel production process mostly revolves around a melt shop, which is a key facility in processing all the raw materials that are made into steel products. Within this shop, the ladle is considered the central vessel for transporting and refining molten metal [1]. Every integrated steel-making plant has a steel melting shop (SMS), which processes hot metal/scrap and raw materials using Electric Arc Furnaces (EAFs) or Basic Oxygen Furnaces (BOFs). After such preliminary operations, the molten metal is placed in the ladle furnace for modifications to its composition [2]. Melt shops, on the other hand, tend to install degassing units as well in order to enhance the cleanliness of the molten metal and remove impure gases. When all the required elements are measured and prepared, the molten metal is then delivered to a continuous caster, where it becomes a slab, a billet, a bloom, etc. [3]. A billet is a partially finished steel product with a smaller cross-sectional area, often utilized in bar and rod production. A bloom is a larger partially finished product, serving as raw material for manufacturing structural members such as beams and rails. A continuous caster is a device that casts molten metal into these partially finished shapes in a continuous process, improving efficiency and minimizing material loss. Safety concerns are paramount in these operations due to the hazardous materials and high temperatures involved. The entire process, which lasts from the packaging of the raw materials until the packaging of the steel products, is mainly based on the control and movement of ladles [4]. Ladles play a critical role in the smooth transport of liquid metal within the melt shop. Usually, ladles are moved on rail transfer cars, or with the aid of cranes to allow for their quick movement from one unit to another inside the SMS [5]. Minimized ladle transport time is very important for the effectiveness of operations and attainment of production targets [6]. The effective utilization of Electric Overhead Travel (EOT) cranes is essential in ensuring the smooth operation of ladle transportation within the melt shop. These cranes serve the purpose of picking up and carrying ladles to and from processing units, such as arc furnaces, ladle furnaces, continuous casters, etc. [7].

Ladle tracking systems play a major role in monitoring and managing the transportation of molten metal between different processing units. The traditional ladle tracking method includes a practical approach of visual monitoring through manual supervision. This approach is labor-intensive and also has a high element of human error, particularly in fast-paced environments where manual tracking becomes difficult. Manually supervising and tracking the ladles and other critical equipment by ground personnel or supervisor raises the probability of injuries, but digitization provides room for automating tracking to minimize human activity in risky zones, thus increasing safety. Also, ladle transportation between the major units is important in melting shop productivity; there is a need to reduce transportation duration in order to avoid delays that are expensive when it comes to casting operations. One of the current methods for monitoring the positions of ladles employs the use of Radio Frequency Identification (RFID) systems, which are costly to maintain and provide low reliability due to the high frequency of malfunctions due to damage caused in very hot and harsh conditions. RFID systems are exposed to various challenges in the harsh environment of steel plants. The high temperatures, typically above 250 °C, cause degradation of the strength of RFID tags, which are normally thermally tolerant to about 120 °C, leading to frequent malfunctioning. Moreover, interference from large steel structures lowers the efficiency of signal transmission and makes the detection of tags inconsistent [8]. The harsh working conditions lead to signal attenuation, difficulty in precise positioning, and reduction in RFID component lifespan. Dust, debris, and slag accumulation on RFID transponders also render them unreadable [9]. Moreover, maintenance of RFID-based tracking systems under such working conditions entails frequent replacement and additional protective insulation, thus increasing the operational costs and lowering their long-term sustainability. These conventional methods of tracking cause delays in decision-making, especially in dynamic real-time movement, but contemporary advanced digital tracking systems enhance tracking, enabling the monitoring of supervisors in controlling the position of the ladle and movement to minimize congestion. Moreover, there is the consideration of energy efficiency, which is of paramount importance in the ladle furnace process, which consumes a great deal of energy, mainly because the molten metal must be maintained at an appropriate temperature. If there is too much cooling when a ladle is being transferred, rework will have to be conducted with reheating; however, if there is too much heat, there will be higher energy costs [10]. Digitalized systems are beneficial in this case because they can use information on the temperatures and positions of the ladles with regard to the planned operations to optimize energy employed in processes in real time.

The evolution of artificial intelligence and machine learning has dynamically reshaped industrial automation, refining predictive maintenance, defect detection, and process monitoring in steel production [11]. The above gaps are addressed in this research paper by proposing two methods for ladle tracking systems using advanced computer vision and optical character recognition (OCR) techniques. To overcome the challenges faced by RFID systems, computer vision (CV) and optical character recognition (OCR) offer a better solution with the use of real-time image processing for ladle tracking without physical contact. Deep learning algorithms like YOLO and Convolutional Neural Networks (CNNs) offer precise object detection, making them suitable for industrial applications [12]. In addition, OCR, with preprocessing methods like contrast enhancement and noise removal, improves the readability of ladle numbers even when there is poor visibility [13]. The proposed method detects ladles, along with their identification numbers, in real time, thereby automating the steel-making process. The proposed solution removes the requirement for human involvement and ensures that it is more reliable and efficient than any conventional method. The main contributions of this paper are as follows:

1.

A computer-vision-based ladle identification system that uses grayscale images of ladles to detect ladles in real time.

2.

A preprocessing-based OCR method that significantly improves the accuracy of ladle number extraction in challenging industrial environments.

3.

A robust methodology for integrating object detection and OCR, which can further be used for any industrial process that requires real-time monitoring and tracking.

The rest of the paper is structured as follows: Section 2 provides a detailed explanation of the methodologies adopted for ladle tracking, Section 3 discusses the results obtained from the proposed methods, and Section 4 concludes the paper by summarizing the key points and potential future work.

2. Proposed Methodology

2.1. Computer-Vision-Based Ladle Identification

This research is focused on a computer-vision-based approach for accurate ladle identification in a steel melting shop (SMS) unit. It highlights the importance of object detection techniques in combination with optical character recognition (OCR) methods to detect ladles along with their unique identification numbers within minimal time. In order to develop an accurate object detection model for ladle number identification, the research utilizes a dataset containing images of steel ladles with clear identification numbers for training the proposed model. The objective of the training phase is to develop the model’s ability to extract important features from ladle images, such as edges, shapes, textures, and patterns, for accurate ladle identification. A prominent feature of the methodology is the use of bounding boxes for the images containing ladles and their respective identification numbers. This has the effect of constraining the region of interest of the OCR system to the location of the ladle number, hence improving ladle number recognition performance as against the typical OCR systems that scan the entire image in search of ladle numbers. The proposed methodology is a combination of an object detection model and OCR system, becoming a two-stage process with ladle detection before number extraction. First, the model detects the two classes, such as the ladle and the number region, in the image and creates bounding boxes around these objects. Then, the OCR system is specifically applied to the box containing the number region to extract the number with more accuracy. The integration of this approach into software helps to improve the ladle handling processes that enhance productivity and operational processes in steel manufacturing industries. Figure 1 displays the workflow of the proposed approach. The object detection process involves the following steps: i. data collection; ii. image labelling; iii. data preprocessing; iv. YAML file; and v. training

(1) Data collection: Data collection involves the process of obtaining a complete dataset of grayscale images of ladles and their corresponding identification numbers. First, it is necessary to determine what classes of objects are going to be detected. In this case, ladles are identified as one target class and numbers as a separate class. The dataset must consist of different images of ladles and their unique identification numbers. These images should differ in the background, lighting and posing, and the size of the objects. Such a collection of images that captures various images of ladles and their numbers helps to build a dataset required to train the object detection model.

(2) Image Labeling: The image labeling process refers to attaching labels on images so as to clarify the source and the specific identification numbers of the ladles therein. An appropriate annotation tool such as the Computer Vision Annotation Tool (CVAT), is used for drawing the bounding boxes around the classes around the ladles and the numbers present within the ladles. Every bounding box is labeled with its relevant class definition, as shown in Figure 2. This process of making annotations is used to develop a labeled dataset upon which the object detection model is trained.

The CVAT was chosen due to its ease of use, ability to handle large datasets, and compatibility with multiple annotation formats appropriate for deep learning models. It has the ability to handle precise bounding box annotations, thus providing high-quality labeled data for YOLOv8 training. It also has automation features, which reduce human annotation time and increase labeling efficiency.

(3) Data Preprocessing: The improvement of the quality and quantity of the dataset largely depends on data preprocessing. This research uses data augmentation methods that are used in enriching datasets by introducing pattern transformations that include rotation, flipping, scaling, and brightness modifications. These adjustments are conducted in perspective to increase the number of available images, thus improving the model’s generalization and robustness. Then, the labeled data is further subdivided into three main datasets: the training dataset, validation dataset, and test dataset. This process is also carried out in the CVAT tool, and a dataset folder is created that contains two subfolders: “images” and “labels”, as shown in Figure 3. The “images” folder consists of the ladle images, and the “labels” folder contains an annotation text file for each image. These annotation text files follow a certain pattern following Equation (1). These files contain the details of each object present in the image, such as the object class ID and coordinates of the bounding boxes, thus providing structured data for training the model.

(object class id)(x center)(y center)(width)(height)

(1)

(4) YAML file: The development of YAML files is essential for configuring YOLOv8 training parameters. The YAML file is central to YOLOv8 training configuration. It contains important parameters like

• path: root directory for storing dataset;
• train, validation, and test: pathways to the training, validation, and testing sets;
• nc: count of object classes (ladle and number);
• names: class names (“ladle”; “number”);
• batch_size: number of images trained per iteration;
• epochs: how many training steps to update model parameters.

These parameters provide organized dataset arrangement and successful model training for precise ladle tracking. In the instance of the provided YAML file in Figure 4, the directory that holds the dataset is specified by the “path” parameter.

(5) Training: Training and validation of the suggested YOLOv8 model were carried out on the cloud-based NVIDIA GPU of Google Colab. Along with this, an HP Victus laptop (AMD Ryzen 7000, RTX 3050 GPU; HP Inc., Palo Alto, CA, USA) was employed for preprocessing and handling the data. The laptop’s mid-end GPU was employed for preprocessing data and annotation in the beginning, and the deep learning model training was carried out using Colab’s cloud GPU. For actual real-time implementation in steel industries, an industrial-grade edge computing platform such as the NVIDIA Jetson Xavier NX can be utilized for efficient inference and tracking.

The ultralytics library is used to install and initialize the YOLOv8 that is used for developing the proposed model. During training phase, the model iteratively learns to detect ladles and their numbers by minimizing box, class, and detection focal losses. Evaluation metrics such as precision (P), recall (R), mean average precision at 50% IoU (mAP50), and mean average precision at 50–95% IoU [mAP50-95] are also monitored over epochs. The model progressively improves its ability over epochs to accurately detect ladles and numbers, achieving high precision and recall. This training and validation process leads to the development of a robust object detection model capable of accurately identifying ladles and their numbers in steel manufacturing industries, as depicted in Figure 5. The model was trained using YOLOv8 with the following hyperparameters:

• Learning rate: 0.001;
• Batch size: 16;
• Epochs: 100;
• Optimizer: Adam;
• Loss function: CIoU loss.

YOLOv8 was selected due to its superior accuracy vs. inference speed and fits perfectly for real-time ladle tracking.

• YOLOv5 → YOLOv8 has a better backbone, better detection of small objects, and better computational efficiency.
• Faster R-CNN → It has very high accuracy but is computationally costly, though, and is not appropriate for application in real-time industrial scenarios.
• SSD (Single Shot MultiBox Detector) → Slightly faster than Faster R-CNN but not quite as good as YOLOv8.

Through providing high detection accuracy with minimal latency, YOLOv8 is best for industrial applications.

2.2. Enhanced Ladle Identification Through Preprocessing and OCR Integration

The proposed method II represents a refined approach for ladle identification, specifically designed to tackle the challenges faced by the initial method in accurately identifying ladles, particularly in harsh environments where the clarity of ladle images may be compromised. This enhanced method incorporates comprehensive preprocessing steps prior to the application of optical character recognition (OCR), acknowledging the limitations inherent in relying exclusively on OCR for real-time ladle tracking. The proposed methodology includes the entire preprocessing pipeline and comprises different techniques like image denoising and contrast enhancement to improve the quality of the images for OCR application. Morphological operations are applied, including the use of erosion and dilation, to refine ladle contours, remove unwanted noise, and improve OCR results. Following the preprocessing of images, OCR is used to effectively extract identification numbers from ladle surfaces as it has better clarity and distinction offered through preprocessing. Such a methodological development aims at the shortcomings of the initial approach by assigning more importance to the preprocessing of the images towards better ladle surface visibility and OCR extraction. In this direction, the proposed method integrates the preprocessing techniques with OCR in a seamless manner to provide a better solution for ladle identification in real time on the SMS units and thus improves productivity and operational efficiency at such facilities. Method II involves the following preprocessing steps: i. grayscale conversion; ii. histogram equalization; iii. blurring and sharpening; iv. thresholding; v. finding and drawing contours; and vi. EasyOCR number extraction.

(1) Grayscale Conversion: The ladle images are converted to grayscale for ease of image processing by eliminating the color information but keeping some relevant details about the shape of the ladles and their identification numbers. Adjusting images to shades of gray is computationally less complex and helps to focus on the important aspects of the following operations by reducing the level of variation in the images only to intensity differences.

(2) Histogram Equalization: Histogram equalization redistributes pixel intensities, striving for contrast enhancement and a more uniformly distributed histogram. It significantly enhances the ladle surface’s visibility by adjusting the intensity range, thus focusing the attention to identification numbers and hence improving OCRs.

(3) Blurring and Sharpening: Median blurring is used to reduce the noise and create a smoother ladle image, especially where light and texture are uneven. This process improves the quality of the image and prepares the image for further stages of processing. On the other hand, edge enhancement is one of the sharpening techniques where the intensity or contrast of edges is enhanced, and even fine details are brought out clearly. This sharpens the edges of the ladle and the identification numbers.

(4) Thresholding: Thresholding techniques are used to segment the ladle numbers into foreground from the background regions as per their intensity levels. Adaptive thresholding calculates dynamic threshold values for various regions of the image such that the ladle numbers are well-separated from the background. A simple binary thresholding further simplifies the image to format it as a binary image in which the ladle features will appear as foreground objects against a uniform background, facilitating contour detection and OCR extraction.

(5) Finding and Drawing Contours: Contour detection algorithms work to identify and outline ladle boundaries as well as identification numbers in the preprocessed image. Tracing the contours of objects enables the precise localization of the ladle unique number, laying a foundation for subsequent OCR extraction. Contours help to provide spatial information necessary for ladle identification as well as tracking.

(6) EasyOCR Number Extraction: Optical character recognition (OCR) algorithms can be described as advanced character recognition systems that read a given text and convert it into a digital form that helps to identify characters with very high precision [13]. OCR algorithms such as EasyOCR are employed for the purpose of extracting identification numbers from the processed images of ladles. Preprocessing techniques such as grayscale conversion, histogram equalization, blurring, thresholding, and contour detection were applied to enhance the clarity of the input ladle image with number “20” before OCR extraction, as illustrated in Figure 6.

3. Results and Discussion

The evaluation of the proposed computer-vision-based ladle identification model was carried out on 169 images consisting of 540 instances contained in the test set while keeping track of the performance over a period of 10 epochs. Training employed the YOLOv8 object detection architecture, and the results showed that the model performance improved progressively as the box loss, class loss, and detection focal loss (DFL) of the network minimized per epoch. To assess the performance of the model, several performance metrics were used: precision (P), recall (R), mean average precision at 50% Intersection over Union (IoU), and mAP over an IoU range of 50% to 95%. The performance of the model attains its peak at the 10th epoch, with a precision of 0.983, recall of 0.998, mAP@50 of 0.994, and mAP@50-95 of 0.826, thus indicating that the model was able to detect ladles and their identification numbers successfully with minimal errors. While testing and evaluating the performance of ladle and number detection on the test dataset, ladle identification performed nearly perfectly, both in terms of precision and recall, whereas number identification exhibited performance with a lower mAP50-95 because some numbers in some images were not necessarily visible and clear. Specifically, ladle detection scored 0.992 in precision, 1.000 in recall, an mAP@50 of 0.995, and an mAP@50-95 of 0.937, while number detection scored precision of 0.974, recall of 0.996, mAP@50 of 0.993, and mAP@50-95 of 0.713. The results of the trained model are provided in

Table 1. Results of YOLOv8 model.

Class	Precision	Recall	mAP50	mAP50-95
Ladle	0.992	1.000	0.995	0.937
Number	0.974	0.996	0.993	0.713

. These results emphasize that the model is sufficient and capable of real-time ladle tracking in challenging environments such as SMS units in steel plants. Once these two classes, namely ladle and number, are identified, OCR can then be applied on the bounding box found with the number class for retrieval of the identification number. These numbers are then tracked in each process in the steel plant to monitor the ladles. The integration of object detection and OCR proved successful in improving detection accuracy, while the slight decline in mAP50-95 for number detection suggests that variations in image quality may still pose challenges. Nonetheless, the high precision and recall values reflect the model’s ability to learn and distinguish between ladles and numbers with increasing accuracy across epochs. Future work may focus on enhancing number detection through improved preprocessing techniques or the use of more advanced OCR algorithms to further refine recognition accuracy under less-than-ideal conditions.

The second proposed method, which includes preprocessing as well as the application of optical character recognition (OCR), further improved ladle identification as it enhanced resolution and readability for the numbers appearing on the surfaces of the ladles. EasyOCR was applied to extract the identification numbers after it passed through all the preprocessing steps: grayscale conversion, histogram equalization, blurring, sharpening, and contour detection. The results show that EasyOCR correctly identifies the identification numbers with high confidence scores. For instance, the input ladle image that contains the number 20 was read correctly by the EasyOCR algorithm with a confidence score of 0.9948. This methodology with the preprocessing enhanced effectively shows the feasibility of applying advanced image enhancement techniques before optical character recognition, especially in cases where the quality of the numbers may be compromised due to harsh industrial environments. The efficiency of preprocessing in improving image quality has significantly contributed to the accuracy of number recognition with OCR. This advancement, therefore, provides a reliable means of real-time monitoring of ladles during the steel production processes. The suggested system is scalable for real-time operation in steel plants. The YOLOv8 model is also optimized for speedy inference, which will be able to handle large quantities of ladle images within the industrial setup. Cloud computing (e.g., Google Colab, AWS, or Azure ML) or edge AI devices (e.g., Jetson Xavier NX, or AGX Orin) ensures that the model can be deployed to various production lines with little delay. This work can continue by studying federated learning strategies in the future, which will enable the system to learn in multiple environments without having data stored centrally.

4. Conclusions

This paper proposes an extremely efficient solution for ladle tracking and identification inside steel plants. The two methodologies proposed include a computer-vision-based approach and an improved preprocessing- and OCR-based approach. In the first method, this work utilizes the YOLOv8 object detection framework, which demonstrated great precision and recall for ladle and number identification in cross-validation across several evaluation metrics. With precision reaching 0.983 and recall at 0.998, the model proved its real-time ladle tracking capability, even in the challenging environments of steel melting shops (SMSs). The second method, integrating preprocessing techniques with EasyOCR, further improved the number extraction accuracy by addressing challenges related to image quality and number visibility. The preprocessing steps significantly improved the quality of the ladle images, completing the extraction with a confidence score of 0.9948, thereby validating the effectiveness of this approach. The results indicate that both techniques are highly suited for the real-time monitoring and tracking of ladles, which is used for improving the operational efficiency, safety, and productivity at steel plants and also offers tremendous advantages in terms of reductions in labor and manual errors, and it improves process automation. Some future research directions may include increasing the OCR accuracy under challenging environments through developing customized deep-learning-based OCR models. The dataset can also be enlarged to have many images of ladles under diverse lighting conditions for better generalization of the models. The integration of this system with crane scheduling algorithms can further optimize the ladle movements within the plant in the direction of maximizing resource utilization. Beyond ladle tracking, this system can be used for other industrial processes requiring real-time object detection and tracking.