Multi-View Industrial Image Super-Resolution via Hierarchical Multi-Scale Data Fusion

Abstract

Machine vision plays a pivotal role in precision engineering for high-precision measurement that relies on high-resolution images. The highly reflective nature of metal surfaces and the need for high-quality images pose significant challenges in image processing. Although existing research has made significant progress in enhancing the resolution of natural images, super-resolution methods specifically tailored for multi-view metal images remain unexplored areas. To fill this gap, this paper focuses on developing a deep learning-based super-resolution algorithm, focusing on detail recovery on under multi-view metal images. The proposed super-resolution model utilizes a hybrid-resolution input that combines light field super-resolution at the image level and reference-based super-resolution at the feature level, demonstrating the effectiveness for achieving a large-scale multi-view metal image super-resolution. An experiment using a public metal object image dataset is conducted, and a comparison has been carried out with Bicubic, LFhybridSR and ERVSR. The proposed method demonstrates superior SSIM and achieves average PSNR improvements of 4.45 dB and 1.18 dB on synthetic data and real-world data. The results demonstrate that the method can improve the resolution and detail representation of metal images in terms of PSNR/SSIM and address the problem of super-resolution in multi-view metal images. Furthermore, applying the proposed SR method as preprocessing reduces the absolute relative error in depth estimation from approximately 0.5 to 0.1.

1. Introduction

The need to increase product precision in contemporary industrial systems has grown with scientific and technological advancements in recent years, giving rise to precision engineering. Since precision engineering promises to produce surfaces with submicron shape accuracy and nanometer surface roughness, surface measurement is critical to improving accuracy and efficiency in precision engineering. When direct measurement is conducted, high-quality photographs of the workpieces are required due to the high standard of the ultra-precision product. However, unaffordable capture costs of high-quality images result in limited image resolution in precision engineering. Therefore, investigation of image processing techniques is urgently desired to balance measurement precision and expense.

Super-resolution (SR) is an image processing technique that aims to produce high-resolution (HR) images from low-resolution (LR) images with the same content [1]. Since it can solve the problems of device limitations and high storage costs for high-quality pictures, SR plays a crucial role in computer vision and emerges as a hot research issue in various industries [2,3]. Based on the widespread applications and powerful development potential of SR technology, SR techniques in theoretical studies and applications have been a hotspot for two or three decades.

The current mainstream theoretical image SR research focuses on single-image SR (SISR). However, SISR is limited to recovering details from only one LR image and usually produces over-smoothed SR results. Multi-view image SR (MVISR) can fuse valuable contents of multiple input images, which helps generate one image containing more clarity and detail [1]. However, MVISR, which hopes to super-resolve the whole set of LR multi-view images, is rarely studied in practical applications. Especially in machining, there are a few SISR studies that have been published, but none for MVISR. In this situation, research on metal MVISR is very crucial and necessary.

In many real-world cases, SR models can generate different HR images from the same LR images since there are no corresponding HR images. The cost of capturing and storing HR multi-view images tends to be higher, making LR and HR pairs hard to obtain in practice. Current research on MVISR focuses on geometric information from other LR images to generate HR images [3]. However, this method cannot reconstruct higher-frequency textures due to the absence of HR images, especially at a large-scale SR. Considering the balance of shooting and SR cost, there is a simple solution to enhance the resolution of multi-view images motivated by light field cameras. A light field image can be regarded as a multi-view image obtained by an array of light field cameras. There is a hybrid imaging system that can capture multiple LR side views with one HR center view in the light field [4]. By multi-resolution image fusion at a large-scale SR, the HR center-view image can transfer high-frequency details to other LR images, resulting in better large-scale SR results. Therefore, it is a vital issue deserving of long-term investigation.

Many researchers have been developing numerous methodologies and algorithms for reconstructing HR images [5]. Deep learning methods can effectively restore high-frequency information in complex images by learning the mapping relationships of LR and HR images. This also enables end-to-end SR, avoiding situations where errors in individual steps affect the overall model results [6]. In addition, some traditional methods try to use patch matches to transfer details from the HR center-view image but suffer from mismatching and limit their performances [7,8,9]. The deep learning method excels very well at calculating the similarity between the reference image and the LR image, which provides a better solution to this challenge [10]. Therefore, deep learning has become the dominant approach in SR, including the hybrid input MVISR.

Existing ideas for hybrid-input MVISR based on deep learning cover three solutions. One is to perform SISR on each LR image. However, it is inevitably limited by the information contained in a single image. In the same scene, an object blurred in one viewpoint image might show more detail in another image. This approach cannot utilize the information in LR images with another view. Other alternatives cover video SR (VSR), which considers each image in the multi-view image as each frame in a video [11]. However, there is still a difference between MVISR and VSR. VSR considers temporal and spatial resolution, while MVISR considers spatial and angular resolution. The third is to utilize complementary information between LR multi-view images by computing and warping disparity information [4,6,12,13]. Although the method can extract complementary information in the input image, it ignores the high-frequency information of HR images. In other words, those methods cannot use the low-frequency or high-frequency information of the hybrid input. In addition, high-frequency textures of workpiece images significantly impact the measurement precision in precision engineering. Therefore, it is critical to remedy the defect of the solution to generate multi-view metal images with realistic and high-resolution textures.

To address this issue, this study adopts reference-based SR with deep learning, which can capture high-frequency information from the HR center-view image and produce multi-view HR images. It adopts an image with a similar high-quality texture to the LR image as a reference and transfers its high-frequency texture to the LR image to enhance the SR results. The mainstream research on reference-based SR focuses on how to transfer high-frequency textures accurately [14,15]. Therefore, a novel multi-view industrial image SR method based on multi-resolution data fusion is proposed to enhance the resolution of multi-view metal images to assist subsequent precision measurement or detection tasks. The significance of this research is presented below.

(1)

The proposed hybrid-resolution multi-view data fusion model architecture helps significantly enhance performance in large-scale SR.

(2)

This study uses reference-based SR to address the neglect of transferring high-frequency workpiece texture in HR images in previous studies, enabling realistic texture generation.

(3)

An SR model estimates complementary information between LR multi-view workpiece images to extract low-frequency contents to capture the overall geometry and silhouette of the object.

To evaluate the quality of the reconstructed industrial images, this paper employed three complementary evaluation metrics. The peak signal-to-noise ratio (PSNR) measures errors at the pixel level, the structural similarity index measure (SSIM) evaluates structure and texture retention of generated images, and the learned perceptual image patch similarity (LPIPS) assesses the quality of visual perception. For experimental validation, the recently released Sim-to-Real industrial metal object dataset [16] was utilized. The dataset uniquely provides metal images from multiple viewpoints, making it highly suitable for the problem setting described in this paper. The experimental results demonstrate that the method proposed in this paper outperforms existing approaches. In synthetic scenarios, the model achieves an average PSNR exceeding 40 dB, representing improvements of approximately 4.45 dB and 8.39 dB over ERVSR and LFhybridSR, respectively. On challenging real-world reflective surfaces, text generated using this method exhibits sharper edges and finer details. In addition, a preliminary exploration was conducted to assess the performance gains of the proposed SR method for depth estimation. The results show a reduction in absolute relative errors (Abs Rel) from approximately 0.5 to 0.1. These findings highlight the practical value of this method for industrial applications.

The rest of this article is structured as follows: Section 2 provides a detailed analysis of current deep learning-based SR methods. The framework of the proposed method and the implementation details are presented in Section 3. Section 4 shows the experimental validation and comparative analysis of the results. Section 5 is a summary.

2. Literature Review

2.1. Preliminaries and Challenges

Multi-view imaging captures scenes from multiple viewpoints and can reconstruct spatial and angular information. In SR literature, the term “resolution” typically refers to spatial resolution, which denotes the clarity and level of detail within an image [1]. In multi-view and light field imaging, spatial resolution is further distinguished from angular resolution. Spatial resolution refers to the pixel density within a single view, while angular resolution corresponds to the number and distribution of different viewpoints [17]. This paper focuses solely on enhancing the spatial resolution of fixed 3 × 3 multi-view arrays, as this directly determines the fidelity of surface details required for precise measurements.

A strategy for balancing resolution quality and hardware costs is to use a hybrid-resolution multi-view system, in which the central reference view is captured at high spatial resolution, while the surrounding views are captured at lower resolution. The hybrid-resolution system uses neighboring HR views to enhance the visual quality of LR images [18]. This approach lies at the intersection of MVISR and reference-based SR. Images captured from different viewpoints can provide richer detail than SISR, and HR reference images supplement the LR input with additional high-frequency information [19]. Nevertheless, applying such hybrid-resolution strategies to industrial metal surfaces still presents practical challenges, such as highly reflective surfaces, low texture content, and large-angle parallax, which will be discussed in Section 3.1.

2.2. Multiple-View Image Super-Resolution

Since Dong et al. [20] proposed an SISR model using convolutional neural networks (CNNs) in 2014, deep learning has become the mainstream method for the image SR reconstruction problem. For MVISR, Cheng et al. [3] proposed MVSRnet, which considers other LR view images as reference images and utilizes geometry information to extract high-frequency textures from other LR view images to enhance the image quality. The LR multi-view images and corresponding depth maps and camera pose information are fed into the geometry-aware reference synthesis module to be aligned and warped to obtain reference images. The high-frequency textures of the generated reference images are transferred to LR input multi-view images through a dynamic high-frequency search network to obtain HR output images. However, it is challenging to perform prior depth estimation for multi-view metal images with large-angle transformations. The high-frequency details of the LR view image are still not clear enough in the case of large-scale SR. In addition, there is much research on multi-view satellite image SR using deep learning-based data fusion including TR-MISR [21] based on a Transformer. However, these methods are fusing multi-view LR images to produce a single HR satellite image. There are also some works on multi-view texture SR without improving the resolution of multi-view LR images [22]. Thus, they are not mentioned here. In addition to those, research on MVISR in recent years has focused on stereo image SR and light field SR.

Stereo image SR is a technique to enhance the quality of LR binocular images. Wang et al. [23] proposed a parallax-attention stereo super-resolution network (PASSRnet) fusing stereo image pair information to improve image resolution. The PASSRnet includes a residual atrous spatial pyramid pooling (ASPP) to extract multi-scale features and a parallax-attention module (PAM) to fuse the features of stereo image pairs. The PAM computes the differences between left- and right-view images and updates the features of the left view by integrating the differences based on the parallax-attention mechanism. Following the PASSRnet, Zhu et al. presented a cross-view capture network (CVCnet) to improve the resolution of left-view images. After stereo image pair feature extraction, a cross-view block is created to explore global and local feature correlations, and spatial perception modules are constructed to enhance the fused features of cross-view images. However, the PASSRnet and CVCnet just used the right-view images for references and only super-resolved the left-view images. Chu et al. [24] proposed an NAFNet-based [25] stereo super-resolution network (NAFSSR), increasing the resolution of both left- and right-view images. The NAFSSR extracted the intra-view features of stereo image pairs and then used stereo cross-attention modules to fuse the cross-view features. But the stereo image has fewer viewpoints compared to other multi-view images.

A light field image captures information about the light in a scene. It not only records the color and brightness in the scene but also contains information about the light direction. It usually consists of images from multiple viewpoints. Light field image SR refers to algorithmic techniques to enhance the spatial resolution of light field images in order to obtain clearer and more detailed image details. Yoon et al. [26] used a deep CNN to enhance the spatial and angular resolution of light field images simultaneously. Stacked images in pairs according to horizontal, vertical and surrounding orientations are fed into different sub-networks. Zhang et al. [27] divided the images from different viewpoints into four groups according to angular directions and input them into different network branches for spatial-resolution enhancement, respectively. Mo et al. [28] introduced a view–channel dual-attention module based on dense skip connections to make full use of discriminative features from views and channels for SR performance improvement. As the diffusion model is being intensely investigated, the SR method based on the diffusion model has also been proposed [29]. These studies involved only LR light field images, with no mention of large-scale light field image SR.

A hybrid camera setup that can capture both LR light field images and HR center-view images has emerged in order to effectively improve the angular resolution of light field images on a large scale while controlling costs. Zhao et al. [4] super-resolved light field images by calculating the disparity between the HR center-view and LR side-view images to transfer high-frequency details to other upsampled view images. Jin et al. [6,13] proposed a method including an SR network branch to fuse high-frequency information and a warp network branch to warp the HR center view image to other side views. Chang et al. [12] used a layered network to extract the coarse alpha maps for position information presentation and then adopted HR center-view images to refine the coarse alpha maps. By rendering and fusing the color information and position information, HR light field images were generated. The hybrid system can effectively achieve good performance on large-scale SR, which is similar to reference-based SR.

2.3. Reference-Based Super-Resolution

Reference-based SR is a technique to enhance the quality of LR images by utilizing one or more HR reference images. As shown in Figure 1, Reference-based SR matches the similar contents of the LR image and the reference image, extracts the high-frequency information of the reference image and fuses it into the LR image, thus generating a clearer and more detailed high-resolution image. Depending on the matching method, it is usually categorized into alignment-based methods and patch matching.

The alignment-based method ensures that the LR image and the HR reference image are in the same coordinate system by spatially aligning them. It typically relies on image alignment techniques to eliminate differences due to viewpoint or motion. Shim et al. [30] introduced deformable convolutions to extract aligned reference features. Zheng et al. [31] performed multi-scale spatial alignment using warping by optical flow and the proposed Crossnet can perform a large-scale SR. Kim et al. [11] proposed an attention-based video SR method, Efficient Reference-based Video Super-Resolution (ERVSR), in which optical flow is used to align the HR center frame with other frames.

Patch matching searches for regions in an HR reference image that are similar to small patches in an LR image. It utilizes similarity computing to select the most appropriate HR block to enhance the image quality. Zhang et al. [32] matched the local patches in the LR and reference images. Yang et al. [10] proposed a novel method, Texture Transformer Network for Image Super-Resolution (TTSR), by introducing a Transformer to reference-based SR. To reduce the computational cost of block matching in TTSR, Lu et al. [14] proposed the matching acceleration and spatial adaptation method using coarse-to-fine matching. Although these methods are usually for the case where the scale factor is ×4, they demonstrate the importance of attention mechanisms in searching similar contents and transferring high-frequency details.

To provide a clearer overview of the related work,

Table 1. Comparative summary of representative multi-view and reference-based SR methods.

Method	Category	Multiple LR Input	Use HR Reference	Strengths	Limitations
Dong et al. [20]	SISR	✕	✕	Simple and fast	Over-smooths low-texture surfaces
MVSRnet [3]	MVISR	✕	✕	Geometry-aware warping; fuses complementary LR views	Requires depth priors; fails under large parallax and reflections
PASSRnet [23]	Stereo SR	✓ (stereo pair)	✕	Parallax attention for cross-view fusion	Limited to two views; no HR texture guidance
LFhybridSR [6]	Light field SR	✓	✓	Hybrid input; attention-guided fusion	Not large parallax
ERVSR [11]	Reference-based video SR	✓ (temporal)	✓	Optical flow alignment; efficient	Temporal assumption mismatched with angular views
TTSR [10]	Reference-based SR	✕	✓	Transformer-based texture transfer	Single view only; Not large-scale

Note: ✓ indicates yes and ✕ indicates no.

summarizes the key characteristics of the representative SR methods discussed above and compares their strengths and weaknesses in multi-view imaging scenarios.

As summarized in Table 1, existing methods involve significant trade-offs. Multi-view methods either lack HR reference images or perform poorly under conditions of large disparity and specular reflections. Reference-based SR methods, on the other hand, are primarily designed for single-image enhancement and lack constraints on angular consistency. In contrast, the method proposed in this paper addresses these shortcomings in the following ways. Firstly, it operates directly on the original multi-view images without the need for pre-computed geometric information. Secondly, it decouples feature-level texture transfer from image-level geometric aggregation. Finally, it employs a hierarchical multi-scale fusion strategy specifically designed for hybrid-resolution industrial inputs. To the best of our knowledge, this is the first large-scale MVISR work targeting industrial metal surfaces.

3. Methodology

3.1. Overview of Model Structure

The untextured and reflective properties of the metal surface make it difficult to extract high-frequency texture features when increasing the image resolution at large scales. At the same time, the large angular parallax between multi-view images makes it difficult to achieve feature matching between images at different viewpoints. In addition, based on the high cost of HR image shooting, multiple LR images at different viewpoints and one HR image with a specific viewpoint are considered as a group. A multi-view image SR method based on multi-resolution fusion is proposed to transfer the high-frequency texture of HR images to LR images from the feature level and the image level to improve the quality of LR images. The model structure is shown in Figure 2.

In this paper, referring to the model structure in [33], while the original structure performs acceptably when performing SR with a scale factor of 4, the model performance is drastically reduced when the scale factor is 8. The original model structure was a multi-layer nesting of SR at image-level (SR2I) modules, and the image obtained after zooming would be directly applied to the next module. In this study, the feature-level search and transfer (STF) module is proposed to be isolated from the SR2I module to obtain features with sizes ×1, ×2, ×4, and ×8. The details are shown in the following section.

Suppose that there are n LR images with different viewpoints $I_{lr}=\{I_{lr}^1, I_{lr}^2, \dots , I_{lr}^n\}$ where the intermediate viewpoint image is denoted as $I_{lr}^c$, and the corresponding HR image $I_{hr}^c$ is obtained. The size of $I_{hr}^c$ is eight times that of $I_{lr}^c$. The HR middle-view image $I_{hr}^c$ is downsampled to obtain a reference image $I_{hr\_2}^c$ with four times the size of the LR image and $I_{hr\_4}^c$ with twice the size of the LR image. The input images $I_{lr}$ along with the HR intermediate-view image $I_{hr}^c$ are fed into the STF module for feature extraction and performing similar detail matching and high-frequency texture transfer at the feature level in order to obtain images $I_{lr2}^{\prime }$, $I_{lr4}^{\prime }$, and $I_{lr8}^{\prime }$ with sizes ×2, ×4, and ×8. Subsequently, the downsampled $I_{hr\_4}^c$ along with the original image $I_{lr}$ is fused and enlarged at the image level in the SR2I module, thus obtaining an image $I_{lr2}^{*}$ with twice the size of $I_{lr}$. Similarly, until super-resolved images $I_{lr8}^{*}$ are obtained with an eight-fold enlargement.

3.2. Search and Transfer at Feature Levels

In the proposed multi-resolution fusion model, high-frequency details are extracted, matched and transferred from the feature level and image level respectively. The STF module presented in Figure 3 implements feature-level similar-content matching and high-frequency texture transfer based on the attention mechanism.

First, unlike feature extraction with shared parameters in TTSR, the STF module utilizes two encoders that do not share parameters for feature extraction. The encoders are trained together with the other structures of the network during training. Both encoders consist of four residual blocks, with the last three blocks having a stride of 2. The size of the resulting feature map is halved to extract image features of different sizes, respectively. Therefore, four features $F_{hr\_c}^1, F_{hr\_c}^2, F_{hr\_c}^4, F_{hr\_c}^8$ with different sizes ×1, ×2, ×4, and ×8 are extracted from the center-view image by the encoder. The multi-view LR image does not need to be halved in size, so only the first block in the encoder is utilized for feature extraction to get features $F_{lr}=\{F_{lr}^1, F_{lr}^2, \dots , F_{lr}^n\}$ with size ×1.

Afterwards, the obtained LR image features $F_{lr}$ and four HR center-view image features $F_{hr\_c}^1, F_{hr\_c}^2, F_{hr\_c}^4, F_{hr\_c}^8$ are fed into the search and transfer module to perform similar content matching and high-frequency texture transfer at the feature level, the details of which are shown in Figure 4. This module utilizes the attention mechanism to perform feature matching and transfer. Consider the LR side-view features $F_{lr}^i$ where $i=1,2,\dots ,n-1$ and $i\ne (n+1)/2$ as the Query and the center-view feature $F_{lr}^c$ where $c=(n+1)/2$ as the Key in the attention mechanism. Similar-content matching is performed by estimating the similarity of the center-view image feature $F_{lr}^c$ with each side-view image feature $F_{lr}^i$. Each feature and the center feature are unfolded as patches for similarity computation and n similarity maps $S_{lr}=\{S_{lr}^1, S_{lr}^2, \dots , S_{lr}^{n-1}\}$ can be obtained.

More specifically, for the ith side view feature $F_{lr}^i$, matching it with the center view feature $F_{lr}^c$ for similar content is done by unfolding them into m patches $P_{lr}=\{P_{lr}^1, P_{lr}^2, \dots , P_{lr}^m\}$ and $P_{lr\_c}=\{P_{lr\_c}^1, P_{lr\_c}^2, \dots , P_{lr\_c}^m\}$. The similarity between the two patches of the side view and the center view is calculated as

$$r_{k,j}=⟨{\displaystyle \frac{P_{lr}^k}{\Vert P_{lr}^k\Vert }},{\displaystyle \frac{P_{lr\_c}^j}{\Vert P_{lr\_c}^j\Vert }}⟩$$

(1)

where $P_{lr}^k$ denotes the kth patch in $P_{lr}$, $P_{lr\_c}^j$ denotes the jth patch in $P_{lr\_c}$, and $r_{k,j}$ represents the similarity scores between $P_{lr}^k$ and $P_{lr\_c}^j$. A set of similarity maps $S_{lr}$ and their corresponding indexes $N_{lr}$ are obtained by

$$S_{lr}^k=\underset{j}{max}{r}_{k,j}$$

(2)

$$N_{lr}^k=\underset{j}{argmax}{r}_{k,j}$$

(3)

where $S_{lr}^k$ belongs to $S_{lr}$ and $N_{lr}^k$ belongs to $N_{lr}$. $N_{lr}$ records the location of the corresponding most relevant feature block and $S_{lr}$ records the specific relevance of this most relevant feature block.

The HR center view feature $F_{hr\_c}^s$ where $s=1, 2, 4, 8$ are then the values in the attention mechanism. With the positions of the relevant feature blocks recorded by $N_{lr}$, the feature blocks at the corresponding positions can be migrated from $F_{hr\_c}^s$ to obtain the transferred texture features. This texture feature map contains high-frequency texture features at the most similar positions in the HR intermediate-view image. Finally, the obtained texture feature maps with different sizes are fed into the decoder with the similarity map and corresponding index map to perform feature fusion for texture enhancements. In the decoder, the similarity map will be dot-multiplied with the textured features at the corresponding locations. Texture information with strong correlation can be given relatively larger weights and texture information with weak correlation can be suppressed by small weights. Thus, high-frequency texture transfer is achieved at the feature level.

This feature-level matching strategy is designed to address the challenges posed by industrial metal surfaces. First, the STF module performs correspondence matching in the learned deep feature space rather than on raw pixel intensities, thereby decoupling the matching process from direct photometric appearance. Second, the similarity map calculated via Equation (2) serves as a soft gating mechanism. In regions where reflections or a lack of texture prevent reliable matching, the similarity scores naturally remain low, thereby suppressing the transfer of erroneous high-frequency textures. Third, matching is performed at the coarsest feature scale, and the resulting indices are reused for the transfer of texture features across all scales. Since each spatial location on the coarser-resolution feature map aggregates information from a larger image region, this design reduces sensitivity to spatial misalignments caused by wide-angle parallax. In this way, high-frequency details are selectively transferred only from positions identified by confident matches and are distributed across multiple scales.

3.3. Super-Resolution at Image Levels

In this section, image-level SR is mainly performed on the LR images. Firstly, the multi-view LR image is directly obtained by upsampling it to twice the original size. Then, the Spatial–Angular Separable (SAS) Convolution blocks in [17] are used for feature extraction and residual map prediction. The feature extraction method is used for spatial–angular feature extraction of light field images [34]. In this paper, feature extraction is performed for multi-view images.

Firstly, the multi-view LR image input is converted into a four-dimensional tensor $L{R}_{x,y,u,v}$ containing spatial (x, y) and angular (u, v) information. Then, a spatial convolution operation is performed on the input to extract spatial features. Spatial convolution is usually performed on the image at each viewpoint and can capture the local features of the image. Next, an angular convolution operation is performed to extract angular features. Angular convolution is performed between different viewpoints to capture the information brought about by changes in viewpoints. After spatial and angular convolution, the feature fusion layer fuses the features of both, aiming to synthesize spatial and angular information. In this way multiple SAS convolutions are stacked to form SAS blocks.

After feature extraction, the residual map is predicted by further convolution to predict the difference between the upsampled image and the HR image. And the residual map is added to the upsampled image to obtain the SR image that integrates complementary information from multiple viewpoints.

Spatial convolution focuses on the spatial structure of the image, while angular convolution focuses on the relationship between different viewpoints. With this separation of convolution, the network can better understand the depth and perspective changes of an image. By combining spatial and angular features, spatial–angular convolution can capture the detailed information of multi-view images more comprehensively. In addition, spatial angle convolution significantly reduces computational complexity by decomposing the convolution operation into spatial and angular components.

The output tensors generated by the STF and SR2I branches have the same spatial dimensions. These two outputs are concatenated along the channel dimension and then passed through a convolutional residual block. This module learns to adaptively fuse high-frequency texture information from the STF with multi-view geometric consistency from the SR2I. This progressive fusion occurs at every upsampling scale.

3.4. Loss Function

Since the network is cascaded to generate SR images of multiple sizes, the loss function is the sum of the loss functions with different sizes, where the sizes are ×2, ×4, and ×8. The loss function used in this study is L1 Charbonnier loss [35]. It is a smooth L1 loss function which is commonly used in image processing. For the generated SR image, the loss function is calculated as

$$\begin{array}{c}L=L_{\times 2}^c+L_{\times 4}^c+L_{\times 8}^c\\ L_{\times s}^c=\sqrt{(I_{sr}-I_{hr}{)}^2+{\epsilon }^2}\end{array}$$

(4)

where $L_{\times 2}^c, L_{\times 4}^c, L_{\times 8}^c$ denote the L1 Charbonnier loss at different sizes, $L_{\times s}^c$ stands for the L1 Charbonnier loss when size is ×s, I_sr, I_hr are the SR output images and corresponding HR images, and ϵ is a small constant used to avoid numerical instability when calculating square roots.

Charbonnier loss exhibits smoother behavior in regions with smaller errors, reducing the risk of vanishing or exploding gradients compared to L1 loss. In addition, Charbonnier loss has better robustness to outliers due to the introduction of the ϵ term. The L1 Charbonnier loss performs better in the face of noise or outliers compared to the L2 loss.

4. Experiment

4.1. Dataset and Data Processing

Since there is a lack of industrial SR image datasets, a newly proposed dataset of industrial metal objects [16] is used to perform experiments. The dataset was constructed for 6D pose estimation but includes multi-view images of different metal objects and scenes. It contains real-world images and synthetic images under different lighting conditions. There were six types of industrial objects photographed, which can be categorized into highly reflective and less reflective objects. The former includes a metal cylinder, a small cube, a large cube, and a cylinder, while the latter includes a red cube with text and a red cylinder. Items are placed on three different carriers, including pallets, boxes, and cardboard. These items are placed in various stacked configurations on three different types of carriers, including pallets, boxes, and cardboard.

Figure 5a shows the real-world image collection setup for this dataset, where multi-view images of different scenes are captured by different cameras mounted on the industrial robot. The multi-view images include 13 camera capture viewpoints, which are on a circle centered on the location where the object is placed, as shown in Figure 5b. Since the 4 images at the inner circle are not the same distance from the center image as the other 8 images, only the center image and the 8 images on the outer circle are taken as the multi-view images for experimental verification in order to facilitate data processing. Multi-view images are captured by different cameras, and RGB data taken by the Intel RealSense L515 camera (Intel Corporation, Santa Clara, CA, USA) were selected for experiment validation.

Highly realistic synthetic data is generated in Unity based on real-world data. HDRI environment maps simulate authentic lighting conditions, while texture synthesis algorithms create object surfaces with details like scratches. Synthetic data is generated based on real-world data captured by the JAI GO-5000-PGE camera (JAI A/S, Copenhagen, Denmark), enriching data diversity by altering lighting and pose. Multi-view images featuring different objects placed on a pallet background were randomly selected for the experiment. Like for real-world data, the 8 images on the outer circle and the central image were chosen.

The information on the processed images is listed in

Table 2. Information on the processed metal images.

Images	Information	Value
Real-world image	Spatial resolution (HR)	1920 × 1080
	Spatial resolution (LR)	240 × 135
Synthetic image	Spatial resolution (HR)	2560 × 2048
	Spatial resolution (LR)	320 × 256
	Angular resolution	3 × 3

below. The spatial resolution of the HR center image and the LR multi-view images for real-world data and synthetic data are presented. Since only 9 images are chosen for experiment validation, the angular resolution of the multi-view images is 3 × 3. Data augmentation operations, including cropping, rotating, and flipping, are performed for training.

4.2. Experiment Results

4.2.1. Ablation Study

Ablation experiments were conducted on some modules in the proposed SR model, including feature extraction, model structure, fusion strategy, and loss function. The SR results of these variants using real-world data are presented in

Table 3. Ablation study results.

Variant	Description	PSNR	SSIM	LPIPS
A	a weight-shared encoder for feature extraction	35.30	0.847	0.226
B	remove the SAS block	35.16	0.842	0.241
C	remove the fusion module	35.17	0.840	0.244
D	L1 loss function	35.33	0.847	0.227
Ours	proposed full model	35.37	0.848	0.221

. Variant A removed the two separate feature extraction encoders in the STF module, replaced by a weight-shared encoder. Variant B removed the SAS block in SR2I, replaced by a residual block for feature extraction. Variant C removed the fusion module and used the SR2I module for image fusion. The upsampled images in the SR2I module are replaced by the output generated by the STF module. Variant D replaced the L1 Charbonnier loss with the L1 loss function. The larger the PSNR and SSIM values are, the better the performance of the model gets. The opposite happens with LPIPS. Our model shows higher performance than Variant A and B, which demonstrates the usefulness of feature extraction. Especially for Variant B, the results show that the SAS feature extraction module can help extract more information from multiple views. The results with Variant A show that the independent encoder consistently improves reconstruction quality. The improvement was more noticeable on the Y channel, where the PSNR increased from 34.73 dB to 34.87 dB. However, independent encoders involve a larger number of parameters than shared designs. There is room for improvement when it comes to scenarios requiring stricter real-time deployment. Variant C shows the fusion model has better results than directly fusing warped features by the SR2I module. The better results than those of Variant D show that the L1 Char loss function can help produce clearer visualization than the L1 loss.

4.2.2. Results with Synthetic Images

In this section, the proposed multi-resolution hybrid model is deployed to enhance the resolution of metal images. The performance of the proposed SR model can be evaluated by comparing the PSNR and SSIM values of this method with the corresponding values obtained using other methods.

Table 4. Comparative quantitative results with synthetic data.

Scenes	Bicubic (PSNR/SSIM)	LFhybridSR (PSNR/SSIM)	ERVSR (PSNR/SSIM)	Ours (PSNR/SSIM)
1	33.06/0.904	33.56/0.882	37.71/0.935	42.25/0.937
2	32.52/0.894	33.05/0.874	36.96/0.926	41.32/0.930
3	31.29/0.875	31.81/0.862	35.36/0.910	39.68/0.924
4	32.93/0.896	33.45/0.874	37.39/0.927	41.84/0.930
5	33.06/0.899	33.53/0.878	37.56/0.930	42.06/0.933
6	32.33/0.885	32.66/0.871	36.71/0.918	41.27/0.929
Mean	32.53/0.892	33.01/0.874	36.95/0.924	41.40/0.930

Note: The best results in every scene are highlighted in bold.

shows the PSNR and SSIM values of the proposed method and the comparison method for the six scenes of the test set. The calculated PSNR and SSIM values are the average of those of all the multi-view images for each scene. To provide a clearer comparison of results, the table displays the PSNR and SSIM values for the image’s Y channel. Other methods employed as comparisons include Bicubic interpolation techniques, a hybrid-input light field SR technique, and a reference-based video SR technique. The comparative methods are presented below.

• LFhybridSR [6]: The method captured the multi-view information of light field images by a hybrid-lens system, which takes an HR image in the center view while taking LR images in other views.
• ERVSR [11]: VSR reconstruction is performed using an HR intermediate frame in the video as a reference image, where optical flow is used to compute the similarity of different frames.

For the objects contained in the six scenes of the test set: 1, 4, and 5 consist of one or more highly reflective metal cubes arranged in various stacked configurations; 2 and 3 contain one and multiple low-reflective red cubes with text; 6 comprises multiple low-reflective red cylinders. Table 3 clearly shows that Scene 3, which contains multiple lettered blocks, yields lower PSNR and SSIM values across all methods compared to other scenes. This demonstrates that more complex scenes pose greater challenges for SR processing. Our proposed model achieved PSNR values exceeding 40 in nearly all scenarios, significantly outperforming other methods. It also yielded the highest SSIM values among all approaches. Particularly in the complex Scene 3, our method exhibits a smaller decline in performance compared to other techniques relative to other scenes. This demonstrates the superior stability of our approach.

The visual results of a side-view image in Scene 3 are presented in Figure 6. For the side-view image, our proposed method generated an RGB image with PSNR, SSIM, and LPIPS values of 35.59, 0.911, and 0.1502. ERVSR achieved values of 35.02, 0.905, and 0.1652, slightly inferior to our model. LFhybridSR performed with a PSNR of 30.50, SSIM of 0.858, and LPIPS of 0.2564, showing a noticeable gap compared to the previous two methods. Bicubic yielded the poorest results, lagging significantly behind the other approaches. This could be seen from the enlarged red and green sections in Figure 6. For the large and clear text in the red block, all methods except Bicubic produced easily recognizable SR images. However, the small text in the dark areas of the green block is mostly unrecognizable. Only the images generated by our model show faint edges of the text, while other methods render them as a blurred mess.

4.2.3. Results with Real Images

For real-world images, the PSNR and SSIM results between generated RGB images and ground-truth images are presented in

Table 5. Comparative quantitative results with real-world data.

Scenes	Bicubic (PSNR/SSIM)	LFhybridSR (PSNR/SSIM)	ERVSR (PSNR/SSIM)	Ours (PSNR/SSIM)
1	33.39/0.755	33.60/0.747	28.56/0.785	34.73/0.814
2	34.36/0.823	34.50/0.808	30.67/0.855	35.88/0.878
3	34.19/0.816	34.21/0.797	30.05/0.847	35.53/0.869
4	35.86/0.841	35.73/0.837	34.59/0.856	36.61/0.870
5	33.86/0.796	33.98/0.781	29.22/0.830	35.24/0.855
6	32.94/0.734	33.14/0.724	27.97/0.769	34.23/0.799
Mean	34.1/0.794	34.19/0.782	30.18/0.823	35.37/0.848

Note: The best results in every scene are highlighted in bold.

. Unlike synthetic data, real-world images feature backgrounds that extend beyond metal plates and incorporate more diverse combinations of objects. The results are listed in Table 5. It is clear that the proposed SR model achieves the highest values for each scene. In contrast, ERVSR achieved the lowest value relatively. This may be due to the fact that the image processing used in processing ERVSR was different from the other methods. The results obtained by LFhybridSR were not very different from those of Bicubic. LFhybridSR had a higher PSNR value, while Bicubic had a higher SSIM value. This may be due to the difference in focus between the PSNR and SSIM metrics. PSNR is a simple pixel difference metric, while SSIM is more concerned with the structural information of the image. In addition, it may also be related to the different multi-view SR images obtained by several methods. Due to the existence of angular difference, it is possible that different methods have different processing performance for different viewpoint images. Therefore, it is necessary to visualize and analyze the different viewpoint images of each scene.

Figure 7 and Figure 8 present the visualization results of center-view images and side-view images in six scenes. Scene 4 has a non-metallic platform in the background. Scene 3 and Scene 5 have metal objects containing textual information.

Figure 7 shows the SR results of the center-view images in three scenes. It is clear that the center-view images generated by LFhybridSR and by the proposed method have the sharpest texture details. This is most evident in the center-view of Scene 3. As can be seen in the green zoomed-in view of Scene 3, the text portion generated by Bicubic is very blurry and the content is not recognizable. The ERVSR can generate text, but only barely recognizes the first line of text information. For the smaller second line of text, ERVSR cannot generate it. In comparison, both lines of text information generated by LFhybridSR and the proposed SR model are clearer. In addition, the strongly illuminated metallic backgrounds produced by both methods also have distinct textures. As for the texture of the metal objects in the other two scenes, it can be noticed from the zoomed-in images that ERVSR produces better images than Bicubic.

Figure 8 presents the SR results of the side-view images in three scenes. The SR reconstruction of the remaining views is more important and more difficult than that of the center view with the HR reference image. This is most evident in the text portion of Scene 5, where the text generation results are more recognizable compared to the weak metallic texture. In the green amplified block of the generated image for Scene 5, there is no method to generate textual information with a clear and easily recognizable text. This is due to the fact that the text in the original image of Scene 5 is smaller and also because the rest of the view has large changes in angle. The images generated by Bicubic are still fuzzy and do not have a clear texture and detail. And the image generated by LFhybridSR is jagged in terms of the edge details of the red items. This is also seen in the other scene images, where the LFhybridSR-generated stick and block objects with metallic luster have jagged edges. On the contrary, ERVSR achieved better performance in the side view. And the proposed SR model has relatively good performance on side-view images. On balance, the proposed SR method achieves relatively better results for both center-view and other-view images. This is reflected both in the quantitative evaluation results and in the visualization results.

It can be noticed that ERVSR achieves better results than Bicubic in both the center and side views. However, according to the results in Table 5, Bicubic performs much better than ERVSR. From this result, it can be inferred that the results obtained are not related to the angular difference of the multi-view images but to the properties of PSNR and SSIM. The values of PSNR and SSIM are not a complete measure of the goodness of the images generated.

4.2.4. Depth Estimation Results with Real Images

To demonstrate the effectiveness of SR methods in improving performance on subsequent tasks, the MiDaS model [36] was used for depth estimation. Specifically, the MiDaS small type is adopted in this section since the inference is fast and it can also run on a CPU. The depth estimation results of real images are presented in

Table 6. Quantitative depth estimation results of real images by MiDaS models.

Input	Abs Rel	δ1	δ2	δ3
SR	0.1024	0.8989	0.9383	0.9521
LR	0.5349	0.4007	0.5951	0.6569

. The Abs Rel value of SR images indicates an average relative error of approximately 10% between predicted depth and actual depth, which is significantly lower than that of LR images. Furthermore, based on the results for δ₁, δ₂, and δ₃, approximately 90%, 94%, and 95% of the predicted pixels satisfy relative errors of less than 1.25, 1.56, and 1.95, respectively. These proportions are also significantly higher than the predictions for LR images. Therefore, the depth estimation results in Table 6 demonstrate that SR processing can help enhance the performance of subsequent depth estimation.

5. Discussion

The experimental results presented in Section 4 demonstrate that the proposed method achieves significant improvements in both standard image quality metrics and downstream task performance. This indicates that the proposed SR method can serve as an effective preprocessing module to enhance the reliability of existing industrial vision pipelines without the need for costly hardware upgrades.

Despite the positive results, this study has several limitations. Firstly, the proposed method assumes a mixed-resolution acquisition scheme. While this configuration is feasible in controlled industrial environments, it may not be directly transferable to other scenarios. For pure-LR multi-view input, the model can be adapted by first applying SISR to a specific view to generate a pseudo-HR reference image, or by exploring blind SR techniques. Secondly, the angular resolution is currently fixed at 3 × 3. In scenarios with only a limited number of input views, the effective receptive field of SAS convolution relying on angular neighborhoods for cross-view aggregation will narrow, potentially preventing the model from fully leveraging inter-view correlations. However, the STF module can still function normally using the central HR reference image. Therefore, performance on sparser or denser viewpoint arrays remains to be explored. Thirdly, the model does not explicitly incorporate prior information about reflectance or physics-based constraints, factors that could further improve the handling of specular highlights. Finally, a direct empirical comparison with state-of-the-art reference-based SR methods was not conducted.

Therefore, future research could explore the following directions. First, extending the framework to handle variable angular resolutions will enhance deployment flexibility. Second, introducing differentiable rendering or physics-based constraints could regularize texture transfer in highly specular regions. Finally, constructing a diverse multi-view dataset of industrial metal images will facilitate more robust training and broader generalization capabilities.

6. Conclusions

In precision engineering, the applications of machine vision are becoming increasingly critical, especially where high-precision measurements of complex shapes and tiny features are required. However, in practice, image resolution limitations often affect performance. There is currently a lack of research on multi-view industrial image enhancement. To address this issue, this paper proposes an innovative algorithm based on deep learning, which aims to fill the current gap in the research of multi-view metal image SR. The designed multi-resolution fusion SR model combines the features of light field SR techniques and those of reference-based SR methods. Spatial and angular feature extraction is conducted to achieve SR at the image level. And multi-scale feature extraction and feature matching based on an attention mechanism are fused for SR at the feature level. To validate the effectiveness of the proposed method, the study uses a publicly available dataset of metal object images for experimentation and evaluation. The experimental results show that the method can significantly improve the resolution and detail performance of metal images at large scales. Furthermore, combining the SR output with a monocular depth estimation network significantly reduces depth errors, confirming that enhanced texture fidelity directly benefits downstream visual tasks. In the future, we will focus on constructing specialized multi-view metal image datasets, combining SR reconstruction with subsequent depth estimation or precision measurement.