1. Introduction
Recent advancements in high-energy-density (HED) beam advanced machining and metal additive manufacturing (AM) have provided the ability to create parts with unique properties that are unavailable to traditional manufacturing [
1,
2,
3,
4,
5,
6], including the production of complex structures with high accuracy and significant weight reduction. Crucial challenges in the quality control of metal AM technologies are related to in situ monitoring, due to a lack of information about defect formation, which affects the reliability and repeatability of the manufacturing process [
7,
8].
Thermal imaging of the processing zone addresses the task of in situ temperature monitoring. The basic approach to temperature monitoring in HED industrial processes is point temperature measurement using a pyrometer [
2,
9,
10]. To achieve temperature gradient mapping, one should proceed with thermal imaging. Infrared (IR) radiometric imaging pyrometry [
11,
12] demonstrates moderate spatial resolution due to larger wavelengths [
13] and requires a thorough calibration process to account for the spectral and temperature dependences of material emissivity and gas absorption in the operational area [
14,
15]. To overcome the need for prior knowledge of emissivity, spectral imaging techniques can be used [
12,
13,
16,
17]. The most common spectral thermal imaging techniques include two-wavelength imaging pyrometry (TWIP) and multispectral (MS) and hyperspectral imaging. TWIP provides good temperature measurement uncertainty for high-temperature processes in metal AM [
18] and may be achieved by combining red and green color channels of an RGB camera [
19]. Despite the relative simplicity and cost effectiveness this technique demonstrates significant temperature uncertainty in the 1000–2000 °C temperature range. Hyperspectral imaging techniques show promise for achieving lower measurement error and eliminating difficulties associated with emissivity dependencies by obtaining the detailed spectra [
20,
21], while requiring the spatial scanning to acquire a two-dimensional temperature map. MS imaging appears to be an intermediate technique providing two-dimensional images in multiple spectral bands without spatial scanning [
22,
23].
One of the simplest solutions within the variety of MS camera layouts [
24] is the multiaperture filtered camera (MAFC) [
25]. It offers relatively high spatial resolution and additional spectral data to enhance the precision of temperature estimation. Meanwhile, this layout maintains the availability of interchangeable spectral filters, while being versatile and offering an adjustable wavelength set to measure spectral intensities distributed along the monotonically increasing part of the Planck curve. This feature provides operational flexibility in dealing with the different observed object temperatures. Since the image sensor (IS) of the MAFC is virtually divided into zones corresponding to the spectral bands, the mentioned MS camera layout involves a specific optical system design. In this work, we describe a basic design concept for an MS MAFC and a general data processing workflow that accounts for the intrinsic properties and residual image distortions inherent to the chosen camera elements before temperature calculation. To provide good flexibility of component choice, the camera’s basic design concept is based on the use of interchangeable, commercially available machine vision lenses equipped with bandpass filters and a custom mirror alignment system. The four-wavelength pixel-wise temperature calculation approach allows for initial temperature estimation with two wavelengths and subsequent fitting of four measured spectral intensities to the theoretical Planck curve by optimizing both temperature and emissivity. To demonstrate the feasibility of the proposed basic design and methods, we developed a prototype of a versatile MS MAFC for thermal imaging applications and experimentally assessed its performance with common thermal imaging test objects.
2. Materials and Methods
2.1. Experimental Setup
2.1.1. Multispectral Camera
Our MS camera design is developed within the general MAFC layout. Essentially, the MAFC generates distinct images in individual spectral channels using an array of lenses that is compact enough to be positioned in front of the IS [
24,
26]. The small size of the individual channel components (lenses and filters) presents significant technological challenges. Adjusting the image scale and working distance necessitates the use of additional objective lenses, and reconfiguring the spectral bands requires disassembling the camera. We present a basic design suitable for finite-conjugate imaging with the MAFC using interchangeable commercially available machine vision lenses and bandpass filters.
The use of the proposed MS MAFC in thermal imaging applications imposes certain limitations on its use at finite working distances due to the mutual parallax of each spectral channel’s objectives. The presence of parallax causes each spectral channel to capture individual scenes that only partially overlap, significantly reducing the effective field of view of the MS camera. The obvious solution is to create a hinged structure that allows the objectives to tilt toward each other, enabling each channel to record the same scene without parallax. However, this solution has several design-related limitations. The aperture ratio of each spectral channel’s objectives depends on their diameter, while the resulting field of view depends on their focal length. Additionally, both the IS and the printed circuit board on which it is mounted have finite dimensions, further limiting the placement and size of the objectives. Therefore, it is essential to select the optimal balance between focal length, aperture ratio, and IS size to design an MS camera capable of capturing objects located at both finite and infinite distances. Based on the above considerations, the selection of objectives and IS was made to achieve the desired image scale at a specific working distance, in accordance with the formulas of geometric optics [
27]:
where
is the linear magnification of the objective of each spectral channel,
is the image size of each spectral channel,
is the object size (linear field of view),
is the distance from the objective to the object (working distance),
is the distance from the objective to the IS.
where
is the focal length of the objective of each spectral channel.
Since the MS camera was developed specifically for thermal imaging applications in AM and HED processes, we selected a linear field of view ranging from 5 to 10 mm, with a minimum working distance of approximately 300 mm for each spectral channel. The camera’s design prioritizes simplicity in manufacturing mirrors and fasteners, as well as ease of assembly and alignment. To achieve this, the IS size needs to be maximized. Based on these parameters, we performed calculations using Equations (1) and (2) and determined that objectives with a focal length of 75 mm were optimal. Subsequent 3D modeling of the MS camera helped us to establish the required objective diameters to eliminate parallax at the specified minimum working distance. The primary components of the prototype and its design are detailed below.
The presented camera prototype contains four individual spectral channels conjugated with a single IS (DMK 33GX264, IMX264 2/3″ sensor, 2448 × 2048 pixels, 3.45 μm pixel size, The Imaging Source, Bremen, Germany). The identical objectives with 75 mm focal length and F#2.8 (VM7528MP5, Guangdong ZLKC Optical Co., Ltd., Guangzhou, China) are equipped with 1″ bandpass spectral filters. Central wavelengths (CWL) and the full width at half maximum (FWHM) of filters’ spectral bands are shown in
Table 1.
Figure 1 illustrates the MS camera layout configuration. The supporting frame assembly SF includes the frame itself and four adjustable pivots for C-mount objectives O and adjustable mirrors AM. The optical axis direction for each individual spectral channel is altered twice using a combination of adjustable and fixed mirrors. Fixed mirrors M are mounted on base B in front of the image sensor IS. To correct for position deviations in the fixed mirrors, we incorporated the capability to finely adjust the position of the adjustable mirrors while keeping the pivots stationary. To minimize stray light, we enclosed the support frame assembly in a custom-made opaque housing. The completed MS camera prototype is depicted in
Figure 2.
2.1.2. Auxiliary Equipment and Calibration Targets
Since the stationary mirrors are near the IS to minimize the distance between the objectives and the IS, they need to be very compact. The ones in the proposed MS camera prototype are attached to the base with glue. To ensure proper alignment during assembly and evaluate displacement due to glue shrinkage we implemented an auxiliary laser-based setup (
Figure 3). The base with attached mirrors is located on a precise rotating stage. Prior to this, the laser O beam axis is aligned with the pair of irises D1, D2. Direction angles for each mirror are determined using the position deviations
of laser spot positions p
1–p
4 observed on a screen S and a distance to the screen. If the optimal alignment is achieved,
= 0. In practice, the determined position deviations may be accounted for during fine alignment of adjustable mirrors. Finally, to overcome uncompensated image rotation and achieve point-to-point correspondence of images obtained with individual spectral channels, we applied camera calibration described in
Section 2.2.
We determined the spatial image transformations associated with correction of spatial distortions using the images of a checkerboard pattern test target. To quantify the dependence of the sensor response on exposure time, a reference source (integrating sphere) with stable radiative output was employed (Spectra-FT-2300-W, Labsphere Inc., North Sutton, NH, USA). The integrating sphere was also implemented for vignetting correction. The spectral response of individual spectral channels was assessed involving images of a diffuse scattering reference target providing uniform reflectance close to 1 across the relevant spectral range illuminated with the tunable light source (broadband light source: XWS-65, ISTEQ, Eindhoven, The Netherlands; monochromator: M266i–IV, Solar Laser Systems, Minsk, Republic of Belarus). Temperature mapping validation required a blackbody source (Metropir Helios, NPL Metropir LLC, Saint Petersburg, Russia, temperature range 800–1500 °C).
2.2. Camera Calibration
The MS data captured by the camera can be distorted due to variations in the actual transmission bands of spectral filters, the optical system characteristics, and the IS response and sensitivity. Each of these factors impacts the resulting spectral curve of the observed object and subsequently reduces the accuracy of temperature estimation. To mitigate this, proper calibration of the MS camera is essential. This process entails calculating correction coefficients for each spectral channel and applying them to adjust the raw data. The primary stages of the correction workflow, developed using reference targets and calibrated laboratory radiation sources, are depicted in
Figure 4.
The IS response describes the relationship between the radiant flux and the pixel intensity values. Distortions caused by the non-linearity and disproportionality of the light-sensitive element and electronic components [
28] introduce systematic errors that reduce the accuracy of spectral measurements, narrow the dynamic range and complicate the comparison of data acquired at different times or using different devices. Several studies [
29,
30] have examined sensor response functions and how they vary with changes in light intensity and exposure time. Due to the pronounced curvature of the Planck distribution in the 550–900 nm spectral range, image acquisition can be performed across a range of exposure times, either for each spectral channel or for different temperature regimes. An integrating sphere was used to provide spatially uniform, broadband illumination across the MS camera’s field of view, allowing us to measure how the IS response changes with exposure time. We acquired a series of images of the sphere’s output port at different exposure settings and normalized the results. This allowed calculating correction coefficients that make the sensor output directly proportional to the incoming radiance. Applying these corrections enables automatic exposure adjustment during real-time monitoring.
The spectral sensitivity of the MS camera is determined by both the sensor’s sensitivity and the transmission characteristics of the spectral filters in each optical channel. These properties are typically assumed to be spatially uniform. Spectral non-uniformities can distort the shape of an object’s spectral signature. It becomes especially critical in spectroscopic applications, such as spectral thermal imaging. To quantify and correct these sensitivity variations, diffuse scattering reference target images were captured. By averaging the signal across the image and accounting for both the measured radiation power and the sensor’s response, we derived the camera’s sensitivity curves (shown in
Figure 4). Correction coefficients were then calculated by integrating under these curves, allowing us to equalize the sensitivity across all channels and ensure consistent spectral measurements.
Optical system vignetting causes a gradual reduction in irradiance from the center of the image toward its boundaries. This spatial non-uniformity introduces systematic errors when reconstructing temperature maps, since accurate temperature measurements depend on precise pixel-level intensity values. Due to vignetting, calculated thermal gradients (like those found in melt pools) can be distorted, potentially leading to incorrect conclusions about the object’s thermal state. To correct for vignetting, the method described in [
31] can be used. It involves capturing images of a spatially uniform radiance source (an integrating sphere) with the MS camera at various intensity levels. By averaging these images and normalizing them to the highest value, a correction matrix is derived. This matrix compensates for vignetting by adjusting the measured data, restoring radiometric uniformity across the entire field of view.
The proposed realization of a finite-conjugate MAFC introduces parallax—a shift in the position of objects between images from different channels due to differences in the optical path geometry. This issue is especially important when precise pixel-level alignment is needed for accurate mapping of physical parameters like temperature. Additional distortions come from the mirror system in the MS camera design, which can cause angular rotation of the images in individual channels. These effects cause spatial misalignment of objects across the spectral images, making it impossible to directly use the data for quantitative analysis, including temperature mapping, without first correcting the geometric distortions. To address this, we captured images of a checkerboard pattern test target using the MS camera. Matching points on the target were manually identified in each image, and rotation and translation matrices [
32] were calculated for each channel relative to a reference channel. The 660 nm channel was chosen as the reference because its image center was closest to the center of the corresponding sensor area.
The algorithms for determining correction coefficients and transformation matrices, along with the data correction procedures, were implemented in Mathworks MATLAB R2024a.
2.3. Temperature Calculation
The application of correction coefficients and transformation matrices to the spectral images allows us to build a three-dimensional MS data cube, where each pixel represents the brightness of a corresponding area on the emitting object. Several methods can then be used to estimate temperature from the emission spectrum: the colorimetric method [
33], Wien’s displacement law [
13], TWIP [
10], and fitting the experimental spectrum to the blackbody radiation curve [
34].
The colorimetric method, which estimates the object’s color temperature, requires only an RGB camera, offering a simple and cost-effective solution. However, this method yields a subjective parameter influenced by both the spectral composition of the emission and the characteristics of the standard observer, resulting in a measured color temperature that may not correspond precisely to the actual thermodynamic temperature. Additionally, it is sensitive to spectral distortions introduced by sensor characteristics, environmental conditions, and color processing algorithms. Wien’s displacement law is applicable only within a temperature range of approximately 3000 to 7000 °C, where the emission peak lies within the visible spectrum; outside this range, it is only useful as a rough estimate.
TWIP requires brightness measurements in only two spectral bands and is robust against partial spectral distortions because it relies on brightness ratios. However, at lower temperatures (around 1000 to 2000 °C), it can produce significant errors due to the small differences in intensity (i.e., the low logarithmic ratio of intensities). Since the camera we developed has four spectral channels, the two-color pyrometry method can be used to provide initial conditions for the blackbody radiation fitting.
The approximation method ensures both versatility and high precision in temperature estimation. While the high quantity of spectral bands is ideal for accurate approximation, it can still be effectively applied using only four spectral channels, provided these channels are evenly distributed across the spectrum to adequately capture the characteristic shape of the Planck radiation curve. In this context, the method reduces to fitting four measured intensity values to the theoretical curve by optimizing the temperature parameter
along with the scaling coefficient
:
where
represents the emission spectrum acquired by the MS camera, averaged either over a specified region or at an individual pixel, while
represents the theoretical Planck radiation curve, with
indexing the spectral channels. Optimization of the temperature parameter and scaling factor was performed using the Lagarias simplex search algorithm [
35], utilizing MATLAB R2024a built-in optimization functions. After optimization, the temperature was converted from Kelvin to Celsius. Initial parameter estimates for the optimization were derived from two-color pyrometry calculations based on the 780 nm and 840 nm spectral channels. This pixel-wise optimization of MS data facilitates high-resolution spatial mapping of the temperature distribution across the observed object.
Emissivity
is a parameter that determines the radiation intensity in comparison to an ideal blackbody. During the optimization process the emissivity value is varied along with the temperature to minimize the difference between the theoretical radiation and the actual measured MS data. In practice, when studying processes with known materials, approximate tabulated emissivity values are used [
36,
37]. Our optimization algorithm allows setting the initial emissivity range guess from tabulated values. This reduces the complexity of the problem, making the calculations faster and improving the stability of the optimization. For the development of an operational temperature estimation algorithm, we applied this approximation.
4. Discussion
Experimental validation highlighted potential advantages and drawbacks of the proposed approach. Although the image resolution decreases due to IS division and cropping during the correction stage, the resulting temperature maps still provide sufficient detail to identify defects and local temperature variations. Differences in temperature values at similar power levels, compared to the results described in [
39], are related to filament degradation and non-uniform filament heating. The spatial offset in the image acquisition area relative to the filament center emphasizes the temperature gradient inherent to this object type at moderate power levels. The experimental technique used in [
39] is based on combining a brightness imaging pyrometer that acquires images in a narrow band within the visible wavelength range with a point-measurement spectrometer. In this case, temperature mapping is achieved with a significant assumption of a graybody with a uniform emissivity spatial distribution. Detailed spectral intensity measurement provides precise temperature estimation at one point, and the temperature map is obtained by comparing the brightness in distinct pixels. A larger temperature error obtained with four-wavelength fitting compared to the values reported in [
39] is associated with reduced spectral information relative to the point-measurement spectrometer. Similarly, line-scanning hyperspectral imaging approaches [
20,
21] demonstrate better temperature precision along with lower temporal resolution. Our prototype acquired 24 fps for 16-bit at full IS resolution with a single frame exposure time of less than 10 milliseconds. The data acquisition speed was limited by the bandwidth connection of the gigabit Ethernet interface. Using the proposed approach, one can achieve higher temporal resolution while monitoring non-stationary objects by selecting a faster IS and limiting the temperature precision.
The mentioned graybody assumption is widespread in common visible and IR imaging pyrometry techniques [
13], limiting the attainable precision in real “non-gray” objects. To address the emissivity dependencies issue, one can refer to published data to obtain an initial estimate [
40,
41] and assume the simplified model [
13,
23]. However, within the context of the defined objectives, this approximation provides a reasonable balance between computational efficiency and required accuracy. The assumption of a known emissivity can lead to inaccuracies, particularly when the material’s surface properties evolve during heating [
42]. The proposed approach allows pixel-wise emissivity fitting along with temperature, although it is time-consuming. To enhance fitting stability and provide the algorithm with an initial guess we introduced an ability to insert a set of characteristic emissivity ranges prior to temperature calculation. Thus, the proposed approach relies on the assumption of a finite set of material types in the field of view during the HED process. Also, using the proposed approach one can choose two of the four individual spectral images as TWIP data to achieve fast temperature estimation with potentially lower precision.
Implementation of four different spectral bands provides the researcher with an ability to implement the desired Planck’s curve fitting methods and improve calculation stability. The analysis of the coefficients of variation demonstrates a moderate degree of variability in the temperature estimates derived by proposed method. At lower temperatures, the coefficients range from 1.13 to 1.36, indicating higher measurement instability and greater fluctuations. As the temperature increases, the coefficients decrease (coefficients drop below 1, from 0.66 to 0.92), suggesting improved precision and reduced variability in the measurements at elevated temperatures. This trend may reflect enhanced measurement accuracy or a reduced sensitivity to external factors at higher temperature levels. To maintain the measurement accuracy at lower temperatures using the proposed approach, one can select bigger central wavelength values of spectral filters and extend the working spectral range in near IR by choosing the InGaAs-based IS if needed. Although reducing the accessibility, an interchangeable filter–IS combination may be achieved without objective replacement by application of broadband apochromatic optical systems [
43,
44,
45]. In addition to simple temperature-regime-dependent reconfiguration, the flexible choice of filter transmission bands provides potential for avoiding the absorption bands of gases and evaporation plume [
13,
19] in HED industrial processes.
Another issue to consider is the angular variations in thermal radiation intensity with respect to the observation direction. The expected location of the thermal imaging device results in off-axis image acquisition. Additionally, the proposed camera design implies individual spectral channel parallax, leading to slightly different observation directions. Thus, the angle between the observation direction of each channel and the surface normal also needs to be considered. Since the emissivity is reported to remain constant in the range of angles between 0° and 40° [
46], the channel directions should not exceed this range to avoid distortions. Assuming the heated object to be a Lambertian emitting surface the intensity variations in different off-axis directions of spectral channels may be taken into account. The alternative solution to minimize angular variations is implementation of a single-objective system based on spectrally resolving detector arrays [
24] with Bayer-type filter arrays. Providing a spatial resolution decrease proportional to the number of spectral channels similar to the MAFC, this approach involves a technologically complex commercially available IS with embedded fixed patterns of spectral filters, significantly increasing the cost relative to a conventional monochrome IS and reducing the flexibility. Since the optimal spectral channel bandwidth is a tradeoff between light throughput and spectral resolution, overlapping spectral bands of Bayer-type filters lead to channel cross-talk identical to RGB cameras that require additional correction [
47].
It is also important to note the complex impact of parallax in the optical system. During the development of our MS camera, we intentionally minimized parallax to mitigate potential distortions caused by differences in viewing angles. However, accounting for the dependence of thermal radiation intensity on the viewing angle [
46], we can increase both the parallax and the stereoscopic baseline (while adhering to the constraints on lens-to-sensor distance) to enable multispectral stereo functionality [
48]. This approach enhances spectral measurement devices by adding the capability to reconstruct the 3D shape of the observed object’s surface.
5. Conclusions
Experimental validation of the prototype developed using the basic design concept and data processing workflow proposed in this work demonstrated competitive temperature uncertainty compared to other thermal imaging techniques, while maintaining the potential for quick and simple reconfiguration to adapt to object parameters (scale, working distance, temperature regime). The primary considerations of the MS MAFC basic design concept, including geometric optics assumptions and alignment recommendations, are presented. The correction workflow enables one to assemble a camera using commercially available components and personally assess and account for their characteristic properties, even without comprehensive datasheet specifications.
Future research will focus on two main aspects: (1) further development of the basic design concept to elucidate the rationale behind the optimal selection of spectral channels’ central wavelengths and bandwidths; (2) conducting in situ experiments to evaluate the effectiveness and practicality of the proposed MS camera design and workflow for HED industrial processes and AM monitoring using the prototype. We will also optimize algorithms to enable real-time temperature mapping.