Search Results (46)

Visual–inertial odometry (VIO) is the primary supporting technology for autonomous systems, but it faces three major challenges: initialization sensitivity, dynamic illumination, and multi-sensor fusion. In order to overcome these challenges, this paper proposes stereo direct sparse visual–inertial odometry with efficient second-order minimization. It is entirely implemented using the direct method, which includes a depth initialization module based on visual–inertial alignment, a stereo image tracking module, and a marginalization module. Inertial measurement unit (IMU) data is first aligned with a stereo image to initialize the system effectively. Then, based on the efficient second-order minimization (ESM) algorithm, the photometric error and the inertial error are minimized to jointly optimize camera poses and sparse scene geometry. IMU information is accumulated between several frames using measurement preintegration and is inserted into the optimization as an additional constraint between keyframes. A marginalization module is added to reduce the computation complexity of the optimization and maintain the information about the previous states. The proposed system is evaluated on the KITTI visual odometry benchmark and the EuRoC dataset. The experimental results demonstrate that the proposed system achieves state-of-the-art performance in terms of accuracy and robustness. Full article

The mathematical model for photometric stereo makes several restricting assumptions, which are often not fulfilled in real-life applications. Specifically, an object surface does not always satisfies Lambert’s cosine law, leading to reflection issues. Moreover, the camera and the light source, in some situations, have to be placed at a close distance from the target, rather than at infinite distance from it. When studying algorithms for these complex situations, it is extremely useful to have at disposal synthetic datasets with known exact solutions, to assert the accuracy of a solution method. The aim of this paper is to present a Matlab package which constructs such datasets on the basis of a chosen exact solution, providing a tool for simulating various real camera/light configurations. This package, starting from the mathematical expression of a surface, or from a discrete sampling, allows the user to build a set of images matching a particular light configuration. Setting various parameters makes it possible to simulate different scenarios, which can be used to investigate the performance of reconstruction algorithms in several situations and test their response to lack of ideality in data. The ability to construct large datasets is particularly useful to train machine learning based algorithms. Full article

With the gradual application of 3D reconstruction technology in underwater scenes, the design of vision-based reconstruction models has become an important research direction for human ocean exploration and development. The underwater laser triangulation method is the most commonly used approach, yet it misses details during the reconstruction of sparse point clouds, which do not meet the requirements of practical applications. On the other hand, existing underwater photometric stereo methods can accurately reconstruct local details of target objects, but they require relative stillness to be maintained between the camera and the target, which is practically difficult to achieve in underwater imaging environments. In this paper, we propose an underwater target reconstruction algorithm that combines laser triangulation and multispectral photometric stereo (MPS) to address the aforementioned practical problems in underwater 3D reconstruction.This algorithm can obtain more comprehensive 3D surface data of underwater objects through mobile measurement. At the same time, we propose to optimize the laser place calibration and laser line separation processes, further improving the reconstruction performance of underwater laser triangulation and multispectral photometric stereo. The experimental results show that our method achieves higher-precision and higher-density 3D reconstruction than current state-of-the-art methods. Full article

In the digital 3D reconstruction of the shapes and surface reflectance of ancient paintings and drawings using Photometric Stereo (PS) techniques, normal integration is a key step. However, difficulties in locating light sources, non-Lambertian surfaces, and shadows make the results of this step inaccurate for such artworks. This paper presents a solution for PS to overcome this problem based on some enhancement of the normal integration process and the accurate measurement of Points of Interest (PoIs). The mutual positions of the LED lights, the camera sensor, and the acquisition plane in two custom-designed stands, are measured in laboratory as a system calibration of the 3D acquisition workflow. After an introduction to the requirements and critical issues arising from the practical application of PS techniques to artworks, and a description of the newly developed PS solution, the measurement process is explained in detail. Finally, results are presented showing how the normal maps and 3D meshes generated using the measured PoIs’ positions, and further minimized using image processing techniques, which significantly limits outliers and improves the visual fidelity of digitized artworks. Full article

Non-collaborative (i.e., reflective, transparent, metallic, etc.) surfaces are common in industrial production processes, where 3D reconstruction methods are applied for quantitative quality control inspections. Although the use or combination of photogrammetry and photometric stereo performs well for well-textured or partially textured objects, it usually produces unsatisfactory 3D reconstruction results on non-collaborative surfaces. To improve 3D inspection performances, this paper investigates emerging learning-based surface reconstruction methods, such as Neural Radiance Fields (NeRF), Multi-View Stereo (MVS), Monocular Depth Estimation (MDE), Gaussian Splatting (GS) and image-to-3D generative AI as potential alternatives for industrial inspections. A comprehensive evaluation dataset with several common industrial objects was used to assess methods and gain deeper insights into the applicability of the examined approaches for inspections in industrial scenarios. In the experimental evaluation, geometric comparisons were carried out between the reference data and learning-based reconstructions. The results indicate that no method can outperform all the others across all evaluations. Full article

In recent years, target inspection has found extensive utilization within the industry, making it crucial to detect defects in industrial products to ensure quality. To address the challenges posed by large brightness differences, attached dirt, and complex backgrounds in saggers, we propose a sagger defect recognition method that integrates deep learning target detection and machine vision feature extraction. This method commences by employing the photometric stereo method to construct a curvature map of the sagger surface, reducing the interference from brightness differences and dirt. Next, an improved YOLOv5s target detection model uses the surface curvature map as an image source for crack detection. The model incorporates the Faster Block module in the backbone network and an efficient coordinate attention mechanism, embedding position information into channel attention to enhance the model’s understanding of crack defects. Finally, the method extracts crack geometry features from the target region, using feature scoring to confirm whether a crack defect is present. Compared with existing methods, this approach provides a new solution for detecting sagger cracks in complex backgrounds. Field applications and test results demonstrate that this method effectively improves the accuracy of sagger crack defect recognition. Full article

This paper addresses image enhancement and 3D reconstruction techniques for dim scenes inside the vacuum chamber of a nuclear fusion reactor. First, an improved multi-scale Retinex low-light image enhancement algorithm with adaptive weights is designed. It can recover image detail information that is not visible in low-light environments, maintaining image clarity and contrast for easy observation. Second, according to the actual needs of target plate defect detection and 3D reconstruction inside the vacuum chamber, a defect reconstruction algorithm based on photometric stereo vision is proposed. To optimize the position of the light source, a light source illumination profile simulation system is designed in this paper to provide an optimized light array for crack detection inside vacuum chambers without the need for extensive experimental testing. Finally, a robotic platform mounted with a binocular stereo-vision camera is constructed and image enhancement and defect reconstruction experiments are performed separately. The results show that the above method can broaden the gray level of low-illumination images and improve the brightness value and contrast. The maximum depth error is less than 24.0% and the maximum width error is less than 15.3%, which achieves the goal of detecting and reconstructing the defects inside the vacuum chamber. Full article

In the process of scratch defect detection in textured materials, there are often problems of low efficiency in traditional manual detection, large errors in machine vision, and difficulty in distinguishing defective scratches from the background texture. In order to solve these problems, we developed an enhanced scratch defect detection system for textured materials based on optimized photometric stereo vision and FFT-Gabor filtering. We designed and optimized a novel hemispherical image acquisition device that allows for selective lighting angles. This device integrates images captured under multiple light sources to obtain richer surface gradient information for textured materials, overcoming issues caused by high reflections or dark shadows under a single light source angle. At the same time, for the textured material, scratches and a textured background are difficult to distinguish; therefore, we introduced a Gabor filter-based convolution kernel, leveraging the fast Fourier transform (FFT), to perform convolution operations and spatial domain phase subtraction. This process effectively enhances the defect information while suppressing the textured background. The effectiveness and superiority of the proposed method were validated through material applicability experiments and comparative method evaluations using a variety of textured material samples. The results demonstrated a stable scratch capture success rate of 100% and a recognition detection success rate of 98.43% ± 1.0%. Full article

Self-supervised monocular depth estimation methods have become the focus of research since ground truth data are not required. Current single-image-based works only leverage appearance-based features, thus achieving a limited performance. Deep learning based multiview stereo works facilitate the research on multi-frame depth estimation methods. Some multi-frame methods build cost volumes and take multiple frames as inputs at the time of test to fully utilize geometric cues between adjacent frames. Nevertheless, low-textured regions, which are dominant in indoor scenes, tend to cause unreliable depth hypotheses in the cost volume. Few self-supervised multi-frame methods have been used to conduct research on the issue of low-texture areas in indoor scenes. To handle this issue, we propose SIM-MultiDepth, a self-supervised indoor monocular multi-frame depth estimation framework. A self-supervised single-frame depth estimation network is introduced to learn the relative poses and supervise the multi-frame depth learning. A texture-aware depth consistency loss is designed considering the calculation of the patch-based photometric loss. Only the areas where multi-frame depth prediction is considered unreliable in low-texture regions are supervised by the single-frame network. This approach helps improve the depth estimation accuracy. The experimental results on the NYU Depth V2 dataset validate the effectiveness of SIM-MultiDepth. The zero-shot generalization studies on the 7-Scenes and Campus Indoor datasets aid in the analysis of the application characteristics of SIM-MultiDepth. Full article

The standard photometric stereo model makes several assumptions that are rarely verified in experimental datasets. In particular, the observed object should behave as a Lambertian reflector, and the light sources should be positioned at an infinite distance from it, along a known direction. Even when Lambert’s law is approximately fulfilled, an accurate assessment of the relative position between the light source and the target is often unavailable in real situations. The Hayakawa procedure is a computational method for estimating such information directly from data images. It occasionally breaks down when some of the available images excessively deviate from ideality. This is generally due to observing a non-Lambertian surface, or illuminating it from a close distance, or both. Indeed, in narrow shooting scenarios, typical, e.g., of archaeological excavation sites, it is impossible to position a flashlight at a sufficient distance from the observed surface. It is then necessary to understand if a given dataset is reliable and which images should be selected to better reconstruct the target. In this paper, we propose some algorithms to perform this task and explore their effectiveness. Full article

Predicting accurate normal maps of objects from two-dimensional images in regions of complex structure and spatial material variations is challenging using photometric stereo methods due to the influence of surface reflection properties caused by variations in object geometry and surface materials. To address this issue, we propose a photometric stereo network called a RMAFF-PSN that uses residual multiscale attentional feature fusion to handle the “difficult” regions of the object. Unlike previous approaches that only use stacked convolutional layers to extract deep features from the input image, our method integrates feature information from different resolution stages and scales of the image. This approach preserves more physical information, such as texture and geometry of the object in complex regions, through shallow-deep stage feature extraction, double branching enhancement, and attention optimization. To test the network structure under real-world conditions, we propose a new real dataset called Simple PS data, which contains multiple objects with varying structures and materials. Experimental results on a publicly available benchmark dataset demonstrate that our method outperforms most existing calibrated photometric stereo methods for the same number of input images, especially in the case of highly non-convex object structures. Our method also obtains good results under sparse lighting conditions. Full article

Multiview stereo (MVS) achieves efficient 3D reconstruction on Lambertian surfaces and strongly textured regions. However, the reconstruction of weakly textured regions, especially planar surfaces in weakly textured regions, still faces significant challenges due to the fuzzy matching problem of photometric consistency. In this paper, we propose a multiview stereo for recovering planar surfaces guided by confidence calculations, resulting in the construction of large-scale 3D models for high-resolution image scenes. Specifically, a confidence calculation method is proposed to express the reliability degree of plane hypothesis. It consists of multiview consistency and patch consistency, which characterize global contextual information and local spatial variation, respectively. Based on the confidence of plane hypothesis, the proposed plane supplementation generates new reliable plane hypotheses. The new planes are embedded in the confidence-driven depth estimation. In addition, an adaptive depth fusion approach is proposed to allow regions with insufficient visibility to be effectively fused into the dense point clouds. The experimental results illustrate that the proposed method can lead to a 3D model with competitive completeness and high accuracy compared with state-of-the-art methods. Full article

Automated inspection technology based on computer vision is now widely used in the manufacturing industry with high speed and accuracy. However, metal parts always appear in high gloss or shadow on the surface, resulting in the overexposure of the captured images. It is necessary to adjust the light direction and view to keep defects out of overexposure and shadow areas. However, it is too tedious to adjust the position of the light direction and view the variety of parts’ geometries. To address this problem, we design a photometric-stereo-based defect detection system (PSBDDS), which combines the photometric stereo with defect detection to eliminate the interference of highlights and shadows. Based on the PSBDDS, we introduce a photometric-stereo-based defect detection framework, which takes images captured in multiple directional lights as input and obtains the normal map through the photometric stereo model. Then, the detection model uses the normal map as input to locate and classify defects. Existing learning-based photometric stereo methods and defect detection methods have achieved good performance in their respective fields. However, photometric stereo datasets and defect detection datasets are not sufficient for training and testing photometric-stereo-based defect detection methods, thus we create a photometric stereo defect detection (PSDD) dataset using our PSBDDS to eliminate gaps between learning-based photometric stereo and defect detection methods. Furthermore, experimental results prove the effectiveness of the proposed PSBBD and PSDD dataset. Full article

Image-based 3D reconstruction has been employed in industrial metrology for micro-measurements and quality control purposes. However, generating a highly-detailed and reliable 3D reconstruction of non-collaborative surfaces is still an open issue. In this paper, a method for generating an accurate 3D reconstruction of non-collaborative surfaces through a combination of photogrammetry and photometric stereo is presented. On one side, the geometric information derived with photogrammetry is used in areas where its 3D measurements are reliable. On the other hand, the high spatial resolution capability of photometric stereo is exploited to acquire a finely detailed topography of the surface. Finally, three different approaches are proposed to fuse both geometric information and high frequency details. The proposed method is tested on six different non-collaborative objects with different surface characteristics. To evaluate the accuracy of the proposed method, a comprehensive cloud-to-cloud comparison between reference data and 3D points derived from the proposed fusion methods is provided. The experiments demonstrated that, despite correcting global deformation up to an average RMSE of less than 0.1 mm, the proposed method recovers the surface topography at the same high resolution as the photometric stereo. Full article

Reflectance Transformation Imaging (RTI) is a non-contact technique which consists in acquiring a set of multi-light images by varying the direction of the illumination source on a scene or a surface. This technique provides access to a wide variety of local surface attributes which describe the angular reflectance of surfaces as well as their local microgeometry (stereo photometric approach). In the context of the inspection of the visual quality of surfaces, an essential issue is to be able to estimate the local visual saliency of the inspected surfaces from the often-voluminous acquired RTI data in order to quantitatively evaluate the local appearance properties of a surface. In this work, a multi-scale and multi-level methodology is proposed and the approach is extended to allow for the global comparison of different surface roughnesses in terms of their visual properties. The methodology is applied on different industrial surfaces, and the results show that the visual saliency maps thus obtained allow an objective quantitative evaluation of the local and global visual properties on the inspected surfaces. Full article