Search Results (76)

With the advancement of agricultural automation technologies, apple-harvesting robots have gradually become a focus of research. As their “perceptual core,” machine vision systems directly determine picking success rates and operational efficiency. However, existing vision systems still exhibit significant shortcomings in target detection and positioning accuracy in complex orchard environments (e.g., uneven illumination, foliage occlusion, and fruit overlap), which hinders practical applications. This study proposes a visual system for apple-harvesting robots based on improved Mask R-CNN and binocular vision to achieve more precise fruit positioning. The binocular camera (ZED2i) carried by the robot acquires dual-channel apple images. An improved Mask R-CNN is employed to implement instance segmentation of apple targets in binocular images, followed by a template-matching algorithm with parallel epipolar constraints for stereo matching. Four pairs of feature points from corresponding apples in binocular images are selected to calculate disparity and depth. Experimental results demonstrate average coefficients of variation and positioning accuracy of 5.09% and 99.61%, respectively, in binocular positioning. During harvesting operations with a self-designed apple-picking robot, the single-image processing time was 0.36 s, the average single harvesting cycle duration reached 7.7 s, and the comprehensive harvesting success rate achieved 94.3%. This work presents a novel high-precision visual positioning method for apple-harvesting robots. Full article

In fields such as military reconnaissance, forest fire prevention, and autonomous driving at night, there is an urgent need for high-precision three-dimensional reconstruction in low-light or night environments. The acquisition of remote sensing data by RGB cameras relies on external light, resulting in a significant decline in image quality and making it difficult to meet the task requirements. The method based on lidar has poor imaging effects in rainy and foggy weather, close-range scenes, and scenarios requiring thermal imaging data. In contrast, infrared cameras can effectively overcome this challenge because their imaging mechanisms are different from those of RGB cameras and lidar. However, the research on three-dimensional scene reconstruction of infrared images is relatively immature, especially in the field of infrared binocular stereo matching. There are two main challenges given this situation: first, there is a lack of a dataset specifically for infrared binocular stereo matching; second, the lack of texture information in infrared images causes a limit in the extension of the RGB method to the infrared reconstruction problem. To solve these problems, this study begins with the construction of an infrared binocular stereo matching dataset and then proposes an innovative perspective projection positional encoding-based transformer method to complete the infrared binocular stereo matching task. In this paper, a stereo matching network combined with transformer and cost volume is constructed. The existing work in the positional encoding of the transformer usually uses a parallel projection model to simplify the calculation. Our method is based on the actual perspective projection model so that each pixel is associated with a different projection ray. It effectively solves the problem of feature extraction and matching caused by insufficient texture information in infrared images and significantly improves matching accuracy. We conducted experiments based on the infrared binocular stereo matching dataset proposed in this paper. Experiments demonstrated the effectiveness of the proposed method. Full article

With the development of industry and society, explosives are widely used in social production as an important industrial product and require transportation. Explosives transport vehicles are susceptible to various objective factors during driving, increasing the risk of transportation. At present, new transport vehicles are generally equipped with intelligent driving monitoring systems. However, for old transport vehicles, the cost of installing such systems is relatively high. To enhance the safety of older explosives transport vehicles, this study proposes a cost-effective intelligent monitoring system using consumer-grade IP cameras and edge computing. The system integrates YOLOv8 for real-time vehicle detection and a novel hybrid ranging strategy combining monocular (fast) and binocular (accurate) techniques to measure distances, ensuring rapid warnings and precise proximity monitoring. An optimized stereo matching workflow reduces processing latency by 23.5%, enabling real-time performance on low-cost devices. Experimental results confirm that the system meets safety requirements, offering a practical, application-specific solution for improving driving safety in resource-limited explosive transport environments. Full article

Three-dimensional information acquisition is crucial for the intelligent control and safe operation of bulk material transportation systems. However, existing visual measurement methods face challenges, including difficult stereo matching due to indistinct surface features, error accumulation in multi-camera calibration, and unreliable depth information fusion. This paper proposes a three-dimensional reconstruction method based on multi-camera hierarchical calibration. The method establishes a measurement framework centered on a core camera, enhances material surface features through speckle structured light projection, and implements a ‘monocular-binocular-multi-camera association’ calibration strategy with global optimization to reduce error accumulation. Additionally, a depth information fusion algorithm based on multi-epipolar geometric constraints improves reconstruction completeness through multi-view information integration. Experimental results demonstrate excellent precision with absolute errors within 1 mm for features as small as 15 mm and relative errors between 0.02% and 2.54%. Compared with existing methods, the proposed approach shows advantages in point cloud completeness, reconstruction accuracy, and environmental adaptability, providing reliable technical support for intelligent monitoring of bulk material transportation systems. Full article

In this research, I introduce a water surface elevation measurement method that combines point cloud processing techniques and stereo vision cameras. While current vision-based water level measurement techniques focus on laboratory measurements or are based on auxiliary devices such as water rulers, I investigated the feasibility of measuring elevation based on images of the water surface. This research implements a monitoring system on-site, comprising a ZED 2i binocular camera (Stereolabs, San Francisco, CA, USA). First, the uncertainty of the camera is evaluated in a real measurement scenario. Then, the water surface images captured by the binocular camera are stereo matched to obtain parallax maps. Subsequently, the results of the binocular camera calibration are utilized to obtain the 3D point cloud coordinate values of the water surface image. Finally, the horizontal plane equation is solved by the RANSAC algorithm to finalize the height of the camera on the water surface. This approach is particularly significant as it offers a non-contact, shore-based solution that eliminates the need for physical water references, thereby enhancing the adaptability and efficiency of water level monitoring in challenging environments, such as remote or inaccessible areas. Within a measured elevation of 5 m, the water level measurement error is less than 2 cm. Full article

As a critical task in underground coal mining, personnel identification and positioning in fully mechanized mining faces are essential for safety. Yet, complex environmental factors—such as narrow tunnels, heavy dust, and uneven lighting—pose significant challenges to accurate detection. In this paper, we propose a personnel detection network, MSS-YOLO, for fully mechanized mining faces based on YOLOv8. By designing a Multi-Scale Edge Enhancement (MSEE) module and fusing it with the C2f module, the performance of the network for personnel feature extraction under high-dust or long-distance conditions is effectively enhanced. Meanwhile, by designing a Spatial Pyramid Shared Conv (SPSC) module, the redundancy of the model is reduced, which effectively compensates for the problem of the max pooling being prone to losing the characteristics of the personnel at long distances. Finally, the lightweight Shared Convolutional Detection Head (SCDH) ensures real-time detection under limited computational resources. The experimental results show that compared to Faster-RCNN, SSD, YOLOv5s6, YOLOv7-tiny, YOLOv8n, and YOLOv11n, MSS-YOLO achieves AP50 improvements of 4.464%, 10.484%, 3.751%, 4.433%, 3.655%, and 2.188%, respectively, while reducing the inference time by 50.4 ms, 11.9 ms, 3.7 ms, 2.0 ms, 1.2 ms, and 2.3 ms. In addition, MSS-YOLO is combined with the SGBM binocular stereo vision matching algorithm to provide a personnel 3D spatial position solution by using disparity results. The personnel location results show that in the measurement range of 10 m, the position errors in the x-, y-, and z-directions are within 0.170 m, 0.160 m, and 0.200 m, respectively, which proves that MSS-YOLO is able to accurately detect underground personnel in real time and can meet the underground personnel detection and localization requirements. The current limitations lie in the reliance on a calibrated binocular camera and the performance degradation beyond 15 m. Future work will focus on multi-sensor fusion and adaptive distance scaling to enhance practical deployment. Full article

Multispectral imaging plays a key role in crop monitoring. A major challenge, however, is spectral band misalignment, which can hinder accurate plant health assessment by distorting the calculation of vegetation indices. This study presents a novel approach for short-range calibration of a multispectral camera, utilizing stereo vision for precise geometric correction of acquired images. By using multispectral camera lenses as binocular pairs, the sensor acquisition distance was estimated, and an alignment model was developed for distances ranging from 500 mm to 1500 mm. The approach relied on selecting the red band image as a reference, while the remaining bands were treated as moving images. The stereo camera calibration algorithm estimated the target distance, enabling the correction of band misalignment through previously developed models. The alignment models were applied to assess the health status of baby leaf crops (Lactuca sativa cv. Maverik) by analyzing spectral indices correlated with chlorophyll content. The results showed that the stereo vision approach used for distance estimation achieved high accuracy, with average reprojection errors of approximately 0.013 pixels (4.485 × 10⁻⁵ mm). Additionally, the proposed linear model was able to explain reasonably the effect of distance on alignment offsets. The overall performance of the proposed experimental alignment models was satisfactory, with offset errors on the bands less than 3 pixels. Despite the results being not yet sufficiently robust for a fully predictive model of chlorophyll content in plants, the analysis of vegetation indices demonstrated a clear distinction between healthy and unhealthy plants. Full article

Accurate estimation of three-dimensional (3D) information from captured images is essential in numerous computer vision applications. Although binocular stereo vision has been extensively investigated for this task, its reliability is conditioned by the baseline between cameras. A larger baseline improves the resolution of disparity estimation but increases the probability of matching errors. This research presents a reliable method for disparity estimation through progressive baseline increases in multiocular vision. First, a robust rectification method for multiocular images is introduced, satisfying epipolar constraints and minimizing induced distortion. This method can improve rectification error by 25% for binocular images and 80% for multiocular images compared to well-known existing methods. Next, a dense disparity map is estimated by stereo matching from the rectified images with the shortest baseline. Afterwards, the disparity map for the subsequent images with an extended baseline is estimated within a short optimized interval, minimizing the probability of matching errors and further error propagation. This process is iterated until the disparity map for the images with the longest baseline is obtained. The proposed method increases disparity estimation accuracy by 20% for multiocular images compared to a similar existing method. The proposed approach enables accurate scene characterization and spatial point computation from disparity maps with improved resolution. The effectiveness of the proposed method is verified through exhaustive evaluations using well-known multiocular image datasets and physical scenes, achieving superior performance over similar existing methods in terms of objective measures. Full article

Currently, research on peg-in-hole (PiH) compliant assembly is predominantly limited to circular pegs and holes, with insufficient exploration of various complex-shaped PiH tasks. Furthermore, the degree of freedom for rotation about the axis of the circular peg cannot be constrained after assembly, and few studies have covered the complete process from autonomous hole-searching to insertion. Based on the above problems, a novel cross-shaped peg and hole design has been devised. The center coordinates of the cross-hole are obtained during the hole-searching process using the three-dimensional reconstruction theory of a binocular stereo vision camera. During the insertion process, 26 contact states of the cross-peg and the cross-hole were classified, and the mapping relationship between the force-moment sensor and relative errors was established based on a backpropagation (BP) neural network, thus completing the task of autonomous PiH assembly. This system avoids hand-guiding, completely realizes the autonomous assembly task from hole-searching to insertion, and can be replaced by other structures of pegs and holes for repeated assembly after obtaining the accurate relative pose between two assembly platforms, which provides a brand-new and unified solution for complex-shaped PiH assembly. Full article

The advancement of unloading technology in combine harvesting is crucial for the intelligent development of agricultural machinery. Accurately measuring material pile height in transport vehicles is essential, as uneven accumulation can lead to spillage and voids, reducing loading efficiency. Relying solely on manual observation for measuring stack height can decrease harvesting efficiency and pose safety risks due to driver distraction. This research applies binocular vision to agricultural harvesting, proposing a novel method that uses a stereo matching algorithm to measure material pile height during harvesting. By comparing distance measurements taken in both empty and loaded states, the method determines stack height. A linear regression model processes the stack height data, enhancing measurement accuracy. A binocular vision system was established, applying Zhang’s calibration method on the MATLAB (R2019a) platform to correct camera parameters, achieving a calibration error of 0.15 pixels. The study implemented block matching (BM) and semi-global block matching (SGBM) algorithms using the OpenCV (4.8.1) library on the PyCharm (2020.3.5) platform for stereo matching, generating disparity, and pseudo-color maps. Three-dimensional coordinates of key points on the piled material were calculated to measure distances from the vehicle container bottom and material surface to the binocular camera, allowing for the calculation of material pile height. Furthermore, a linear regression model was applied to correct the data, enhancing the accuracy of the measured pile height. The results indicate that by employing binocular stereo vision and stereo matching algorithms, followed by linear regression, this method can accurately calculate material pile height. The average relative error for the BM algorithm was 3.70%, and for the SGBM algorithm, it was 3.35%, both within the acceptable precision range. While the SGBM algorithm was, on average, 46 ms slower than the BM algorithm, both maintained errors under 7% and computation times under 100 ms, meeting the real-time measurement requirements for combine harvesting. In practical operations, this method can effectively measure material pile height in transport vehicles. The choice of matching algorithm should consider container size, material properties, and the balance between measurement time, accuracy, and disparity map completeness. This approach aids in manual adjustment of machinery posture and provides data support for future autonomous master-slave collaborative operations in combine harvesting. Full article

Vessel-mounted cranes operate in complex marine environments, where precise measurement of cargo positions and attitudes is a key technological challenge to ensure operational stability and safety. This study introduces an integrated measurement system that combines vision and inertial sensing technologies, utilizing a stereo camera and two inertial measurement units (IMUs) to capture cargo motion in five degrees of freedom (DOF). By merging data from the stereo camera and IMUs, the system accurately determines the cargo’s position and attitude relative to the camera. The specific methodology is introduced as follows: First, the YOLO model is adopted to identify targets in the image and generate bounding boxes. Then, using the principle of binocular disparity, the depth within the bounding box is calculated to determine the target’s three-dimensional position in the camera coordinate system. Simultaneously, the IMU measures the attitude of the cargo, and a Kalman filter is applied to fuse the data from the two sensors. Experimental results indicate that the system’s measurement errors in the x, y, and z directions are less than 2.58%, 3.35%, and 3.37%, respectively, while errors in the roll and pitch directions are 3.87% and 5.02%. These results demonstrate that the designed measurement system effectively provides the necessary motion information in 5-DOF for vessel-mounted crane control, offering new approaches for pose detection of marine cranes and cargoes. Full article

Despite the considerable success of deep learning methods in stereo matching for binocular images, the generalizability and robustness of these algorithms, particularly under challenging conditions such as occlusions or degraded infrared textures, remain uncertain. This paper presents a novel deep-learning-based depth optimization method that obviates the need for large infrared image datasets and adapts seamlessly to any specific infrared camera. Moreover, this adaptability extends to standard binocular images, allowing the method to work effectively on both infrared and visible light stereo images. We further investigate the role of infrared textures in a deep learning framework, demonstrating their continued utility for stereo matching even in complex lighting environments. To compute the matching cost volume, we apply the multi-scale census transform to the input stereo images. A stacked sand leak subnetwork is subsequently employed to address the matching task. Our approach substantially improves adversarial robustness while maintaining accuracy on comparison with state-of-the-art methods which decrease nearly a half in EPE for quantitative results on widely used autonomous driving datasets. Furthermore, the proposed method exhibits superior generalization capabilities, transitioning from simulated datasets to real-world datasets without the need for fine-tuning. Full article

Camouflage is a natural or artificial process that prevents an object from being detected, while camouflage breaking is a countering process for the identification of the concealed object. We report that a perfectly camouflaged object can be retrieved from the background and detected with stereo-vision-assisted three-dimensional (3D) imaging. The analysis is based on a binocular neuron energy model applied to general 3D settings. We show that a perfectly concealed object with background interference can be retrieved with vision stereoacuity to resolve the hidden structures. The theoretical analysis is further tested and demonstrated with distant natural images taken by a drone camera, processed with a computer and displayed using autostereoscopy. The recovered imaging is presented with the removal of background interference to demonstrate the general applicability for camouflage breaking with stereo imaging and sensing. Full article

This paper addresses image enhancement and 3D reconstruction techniques for dim scenes inside the vacuum chamber of a nuclear fusion reactor. First, an improved multi-scale Retinex low-light image enhancement algorithm with adaptive weights is designed. It can recover image detail information that is not visible in low-light environments, maintaining image clarity and contrast for easy observation. Second, according to the actual needs of target plate defect detection and 3D reconstruction inside the vacuum chamber, a defect reconstruction algorithm based on photometric stereo vision is proposed. To optimize the position of the light source, a light source illumination profile simulation system is designed in this paper to provide an optimized light array for crack detection inside vacuum chambers without the need for extensive experimental testing. Finally, a robotic platform mounted with a binocular stereo-vision camera is constructed and image enhancement and defect reconstruction experiments are performed separately. The results show that the above method can broaden the gray level of low-illumination images and improve the brightness value and contrast. The maximum depth error is less than 24.0% and the maximum width error is less than 15.3%, which achieves the goal of detecting and reconstructing the defects inside the vacuum chamber. Full article

Infrared binocular cameras, leveraging their distinct thermal imaging capabilities, are well-suited for visual measurement and 3D reconstruction in challenging environments. The precision of camera calibration is essential for leveraging the full potential of these infrared cameras. To overcome the limitations of traditional calibration techniques, a novel method for calibrating infrared binocular cameras is introduced. By creating a virtual target plane that closely mimics the geometry of the real target plane, the method refines the feature point coordinates, leading to enhanced precision in infrared camera calibration. The virtual target plane is obtained by inverse projecting the centers of the imaging ellipses, which are estimated at sub-pixel edge, into three-dimensional space, and then optimized using the RANSAC least squares method. Subsequently, the imaging ellipses are inversely projected onto the virtual target plane, where its centers are identified. The corresponding world coordinates of the feature points are then refined through a linear optimization process. These coordinates are reprojected onto the imaging plane, yielding optimized pixel feature points. The calibration procedure is iteratively performed to determine the ultimate set of calibration parameters. The method has been validated through experiments, demonstrating an average reprojection error of less than 0.02 pixels and a significant 24.5% improvement in calibration accuracy over traditional methods. Furthermore, a comprehensive analysis has been conducted to identify the primary sources of calibration error. Ultimately, this achieves an error rate of less than 5% in infrared stereo ranging within a 55-m range. Full article