Search Results (98)

With the advancement of agricultural automation technologies, apple-harvesting robots have gradually become a focus of research. As their “perceptual core,” machine vision systems directly determine picking success rates and operational efficiency. However, existing vision systems still exhibit significant shortcomings in target detection and positioning accuracy in complex orchard environments (e.g., uneven illumination, foliage occlusion, and fruit overlap), which hinders practical applications. This study proposes a visual system for apple-harvesting robots based on improved Mask R-CNN and binocular vision to achieve more precise fruit positioning. The binocular camera (ZED2i) carried by the robot acquires dual-channel apple images. An improved Mask R-CNN is employed to implement instance segmentation of apple targets in binocular images, followed by a template-matching algorithm with parallel epipolar constraints for stereo matching. Four pairs of feature points from corresponding apples in binocular images are selected to calculate disparity and depth. Experimental results demonstrate average coefficients of variation and positioning accuracy of 5.09% and 99.61%, respectively, in binocular positioning. During harvesting operations with a self-designed apple-picking robot, the single-image processing time was 0.36 s, the average single harvesting cycle duration reached 7.7 s, and the comprehensive harvesting success rate achieved 94.3%. This work presents a novel high-precision visual positioning method for apple-harvesting robots. Full article

Pose measurement of non-cooperative targets in space is one of the key technologies in space missions. However, most existing methods simulate well-lit environments and do not consider the degradation of algorithms in low-light conditions. Additionally, due to the limited computing capabilities of space platforms, there is a higher demand for real-time processing of algorithms. This paper proposes a real-time pose measurement method based on binocular vision that is suitable for low-light environments. Firstly, the traditional point feature extraction algorithm is adaptively improved based on lighting conditions, greatly reducing the impact of lighting on the effectiveness of feature point extraction. By combining point feature matching with epipolar constraints, the matching range of feature points is narrowed down to the epipolar line, significantly improving the matching speed and accuracy. Secondly, utilizing the structural information of the spacecraft, line features are introduced and processed in parallel with point features, greatly enhancing the accuracy of pose measurement results. Finally, an adaptive weighted multi-feature pose fusion method based on lighting conditions is introduced to obtain the optimal pose estimation results. Simulation and physical experiment results demonstrate that this method can obtain high-precision target pose information in a real-time and stable manner, both in well-lit and low-light environments. Full article

In this research, I introduce a water surface elevation measurement method that combines point cloud processing techniques and stereo vision cameras. While current vision-based water level measurement techniques focus on laboratory measurements or are based on auxiliary devices such as water rulers, I investigated the feasibility of measuring elevation based on images of the water surface. This research implements a monitoring system on-site, comprising a ZED 2i binocular camera (Stereolabs, San Francisco, CA, USA). First, the uncertainty of the camera is evaluated in a real measurement scenario. Then, the water surface images captured by the binocular camera are stereo matched to obtain parallax maps. Subsequently, the results of the binocular camera calibration are utilized to obtain the 3D point cloud coordinate values of the water surface image. Finally, the horizontal plane equation is solved by the RANSAC algorithm to finalize the height of the camera on the water surface. This approach is particularly significant as it offers a non-contact, shore-based solution that eliminates the need for physical water references, thereby enhancing the adaptability and efficiency of water level monitoring in challenging environments, such as remote or inaccessible areas. Within a measured elevation of 5 m, the water level measurement error is less than 2 cm. Full article

As a critical task in underground coal mining, personnel identification and positioning in fully mechanized mining faces are essential for safety. Yet, complex environmental factors—such as narrow tunnels, heavy dust, and uneven lighting—pose significant challenges to accurate detection. In this paper, we propose a personnel detection network, MSS-YOLO, for fully mechanized mining faces based on YOLOv8. By designing a Multi-Scale Edge Enhancement (MSEE) module and fusing it with the C2f module, the performance of the network for personnel feature extraction under high-dust or long-distance conditions is effectively enhanced. Meanwhile, by designing a Spatial Pyramid Shared Conv (SPSC) module, the redundancy of the model is reduced, which effectively compensates for the problem of the max pooling being prone to losing the characteristics of the personnel at long distances. Finally, the lightweight Shared Convolutional Detection Head (SCDH) ensures real-time detection under limited computational resources. The experimental results show that compared to Faster-RCNN, SSD, YOLOv5s6, YOLOv7-tiny, YOLOv8n, and YOLOv11n, MSS-YOLO achieves AP50 improvements of 4.464%, 10.484%, 3.751%, 4.433%, 3.655%, and 2.188%, respectively, while reducing the inference time by 50.4 ms, 11.9 ms, 3.7 ms, 2.0 ms, 1.2 ms, and 2.3 ms. In addition, MSS-YOLO is combined with the SGBM binocular stereo vision matching algorithm to provide a personnel 3D spatial position solution by using disparity results. The personnel location results show that in the measurement range of 10 m, the position errors in the x-, y-, and z-directions are within 0.170 m, 0.160 m, and 0.200 m, respectively, which proves that MSS-YOLO is able to accurately detect underground personnel in real time and can meet the underground personnel detection and localization requirements. The current limitations lie in the reliance on a calibrated binocular camera and the performance degradation beyond 15 m. Future work will focus on multi-sensor fusion and adaptive distance scaling to enhance practical deployment. Full article

Multispectral imaging plays a key role in crop monitoring. A major challenge, however, is spectral band misalignment, which can hinder accurate plant health assessment by distorting the calculation of vegetation indices. This study presents a novel approach for short-range calibration of a multispectral camera, utilizing stereo vision for precise geometric correction of acquired images. By using multispectral camera lenses as binocular pairs, the sensor acquisition distance was estimated, and an alignment model was developed for distances ranging from 500 mm to 1500 mm. The approach relied on selecting the red band image as a reference, while the remaining bands were treated as moving images. The stereo camera calibration algorithm estimated the target distance, enabling the correction of band misalignment through previously developed models. The alignment models were applied to assess the health status of baby leaf crops (Lactuca sativa cv. Maverik) by analyzing spectral indices correlated with chlorophyll content. The results showed that the stereo vision approach used for distance estimation achieved high accuracy, with average reprojection errors of approximately 0.013 pixels (4.485 × 10⁻⁵ mm). Additionally, the proposed linear model was able to explain reasonably the effect of distance on alignment offsets. The overall performance of the proposed experimental alignment models was satisfactory, with offset errors on the bands less than 3 pixels. Despite the results being not yet sufficiently robust for a fully predictive model of chlorophyll content in plants, the analysis of vegetation indices demonstrated a clear distinction between healthy and unhealthy plants. Full article

With the ever-increasing volume of maritime traffic, the risks of ship navigation are becoming more significant, making the use of advanced multi-source perception strategies and AI technologies indispensable for obtaining information about ship navigation status. In this paper, first, the ship tracking system was optimized using the Bi-YOLO network based on the C2f_BiFormer module and the OC-SORT algorithms. Second, to extract the visual trajectory of the target ship without a reference object, an absolute position estimation method based on binocular stereo vision attitude information was proposed. Then, a perception data fusion framework based on ship spatio-temporal trajectory features (ST-TF) was proposed to match GPS-based ship information with corresponding visual target information. Finally, AR technology was integrated to fuse multi-source perceptual information into the real-world navigation view. Experimental results demonstrate that the proposed method achieves a mAP0.5:0.95 of 79.6% under challenging scenarios such as low resolution, noise interference, and low-light conditions. Moreover, in the presence of the nonlinear motion of the own ship, the average relative position error of target ship visual measurements is maintained below 8%, achieving accurate absolute position estimation without reference objects. Compared to existing navigation assistance, the AR-based navigation assistance system, which utilizes ship ST-TF-based perception data fusion mechanism, enhances ship traffic situational awareness and provides reliable decision-making support to further ensure the safety of ship navigation. Full article

Developing accurate, non-destructive, and automated methods for monitoring the phenotypic traits of rapeseed is crucial for improving yield and quality in modern agriculture. We used a line laser binocular stereo vision technology system to obtain the three-dimensional (3D) point cloud data of different rapeseed varieties (namely Qinyou 7, Zheyouza 108, and Huyou 039) at the seedling stage, and the phenotypic traits of rapeseed were extracted from those point clouds. After pre-processing the rapeseed point clouds with denoising and segmentation, the plant height, leaf length, leaf width, and leaf area of the rapeseed in the seedling stage were extracted by a series of algorithms and were evaluated for accuracy with the manually measured values. The following results were obtained: the R² values for plant height data between the extracted values of the 3D point cloud and the manually measured values reached 0.934, and the RMSE was 0.351 cm. Similarly, the R² values for leaf length of the three kinds of rapeseed were all greater than 0.95, and the RMSEs for Qinyou 7, Zheyouza 108, and Huyou 039 were 0.134 cm, 0.131 cm, and 0.139 cm, respectively. Regarding leaf width, R² was greater than 0.92, and the RMSEs were 0.151 cm, 0.189 cm, and 0.150 cm, respectively. Further, the R² values for leaf area were all greater than 0.98 with RMSEs of 0.296 cm², 0.231 cm² and 0.259 cm², respectively. The results extracted from the 3D point cloud are reliable and have high accuracy. These results demonstrate the potential of 3D point cloud technology for automated, non-destructive phenotypic analysis in rapeseed breeding programs, which can accelerate the development of improved varieties. Full article

Point cloud segmentation is necessary for obtaining highly precise morphological traits in plant phenotyping. Although a huge development has occurred in point cloud segmentation, the segmentation of point clouds from complex plant leaves still remains challenging. Rapeseed leaves are critical in cultivation and breeding, yet traditional two-dimensional imaging is susceptible to reduced segmentation accuracy due to occlusions between plants. The current study proposes the use of binocular stereo-vision technology to obtain three-dimensional (3D) point clouds of rapeseed leaves at the seedling and bolting stages. The point clouds were colorized based on elevation values in order to better process the 3D point cloud data and extract rapeseed phenotypic parameters. Denoising methods were selected based on the source and classification of point cloud noise. However, for ground point clouds, we combined plane fitting with pass-through filtering for denoising, while statistical filtering was used for denoising outliers generated during scanning. We found that, during the seedling stage of rapeseed, a region-growing segmentation method was helpful in finding suitable parameter thresholds for leaf segmentation, and the Locally Convex Connected Patches (LCCP) clustering method was used for leaf segmentation at the bolting stage. Furthermore, the study results show that combining plane fitting with pass-through filtering effectively removes the ground point cloud noise, while statistical filtering successfully denoises outlier noise points generated during scanning. Finally, using the region-growing algorithm during the seedling stage with a normal angle threshold set at 5.0/180.0* M_PI and a curvature threshold set at 1.5 helps to avoid the under-segmentation and over-segmentation issues, achieving complete segmentation of rapeseed seedling leaves, while the LCCP clustering method fully segments rapeseed leaves at the bolting stage. The proposed method provides insights to improve the accuracy of subsequent point cloud phenotypic parameter extraction, such as rapeseed leaf area, and is beneficial for the 3D reconstruction of rapeseed. Full article

Icing on wind turbine blades in cold and humid weather has become a detrimental factor limiting their efficient operation, and traditional methods for detecting blade icing have various limitations. Therefore, this paper proposes a non-contact ice volume estimation method based on binocular vision and improved image processing algorithms. The method employs a stereo matching algorithm that combines dynamic windows, multi-feature fusion, and reordering, integrating gradient, color, and other information to generate matching costs. It utilizes a cross-based support region for cost aggregation and generates the final disparity map through a Winner-Take-All (WTA) strategy and multi-step optimization. Subsequently, combining image processing techniques and three-dimensional reconstruction methods, the geometric shape of the ice is modeled, and its volume is estimated using numerical integration methods. Experimental results on volume estimation show that for ice blocks with regular shapes, the errors between the measured and actual volumes are 5.28%, 8.35%, and 4.85%, respectively; for simulated icing on wind turbine blades, the errors are 5.06%, 6.45%, and 9.54%, respectively. The results indicate that the volume measurement errors under various conditions are all within 10%, meeting the experimental accuracy requirements for measuring the volume of ice accumulation on wind turbine blades. This method provides an accurate and efficient solution for detecting blade icing without the need to modify the blades, making it suitable for wind turbines already in operation. However, in practical applications, it may be necessary to consider the impact of illumination and environmental changes on visual measurements. Full article

Accurate estimation of three-dimensional (3D) information from captured images is essential in numerous computer vision applications. Although binocular stereo vision has been extensively investigated for this task, its reliability is conditioned by the baseline between cameras. A larger baseline improves the resolution of disparity estimation but increases the probability of matching errors. This research presents a reliable method for disparity estimation through progressive baseline increases in multiocular vision. First, a robust rectification method for multiocular images is introduced, satisfying epipolar constraints and minimizing induced distortion. This method can improve rectification error by 25% for binocular images and 80% for multiocular images compared to well-known existing methods. Next, a dense disparity map is estimated by stereo matching from the rectified images with the shortest baseline. Afterwards, the disparity map for the subsequent images with an extended baseline is estimated within a short optimized interval, minimizing the probability of matching errors and further error propagation. This process is iterated until the disparity map for the images with the longest baseline is obtained. The proposed method increases disparity estimation accuracy by 20% for multiocular images compared to a similar existing method. The proposed approach enables accurate scene characterization and spatial point computation from disparity maps with improved resolution. The effectiveness of the proposed method is verified through exhaustive evaluations using well-known multiocular image datasets and physical scenes, achieving superior performance over similar existing methods in terms of objective measures. Full article

Binocular vision may serve as a good model for research on awareness. Binocular summation (BS) can be defined as the superiority of binocular over monocular visual performance. Early studies of BS found an improvement of a factor of about 1.4 (empirically), leading to models suggesting a quadratic summation of the two monocular inputs (√2). Neural interaction modulates a target’s visibility within the same eye or between eyes (facilitation or suppression). Recent results indicated that at a closely flanked stimulus, BS is characterized by instability; it relies on the specific order in which the stimulus condition is displayed. Otherwise, BS is stable. These results were revealed in experiments where the tested eye was open, whereas the other eye was occluded (mono-optic glasses, blocked presentation); thus, the participants were aware of the tested eye. Therefore, in this study, we repeated the same experiments but utilized stereoscopic glasses (intermixed at random presentation) to control the monocular and binocular vision, thus potentially eliminating awareness of the tested condition. The stimuli consisted of a central vertically oriented Gabor target and high-contrast Gabor flankers positioned in two configurations (orthogonal or collinear) with target–flanker separations of either two or three wavelengths (λ), presented at four different presentation times (40, 80, 120, and 200 ms). The results indicate that when utilizing stereoscopic glasses and mixing the testing conditions, the BS is normal, raising the possibility that awareness may be involved. Full article

Currently, research on peg-in-hole (PiH) compliant assembly is predominantly limited to circular pegs and holes, with insufficient exploration of various complex-shaped PiH tasks. Furthermore, the degree of freedom for rotation about the axis of the circular peg cannot be constrained after assembly, and few studies have covered the complete process from autonomous hole-searching to insertion. Based on the above problems, a novel cross-shaped peg and hole design has been devised. The center coordinates of the cross-hole are obtained during the hole-searching process using the three-dimensional reconstruction theory of a binocular stereo vision camera. During the insertion process, 26 contact states of the cross-peg and the cross-hole were classified, and the mapping relationship between the force-moment sensor and relative errors was established based on a backpropagation (BP) neural network, thus completing the task of autonomous PiH assembly. This system avoids hand-guiding, completely realizes the autonomous assembly task from hole-searching to insertion, and can be replaced by other structures of pegs and holes for repeated assembly after obtaining the accurate relative pose between two assembly platforms, which provides a brand-new and unified solution for complex-shaped PiH assembly. Full article

The advancement of unloading technology in combine harvesting is crucial for the intelligent development of agricultural machinery. Accurately measuring material pile height in transport vehicles is essential, as uneven accumulation can lead to spillage and voids, reducing loading efficiency. Relying solely on manual observation for measuring stack height can decrease harvesting efficiency and pose safety risks due to driver distraction. This research applies binocular vision to agricultural harvesting, proposing a novel method that uses a stereo matching algorithm to measure material pile height during harvesting. By comparing distance measurements taken in both empty and loaded states, the method determines stack height. A linear regression model processes the stack height data, enhancing measurement accuracy. A binocular vision system was established, applying Zhang’s calibration method on the MATLAB (R2019a) platform to correct camera parameters, achieving a calibration error of 0.15 pixels. The study implemented block matching (BM) and semi-global block matching (SGBM) algorithms using the OpenCV (4.8.1) library on the PyCharm (2020.3.5) platform for stereo matching, generating disparity, and pseudo-color maps. Three-dimensional coordinates of key points on the piled material were calculated to measure distances from the vehicle container bottom and material surface to the binocular camera, allowing for the calculation of material pile height. Furthermore, a linear regression model was applied to correct the data, enhancing the accuracy of the measured pile height. The results indicate that by employing binocular stereo vision and stereo matching algorithms, followed by linear regression, this method can accurately calculate material pile height. The average relative error for the BM algorithm was 3.70%, and for the SGBM algorithm, it was 3.35%, both within the acceptable precision range. While the SGBM algorithm was, on average, 46 ms slower than the BM algorithm, both maintained errors under 7% and computation times under 100 ms, meeting the real-time measurement requirements for combine harvesting. In practical operations, this method can effectively measure material pile height in transport vehicles. The choice of matching algorithm should consider container size, material properties, and the balance between measurement time, accuracy, and disparity map completeness. This approach aids in manual adjustment of machinery posture and provides data support for future autonomous master-slave collaborative operations in combine harvesting. Full article

Vessel-mounted cranes operate in complex marine environments, where precise measurement of cargo positions and attitudes is a key technological challenge to ensure operational stability and safety. This study introduces an integrated measurement system that combines vision and inertial sensing technologies, utilizing a stereo camera and two inertial measurement units (IMUs) to capture cargo motion in five degrees of freedom (DOF). By merging data from the stereo camera and IMUs, the system accurately determines the cargo’s position and attitude relative to the camera. The specific methodology is introduced as follows: First, the YOLO model is adopted to identify targets in the image and generate bounding boxes. Then, using the principle of binocular disparity, the depth within the bounding box is calculated to determine the target’s three-dimensional position in the camera coordinate system. Simultaneously, the IMU measures the attitude of the cargo, and a Kalman filter is applied to fuse the data from the two sensors. Experimental results indicate that the system’s measurement errors in the x, y, and z directions are less than 2.58%, 3.35%, and 3.37%, respectively, while errors in the roll and pitch directions are 3.87% and 5.02%. These results demonstrate that the designed measurement system effectively provides the necessary motion information in 5-DOF for vessel-mounted crane control, offering new approaches for pose detection of marine cranes and cargoes. Full article

Camouflage is a natural or artificial process that prevents an object from being detected, while camouflage breaking is a countering process for the identification of the concealed object. We report that a perfectly camouflaged object can be retrieved from the background and detected with stereo-vision-assisted three-dimensional (3D) imaging. The analysis is based on a binocular neuron energy model applied to general 3D settings. We show that a perfectly concealed object with background interference can be retrieved with vision stereoacuity to resolve the hidden structures. The theoretical analysis is further tested and demonstrated with distant natural images taken by a drone camera, processed with a computer and displayed using autostereoscopy. The recovered imaging is presented with the removal of background interference to demonstrate the general applicability for camouflage breaking with stereo imaging and sensing. Full article