Search Results (776)

Efficient object detection is crucial to real-time monitoring applications such as autonomous driving or security systems. Modern RGB cameras can produce high-resolution images for accurate object detection. However, increased resolution results in increased network latency and power consumption. To minimise this latency, Convolutional Neural Networks (CNNs) often have a resolution limitation, requiring images to be down-sampled before inference, causing significant information loss. Event-based cameras are neuromorphic vision sensors with high temporal resolution, low power consumption, and high dynamic range, making them preferable to regular RGB cameras in many situations. This project proposes the fusion of an event-based camera with an RGB camera to mitigate the trade-off between temporal resolution and accuracy, while minimising power consumption. The cameras are calibrated to create a multi-modal stereo vision system where pixel coordinates can be projected between the event and RGB camera image planes. This calibration is used to project bounding boxes detected by clustering of events into the RGB image plane, thereby cropping each RGB frame instead of down-sampling to meet the requirements of the CNN. Using the Common Objects in Context (COCO) dataset evaluator, the average precision (AP) for the bicycle class in RGB scenes improved from 21.08 to 57.38. Additionally, AP increased across all classes from 37.93 to 46.89. To reduce system latency, a novel object detection approach is proposed where the event camera acts as a region proposal network, and a classification algorithm is run on the proposed regions. This achieved a 78% improvement over baseline. Full article

Precision irrigation plays a crucial role in managing crop production in a sustainable and environmentally friendly manner. This study builds on the results of the GreenWaterDrone project, aiming to estimate, in real time, the actual water requirements of crop fields using the crop water stress index, integrating infrared canopy temperature, air temperature, relative humidity, and thermal and near-infrared imagery. To achieve this, a state-of-the-art aerial micrometeorological station (AMMS), equipped with an infrared thermal sensor, temperature–humidity sensor, and advanced multispectral and thermal cameras is mounted on an unmanned aerial system (UAS), thus minimizing crop field intervention and permanently installed equipment maintenance. Additionally, data from satellite systems and ground micrometeorological stations (GMMS) are integrated to enhance and upscale system results from the local field to the regional level. The research was conducted over two years of pilot testing in the municipality of Trifilia (Peloponnese, Greece) on pilot potato and watermelon crops, which are primary cultivations in the region. Results revealed that empirical irrigation applied to the rhizosphere significantly exceeded crop water needs, with over-irrigation exceeding by 390% the maximum requirement in the case of potato. Furthermore, correlations between high-resolution remote and proximal sensors were strong, while associations with coarser Landsat 8 satellite data, to upscale the local pilot field experimental results, were moderate. By applying a comprehensive model for upscaling pilot field results, to the overall Trifilia region, project findings proved adequate for supporting sustainable irrigation planning through simulation scenarios. The results of this study, in the context of the overall services introduced by the project, provide valuable insights for farmers, agricultural scientists, and local/regional authorities and stakeholders, facilitating improved regional water management and sustainable agricultural policies. Full article

This paper examines the integration of markerless augmented reality (AR) within the da Vinci Surgical Robot, utilizing artificial intelligence (AI) for improved precision. The main challenge in creating AR for these systems is the small size (5 mm diameter) of the cameras used. Traditional camera-calibration approaches produce significant errors when used for miniature cameras. Further, the use of external markers can be obstructive and inaccurate in dynamic surgical environments. The study focuses on overcoming these limitations of traditional AR methods by employing advanced neural networks for camera calibration and real-time image processing. We demonstrate the use of a dense neural network to reduce the total projection error by directly learning the mapping of a 3D point to a 2D image plane. The results show a median error of 7 pixels (1.4 mm) when using a neural network, as compared to an error of 50 pixels (10 mm) when using a more traditional approach involving camera calibration and robot kinematics. This approach not only enhances the accuracy of AR for surgical procedures but also offers a more seamless integration with existing robotic platforms. These research findings underscore the potential of AI in revolutionizing AR applications in medical robotics and other teleoperated systems, promising efficient and safer interventions. Full article

It is projected that the world population will quadruple over the next century, and to meet future food demands, agricultural production will need to increase by 70%. Therefore, there has been a transition from traditional farming methods to autonomous modern agriculture. One such modern technique is aeroponic farming, in which plants are grown without soil under controlled and hygienic conditions. In aeroponic farming, plants are significantly less affected by climatic conditions, infectious diseases, and biotic and abiotic stresses, such as pest infestations. Additionally, this method can reduce water, nutrient, and pesticide usage by 98%, 60%, and 100%, respectively, while increasing the yield by 45–75% compared to traditional farming. In this study, a three-dimensional industrial design of an innovative aeroponic plant growth chamber was presented for use by individuals, researchers, and professional growers. The proposed chamber design is modular and open to further innovation. Unlike existing chambers, it includes load cells that enable real-time monitoring of the fresh weight of the plant. Furthermore, cameras were integrated into the chamber to track plant growth and changes over time and weight. Additionally, RGB power LEDs were placed on the inner ceiling of the chamber to provide an optimal lighting intensity and spectrum based on the cultivated plant species. A customizable chamber design was introduced, allowing users to determine the growing tray and nutrient nozzles according to the type and quantity of plants. Finally, system models were developed for temperature control of the chamber. Temperature control was implemented using a proportional-integral-derivative controller optimized with particle swarm optimization, radial movement optimization, differential evolution, and mayfly optimization algorithms for the gain parameters. The simulation results indicate that the temperatures of the growing and feeding chambers in the cabinet reached a steady state within 260 s, with an offset error of no more than 0.5 °C. This result demonstrates the accuracy of the derived model and the effectiveness of the optimized controllers. Full article

The determination of precise and reliable interior (IO) and relative (RO) orientation parameters for thermal infrared (TIR) cameras is critical for their subsequent use in photogrammetric processes. Although 2D calibration boards have become the predominant approach for TIR geometric calibration, these targets are susceptible to projective coupling and often introduce error through manual construction methods, necessitating the development of 3D targets tailored to TIR geometric calibration. Therefore, this paper evaluates TIR geometric calibration results obtained from 2D board and 3D field calibration approaches, documenting the construction, observation, and calculation of IO and RO parameters. This includes a comparative analysis of values derived from three popular commercial software packages commonly used for geometric calibration: MathWorks’ MATLAB, Agisoft Metashape, and Photometrix’s Australis. Furthermore, to assess the validity of derived parameters, two InfraRed Thermography 3D-Data Fusion (IRT-3DDF) methods are developed to model historic building façades and medieval frescoes. The results demonstrate the success of the proposed 3D field calibration targets for the calculation of both IO and RO parameters tailored to photogrammetric data fusion. Additionally, a novel combined TIR-RGB bundle block adjustment approach demonstrates the success of applying ‘out-of-the-box’ deep-learning neural networks for multi-modal image matching and thermal modelling. Considerations for the development of TIR geometric calibration approaches and the evolution of proposed IRT-3DDF methods are provided for future work. Full article

The health of road tunnel linings directly impacts traffic safety and requires regular inspection. Appearance defects on tunnel linings can be measured through images scanned by cameras mounted on a car to avoid disrupting traffic. Existing tunnel lining mobile scanning methods often fail in image stitching due to the lack of corresponding feature points in the lining images, or require complex, time-consuming algorithms to eliminate stitching seams caused by the same issue. This paper proposes a mobile scanning method aided by collimated lasers, which uses lasers as corresponding points to assist with image stitching to address the problems. Additionally, the lasers serve as structured light, enabling the measurement of image projection relationships. An inspection car was developed based on this method for the experiment. To ensure operational flexibility, a single checkerboard was used to calibrate the system, including estimating the poses of lasers and cameras, and a Laplace kernel-based algorithm was developed to guarantee the calibration accuracy. Experiments show that the performance of this algorithm exceeds that of other benchmark algorithms, and the proposed method produces nearly seamless, measurable tunnel lining images, demonstrating its feasibility. Full article

Sound source localization represents one of the major challenges for soundscapes due to the dynamicity of a large variety of signals. Many applications are found related to ecosystems to study the migration process of birds and animals other than other terrestrial environments to survey wildlife. Other applications on sound recording are supported by sensors to detect animal movement. This paper deals with the immersive 3D soundscape by using a multi-channel spherical microphone probe, in combination with a 360° camera. The soundscape has been carried out in three Italian squares across the city of Parma. The acoustic maps obtained from the data processing detect the directivity of dynamic sound sources as typical of an urban environment. The analysis of the objective environmental parameters (like loudness, roughness, sharpness, and prominence) was conducted alongside the investigations on the historical importance of Italian squares as places for social inclusivity. A dedicated listening playback is provided by the AGORA project with a portable listening room characterized by modular unit of soundbars. Full article

In target detection tasks equipped with depth sensors, it is crucial to adopt the point cloud pretreatment process, which is directly related to the quality of the obtained three-dimensional model of the target. However, there are few methods that can be combined with common preprocessing methods to quickly process ToF camera output. In real-life experiments, the common method is to adopt multiple types of preprocessing methods and adjust parameters separately. We proposed PcBD, a method that integrates outlier removal, boundary detection, and smooth sliders. PcBD does not limit the number of input points, and can remove outliers and predict smooth projection boundaries at one time while ensuring that the total number of points remains unchanged. We also introduced Bound57, a benchmark dataset that contains point clouds with synthetic noise, outliers, and projected boundary labels. Experimental results show that PcBD performs significantly better than state-of-the-art methods in various de-noising and boundary detection tasks. Full article

To address the challenges of dynamic object interference and redundant information representation in map construction for indoor dynamic environments, this paper proposes an indoor dynamic environment mapping method based on semantic fusion and hierarchical filtering. First, prior dynamic object masks are obtained using the YOLOv8 model, and geometric constraints between prior static objects and dynamic regions are introduced to identify non-prior dynamic objects, thereby eliminating all dynamic features (both prior and non-prior). Second, an initial semantic point cloud map is constructed by integrating prior static features from a semantic segmentation network with pose estimates from an RGB-D camera. Dynamic noise is then removed using statistical outlier removal (SOR) filtering, while voxel filtering optimizes point cloud density, generating a compact yet texture-rich semantic dense point cloud map with minimal dynamic artifacts. Subsequently, a multi-resolution semantic octree map is built using a recursive spatial partitioning algorithm. Finally, point cloud poses are corrected via Transform Frame (TF) transformation, and a 2D traversability grid map is generated using passthrough filtering and grid projection. Experimental results demonstrate that the proposed method constructs multi-level semantic maps with rich information, clear structure, and high reliability in indoor dynamic scenarios. Additionally, the map file size is compressed by 50–80%, significantly enhancing the reliability of mobile robot navigation and the efficiency of path planning. Full article

LiDAR and camera sensors are widely utilized in autonomous vehicles (AVs) and robotics due to their complementary sensing capabilities—LiDAR provides precise depth information, while cameras capture rich visual context. However, effective multi-sensor fusion remains challenging due to discrepancies in resolution, data format, and viewpoint. In this paper, we propose a robust pattern matching algorithm that leverages singular value decomposition (SVD) and gradient descent (GD) to align geometric features—such as object contours and convex hulls—across LiDAR and camera modalities. Unlike traditional calibration methods that require manual targets, our approach is targetless, extracting matched patterns from projected LiDAR point clouds and 2D image segments. The algorithm computes the optimal transformation matrix between sensors, correcting misalignments in rotation, translation, and scale. Experimental results on a vehicle-mounted sensing platform demonstrate an alignment accuracy improvement of up to 85%, with the final projection error reduced to less than 1 pixel. This pattern-based SVD-GD framework offers a practical solution for maintaining reliable cross-sensor alignment under calibration drift, enabling real-time perception systems to operate robustly without recalibration. This method provides a practical solution for maintaining reliable sensor fusion in autonomous driving applications subject to long-term calibration drift. Full article

Recent advancements in cooperative 3D object detection have demonstrated significant potential for enhancing autonomous driving by integrating roadside infrastructure data. However, deploying comprehensive LiDAR-based cooperative perception systems remains prohibitively expensive and requires precisely annotated 3D data to function robustly. This paper proposes an improved multi-modal method integrating LiDAR-based shape references into a previously mono-camera-based semantic vertex reconstruction framework to enable robust and cost-effective monocular and cooperative pose estimation after the reconstruction. A novel camera–LiDAR loss function that combines re-projection loss from a multi-view camera system alongside LiDAR shape constraints is proposed. Experimental evaluations conducted on the Argoverse dataset and real-world experiments demonstrate significantly improved shape reconstruction robustness and accuracy, thereby improving pose estimation performance. The effectiveness of the algorithm is proven through a real-world smart valet parking application, which is evaluated in our university parking area with real vehicles. Our approach allows accurate 6DOF pose estimation using an inexpensive IP camera without requiring context-specific training, thereby advancing the state of the art in monocular and cooperative image-based vehicle localization. Full article

In fields such as military reconnaissance, forest fire prevention, and autonomous driving at night, there is an urgent need for high-precision three-dimensional reconstruction in low-light or night environments. The acquisition of remote sensing data by RGB cameras relies on external light, resulting in a significant decline in image quality and making it difficult to meet the task requirements. The method based on lidar has poor imaging effects in rainy and foggy weather, close-range scenes, and scenarios requiring thermal imaging data. In contrast, infrared cameras can effectively overcome this challenge because their imaging mechanisms are different from those of RGB cameras and lidar. However, the research on three-dimensional scene reconstruction of infrared images is relatively immature, especially in the field of infrared binocular stereo matching. There are two main challenges given this situation: first, there is a lack of a dataset specifically for infrared binocular stereo matching; second, the lack of texture information in infrared images causes a limit in the extension of the RGB method to the infrared reconstruction problem. To solve these problems, this study begins with the construction of an infrared binocular stereo matching dataset and then proposes an innovative perspective projection positional encoding-based transformer method to complete the infrared binocular stereo matching task. In this paper, a stereo matching network combined with transformer and cost volume is constructed. The existing work in the positional encoding of the transformer usually uses a parallel projection model to simplify the calculation. Our method is based on the actual perspective projection model so that each pixel is associated with a different projection ray. It effectively solves the problem of feature extraction and matching caused by insufficient texture information in infrared images and significantly improves matching accuracy. We conducted experiments based on the infrared binocular stereo matching dataset proposed in this paper. Experiments demonstrated the effectiveness of the proposed method. Full article

Wide-field-of-view imaging using remote-sensing cameras is of great significance in the study of polar environments. However, because of the drastic change in the direction of Earth’s rotation velocity near the polar regions, image-shift analysis and image quality changes in polar images by large wide-space time-delayed integration (TDI) cameras are poorly understood. Therefore, in this study, a novel velocity projection method was used to obtain a mathematical model of the image-shift velocity field. A quantitative analysis of the simulation showed that the anisotropy of the instantaneous image-shift velocity field varied significantly from low to high latitudes, and rapidly decreased to zero at a very low instantaneous point when there was no anisotropy. After correcting for the camera travelling frequency and bias angle, the value of the modulation transfer function at the edges decreased by less than 5% at 196 levels of integration. Thus, a theoretical basis was provided for using a large wide-space TDI camera to photograph high latitudes. The findings of this study provide a theoretical reference for the current large field-of-view space cameras to obtain high-latitude target information for the edge fuzzy degradation problem. Full article

Plant 3D reconstruction plays a critical role in precision agriculture and plant growth monitoring, yet it faces challenges such as complex background interference, difficulties in capturing intricate plant structures, and a slow reconstruction speed. In this study, we propose PlantSamGaussianReconstruction (PSGR), a novel method that integrates Grounding SAM with 3D Gaussian Splatting (3DGS) techniques. PSGR employs Grounding DINO and SAM for accurate plant–background segmentation, utilizes algorithms such as Scale-Invariant Feature Transform (SIFT) for camera pose estimation and sparse point cloud generation, and leverages 3DGS for plant reconstruction. Furthermore, a 3D–2D projection-guided optimization strategy is introduced to enhance segmentation precision. The experimental results of various multi-view plant image datasets demonstrate that PSGR effectively removes background noise under diverse environments, accurately captures plant details, and achieves peak signal-to-noise ratio (PSNR) values exceeding 30 in most scenarios, outperforming the original 3DGS approach. Moreover, PSGR reduces training time by up to 26.9%, significantly improving reconstruction efficiency. These results suggest that PSGR is an efficient, scalable, and high-precision solution for plant modeling. Full article

Fire is a destructive hazard impacting residential, industrial, and forested environments. Once ignited, fire becomes difficult to control, and recovery efforts are often extensive. Therefore, early detection is critical for effective firefighting. This study presents a mobile robotic system designed for early fire detection, integrating a Raspberry Pi, RGB (red, green and blue), and night vision-NIR (near infrared reflectance) cameras. A four-stage hybrid-cascade machine learning model was developed by combining state-of-the-art (SotA) models separately trained on RGB and NIR images. The system accounts for both daytime and nighttime conditions, achieving F1 scores of 96.7% and 95.9%, respectively, on labeled fire/non-fire datasets. Unlike previous single-stage or two-stage vision pipelines, our work delivers a lightweight four-stage hybrid cascade that jointly fuses RGB and NIR imagery, integrates temporal consistency via ConvLSTM, and projects a robot-centric “safe-approach distance” in real time, establishing a novel edge-level solution for mobile robotic fire detection. Based on real-life test results, the robotic system with this new hybrid-cascade model could detect the fire source from a safe distance of 500 mm and with notably higher accuracy compared to structures with other models. Full article