Search Results (47)

Accurate geospatial data are essential for intelligent transportation systems and automated reporting applications, as location precision directly impacts safety analysis and decision-making. GPS devices are now routinely employed by law enforcement officers when filing vehicle crash reports, yet our investigation reveals that significant data quality issues persist. The high apparent precision of GPS coordinates belies their actual accuracy as we find that approximately 20% of crash sites need correction—results consistent with existing research. To address this challenge, we present a novel credibility scoring and correction algorithm that leverages a state-of-the-art multimodal large language model (LLM) capable of integrated visual and textual reasoning. Our framework synthesizes information from structured coordinates, crash diagrams, and narrative text, employing advanced artificial intelligence techniques for comprehensive geospatial validation. In addition to the LLM, our system incorporates open geospatial data from Overture Maps, an emerging collaborative mapping initiative, to enhance the spatial accuracy and robustness of the validation process. This solution was developed as part of research leading to a patent for autonomous vehicle routing systems that require high-precision crash location data. Applied to a dataset of 5000 crash reports, our approach systematically identifies records with location discrepancies requiring correction. By uniting the latest developments in multimodal AI and open geospatial data, our solution establishes a foundation for intelligent data validation in electronic reporting systems, with broad implications for automated infrastructure management and autonomous vehicle applications. Full article

This study investigates how lighting conditions influence emotional and neural responses in a standardized, simulated museum environment. A multimodal evaluation framework combining subjective and objective measures was used. Thirty-two international students assessed their viewing experiences using 14 semantic differential descriptors, while real-time EEG signals were recorded via the EMOTIV EPOC X device. Spectral energy analyses of the α, β, and θ frequency bands were conducted, and a θα energy ratio combined with γ coefficients was used to model attention and comfort levels. The results indicated that high illuminance (300 lx) and high correlated color temperature (4000 K) significantly enhanced both attention and comfort. Art majors showed higher attention levels than engineering majors during short-term viewing. Among four regression models, the backpropagation (BP) neural network achieved the highest predictive accuracy (R² = 88.65%). These findings provide empirical support for designing culturally inclusive museum lighting and offer neuroscience-informed strategies for promoting the global dissemination of traditional Chinese culture, further supported by retrospective interview insights. Full article

The study’s aim was to evaluate electroretinographic (ERG) alterations in Fragile X syndrome (FXS), FMR1 premutation carriers, and controls, and to explore correlations with peripheral blood FMRP expression levels and behavioral outcomes. ERG recordings were obtained using a handheld device across three stimulus protocols in 43 premutation carriers, 39 individuals with FXS, and 23 controls. Peripheral blood FMRP expression levels were quantified using TR-FRET (Time-Resolved Fluorescence Resonance Energy Transfer). Correlations were assessed with cognitive and behavioral measures including IQ (Intelligence Quotient), ABC_FX (Aberrant Behavior Checklist for Fragile X Syndrome), SNAP-IV (Swanson, Nolan, and Pelham Teacher and Parent Rating Scale), SEQ (Sensory Experiences Questionnaire), ADAMS (Anxiety, Depression, and Mood Scale), and the Vineland III Adaptive Behavior Scale standard score. Significant group differences were observed in multiple ERG parameters, particularly in 2 Hz b-wave amplitude (p = 0.0081), 2 Hz b-wave time to peak (p = 0.0164), 28.3 Hz flash combined amplitude (p = 0.0139), 3.4 Hz red/blue flash b-wave amplitude (p = 0.0026), and PhNR amplitude (p = 0.0026), indicating both outer and inner retinal dysfunction in FXS and premutation groups. Despite high test–retest reliability for ERG (ICC range = 0.71–0.92) and FMRP (ICC = 0.70), no correlation was found between ERG metrics and FMRP or behavioral measures. However, FMRP levels strongly correlated with IQ (ρ = 0.69, p < 0.0001) and inversely with behavioral impairment [ABC_FX (ρ = −0.47, p = 0.0041), SNAP-IV (ρ = −0.48, p = 0.0039), SEQ (ρ = −0.43, p = 0.0146), and the Vineland III standard score (ρ = 0.56, p = 0.0019)]. ERG reveals distinct retinal functional abnormalities in FMR1-related conditions but does not correlate with peripheral FMRP expression levels, highlighting the need for multimodal biomarkers integrating radiological, physiological, behavioral, and molecular measures. Full article

Timely and accurate identification of plant diseases is critical to mitigating crop losses and enhancing yield in precision agriculture. This paper proposes AgriFusionNet, a lightweight and efficient deep learning model designed to diagnose plant diseases using multimodal data sources. The framework integrates RGB and multispectral drone imagery with IoT-based environmental sensor data (e.g., temperature, humidity, soil moisture), recorded over six months across multiple agricultural zones. Built on the EfficientNetV2-B4 backbone, AgriFusionNet incorporates Fused-MBConv blocks and Swish activation to improve gradient flow, capture fine-grained disease patterns, and reduce inference latency. The model was evaluated using a comprehensive dataset composed of real-world and benchmarked samples, showing superior performance with 94.3% classification accuracy, 28.5 ms inference time, and a 30% reduction in model parameters compared to state-of-the-art models such as Vision Transformers and InceptionV4. Extensive comparisons with both traditional machine learning and advanced deep learning methods underscore its robustness, generalization, and suitability for deployment on edge devices. Ablation studies and confusion matrix analyses further confirm its diagnostic precision, even in visually ambiguous cases. The proposed framework offers a scalable, practical solution for real-time crop health monitoring, contributing toward smart and sustainable agricultural ecosystems. Full article

The growing implementation of digital platforms and mobile devices in educational environments has generated the need to explore new approaches for evaluating the learning experience beyond traditional self-reports or instructor presence. In this context, the NPFC-Test dataset was created from an experimental protocol conducted at the Experiential Classroom of the Institute for the Future of Education. The dataset was built by collecting multimodal indicators such as neuronal, physiological, and facial data using a portable EEG headband, a medical-grade biometric bracelet, a high-resolution depth camera, and self-report questionnaires. The participants were exposed to a digital test lasting 20 min, composed of audiovisual stimuli and cognitive challenges, during which synchronized data from all devices were gathered. The dataset includes timestamped records related to emotional valence, arousal, and concentration, offering a valuable resource for multimodal learning analytics (MMLA). The recorded data were processed through calibration procedures, temporal alignment techniques, and emotion recognition models. It is expected that the NPFC-Test dataset will support future studies in human–computer interaction and educational data science by providing structured evidence to analyze cognitive and emotional states in learning processes. In addition, it offers a replicable framework for capturing synchronized biometric and behavioral data in controlled academic settings. Full article

While educational excursions are widely acknowledged to enhance student learning through immersive, real-world experiences, there is limited research on how students can best capture and retain knowledge during such activities. Traditional note-taking methods, such as pen and paper or digital devices, may be inadequate for recording spatial or multimodal information encountered in these dynamic environments. With the emergence of mixed reality (MR) technologies, there is an opportunity to explore spatial, immersive note-taking that aligns with the dynamic nature of field-based learning. This study compares the effectiveness of mixed reality, pen and paper, and digital note-taking during educational excursions. A total of 50 participants in grades 7 through 12 used the Apple Vision Pro headset for mixed reality notes, mobile phones for digital notes, and clipboards paired with a pen and paper for traditional notes. The information encountered was categorised as physical, textual, or video-based. The effectiveness was evaluated through three measures: content extracted and organised in notes, post-activity quizzes on retention and critical thinking, and participant feedback. For physical information, mixed reality significantly improved the content extraction and retention. For textual information, mixed reality yielded more content, but pen and paper outperformed it in terms of organisation. Statistically, all the note-taking methods were equally effective in the remaining aspects. Although mixed reality shows potential to be integrated into educational excursions, participant feedback highlighted discomfort with the headset, suggesting that mixed reality should complement, not replace, traditional approaches. Full article

Continuous monitoring of glucose levels is important for diabetes management and prevention. While traditional glucose monitoring methods are often invasive and expensive, recent approaches using machine learning (ML) models have explored non-invasive alternatives—but many still depend on manually logged food intake and activity, which is burdensome and impractical for everyday use. In this study, we propose a novel approach that eliminates the need for manual input by utilizing only passively collected, automatically recorded multi-modal data from non-invasive wearable sensors. This enables practical and continuous glucose prediction in real-world, free-living environments. We used the BIG IDEAs Lab Glycemic Variability and Wearable Device Data (BIGIDEAs) dataset, which includes approximately 26,000 CGM readings, simultaneous ly collected wearable data, and demographic information. A total of 236 features encompassing physiological, behavioral, circadian, and demographic factors were constructed. Feature selection was conducted using random-forest-based importance analysis to select the most relevant features for model training. We evaluated the effectiveness of various ML regression techniques, including linear regression, ridge regression, random forest regression, and XGBoost regression, in terms of prediction and clinical accuracy. Biological sex, circadian rhythm, behavioral features, and tonic features of electrodermal activity (EDA) emerged as key predictors of glucose levels. Tree-based models outperformed linear models in both prediction and clinical accuracy. The XGBoost (XR) model performed best, achieving an R-squared of 0.73, an RMSE of 11.9 mg/dL, an NRMSE of 0.52 mg/dL, a MARD of 7.1%, and 99.4% of predictions falling within Zones A and B of the Clarke Error Grid. This study demonstrates the potential of combining feature engineering and tree-based ML regression techniques for continuous glucose monitoring using wearable sensors. Full article

As urban traffic complexity increases, realistic multimodal simulation environments are essential for evaluating transportation safety and human behavior. This study introduces a novel multimodal, multi-participant co-simulation framework designed to comprehensively model interactions between drivers, bicyclists, and pedestrians. The framework integrates CARLA, a high-fidelity driving simulator, with PTV Vissim, a widely used microscopic traffic simulation tool. This integration was achieved through the development of custom scripts in Python and C++ that enable real-time data exchange and synchronization between the platforms. Additionally, physiological sensors, including heart rate monitors, electrodermal activity sensors, and EEG devices, were integrated using Lab Streaming Layer to capture physiological responses under different traffic conditions. Three experimental case studies validate the system’s capabilities. In the first, cyclists showed a significant rightward lane shift (from 0.94 m to 1.14 m, $p<0.00001$) and elevated heart rates (69.45 to 72.75 bpm, $p<0.00001$) in response to overtaking vehicles. In the second, pedestrians exhibited more conservative gap acceptance behavior at 50 mph vs. 30 mph (gap acceptance time: 3.70 vs. 3.18 s, $p<0.00001$), with corresponding increases in HR (3.54 bpm vs. 1.91 bpm post-event). In the third case study, mean vehicle speeds recorded during simulated driving were compared with real-world field data along urban corridors, demonstrating strong alignment and validating the system’s ability to reproduce realistic traffic conditions. These findings demonstrate the system’s effectiveness in capturing dynamic, real-time human responses and provide a foundation for advancing human-centered, multimodal traffic research. Full article

We present a single-capture multimodal bright-field (BF) and quantitative phase imaging (QPI) approach that enables the analysis of large, connected, or extended samples, such as confluent cell layers or tissue sections. The proposed imaging concept integrates a fiber-optic Mach–Zehnder interferometer-based off-axis digital holographic microscopy (DHM) with an inverted commercial optical BF microscope. Utilizing 8-bit grayscale dynamic range multiplexing, we simultaneously capture both BF images and digital holograms, which are then demultiplexed numerically via Fourier filtering, phase aberration compensation, and weighted image subtraction procedures. Compared to previous BF-DHM systems, our system avoids synchronization challenges caused by multiple image recording devices, improves acquisition speed, and enhances versatility for fast imaging of large, connected, and rapidly moving samples. Initially, we perform a systematic characterization of the system’s multimodal imaging performance by optimizing numerical as well as coherent and incoherent illumination parameters. Subsequently, the application capabilities are evaluated by multimodal imaging of living cells. The results highlight the potential of single-capture BF-DHM for fast biomedical imaging. Full article

Wearable devices for continuous health monitoring in humans are constantly evolving, yet the signal quality may be improved by optimizing electrode placement. While the commonly used locations to measure electrodermal activity (EDA) are at the fingers or the wrist, alternative locations, such as the torso, need to be considered when applying an integrated multimodal approach of concurrently recording multiple bio-signals, such as the monitoring of visceral pain symptoms like those related to irritable bowel syndrome (IBS). This study aims to quantitatively determine the EDA signal quality at four torso locations (mid-chest, upper abdomen, lower back, and mid-back) in comparison to EDA signals recorded from the fingers. Concurrent EDA signals from five body locations were collected from twenty healthy participants as they completed a Stroop Task and a Cold Pressor task that elicited salient autonomic responses. Mean skin conductance (meanSCL), non-specific skin conductance responses (NS.SCRs), and sympathetic response (TVSymp) were derived from the torso EDA signals and compared with signals from the fingers. Notably, TVSymp recorded from the mid-chest location showed significant changes between baseline and Stroop phase, consistent with the TVSymp recorded from the fingers. A high correlation (0.77–0.83) was also identified between TVSymp recorded from the fingers and three torso locations: mid-chest, upper abdomen, and lower back locations. While the fingertips remain the optimal site for EDA measurement, the mid-chest exhibited the strongest potential as an alternative recording site, with the upper abdomen and lower back also demonstrating promising results. These findings suggest that torso-based EDA measurements have the potential to provide reliable measurement of sympathetic neural activities and may be incorporated into a wearable belt system for multimodal monitoring. Full article

This study presents a methodological framework for predicting soil organic carbon (SOC) using laboratory spectral recordings from a handheld near-infrared (NIR, 1350–2550 nm) device combined with open geospatial data derived from remote sensing sensors related to landform, climate, and vegetation. Initial experiments proved the superiority of convolutional neural networks (CNNs) using only spectral data captured by the low-cost spectral devices reaching an $R^2$ of 0.62, RMSE of 0.31 log-SOC, and an RPIQ of 1.87. Furthermore, the incorporation of geo-covariates with Neo-Spectra data substantially enhanced predictive capabilities, outperforming existing approaches. Although the CNN-derived spectral features had the greatest contribution to the model, the geo-covariates that were most informative to the model were primarily the rainfall data, the valley bottom flatness, and the snow probability. The results demonstrate that hybrid modeling approaches, particularly using CNNs to preprocess all features and fit prediction models with Extreme Gradient Boosting trees, CNN-XGBoost, significantly outperformed traditional machine learning methods, with a notable RMSE reduction, reaching an $R^2$ of 0.72, and an RPIQ of 2.17. The findings of this study highlight the effectiveness of multimodal data integration and hybrid models in enhancing predictive accuracy for SOC assessments. Finally, the application of interpretable techniques elucidated the contributions of various climatic and topographical factors to predictions, as well as spectral information, underscoring the complex interactions affecting SOC variability. Full article

In the domain of animal management, the technology for individual livestock identification is in a state of continuous evolution, encompassing objectives such as precise tracking of animal activities, optimization of vaccination procedures, effective disease control, accurate recording of individual growth, and prevention of theft and fraud. These advancements are pivotal to the efficient and sustainable development of the livestock industry. Recently, visual livestock biometrics have emerged as a highly promising research focus due to their non-invasive nature. This paper aims to comprehensively survey the techniques for individual livestock identification based on computer vision methods. It begins by elucidating the uniqueness of the primary biometric features of livestock, such as facial features, and their critical role in the recognition process. This review systematically overviews the data collection environments and devices used in related research, providing an analysis of the impact of different scenarios on recognition accuracy. Then, the review delves into the analysis and explication of livestock identification methods, based on extant research outcomes, with a focus on the application and trends of advanced technologies such as deep learning. We also highlight the challenges faced in this field, such as data quality and algorithmic efficiency, and introduce the baseline models and innovative solutions developed to address these issues. Finally, potential future research directions are explored, including the investigation of multimodal data fusion techniques, the construction and evaluation of large-scale benchmark datasets, and the application of multi-target tracking and identification technologies in livestock scenarios. Full article

The detection of human activities is an important step in automated systems to understand the context of given situations. It can be useful for applications like healthcare monitoring, smart homes, and energy management systems for buildings. To achieve this, a sufficient data basis is required. The presented dataset contains labeled recordings of 25 different activities of daily living performed individually by 14 participants. The data were captured by five multisensors in supervised sessions in which a participant repeated each activity several times. Flawed recordings were removed, and the different data types were synchronized to provide multi-modal data for each activity instance. Apart from this, the data are presented in raw form, and no further filtering was performed. The dataset comprises ambient audio and vibration, as well as infrared array data, light color and environmental measurements. Overall, 8615 activity instances are included, each captured by the five multisensor devices. These multi-modal and multi-channel data allow various machine learning approaches to the recognition of human activities, for example, federated learning and sensor fusion. Full article

There have been multiple technological advancements that promise to gradually enable devices to measure and record signals with high resolution and accuracy in the domain of brain–computer interfaces (BCIs). Multimodal BCIs have been able to gain significant traction given their potential to enhance signal processing by integrating different recording modalities. In this review, we explore the integration of multiple neuroimaging and neurophysiological modalities, including electroencephalography (EEG), magnetoencephalography (MEG), functional magnetic resonance imaging (fMRI), electrocorticography (ECoG), and single-unit activity (SUA). This multimodal approach leverages the high temporal resolution of EEG and MEG with the spatial precision of fMRI, the invasive yet precise nature of ECoG, and the single-neuron specificity provided by SUA. The paper highlights the advantages of integrating multiple modalities, such as increased accuracy and reliability, and discusses the challenges and limitations of multimodal integration. Furthermore, we explain the data acquisition approaches for each of these modalities. We also demonstrate various software programs that help in extracting, cleaning, and refining the data. We conclude this paper with a discussion on the available literature, highlighting recent advances, challenges, and future directions for each of these modalities. Full article

Researchers have attempted to control robotic hands and prostheses through biosignals but could not match the human hand. Surface electromyography records electrical muscle activity using non-invasive electrodes and has been the primary method in most studies. While surface electromyography-based hand motion decoding shows promise, it has not yet met the requirements for reliable use. Combining different sensing modalities has been shown to improve hand gesture classification accuracy. This work introduces a multimodal bracelet that integrates a 24-channel force myography system with six commercial surface electromyography sensors, each containing a six-axis inertial measurement unit. The device’s functionality was tested by acquiring muscular activity with the proposed device from five participants performing five different gestures in a random order. A random forest model was then used to classify the performed gestures from the acquired signal. The results confirmed the device’s functionality, making it suitable to study sensor fusion for intent detection in future studies. The results showed that combining all modalities yielded the highest classification accuracies across all participants, reaching $92.3\pm 2.6\%$ on average, effectively reducing misclassifications by 37% and 22% compared to using surface electromyography and force myography individually as input signals, respectively. This demonstrates the potential benefits of sensor fusion for more robust and accurate hand gesture classification and paves the way for advanced control of robotic and prosthetic hands. Full article