Search Results (18)

Nuclear Fuel Reprocessing literature contains critical experimental parameters, safety information, theoretical relations, and process data that are highly heterogeneous and subject to strict logical constraints. Manually interpreting complex charts and handling tedious database schema mappings imposes a high cognitive load on experts. Although existing Large Multimodal Models (LMMs) have demonstrated strong potential in information extraction, they often face engineering bottlenecks—such as poor structural compliance and a tendency to confuse entity logic—when dealing with domain databases containing complex foreign key constraints. To address this, we propose GenForge, a schema-aware extraction framework. By taking the target database schema as an explicit constraint, GenForge achieves automatic task decomposition and formatting self-correction via a “Generation–Execution–Reflection–Reforging” iterative loop. Additionally, a Local ID mechanism is introduced to ensure data lineage consistency. We evaluated GenForge on four internal evaluation corpora from nuclear fuel reprocessing literature, each aligned with a distinct database schema: Safety Event and Causal Context Extraction Schema, Property-Condition Data Extraction Schema, Model-Parameter Association Schema, and Process Topology and Stream Mapping Schema. On the independent test set, GenForge achieved 88.0% precision, 83.0% recall, and a 98.6% Schema Compliance Rate (SCR). These results indicate that GenForge, as an expert-assisted framework, reduces the need for manual JSON debugging and supports practical schema-constrained knowledge extraction under four schema-specific evaluation settings within the Nuclear Fuel Reprocessing domain. Full article

Visual quality assessment is entering a new frontier as media evolve from static images to temporally dynamic videos and 3D content. These visual signals are typically captured by sensing devices such as cameras and depth sensors, whose acquisition characteristics significantly influence perceptual quality. Traditional quality models, including distortion-centric and regression-based approaches, perform well on conventional degradations but struggle to evaluate higher-level attributes such as semantic plausibility and structural coherence in modern AI-generated and multimodal scenarios. The emergence of large multimodal models (LMMs), including vision–language models (VLMs) and multimodal large language models (MLLMs), reshapes the evaluation paradigm by enabling semantic grounding, instruction-driven assessment, and explainable reasoning. This survey presents a unified perspective on visual quality assessment for sensor-captured visual data across image, video, and 3D modalities. We review conventional deep learning approaches and recent LMM-based methods, highlighting how multimodal fusion and language-conditioned reasoning transform quality assessment from scalar prediction to perceptual intelligence. Finally, we discuss key challenges and future opportunities for building efficient, robust, and sensor-aware visual quality assessment systems. Full article

High-fidelity simulation testing is a critical component in ensuring the safety and reliability of autonomous driving systems. However, traditional methods for constructing simulation scenarios face two major bottlenecks. First, acquiring realistic road network topologies that adhere to physical and traffic rules is expensive. Second, the manual placement of scenario elements (e.g., vehicles and pedestrians) is a time-consuming and labor-intensive process, which struggles to meet the demands of large-scale and diverse testing. To address these challenges, this paper proposes an efficient and automated simulation scenario generation method and toolchain. The proposed approach begins by extracting road network topologies from real-world data sources (e.g., open map datasets) and then uses specialized tools, such as RoadRunner, to automatically assign traffic semantics and rules. The key innovation lies in leveraging the powerful image-text understanding capabilities of large multimodal models (LMMs) to analyze road network images and textual descriptions, generating a semantic heatmap that represents the spatial distribution probabilities of scenario elements. This heatmap guides the procedural content generation (PCG) process, enabling the intelligent and scalable deployment of traffic participants. Experimental results demonstrate that the proposed method can efficiently generate large-scale, high-fidelity, and cost-effective simulation scenarios. The generated scenarios not only maintain realism in topology and traffic rules but also feature rich perception and interaction capabilities. Furthermore, based on this method, we have constructed and released a novel simulation dataset tailored for training perception algorithms, further validating the practical value and advancement of the toolchain. Full article

Low-light image enhancement (LLIE) remains challenging due to severe degradation of high-frequency structures and semantic ambiguity under extreme darkness. Although existing methods achieve satisfactory brightness recovery, they often suffer from structural inconsistency and semantic drift, as diverse scenes are typically processed with uniform enhancement strategies or static text prompts. To address these issues, we propose a Multi-Modal Structural and Semantic-Adaptive Network (MSSA-Net) under a structure-anchored paradigm. First, we design a Multi-Scale Self-Refinement Block (MSRB) to enhance degraded visible representations through multi-scale feature extraction and progressive refinement. Meanwhile, a pseudo-infrared structural prior derived from the input image is introduced to provide noise-insensitive geometric cues. These cues are extracted via a Structure-Guided Cross-Attention (SGCA) module to produce structure-dominant features. The refined visible features and structural features are then adaptively integrated through an adaptive residual fusion (ARF) module to achieve balanced restoration. Furthermore, we develop a Large Multi-modal Model (LMM)-Driven Scene-Adaptive Attention mechanism that generates instance-aware scene tags from a coarse preview and injects semantic embeddings into visual features. Extensive experiments demonstrate that MSSA-Net improves structural fidelity, brightness recovery, and semantic naturalness across multiple benchmarks. Full article

Background: Accurate dietary assessment is vital for preventing malnutrition in aging populations, particularly in home-care settings. Although Large Multimodal Models (LMMs) for nutrient estimation are evolving, their nutrient-specific accuracy requires rigorous validation. Methods: Fifteen standardized hospital meals were photographed under controlled conditions (90-degree angle, 500 lux). Ground truth values were determined by direct weighing. Estimates for energy and macronutrients were performed by 10 registered dietitians (RDs) and 10 AI models (including ChatGPT-4o and Gemini 1.5 Pro). Accuracy was assessed using Pearson’s correlation, Mean Absolute Error (MAE), and Bland–Altman analysis to quantify systematic bias. Results: For energy and carbohydrates, RDs and top-performing AI models (notably ChatGPT-4o and Gemini 1.5 Pro) demonstrated practical accuracy (r > 0.8, frequently within ±10% range). However, accuracy for protein and lipids was significantly lower across all AI models. Specifically, all AI models exhibited a substantial systematic overestimation of lipids (Mean Bias > +20%, p < 0.01), highlighting a critical “invisible nutrient” bias. Conclusions: Current AI tools show potential for caloric and carbohydrate monitoring but struggle with lipid and protein density. These findings emphasize the need for human–AI collaboration (“human-in-the-loop”) and the integration of cooking metadata to improve clinical utility in geriatric nutrition. Full article

Precise drug target discovery is pivotal to mitigating the escalating costs and high attrition rates that characterize pharmaceutical research and development. Given that traditional single-omics methods often fail to elucidate the systemic complexity of human diseases, deep learning (DL)-enabled multi-omics integration has emerged as a transformative frontier. This review systematically summarizes the advancements in DL-driven multi-omics integration for drug target discovery. First, the multi-omics data foundation and integration strategies are delineated, followed by an exploration of the DL architectures utilized for processing such data. Subsequently, the efficacy of DL-driven multi-omics integration is examined regarding the identification of novel disease drivers, prediction of synthetic lethality interactions, and prioritization of therapeutic targets. Finally, addressing persistent challenges related to data sparsity, model interpretability, and target druggability and validation hurdles, emerging opportunities driven by Generative AI, Large Multimodal Models (LMMs), Explainable AI (XAI), and multidimensional feasibility assessment frameworks are discussed in the context of advancing precision medicine. Full article

Wildfires are environmental hazards with severe ecological, social, and economic impacts. Wildfires devastate ecosystems, communities, and economies worldwide, with rising frequency and intensity driven by climate change, human activity, and environmental shifts. Analyzing wildfire insights such as detection, predictive patterns, and risk assessment enables proactive response and long-term prevention. However, most of the existing approaches have been focused on isolated processing of data, making it challenging to orchestrate cross-modal reasoning and transparency. This study proposed a novel orchestrator-based multi-agent system (MAS), with the aim of transforming multimodal environmental data into actionable intelligence for decision making. We designed a framework to utilize Large Multimodal Models (LMMs) augmented by structured prompt engineering and specialized Retrieval-Augmented Generation (RAG) pipelines to enable transparent and context-aware reasoning, providing a cutting-edge Visual Question Answering (VQA) system. It ingests diverse inputs like satellite imagery, sensor readings, weather data, and ground footage and then answers user queries. Validated by several public datasets, the system achieved a precision of 0.797 and an F1-score of 0.736. Thus, powered by Agentic AI, the proposed, human-centric solution for wildfire management, empowers firefighters, governments, and researchers to mitigate threats effectively. Full article

Background/Objectives: AI models such as ChatGPT-4 have shown strong performance in dermatology; however, the diagnostic value of AI-generated clinical image descriptions remains underexplored. This study assesses whether ChatGPT-4’s image descriptions can support accurate dermatologic diagnosis and evaluates their potential integration into the Electronic Medical Record (EMR) system. Materials & Methods: In this Exploratory cross-sectional study, we analyzed images and descriptions from teledermatology consultations conducted between December 2023 and February 2024. ChatGPT-4 generated clinical descriptions for each image, which two senior dermatologists then used to formulate differential diagnoses. Diagnoses based on ChatGPT-4’s output were compared to those derived from the original clinical notes written by teledermatologists. Concordance was categorized as Top1 (exact match), Top3 (correct within top three), Partial, or No match. Results: The study included 154 image descriptions from 67 male and 87 female patients, aged 0 to 93 years. ChatGPT-4 descriptions averaged 74.3 ± 33.1 words, compared to 7.9 ± 3.0 words for teledermatologists. At least one of the two dermatologists achieved a Top 3 concordance rate of 82.5% using ChatGPT-4’s descriptions and 85.3% with teledermatologist descriptions. Conclusions: Preliminary findings highlight the potential integration of ChatGPT-4-generated descriptions into EMRs to enhance documentation. Although AI descriptions were longer, they did not enhance diagnostic accuracy, and expert validation remained essential. Full article

Large Language Models (LLMs) and Multimodal Models (LMMs) are significantly influencing scientific research, including archaeology—a discipline dealing with uniquely complex, multimodal data. This comprehensive review systematically examines recent (2023+) applications of LLMs/LMMs in archaeology, covering ancient texts, artifacts, field data, and knowledge graphs. While current applications are often exploratory and fragmented, they demonstrate substantial potential for addressing long-standing archaeological challenges such as data heterogeneity, knowledge integration, and interpretive complexity. We argue that archaeology serves as a valuable “proving ground” for next-generation AI technologies due to its distinctive data characteristics (multimodal heterogeneity, sparsity, uncertainty) and high demands for robust knowledge reasoning and interpretability. This review critically analyzes technical approaches including fine-tuning methods (LoRA, PEFT), retrieval-augmented generation (RAG), and recent advances (RAFT, LongRoPE, Phi-3-mini) that enable efficient local deployment. We examine data, knowledge, technological, and ethical challenges, distinguishing between issues generic to machine learning and those specific to Transformer-based LLMs in heritage contexts. This review concludes by identifying prioritized future research directions for integrated “AI Archaeology,” emphasizing responsible AI principles and human-in-the-loop frameworks. This study offers cross-disciplinary insights for fostering deep, synergistic, and ethically sound AI integration within archaeological science and broader cultural heritage applications. Full article

Remote sensing visual question answering (RSVQA) involves interpreting complex geospatial information captured by satellite imagery to answer natural language questions, making it a vital tool for observing and analyzing Earth’s surface without direct contact. Although numerous studies have addressed RSVQA, most have focused primarily on answer accuracy, often overlooking the underlying reasoning capabilities required to interpret spatial and contextual cues in satellite imagery. To address this gap, this study presents a comprehensive evaluation of four large multimodal models (LMMs) as follows: GPT-4o, Grok 3, Gemini 2.5 Pro, and Claude 3.7 Sonnet. We used a curated subset of the EarthVQA dataset consisting of 100 rural images with 29 question–answer pairs each and 100 urban images with 42 pairs each. We developed the following three task-specific frameworks: (1) Zero-GeoVision, which employs zero-shot prompting with problem-specific prompts that elicit direct answers from the pretrained knowledge base without fine-tuning; (2) CoT-GeoReason, which enhances the knowledge base with chain-of-thought prompting, guiding it through explicit steps of feature detection, spatial analysis, and answer synthesis; and (3) Self-GeoSense, which extends this approach by stochastically decoding five independent reasoning chains for each remote sensing question. Rather than merging these chains, it counts the final answers, selects the majority choice, and returns a single complete reasoning chain whose conclusion aligns with that majority. Additionally, we designed the Geo-Judge framework to employ a two-stage evaluation process. In Stage 1, a GPT-4o-mini-based LMM judge assesses reasoning coherence and answer correctness using the input image, task type, reasoning steps, generated model answer, and ground truth. In Stage 2, blinded human experts independently review the LMM’s reasoning and answer, providing unbiased validation through careful reassessment. Focusing on Self-GeoSense with Grok 3, this framework achieves superior performance with 94.69% accuracy in Basic Judging, 93.18% in Basic Counting, 89.42% in Reasoning-Based Judging, 83.29% in Reasoning-Based Counting, 77.64% in Object Situation Analysis, and 65.29% in Comprehensive Analysis, alongside RMSE values of 0.9102 in Basic Counting and 1.0551 in Reasoning-Based Counting. Full article

In recent years, large multimodal models (LMMs), such as ChatGPT 4o and DeepSeek R1—artificial intelligence systems capable of multimodal (e.g., image and text) human–computer interaction—have gained traction in industrial and civil engineering applications. Concurrently, insufficient real-world drone-view data (specifically close-distance, high-resolution imagery) for civil engineering scenarios has heightened the importance of artificially generated content (AIGC) or synthetic data as supplementary inputs. AIGC is typically produced via text-to-image generative models (e.g., Stable Diffusion, DALL-E) guided by user-defined prompts. This study leverages LMMs to interpret key parameters for drone-based image generation (e.g., color, texture, scene composition, photographic style) and applies prompt engineering to systematize these parameters. The resulting LMM-generated prompts were used to synthesize training data for a You Only Look Once version 8 segmentation model (YOLOv8-seg). To address the need for detailed crack-distribution mapping in low-altitude drone-based monitoring, the trained YOLOv8-seg model was evaluated on close-distance crack benchmark datasets. The experimental results confirm that LMM-prompted AIGC is a viable supplement for low-altitude drone crack monitoring, achieving >80% classification accuracy (images with/without cracks) at a confidence threshold of 0.5. Full article

Recent advances in large multimodal models (LMMs) have opened new opportunities for multitask recognition from remote sensing images. However, existing approaches still face challenges in effectively recognizing the complex geospatial characteristics of regions such as Japan, where its location along the seismic belt leads to highly diverse urban environments and cityscapes that differ from those in other regions. To overcome these challenges, we propose the GeoJapan Fusion Framework (GFF), a multimodal architecture that integrates a large language model (LLM) and a vision–language model (VLM) and strengthens multimodal alignment ability through an in-context learning mechanism to support multitask recognition for Japanese remote sensing images. The GFF also incorporates a cross-modal feature fusion mechanism with low-rank adaptation (LoRA) to enhance representation alignment and enable efficient model adaptation. To facilitate the construction of the GFF, we construct the GeoJapan dataset, which comprises a substantial collection of high-quality Japanese remote sensing images, designed to facilitate multitask recognition using LMMs. We conducted extensive experiments and compared our method with state-of-the-art LMMs. The experimental results demonstrate that GFF outperforms previous approaches across multiple tasks, demonstrating its promising ability for multimodal multitask remote sensing recognition. Full article

Face-to-face interaction is one of the most natural forms of human communication. Unsurprisingly, Video Conferencing (VC) Applications have experienced a significant rise in demand over the past decade. With the widespread availability of cellular devices equipped with high-resolution cameras, Instant Messaging Video Call Applications (IMVCAs) now constitute a substantial portion of VC communications. Given the multitude of IMVCA options, maintaining a high Quality of Experience (QoE) is critical. While content providers can measure QoE directly through end-to-end connections, Internet Service Providers (ISPs) must infer QoE indirectly from network traffic—a non-trivial task, especially when most traffic is encrypted. In this paper, we analyze a large dataset collected from WhatsApp IMVCA, comprising over 25,000 s of VC sessions. We apply four Machine Learning (ML) algorithms and a Large Multimodal Model (LMM)-based agent, achieving mean errors of 4.61%, 5.36%, and 13.24% for three popular QoE metrics: BRISQUE, PIQE, and FPS, respectively. Full article

In sports training, personalized skill assessment and feedback are crucial for athletes to master complex movements and improve performance. However, existing research on skill transfer predominantly focuses on skill evaluation through video analysis, addressing only a single facet of the multifaceted process required for skill acquisition. Furthermore, in the limited studies that generate expert comments, the learner’s skill level is predetermined, and the spatial-temporal information of human movement is often overlooked. To address this issue, we propose a novel approach to generate skill-level-aware expert comments by leveraging a Large Multimodal Model (LMM) and spatial-temporal motion features. Our method employs a Spatial-Temporal Attention Graph Convolutional Network (STA-GCN) to extract motion features that encapsulate the spatial-temporal dynamics of human movement. The STA-GCN classifies skill levels based on these motion features. The classified skill levels, along with the extracted motion features (intermediate features from the STA-GCN) and the original sports video, are then fed into the LMM. This integration enables the generation of detailed, context-specific expert comments that offer actionable insights for performance improvement. Our contributions are twofold: (1) We incorporate skill level classification results as inputs to the LMM, ensuring that feedback is appropriately tailored to the learner’s skill level; and (2) We integrate motion features that capture spatial-temporal information into the LMM, enhancing its ability to generate feedback based on the learner’s specific actions. Experimental results demonstrate that the proposed method effectively generates expert comments, overcoming the limitations of existing methods and offering valuable guidance for athletes across various skill levels. Full article

Large Multimodal Models are emerging general AI models capable of processing and analyzing diverse data streams, including text, imagery, and sequential data. This paper explores the possibility of exploiting multimodality to develop more interpretable AI-based predictive tools for the water sector, with a first application for sewer defect detection from CCTV imagery. To this aim, we test the zero-shot generalization performance of three generalist large language-vision models for binary sewer defect detection on a subset of the SewerML dataset. We compared the LMMs against a state-of-the-art unimodal Deep Learning approach which has been trained and validated on >1 million SewerML images. Unsurprisingly, the chosen benchmark showcases the best performances, with an overall F1 Score of 0.80. Nonetheless, OpenAI GPT4-V demonstrates relatively good performances with an overall F1 Score of 0.61, displaying equal or better results than the benchmark for some defect classes. Furthermore, GPT4-V often provides text descriptions aligned with the provided prediction, accurately describing the rationale behind a certain decision. Similarly, GPT4-V displays interesting emerging behaviors for trustworthiness, such as refusing to classify images that are too blurred or unclear. Despite the significantly lower performance from the open-source models CogVLM and LLaVA, some preliminary successes suggest good potential for enhancement through fine-tuning, agentic workflows, or retrieval-augmented generation. Full article