Review Reports - CA-VLN: Collaborative Agents in MLLM-Powered Visual-Language Navigation

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

This paper addresses the problem of visual-language navigation in indoor environments and proposes an architecture where semantic scene understanding from a multimodal language model is combined with a memory of the traversed path. A key strength of the work is the thoughtful integration of knowledge, navigation history, and a direction-selection mechanism, which enables the agent to make more informed decisions in complex conditions. The authors report consistent performance improvements on widely used benchmark datasets for this task, especially in unseen environments. However, the following comments should be addressed:

In section 3.3 (Multimodal Fusion), the description of the feature fusion mechanism lacks a clear explanation of the exact sequence of operations used to combine visual features, semantic information, and historical context. To improve clarity and reproducibility, it would be helpful to add pseudocode that shows step-by-step how candidate features are constructed and how they are fused into the final representation used for action selection.
In section 3.4 (Hierarchical History Agent), it is not sufficiently clear how the size of the accumulated memory is managed as the agent progresses through the environment. The paper does not explain which records are stored, what is considered important, or what rules are used to update or discard memory. It is also unclear how this design affects computational cost for long trajectories. For better transparency, the authors should describe the criteria for selecting and removing memory elements and report how performance changes with different memory sizes, including a comparison with simpler ways of modeling history.
In the experimental section, since the performance gains over strong baselines are relatively small, the paper would benefit from statistical evidence of robustness, for example through multiple runs with different random seeds or confidence intervals. In addition, there is no analysis of the computational cost of the proposed architecture, which is important given the use of a multimodal model and memory mechanisms.
Although the datasets are based on scans of real indoor environments, the method is not evaluated in a physical setting. The authors should more explicitly discuss how well the approach is expected to transfer to real-world scenarios.

Minor comments:

In Table 1, the baseline name is misspelled as “KREM [48]”. Please check and fix similar inconsistencies throughout the paper.
In Figure 4, there is a typo in “Konwledge embedding”.
The quality/resolution of Figure 6 should be improved.

Author Response

Comment: 1. In section 3.3 (Multimodal Fusion), the description of the feature fusion mechanism lacks a clear explanation of the exact sequence of operations used to combine visual features, semantic information, and historical context. To improve clarity and reproducibility, it would be helpful to add pseudocode that shows step-by-step how candidate features are constructed and how they are fused into the final representation used for action selection.

Response: We agree that the fusion mechanism involves complex interactions that require a more precise description. To address this:

We have rewritten Section 3.3 to explicitly describe the sequence: (1) Visual features are first enhanced by the Knowledge-aware Visual Semantic Interactor (KVSI); (2) These are concatenated with instruction embeddings; (3) The Adaptive View Weighting Mechanism computes attention scores to fuse these features.

We have added a pseudocode algorithm (Appendix B Algorithm of Multi-modal Fusion and Action Decision) in the revised manuscript, the calculation of attention scores si , and the weighted aggregation process.

Revision in Manuscript: Please refer to the newly added Algorithm A1 in Appendix B and the refined mathematical formulation in Equations 3-8.

Comment: 2. In section 3.4 (Hierarchical History Agent), it is not sufficiently clear how the size of the accumulated memory is managed as the agent progresses through the environment. The paper does not explain which records are stored, what is considered important, or what rules are used to update or discard memory. It is also unclear how this design affects computational cost for long trajectories. For better transparency, the authors should describe the criteria for selecting and removing memory elements and report how performance changes with different memory sizes, including a comparison with simpler ways of modeling history.

Response: We appreciate this insightful suggestion regarding memory management.

Memory Management: We have updated Section 3.4 to clarify that our Hierarchical History Agent employs a incremental update mechanism. To prevent memory explosion during hovering or repetitive movements, the historical memory is built incrementally rather than storing every individual view. A new node is added to the memory graph only when the agent advances to a new observation point. At each new viewpoint, the system generates and integrates an interaction history, which updates the corresponding historical topology map. This update includes visual feature descriptions, spatial states (e.g., ”corridor” or ”bedroom”), and logical connections to previously recorded points. These data points are then incrementally integrated into the current memory to maintain an efficient and accurate spatial representation. To clarify the meaning of these elements, we provide an example in Figure 5.

Discard Memory: History memory where nodes are not actively discarded but efficiently indexed. Inspired by the design philosophy of Retrieval-Augmented Generation(RAG), our efficient retrieval is less constrained by storage limitations, and the retrieval algorithm incurs lower time overhead. For extremely long trajectories (which are rare in R2R/REVERIE), in standard benchmarks, the graph size remains manageable (avg. 10-15 nodes). No memory collapse was observed on the public datasets.

Comparison with different history modeling approaches: We have conducted additional experiments comparing our method with other approaches in the revision. The comparison results are summarized in the table below (see Table 7). Our method outperforms other approaches. The improvement primarily stems from our ability to selectively retrieve task-relevant information while preventing memory explosion. We have added these experimental results in Section 4.5.

Revision in Manuscript: Please refer to the newly added Figure 5 in Section3.4.1 and the refined mathematical formulation in Equations 10-12.

Comment: 3. In the experimental section, since the performance gains over strong baselines are relatively small, the paper would benefit from statistical evidence of robustness, for example through multiple runs with different random seeds or confidence intervals. In addition, there is no analysis of the computational cost of the proposed architecture, which is important given the use of a multimodal model and memory mechanisms.

Response: We acknowledge that SOTA improvements in VLN are becoming incremental. However, our contribution emphasizes robustness.

We have conducted the experiments with 3 different random seeds and reported the mean and standard deviation in Table A1 in Appendix C of the revised manuscript. The variance is low (±0.4%), indicating stability.

We added a subsection “Computational Complexity Analysis” in Appendix A. Regarding computational cost: Compared to the baseline DUET, our model adds the Knowledge Agent. Inference time increases by approximately 15% per step due to the MLLM query, but the Success weighted by Path Length (SPL) improves, meaning the agent takes more efficient paths, partially offsetting the per-step latency.

Revision in Manuscript: We added standard deviations to Table A1 in Appendix C and a “Computational Complexity Analysis” in Appendix A.

Comment: 4. Although the datasets are based on scans of real indoor environments, the method is not evaluated in a physical setting. The authors should more explicitly discuss how well the approach is expected to transfer to real-world scenarios.

Response: We acknowledge this limitation. Our real laboratory platform is still under construction, which will be shown in subsequent work. While our experiments are performed on Matterport3D (which consists of real-world scans), a physical robot introduces noise. Our use of semantic abstractions rather than raw pixels makes the model theoretically more robust. The high-level reasoning (“Find the kitchen”) transfers better than low-level texture matching.

Revision in Manuscript: We have expanded the Conclusion to explicitly discuss the “Sim-to-Real Gap,” highlighting that the semantic reasoning module serves as a bridge for future physical deployment.

Comment: 5 - 7. Minor comments (Typos, Figure resolution).

In Table 1, the baseline name is misspelled as “KREM [48]”. Please check and fix similar inconsistencies throughout the paper.
In Figure 4, there is a typo in “Konwledge embedding”.
The quality/resolution of Figure 6 should be improved.

Response: We apologize for these oversight errors.

Fixed the citation style for “KREM [48]” and checked all references.
Corrected “Konwledge” to “Knowledge” in Figure 4.
Replaced Figure 6 (new Figure 7) with a high-resolution vector graphic (PDF format) to ensure clarity when zoomed in.

Reviewer 2 Report

Comments and Suggestions for Authors

Summary

This paper studies Visual Language Navigation with a focus on generalization to unseen environments. It proposes CA-VLN with two agents. A Knowledge Reasoning Agent uses a multimodal large language model to produce step wise guidance and semantic entities from the instruction. A Hierarchical History Agent maintains a hierarchical trajectory representation and an episodic graph memory for long horizon navigation. A fusion module combines these signals with visual observations to choose actions. Experiments on R2R, REVERIE, and SOON report improvements over a strong baseline.

Strengths

Targets two core VLN difficulties, instruction grounding and long horizon history.
Uses an MLLM to generate structured intermediate signals instead of direct action control.
Adds explicit memory via hierarchical summaries and a graph based episodic module.
Evaluated on multiple benchmarks including REVERIE with grounding metrics.
Includes ablations and Top K sensitivity to support the design choices.

Weaknesses

The proposed pipeline is heavy relative to the strength of the empirical story. The paper combines two agents with additional fusion and memory machinery, yet it also acknowledges that improvements are marginal in places. This makes the cost benefit tradeoff unclear and weakens the case that the added complexity is necessary.
The training setup leverages additional supervision that may not transfer to realistic settings. The instruction enhancement procedure uses annotated ground truth trajectories to generate refined step wise instructions and entities, which are then used as extra signals for training. This reduces the plausibility of the approach in scenarios where such trajectory annotations are not available.
The knowledge retrieval component is not described in enough detail to evaluate what knowledge is actually being injected. The method mentions CLIP retrieval of related knowledge, but the paper does not clearly define the knowledge pool and the exact retrieved content used at inference. As a result, it is difficult to assess whether improvements come from reasoning, from implicit additional data, or from retrieval heuristics.
The approach appears brittle with respect to retrieval and the Top K setting. The paper reports degradation when Top K becomes larger, attributing it to redundant information and hallucination errors, and notes that some tasks are especially sensitive to redundancy. This suggests the method may require careful tuning and stable retrieval quality to perform reliably.
There are minor reporting inconsistencies that reduce confidence in careful execution. For example, the text mixes Top N and Top K terminology in one of the key figure captions, and a baseline value referenced in the discussion does not match the baseline shown in the corresponding table.

The paper proposes a coherent dual agent approach that combines MLLM based knowledge signals with explicit history memory for VLN, supported by multi benchmark evaluation. However, due to the high complexity relative to the gains and remaining clarity issues around knowledge retrieval and supervision assumptions, I recommend Major Revision.

Author Response

Comment: 1. The proposed pipeline is heavy relative to the strength of the empirical story. The paper combines two agents with additional fusion and memory machinery, yet it also acknowledges that improvements are marginal in places. This makes the cost benefit tradeoff unclear and weakens the case that the added complexity is necessary.

Response: We appreciate the reviewer’s perspective on architectural complexity. We respectfully argue that the increased depth is a necessary evolution to bridge the gap between simple ”instruction following” and ”true spatial reasoning.”

Alignment with SOTA Trends: The shift towards utilizing MLLMs as ”Reasoning Agents” or ”World Models” is rapidly becoming the standard in Embodied AI to handle unseen environments. Recent premier works, such as NavCoT (TPAMI 2025) and CoT-VLA (CVPR 2025), have demonstrated that decoupling high-level reasoning (via LLMs) from low-level perception significantly improves generalization, even if it introduces architectural overhead.

Necessity of Dual-Agents: Our dual-agent design mirrors the ”System 1 (Global/Coarse) and System 2 (Local/Fine)” paradigm. The History Agent handles immediate, grounded perception, while the Knowledge Agent leverages MLLM’s world knowledge for deliberative planning.

Trade-off Justification: While the absolute performance gain on seen validation sets may seem marginal, the key contribution is the robustness in Unseen environments (as shown in Table 1). Simple end-to-end models often overfit to training layouts; our ”heavier” pipeline effectively regularizes the agent by forcing it to rely on semantic consistency rather than visual memorization.

Revision in Manuscript: We added a “Computational Complexity Analysis” in Appendix A.

Comment: 2. The training setup leverages additional supervision that may not transfer to realistic settings. The instruction enhancement procedure uses annotated ground truth trajectories to generate refined step wise instructions and entities, which are then used as extra signals for training. This reduces the plausibility of the approach in scenarios where such trajectory annotations are not available.

Response: We wish to clarify a potential misunderstanding regarding the usage of Ground Truth (GT).

During Training Only: The GT trajectories are used strictly for data augmentation (to generate ”Golden Instructions” via MLLM) during the training phase. This acts as a form of curriculum learning to teach the agent optimal reasoning paths.

During Inference/Testing: The knowledge agent does not have access to GT trajectories. The agent generates knowledge using only the original instruction, real-time visual observations, and the trajectory history of the current step. There is no fundamental difference between the two processes. The only distinction is that the knowledge generated during the training phase is more accurate due to the use of ground-truth labels. Therefore, our method is fully plausible for realistic deployment where agents must explore unknown environments without prior trajectory annotations.

Revision in Manuscript: We have clarified this distinction in Section 3.2.1 to ensure readers understand that GT is strictly for training supervision.

Comment: 3. The knowledge retrieval component is not described in enough detail to evaluate what knowledge is actually being injected. The method mentions CLIP retrieval of related knowledge, but the paper does not clearly define the knowledge pool and the exact retrieved content used at inference. As a result, it is difficult to assess whether improvements come from reasoning, from implicit additional data, or from retrieval heuristics.

Response: We appreciate the reviewer’s valuable feedback. To clarify the knowledge retrieval mechanism, we have added a detailed description in Section 3.2.1 of the revised manuscript. Specifically, our ”knowledge pool” is not a static external database, but a dynamic sequence generated based on world knowledge of the Knowledge Agent. It can be understood that the world knowledge of MLLMs serves as our ”knowledge pool”, and a separate sub-knowledge pool is dynamically generated for each task during navigation. To utilize this generated knowledge, We use the CLIP text encoder to embed the current instruction and retrieve the Top-K most semantically related facts (e.g., ”Dining table is near the kitchen”) via cosine similarity to augment the visual features.

Revision in Manuscript: Added details about the Knowledge Pool construction and CLIP retrieval metric in Section 3.2.1.

Comment: 4. The approach appears brittle with respect to retrieval and the Top K setting. The paper reports degradation when Top K becomes larger, attributing it to redundant information and hallucination errors, and notes that some tasks are especially sensitive to redundancy. This suggests the method may require careful tuning and stable retrieval quality to perform reliably.

Response: We agree that retrieving too much irrelevant information (large K) introduces noise. The results of the Top-k experiments align with our expectations. The value of k cannot increase indefinitely. There is an peak, followed by a gradual decline in performance. In real-world, the amount of information relevant to the current action instruction is finite. Therefore, we employ a retrieval module to perform an initial filtering of redundant information, followed by a fusion module to achieve activation. This is why we introduced the Multimodal Fusion and Action Prediction(Section 3.3), which utilizes an attention mechanism. The attention weights allow the model to dynamically learn to ignore irrelevant retrieved knowledge. Our ablation study shows that while performance drops if K > 5, the drop is gradual rather than catastrophic, proving the fusion module’s robustness against noisy retrieval.

Comment: 5. There are minor reporting inconsistencies that reduce confidence in careful execution. For example, the text mixes Top N and Top K terminology in one of the key figure captions, and a baseline value referenced in the discussion does not match the baseline shown in the corresponding table.

Response: Thank you for spotting this. We have standardized the terminology to ”Top-K” throughout the manuscript, figures, and captions.

Reviewer 3 Report

Comments and Suggestions for Authors

Overall Evaluation

The paper proposes a novel visual-language navigation (VLN) framework based on multimodal large language models (MLLMs). Through a formalized dual-agent architecture and extensive experimental validation, it addresses core challenges in VLN—generalization to unseen environments, shallow multimodal feature fusion, and insufficient historical context—without relying on traditional end-to-end deep reinforcement learning methods. The manuscript comprises five main sections and a reference list. Part 1 (Introduction) briefly outlines the study's key contributions, including: 1) Knowledge Reasoning Agent: Leverages MLLM-derived world knowledge for advanced semantic reasoning, enriching action prediction with contextual understanding through knowledge-enhanced instruction generation and key entity extraction; 2) Hierarchical History Agent: Constructs a detailed episodic memory mechanism enabling long-term planning and navigation retracing through Hierarchical History Retrieval (HHR) and History Enhancement Module (HEM); 3) Knowledge-Guided Multimodal Fusion Module: Integrates instruction-guided feature fusion (IGFF) and knowledge-aware visual-semantic interactors (KVSI) to dynamically reweight information from diverse perspectives, bridging the domain gap between MLLMs and VLNs. Section 2 (Related Work) outlines the current state of development in the VLN domain. The authors categorize prior work into three groups: VLN generalization approaches, LLM-based VLN methods, and knowledge-based VLN methods, briefly introducing subsequent sections. Part 3 (Materials and Methods) details the overall research workflow, implementation of the knowledge reasoning agent (including knowledge-enhanced instruction generation and entity-guided feature enhancement), multimodal fusion module (comprising IGFF and KVSI), and hierarchical history agent implementation (including HHR and HEM). Section 4 (Experiments) comprehensively evaluates the algorithms from Section 3 across three representative benchmark datasets (R2R, REVERIE, SOON), covering comparisons with existing SOTA methods, ablation studies, and qualitative analysis. Finally, Section 5 (Conclusions) summarizes the theoretical and experimental findings. The authors conclude that current VLN methods based on MLLMs achieve promising results in leveraging world knowledge and episodic memory, yet face challenges in visually sparse environments—an area warranting future research. The manuscript is exceptionally well-presented, featuring clear theoretical exposition, detailed experimental results, and high-quality illustrations.

Comments

1. Abbreviations are typically expanded upon their first mention in the text, though this is not consistently followed in the manuscript. For example:

The abbreviation “DDN” first appears on page 3 (line 122) without expansion; similar instances occur throughout.

2. In this review, the authors propose a novel CA-VLN framework. The primary innovation lies in overcoming limitations such as generalization capability, multimodal fusion, and historical context gaps by leveraging MLLMs' world knowledge and dual-agent collaboration mechanisms—without relying on traditional end-to-end deep learning paradigms. This significantly enhances visual-language navigation performance in unseen environments. In my view, this topic is highly relevant to the research domain as it addresses core bottlenecks in deploying VLN in real-world dynamic settings.

3. Compared to other published materials, it simultaneously integrates both knowledge reasoning and episodic memory dimensions. In the manuscript, the authors review only some of the most prominent comparative methods (e.g., DUET, KERM, ACME, NaviLLM) and plan to explore more efficient visual feature extraction to complement knowledge-driven approaches in future research (see conclusion). In my view, the conclusions align with the presented evidence and arguments, address the main issues, and the references section clearly reflects the level of research conducted.

4. Collaboration Mechanism Between Knowledge Agent and History Agent:

Section 3.1 and Figure 2 describe the dual-agent architecture where the Knowledge Agent generates knowledge-enhanced guidance and the History Agent maintains episodic memory. Please clarify: What are the specific conditions or thresholds for determining when to prioritize knowledge-based reasoning versus historical memory? Is the interaction automatic through the Multimodal Fusion Module or manually weighted? Does this collaboration cause decision conflicts or latency issues in real-time navigation? Did experiments quantitatively analyze the contribution balance of both agents under different instruction complexity levels? How is the error propagation controlled when the Knowledge Agent generates incorrect semantic entities in visually sparse environments?

5. I am surprised by the minimal mention of the authors' own publications in the manuscript. I am unclear on the reason—it may stem from this being a novel research direction, or the authors having no prior publications in this specific domain. I believe the figures in the manuscript adequately reflect the results of the conducted research, and the ablation experiments are rigorously designed to effectively validate the contributions of each module.

Author Response

Comment: 1. Abbreviations are typically expanded upon their first mention in the text, though this is not consistently followed in the manuscript. For example:

The abbreviation “DDN” first appears on page 3 (line 122) without expansion; similar instances occur throughout.

Response: We have carefully proofread the paper. ”DDN” (Deep Dynamic Network) and other abbreviations are now defined at their first occurrence.

Comment: 2.-3. In this review, the authors propose a novel CA-VLN framework. The primary innovation lies in overcoming limitations such as generalization capability, multimodal fusion, and historical context gaps by leveraging MLLMs' world knowledge and dual-agent collaboration mechanisms—without relying on traditional end-to-end deep learning paradigms. This significantly enhances visual-language navigation performance in unseen environments. In my view, this topic is highly relevant to the research domain as it addresses core bottlenecks in deploying VLN in real-world dynamic settings.

Compared to other published materials, it simultaneously integrates both knowledge reasoning and episodic memory dimensions. In the manuscript, the authors review only some of the most prominent comparative methods (e.g., DUET, KERM, ACME, NaviLLM) and plan to explore more efficient visual feature extraction to complement knowledge-driven approaches in future research (see conclusion). In my view, the conclusions align with the presented evidence and arguments, address the main issues, and the references section clearly reflects the level of research conducted.

Response: We sincerely thank the reviewer for the highly positive and professional assessment of our CA-VLN framework.

Comment: 4. Collaboration Mechanism Between Knowledge Agent and History Agent:

Response: We thank the reviewer for these in-depth questions regarding the internal dynamics of our dual-agent architecture. We have added clarifications in Section 3.1 and Section 3.3. We address your specific questions below:

Conditions for Prioritization (Coarse vs. Fine): The collaboration is simultaneous and softweighted, not a hard switch based on thresholds. For example, if the instruction is ”Turn left at the kitchen,” the Knowledge Agent’s weight typically increases (object recognition). If the instruction is ”Go back the way you came,” the History Agent’s weight increases. In the revised manuscript, we have added Algorithm A1 (Appendix B Algorithm of Multi-modal Fusion and Action Decision), which details the step-by-step process of feature construction and fusion, as detailed in Algorithm A1 .

Automatic Interaction: The interaction is fully automatic via the Multimodal Fusion Module. We have supplemented Algorithm A1 with specific details showing how the attention weights are dynamically generated to integrate features, removing the need for manual weight tuning.
Conflicts and Latency: We employ a weighted fusion strategy at the decision stage, as shown in Algorithm A1. This mechanism avoids ”decision conflicts” because it produces a single fused vector for the policy network. This probabilistic combination effectively smooths out discrepancies between agents, avoiding hard decision conflicts. Latency: The network framework utilizes a dual-stream parallel architecture. Therefore, the collaboration itself does not introduce significant latency, ensuring the system remains efficient.

Instruction Complexity Analysis: We agree that quantitatively analyzing agent contribution under different instruction complexity levels is a valuable research direction. However, since there is currently no standardized quantitative metric to measure ”instruction complexity” in existing VLN datasets, we plan to explore this in future work.

Error Propagation Control: We acknowledge that the Knowledge Agent may generate incorrect semantics in visually sparse environments. To control error propagation, we employ three strategies:

Attention and Gating. We use attention mechanisms and gating units to down-weight lowconfidence or conflicting information.

Temporal Filtering. Even if an error affects the current step, the relevance retrieval mechanism in the subsequent navigation step serves as a secondary filter, reducing the probability of the error propagating further.

Comment: 5. I am surprised by the minimal mention of the authors' own publications in the manuscript. I am unclear on the reason—it may stem from this being a novel research direction, or the authors having no prior publications in this specific domain. I believe the figures in the manuscript adequately reflect the results of the conducted research, and the ablation experiments are rigorously designed to effectively validate the contributions of each module.

Response: As this is a relatively novel direction for our specific research group, the majority of references are from the broader community to position our work within the state-of-the-art. But we are not completely unrelated, the group’s research directions are all grounded in multimodality and Large Language Models.

Round 2

Reviewer 1 Report

Comments and Suggestions for Authors

The authors have addressed all comments appropriately, and the manuscript has been substantially improved

Reviewer 2 Report

Comments and Suggestions for Authors

The revised version clarifies the role of ground truth trajectories as training only augmentation, improves the explanation of the knowledge pool and retrieval, and fixes the Top K terminology, with an added complexity discussion; remaining concerns are mostly about tighter specification and stronger empirical support rather than correctness.