Reasoning Together: Designing and Evaluating MLLM Team Strategies for Multimodal Quiz Questions

Round 1

Reviewer 1 Report

Comments and Suggestions for Authors

The paper proposes a nine-dimensional design space and a seven-stage reasoning pipeline for MLLM. A collaborative MLLM framework was proposed and evaluated for six strategies on a 1,170-question multimodal dataset. All team configurations outperform individual models with reasoning sharing producing the most consistent achievement. The best strategy achieves 31.2% accuracy demonstrating that effective collaboration can substantially improve MLLM reasoning performance. However, the following concerns should be addressed before publication. In the abstract, the writing of percentage should be uniform for example, on line 14 " 20 percentage " and line 13 " achieved 31.2% " pls unify throughout the paper.  Figure 1 have three components, none of them is label, so label each and provide simple description for each at the Figure 1 title. Why only one image on Figure 1? Move the source code link to end of abstract. The LR Section 2 should be rewrite to critique on the previous approaches. Itemize section 3.3. Explain the data preprocessing stages in section 3.4. refer to " ….while the overall best resultis additionally underlined …"In Table 3 and other Tables no value is underlined, check it out. Extend the discussion by adding the implication of the results to theory and practice. The conclusion should be rewrite to make it a block without the research questions. References are updated.

Author Response

Comment 1. In the abstract, the writing of percentage should be uniform for example, on line 14 "20 percentage" and line 13 "achieved 31.2%" pls unify throughout the paper.
Response: Thank you for drawing our attention to the consistency of the percentage notation. We have checked its use throughout the manuscript. In the Abstract, “31.2%” expresses the achieved accuracy, whereas “20 percentage points” expresses the absolute improvement over the corresponding single-model baseline. We have retained these distinct units to convey the two quantities accurately and confirmed that percentages and percentage points are used consistently throughout the manuscript.

Comment 2. Figure 1 have three components, none of them is label, so label each and provide simple description for each at the Figure 1 title.
Response: Thank you for this helpful suggestion. We have labeled the three components of Figure 1 as (a), (b), and (c) and expanded the caption to briefly describe each component: the multimodal question, the individual models’ responses, and the collaborative team response. These changes clarify the structure of the figure and the relationship between its components.

Comment 3. Why only one image on Figure 1?
Response: Thank you for this comment. Figure 1 illustrates a single multimodal quiz question, which contains one accompanying image. The same question and image are provided to all models in both the individual and collaborative settings. The image is therefore displayed once in panel (a), while panels (b) and (c) illustrate the individual responses and the collaborative workflow, respectively. We have clarified this shared-input arrangement in the figure caption.

Comment 4. Move the source code link to end of abstract.
Response: Thank you for this suggestion. We have moved the link to the source code and prompts from the Introduction to the end of the Abstract.

Comment 5. The LR Section 2 should be rewrite to critique on the previous approaches.
Response: Thank you for this constructive suggestion. We have revised Section 2 to strengthen its critical discussion of previous approaches. Specifically, we discuss the limitations of aggregation without rationale exchange, the risks of reinforcing incorrect hypotheses during debate, and the need to assess whether findings from text-based benchmarks transfer to multimodal quiz questions. We also examine the dependence of specialized architectures on communicated evidence and domain-specific design choices, as well as the difficulty of attributing their gains to individual components. Finally, we compare the evaluation goals of existing multimodal benchmarks and clarify both the contribution and the generalizability limitations of our study. The revised and added passages are highlighted in blue.

Comment 6. Itemize section 3.3.
Response: Thank you for this suggestion. We have reformatted the six strategy descriptions in Section 3.3 as a bulleted list, with each strategy name shown in bold, to make the presentation clearer and easier to follow.

Comment 7. Explain the data preprocessing stages in section 3.4.
Response: Thank you for this comment. We have added a brief description of the data preprocessing stages in Section 3.4, covering the extraction of questions with supplementary handout material, organization into JSON records, duplicate removal, local downloading and Base64 encoding of images for submission to the models.

Comment 8. Refer to "….while the overall best results additionally underlined …" In Table 3 and other Tables no value is underlined, check it out.
Response: Thank you for pointing out this issue. To avoid ambiguity, we have simplified the highlighting convention in Table 3 by removing the additional underlining and the corresponding statement from the caption. Boldface identifies the highest score in each column, while the overall best result is explicitly identified in the accompanying text.

Comment 9. Extend the discussion by adding the implication of the results to theory and practice.
Response: Thank you for this valuable suggestion. We have added a subsection entitled “5.6. Implications for Theory and Practice” to the end of Discussion. It outlines implications for evaluating the generation, preservation, and selection of candidate hypotheses, and identifies practical directions for candidate retention and verification.

Comment 10. The conclusion should be rewrite to make it a block without the research questions.
Response: Thank you for this suggestion. We have removed the separately formatted questions and the list structure from the Conclusions, presenting the findings as connected prose organized into paragraphs. Brief references to RQ1–RQ4 remain within the text to preserve the link to the study objectives, without repeating the questions themselves.

Comment 11. References are updated.
Response: Thank you for this comment. We have rechecked the bibliography for currency and consistency. Of the 31 references, 28 (90.3%) are dated 2025–2026. The three earlier references describe the MultiModalQA and ManyModalQA benchmarks and the ClusterLLM method and have been retained because they directly support the discussion of existing benchmarks and the dataset annotation procedure.

Reviewer 2 Report

Comments and Suggestions for Authors

Review

The article investigates whether teams composed of several multimodal language models can answer difficult quiz questions more effectively than single models, and which collaboration mechanisms yield the greatest improvement. The authors compare six team strategies and show that the largest gains come from sharing rationales and from limited debate between models.

 

Strengths:

1. Large and well-designed experiment: 1,170 questions, 6 different MLLMs, and several compared team strategies.
2. Good use of matched configurations, allowing the effects of voting, captain-based aggregation, rationale sharing, and an extra discussion round to be isolated.
3. Interesting analysis of how the team actually works: the paper shows that debate mainly strengthens or introduces correct hypotheses before final aggregation.

 

Weaknesses:

1. The novelty of the 9-dimensional design space and 7-stage pipeline is limited — they look more like an organisation of known elements than a new method.
2. No analysis of computational cost: the best strategies require more model calls, tokens, and time, so accuracy alone does not give the full picture.
3. The best team still loses to a strong single model, GPT-5.5: 31.20% vs 35.47%.
4. Limited generalisability — only one Russian-language quiz benchmark with a specific cultural character.
5. The term "reasoning sharing" is too strong: in practice, the models exchange short rationales, not their internal reasoning process.
6. No comparison with other contemporary strong multi-agent or debate strategies on the same benchmark; most baselines are variants designed by the authors themselves.

 

Major remarks

1. The paper needs to be clearer about what is new in the proposed design space and generic pipeline.

The nine-dimensional design space and seven-stage pipeline are reasonable and useful as an organising framework, but many of their components appear to systematise already known elements of multi-agent reasoning rather than introduce a fundamentally new methodology. The authors should state more precisely what is novel in these two contributions and how they differ from existing taxonomies or architectural decompositions.

 

2. The comparison between strategies does not account for computational cost.

The strongest strategy, Talkative Debate, uses substantially more model calls and exchanged content than simpler baselines. Therefore, higher accuracy alone does not provide a complete comparison. The paper should include at least a discussion, and preferably an analysis, of cost in terms of model calls, tokens, latency, or API cost. This is particularly important because additional rounds show sharply diminishing returns.

 

3. The practical significance of the results should be discussed more carefully.

The best team configuration achieves 31.20% accuracy, whereas the GPT-5.5 single-model reference reaches 35.47%. Thus, the team approach improves substantially over the six constituent models but does not outperform the stronger single-model reference. This should be discussed explicitly in terms of the trade-off between collaboration complexity and model strength.

 

4. The paper uses the term "reasoning sharing" too broadly.

The exchanged information is a short generated rationale rather than access to the models' internal reasoning. The terminology should therefore be made more precise, e.g. "rationale sharing" or "explanation sharing", unless the authors justify the stronger term. This is especially relevant because the prompts explicitly prohibit internal reasoning while requesting a brief rationale field.

 

5. Generalisability remains limited.

All main conclusions are based on a single Russian-language quiz benchmark with specific cultural, linguistic, and answer-format characteristics. The authors acknowledge this limitation, but the broader claims concerning MLLM team strategies should be moderated accordingly.

 

6. The strongest contribution may be the mechanism analysis rather than the framework itself.

The candidate-transition analysis is particularly valuable: debate mainly improves performance by preserving, strengthening, or introducing correct hypotheses before final aggregation. This finding deserves stronger emphasis, because it provides a more specific and informative contribution than the general observation that collaboration improves accuracy.

 

To sum up, Recommendation:

Major Revision

The study is solid and potentially publishable, but the claims of novelty should be sharpened and the lack of cost-aware evaluation should be addressed.

 

Author Response

Comment 1. The paper needs to be clearer about what is new in the proposed design space and generic pipeline. The nine-dimensional design space and seven-stage pipeline are reasonable and useful as an organising framework, but many of their components appear to systematise already known elements of multi-agent reasoning rather than introduce a fundamentally new methodology. The authors should state more precisely what is novel in these two contributions and how they differ from existing taxonomies or architectural decompositions.
Response: Thank you for this thoughtful comment. We agree that the individual components of the design space and pipeline build on established elements of multi-agent reasoning, and that their specific contribution needed clearer articulation.
We have revised the contribution statement in the Introduction and the explanations in Sections 3.1 and 3.2. The revised text identifies the framework’s contribution as the joint organization of established design choices and their explicit mapping to workflow stages. Section 3.1 positions this descriptive scheme in relation to the decision-protocol study of Kaesberg et al. (2025) and the specialized agent architecture of MAMMQA. Section 3.2 explains how the two views complement each other: the design space specifies the choices defining a team strategy, while the pipeline shows where these choices enter its execution. We also clarify that the six evaluated strategies instantiate a selected subset of the design choices, supporting focused comparisons of aggregation, rationale sharing, and additional interaction.

Comment 2. The comparison between strategies does not account for computational cost. The strongest strategy, Talkative Debate, uses substantially more model calls and exchanged content than simpler baselines. Therefore, higher accuracy alone does not provide a complete comparison. The paper should include at least a discussion, and preferably an analysis, of cost in terms of model calls, tokens, latency, or API cost. This is particularly important because additional rounds show sharply diminishing returns.
Response: Thank you for this comment. We have added Section 5.1, “Inference Resource Usage and Accuracy Trade-offs,” at the beginning of the Discussion, together with Table 5 reporting model calls and mean input and output tokens per question. These counts include image tokens and all team-member calls required to produce the final answer. The analysis focuses on Gemma-4, the best-performing captain under five of the six team strategies, whose strategy ranking also matches the ranking of mean accuracies across all six captains.
The new subsection explicitly relates accuracy gains to resource requirements. In particular, extending Talkative Debate by one round increases calls from 13 to 19 and input and output token usage by approximately 52% and 49%, respectively, with a net gain of only one correct answer (approximately 0.08 percentage points). This illustrates the sharply diminishing returns of additional interaction. We also clarify that call and token counts characterize inference workload, while latency and monetary cost depend on the models, providers, and execution environment.

Comment 3. The practical significance of the results should be discussed more carefully. The best team configuration achieves 31.20% accuracy, whereas the GPT-5.5 single-model reference reaches 35.47%. Thus, the team approach improves substantially over the six constituent models but does not outperform the stronger single-model reference. This should be discussed explicitly in terms of the trade-off between collaboration complexity and model strength.
Response: Thank you for highlighting this distinction. We have added a paragraph to “5.6. Implications for Theory and Practice” explicitly discussing the trade-off between collaboration complexity and individual-model strength. We state that the best team achieves 31.20% accuracy, exceeding all six constituent models but remaining 4.27 percentage points below the GPT-5.5 single-model reference (35.47%). We clarify that collaboration improves the capabilities of the evaluated model pool, while a stronger individual model achieves higher accuracy with a simpler inference workflow. The revised discussion identifies model availability, resource requirements, and deployment constraints as relevant considerations when choosing between these approaches.

Comment 4. The paper uses the term "reasoning sharing" too broadly. The exchanged information is a short generated rationale rather than access to the models' internal reasoning. The terminology should therefore be made more precise, e.g. "rationale sharing" or "explanation sharing", unless the authors justify the stronger term. This is especially relevant because the prompts explicitly prohibit internal reasoning while requesting a brief rationale field.
Response: Thank you for pointing out this distinction. We agree that “rationale sharing” more accurately describes the information exchanged in our experiments. We have revised the terminology throughout the manuscript, including Figures 1, 3, and 4. We have also clarified in the opening paragraph of Section 3.3 that a rationale is a brief model-generated explanation accompanying an answer, rather than a record of the model’s internal reasoning process. This clarification aligns the terminology with the experimental prompts, which request brief rationales while explicitly excluding internal reasoning.

Comment 5. Generalisability remains limited. All main conclusions are based on a single Russian-language quiz benchmark with specific cultural, linguistic, and answer-format characteristics. The authors acknowledge this limitation, but the broader claims concerning MLLM team strategies should be moderated accordingly.
Response: Thank you for this comment. We agree that the scope of the empirical conclusions should be stated more precisely. We have revised the Abstract, Introduction, Results, Discussion, and Conclusions to explicitly relate the findings to MChGK and the evaluated model configurations. We have also clarified that the additional-round and candidate-transition analyses concern the configuration with Gemma-4 as captain. Together with the existing limitations paragraph addressing the benchmark’s linguistic, cultural, and answer-format characteristics, these revisions distinguish the demonstrated benefits from their potential applicability to other settings, which requires further evaluation.

Comment 6. The strongest contribution may be the mechanism analysis rather than the framework itself. The candidate-transition analysis is particularly valuable: debate mainly improves performance by preserving, strengthening, or introducing correct hypotheses before final aggregation. This finding deserves stronger emphasis, because it provides a more specific and informative contribution than the general observation that collaboration improves accuracy.
Response: Thank you for highlighting the value of the mechanism analysis. We agree that understanding how collaboration improves the candidate-answer set provides a more specific contribution than the overall accuracy gains alone. We have therefore moved this analysis ahead of the aggregate performance findings in the contribution list and expanded its description to state the principal finding explicitly. For Talkative Debate with Gemma-4 as captain, successful outcomes primarily depended on preserving or introducing correct hypotheses and strengthening their support before final aggregation, whereas correct final answers were rare when no correct candidate remained after discussion. This revision gives the analysis greater prominence, complementing its presentation in the Abstract and the Conclusions under RQ4.

Round 2

Reviewer 2 Report

Comments and Suggestions for Authors

In the Conclusions, replace “introduced a unified framework” with “proposed a unified descriptive framework” to match the revised positioning of the contribution. 

For consistency with the new “rationale sharing” terminology, consider renaming “Reasoning-chain errors” to “Rationale-chain errors” or “Inference-chain errors”.