ActivAsk: Free-Energy-Guided Clarification for Robotic Grasping Under Ambiguous Instructions

Abstract

Service robots often receive natural language instructions in changing workspaces where multiple visible objects may match one description. Relying on detector confidence, random selection, or direct vision–language model (VLM) prediction can lead to a wrong action. This paper presents ActivAsk, a zero-shot framework for resolving referential ambiguity before robotic grasping. ActivAsk constructs open-vocabulary candidates from red-green-blue-depth (RGB-D) input, asks candidate-grounded yes/no questions when needed, updates the candidate state from the user’s answer, and grasps after target resolution. It selects among VLM-proposed candidate partitions using an expected free energy (EFE) criterion motivated by active inference; with neutral response preferences, this reduces to information gain over candidate partitions. Offline experiments showed that interactive clarification improved target accuracy from about 53–54% for noninteractive baselines to about 90–92%. ActivAsk matched the best interactive accuracy (92.13%) while asking 15.47–19.71% fewer questions on asked trials and 21.43–23.88% fewer for ambiguous instructions. In online real robot experiments, ActivAsk achieved 92.98% target selection accuracy and 87.72% full correct object grasp success; unresolved or wrong targets were not physically executed after operator-controlled verification and were counted as task failures.

1. Introduction

Natural language instruction is a natural way for people to communicate with home and service robots. A user can say “bring me my cup” or “give me the spoon” without specifying object identifiers, image coordinates, grasp poses, or robot commands. This reduces interaction effort and can make assistance feel more natural and unobtrusive, especially for non-expert users in everyday environments where objects, layouts, and user needs change frequently. Tabletop grasping is a suitable and important target task for this problem because many service-robot behaviors begin with identifying and grasping the intended object. Fetching, handover, object placement, cleaning, and longer-horizon household assistance all depend on this first step: the robot must understand which physical object the user is referring to before it can act.

Natural language instructions also bring referential ambiguity. When people speak to each other, communication often relies on shared context, such as ownership, previous dialogue, visual attention, task history, or background knowledge that is not explicitly stated [1]. A command such as “pick up my cup” may be clear between people who share this context, but it can be ambiguous to a robot that has never helped the user before, cannot infer object ownership, cannot identify which person issued the command, or cannot connect the expression to a unique object in the current scene. The robot may understand the object category, yet still not know which physical instance is intended. Many robot systems implicitly assume that the given instruction is clear enough for execution and map the instruction directly to a target or action. For referentially ambiguous commands, this assumption can be problematic. A system may select the highest-confidence detection, randomly select one candidate, or ask a vision–language model (VLM) to directly output a target. These strategies can produce a visually plausible but insufficiently grounded answer while the intended target is still ambiguous. The result may be a wrong object grasp, a failed task, or unnecessary robot motion; in poorly constrained settings, acting while ambiguity remains may also create unsafe behavior near the user, nearby objects, or the robot workspace.

Recent work has made strong progress in connecting language, vision, and robot action. Language-conditioned manipulation systems combine natural language commands with visual manipulation policies [2,3], use large language models for planning and control generation [4,5,6], and scale vision–language–action learning to broader task distributions [7,8,9]. These methods substantially expand what a robot can execute from language, although they generally begin from a goal or target that is sufficiently specified for action. Closed-set detectors operate over predefined categories [10], whereas open-vocabulary grounding localizes objects from flexible category names and referring expressions [11,12,13,14,15]. Promptable segmentation and RGB-D manipulation methods provide reusable components for object masks, geometric reasoning, and physical execution [16,17,18,19]. Grounding confidence measures compatibility between a visual region and the supplied language. The user’s intention remains latent. When several objects satisfy the same instruction, the grounding output should remain a set of plausible candidates until clarification identifies the intended referent.

When perception and the initial instruction leave several referents plausible, interactive clarification can provide the missing evidence. Earlier work has studied robot-initiated questions, dialogue-based grounding, and interactive object selection [20,21,22,23,24]. Open-ended approaches allow the user to introduce new attributes, spatial relations, or corrections through free-form language [21,22,24], potentially resolving ambiguity in one response, although the system must interpret and ground that response again. Constrained and model-based approaches use attributes, explicit alternatives, yes/no questions, pointing, or predefined action choices to obtain more structured observations and support more stable belief or state updates [20,23,25,26]. More recent uncertainty-aware and VLM-based systems determine when assistance is needed, reason jointly over language and visual context, generate clarification questions, or learn multi-turn ask-and-act behavior [27,28,29,30]. These studies establish the value of clarification before action. Explicit comparison of several candidate questions according to their expected reduction of the remaining ambiguity remains less developed.

The remaining challenge is to make this clarification efficient and controllable. The robot must preserve the currently plausible referents, ensure that each possible response corresponds to a valid state update, and select a question according to its expected reduction of the remaining ambiguity. Open-ended interaction offers flexibility but requires repeated language grounding. Constrained methods provide controlled updates but may rely on predefined question or action spaces. Recent VLM-based approaches commonly place ambiguity reasoning, question generation, and question choice within the same prompted or learned model. Tabletop grasping provides an end-to-end setting in which unresolved ambiguity immediately affects physical action. The same decision problem arises whenever a robot must resolve a referent through clear, controlled, and efficient clarification before acting.

To address this gap, we propose ActivAsk, a zero-shot active clarification framework, based on expected free energy (EFE), for robotic grasping under ambiguous natural language instructions. As illustrated in Figure 1, ActivAsk starts from an ambiguous user instruction and first constructs a set of plausible target candidates through open-vocabulary grounding. When multiple candidates remain plausible, the system enters a clarification loop before any object is selected. A VLM proposes candidate-grounded yes/no questions, and each question contains both user-facing text and a partition of the current candidate set into yes and no branches. ActivAsk then selects an informative question, updates the candidate state according to the user’s answer, and repeats this process until a unique target is resolved or no valid target remains. The robot executes through a general RGB-D grasping pipeline only after ambiguity has been resolved, so the clarification process acts as a decision layer before physical motion.

Figure 1. Conceptual overview of ActivAsk. An ambiguous instruction can match several objects in the robot workspace. ActivAsk constructs plausible candidates, asks candidate-grounded yes/no clarification questions, selects an informative question using an expected free energy (EFE) rule, updates the candidate state from the user answer, and executes the grasp only after the intended target is resolved.

In ActivAsk, the EFE criterion improves clarification efficiency. A clarification question is treated as an epistemic action because its answer provides information about the hidden intended target. The VLM proposes candidate-grounded yes/no questions, and ActivAsk ranks their partitions according to the expected reduction of target uncertainty. The criterion is motivated by active inference [31,32,33,34,35] and its application to robotics [36]. Under the response-neutral setting, yes and no receive equal preference, so the deployed ranking reduces to information gain over candidate partitions. Section 2 places this mechanism in the literature, and Section 3.4.3 gives the formal definition.

ActivAsk serves as an efficient ambiguity-resolution layer between open-vocabulary candidate grounding and RGB-D grasp execution. The detector, VLM, and grasp planner remain modular components: perception provides plausible candidates, clarification resolves the intended referent, and grasping begins after target resolution. This positioning makes ActivAsk complementary to existing perception, language-conditioned manipulation, interactive ambiguity resolution, and robotic grasping methods.

Within this scope, the main contributions of this paper are as follows:

• We introduce a new candidate-grounded yes/no clarification representation for zero-shot robotic grasping under referential ambiguity. Each VLM-generated question is explicitly represented as a partition of the current candidate state, allowing the system to evaluate a proposal before presentation and separating policy selection from the VLM’s direct choice.
• Building on this representation, we propose an EFE-derived question-selection policy that evaluates VLM-generated candidate partitions according to their expected reduction in target uncertainty and selects the most informative valid question before it is presented to the user.
• We integrate the ambiguity-resolution layer with a real Stretch 3 grasping pipeline to examine whether improved target decisions transfer to correct object grasping.
• We analyze the remaining failure modes across candidate generation, clarification, experimental verification, and physical execution to identify bottlenecks in the complete system.

These contributions correspond one-to-one to the four research questions examined in Section 4. In the offline experiment, noninteractive baselines achieved about 53–54% target resolution accuracy, whereas interactive clarification methods achieved about 90–92%; ActivAsk maintained the best interactive accuracy while asking fewer questions, especially for ambiguous instructions. In the online experiment, the improved target decisions led to higher correct object grasping success under the same RGB-D grasping pipeline.

2. Related Work

Language-conditioned manipulation has expanded the range of tasks that robots can execute from natural language input. CLIPort and VIMA learn manipulation behavior from demonstrations or multimodal task prompts [2,3]. SayCan combines language-model likelihoods with learned affordances, Code as Policies generates executable programs over perception and control interfaces, and VoxPoser constructs spatial value maps for motion generation [4,5,6]. RT-1, PaLM-E, and RT-2 further integrate language, visual observations, and robot actions at larger scales [7,8,9]. These systems differ in training data, planning interface, and action representation. Their main focus is execution after the task and relevant referents have been specified. Improving the downstream policy alone does not resolve an instruction that remains compatible with several physical objects.

Open-vocabulary grounding addresses the complementary problem of connecting flexible language with visual regions. Closed-set detectors operate over predefined categories [10], whereas vision–language representations and open-vocabulary detectors support broader textual queries [11,12]. MDETR and GLIP align image regions with words or referring expressions, and GroundingDINO produces open-set detections from category names and descriptive phrases [13,14,15]. Promptable segmentation and RGB-D manipulation methods can subsequently convert a selected region into a mask, point cloud, grasp pose, or executable trajectory [16,17,18,19]. These components provide candidate objects and physical geometry. Their scores measure compatibility with the supplied expression, leaving the user’s intended referent unresolved when several visual matches are valid.

When grounding leaves several valid matches, interactive robotics can obtain the missing information from the user. Earlier work examined informative robot questions and dialogue-based updates to grounded language models [20,21]. Open-ended systems such as INGRESS and related interactive grounding approaches allow users to introduce attributes or relations that were not anticipated in advance [22,24]. This expressive response channel may resolve ambiguity quickly. Each new utterance still requires interpretation and grounding and may itself be vague, incomplete, or ambiguous. Constrained and model-based approaches use predefined attributes, alternatives, spoken or pointing actions, inverse-semantics requests, or explicit belief-state models [20,23,25,26]. They support more controlled updates, although their question spaces may be predefined and a valid response structure does not determine which question will be most informative.

Recent language and vision–language models extend these interaction strategies to broader forms of ambiguity. CLARA reasons about whether a command is clear, ambiguous, or infeasible through a prompted language-model pipeline [27]. AmbResVLM jointly grounds task-relevant objects, detects ambiguity, generates natural language questions, and interprets user feedback from visual and linguistic context [28]. Ask-to-Clarify studies multi-turn ambiguity resolution, while related embodied help-seeking approaches learn when communication should precede execution [29,30]. These methods support proactive ambiguity resolution. In most of them, ambiguity reasoning, question generation, and question choice remain within the same learned or prompted model; several alternatives are not explicitly exposed and compared according to their predicted effects on the remaining referents. Comparing several valid questions requires an explicit valuation criterion. Active inference provides a general framework for evaluating actions through their expected epistemic and pragmatic consequences [31,32,33,34,35,36]. A clarification question can be treated as an epistemic action because its answer changes the robot’s belief over the hidden intended referent. Classical information gain similarly evaluates an experiment through the uncertainty expected to be removed by its outcome [37]. Under the uniform belief, deterministic binary-response model, and neutral response preference used here, the deployed EFE score and information gain rank candidate partitions identically. The EFE formulation provides a broader objective in which nonuniform intention beliefs, answer reliability, interaction cost, user preferences, and execution-related consequences can later be represented.

The unresolved issue is how to value alternative clarification questions after ambiguity has been recognized. Open-ended interaction supports rich information but requires repeated grounding; constrained methods provide stable updates but often depend on predefined alternatives; and VLM-based systems leave the effects of alternative questions implicit. ActivAsk preserves open-vocabulary grounding results as an explicit candidate state, requires each VLM-generated yes/no question to define a validated partition, and applies an external EFE-derived criterion before presenting one proposal. The candidate state separates visual compatibility from user intention, the validated branches provide a controlled update, and the external selector distinguishes question valuation from language generation. The framework is designed to resolve referential ambiguity with fewer unnecessary interactions before action.

3. Materials and Methods

3.1. Problem Formulation and Assumptions

We consider zero-shot robotic grasping from a meaningful natural language instruction that may refer to more than one visible object. The instruction is assumed to describe an executable object-level task, but it may not uniquely identify the intended physical target. For example, an instruction such as “pick up my cup” may be compatible with several visible cups. This creates referential ambiguity: the task category is understandable, but the intended referent remains hidden from the robot.

Each trial provides a natural language instruction u and an RGB-D observation

$$o=(I,D,K,T_{\mathrm{wc}}),$$

(1)

where I is the RGB image, D is the aligned depth image, K is the camera intrinsic matrix, and $T_{\mathrm{wc}}$ denotes the rigid transform from the camera frame $\mathrm{c}$ to the robot planning (world) frame $\mathrm{w}$. The user’s intended target is represented as an unobserved discrete variable z. The robot observes the instruction, image, depth, and candidate detections, but it does not directly observe which candidate object the user intends.

Open-vocabulary perception converts the instruction and observation into a finite candidate set

$$\mathcal{C}={\left\{c_i\right\}}_{i=1}^N.$$

(2)

Each candidate $c_i$ contains an internal identifier, an overlay display identifier, an object label ${\ell }_i$, a detector confidence score $s_i$, and an image-space bounding box $b_i=[x_{1i},y_{1i},x_{2i},y_{2i}]$. The overlay display identifier is used for visualization and diagnostics, but it is excluded from user-facing clarification questions. If the intended object is included in the candidate set, then $z\in \mathcal{C}$. If the intended object is absent from $\mathcal{C}$, later clarification cannot recover it.

The clarification state at round t is a plausible candidate set $S_t\subseteq \mathcal{C}$. Initially, $S_0$ contains the candidates that visually and semantically match the instruction after candidate construction. The initial state defines three cases. If $|S_0| =0$, the trial fails at candidate construction because no target result is available. If $|S_0| =1$, the target is already resolved. If $|S_0| >1$, the instruction remains referentially ambiguous and the clarification loop is activated.

Clarification is formulated as a sequence of candidate state updates. A yes/no question $q_t$ induces two branches over the current candidate state: $Y\left(q_t\right)$, the candidates retained after a yes answer, and $N\left(q_t\right)$, the candidates retained after a no answer. A valid question must satisfy

$$Y\left(q_t\right)\cap N\left(q_t\right)=\varnothing ,Y\left(q_t\right)\cup N\left(q_t\right)=S_t.$$

(3)

After receiving the user response $r_t\in \{\mathrm{yes},\mathrm{no}\}$, the system updates $S_t$ to the corresponding branch. The clarification goal is to obtain a unique resolved target $\widehat{z}$ while using as few user-facing questions as possible.

We consider a zero-shot setting: no task-specific training is performed for the detector, the VLM, the question-selection policy, or the grasping policy, and calibration only selects operating thresholds and execution parameters. We assume one intended target per instruction, a visible target within the robot workspace, and a cooperative yes/no response channel. The observation model treats each response as corresponding to the intended target. Missed detections, incorrect labels, unclear wording, invalid VLM-generated partitions, and physical grasp failures remain possible system-level failure modes.

3.2. Proposed System Overview

As shown in Figure 2, ActivAsk consists of three main components: perception, the clarification loop, and execution. The perception component constructs a finite candidate set from the natural language instruction and RGB-D observation. The clarification loop resolves ambiguity by selecting informative candidate-grounded yes/no questions using the EFE-based criterion and updating the candidate state according to the user’s answer. The execution component applies a general-purpose RGB-D grasping policy after a unique target has been resolved. In this architecture, ActivAsk acts as a layer for resolving ambiguity before action, placed between open-vocabulary candidate grounding and physical grasp execution.

Figure 2. System overview of ActivAsk. Perception constructs a candidate state from the instruction and RGB-D observation. If multiple candidates remain, the clarification loop repeatedly proposes VLM-generated yes/no questions, selects one with the EFE-based criterion, and updates the candidate state from the user’s answer until a unique target is resolved or no candidate remains. Execution applies the shared RGB-D grasping pipeline after target resolution. The star icon and the superscript star in $q_t^{★}$ mark the selected question; cross-marks indicate failed branches; CP = conformal prediction, SAM = Segment Anything Model, and IK = inverse kinematics. The “correct” and “wrong” labels indicate evaluation outcomes, not information available to the policy during inference.

The architecture separates the main responsibilities of the complete system. Perception determines whether plausible target candidates are available. Clarification determines whether the intended referent can be reduced to a single candidate with a small number of user-facing questions. Execution determines whether the resolved target can be physically grasped. The following subsections follow the same structure as Figure 2: Section 3.3 describes zero-shot candidate construction, Section 3.4 describes the ActivAsk clarification loop, and Section 3.5 describes RGB-D grasp planning and Stretch 3 execution.

3.3. Perception: Open-Vocabulary Candidate Construction

The perception component receives the instruction u and the RGB-D observation o, and it outputs the finite candidate set $\mathcal{C}$ together with object labels, detector confidence scores, and image-space boxes. Its role is high-recall candidate construction: perception should preserve plausible visual hypotheses that may match the user’s instruction, while the clarification loop later decides which remaining hypothesis refers to the intended object.

The detector query is formed from the object noun and visible modifiers in the instruction. The object noun defines the category-level search space, while modifiers such as color, size, material, or spatial relation can be included when they are visually grounded in the image. For example, a color-specific instruction may first be queried as a modified phrase such as “orange cup”, whereas a more underspecified instruction may use a broader object-level query such as “cup”. If a specific query returns an overly narrow set or risks eliminating plausible same category alternatives, the system uses the broader category query to retain a larger candidate set. This is important in referential ambiguity: a phrase such as “my cup” may require all visible cups to remain available, and even a phrase such as “orange cup” may still correspond to multiple orange cups in the scene. The perception stage avoids treating the most specific detector result as the final answer.

This recall-first design may leave extra candidates for clarification, but it preserves the possibility of resolving the intended target through later questions. By contrast, if the intended target is absent from $\mathcal{C}$, the clarification loop can only choose among incorrect hypotheses.

GroundingDINO implements the open-vocabulary candidate generator [15]. A detection proposal is retained when its confidence score satisfies $s_i\ge \tau$, with $\tau =0.38$. The detector also uses a text threshold of 0.30, a non-maximum suppression (NMS) threshold of 0.50 on intersection over union (IoU), and at most 100 detections per image. Each retained proposal becomes a candidate $c_i$ with label ${\ell }_i$, score $s_i$, and box $b_i$. Boxes are used for visualization, candidate bookkeeping, and downstream target cropping; detector confidence is stored as metadata but is not treated as evidence that the candidate is the user’s intended target.

The initial candidate state $S_0$ is then derived from retained proposals that visually and semantically match the instruction. Its possible outcomes correspond to the three initial-state branches defined in Section 3.1 and the perception branch in Figure 2. Calibration images did not overlap with offline or online evaluation scenes. Appendix A reports the threshold sweep, the conformal-prediction rationale for the threshold choice, query expansion, and threshold-dependent recall, precision, F1, and false positives.

3.4. Clarification Loop

3.4.1. Structured VLM Question Proposal

The structured proposal module produces candidate-grounded clarification actions when the current state remains ambiguous. It receives the current candidate state $S_t$, the candidate overlay image, the user instruction, candidate metadata, and the dialogue history. Its output is a finite set ${\mathcal{Q}}_t$ of yes/no questions, where each question includes both user-facing text and yes/no branches over candidates. The binary response channel supports deterministic updates and avoids grounding a new free-form answer after every user response.

Xiaomi MiMo Vision–Language (MiMo-VL-7B-RL-2508) is used as the VLM proposal generator [38]. It generates candidate-grounded clarification actions for downstream policy evaluation. The EFE objective remains outside the proposal prompt, and the policy selects among valid proposals. Dialogue history helps the model avoid repeating answered questions, while candidate metadata and the overlay image expose visible distinctions among the remaining objects.

Each structured proposal contains human-facing text and candidate-level yes/no branches. These branches specify which targets remain plausible after either answer, allowing the belief update and EFE score to operate on explicit candidate partitions while avoiding unconstrained dialogue. Table 1 summarizes the required fields.

Table 1. Compact structured clarification schema used by the vision–language model (VLM) interface.

Field	Meaning
Question text	Human-facing yes/no question written only in terms of visible object properties.
Yes candidates	Candidates retained if the user answers yes.
No candidates	Candidates retained if the user answers no.
Validity flags	Checks that the yes/no branches are exhaustive, disjoint, and limited to current candidates.
Chosen index	Used only by the VLM-selected baseline to mark the proposal it judges most useful.

The proposal interface uses the same base system prompt, image overlay, candidate metadata, current state, dialogue history, decoding parameters, output schema, and candidate-grounding requirements across interactive policies. A short policy-specific instruction identifies the downstream selection rule, while question-generation guidance remains in the shared system prompt. The proposal count is an implementation parameter; the figures show four proposals as an example, while the formulation only requires a finite set ${\mathcal{Q}}_t$. Appendix B provides the prompt, policy instructions, validation rules, and controlled protocol comparison.

3.4.2. Candidate-State Update and Validation

The update and validation module is responsible for turning a proposed clarification question into a well-defined state transition. It receives a proposed question q, its candidate branches, the current state $S_t$, and the user response $r_t$. It outputs either a validated next state $S_{t+1}$ or a failed target resolution outcome if the branches are invalid or no candidate remains. For every proposed question, the VLM must produce two candidate subsets, $Y\left(q\right)$ and $N\left(q\right)$, satisfying Equation (3). In this sense, a question is not merely a text string; it is a semantic partition of the current plausible set.

The text shown to the user asks about visible object properties such as color, size, shape, material, handle, lid, left/right position, or relation to another object. It must not refer to detector scores, raw bounding-box coordinates, thresholds, IoU values, or candidate identifiers, because these internal system variables are not natural user-facing descriptors.

Before a proposed question can update the trial, the system checks that the counts match the candidate lists, the lists contain only current plausible candidates, the yes and no sets are disjoint and exhaustive, and the wording satisfies the user-facing constraints. These automated checks establish structural validity and wording compliance; semantic correctness requires a separate visual assessment of the description and branch assignments. The checks are necessary because the policy score assumes that each candidate belongs to exactly one response branch. If a proposal omits a candidate, assigns a candidate to both branches, or refers to an internal identifier instead of a visible property, then the answer cannot define a reliable state update. Invalid proposals are repaired when the correction is unambiguous; otherwise, they are replaced by rule-based fallback questions generated from visible candidate labels and positions. Once a valid question is selected and answered, the update is deterministic:

$$S_{t+1}=\left\{\begin{array}{cc}{S}_t\cap Y\left(q_t\right),\hfill & r_t=\mathrm{yes},\hfill \\ S_t\cap N\left(q_t\right),\hfill & r_t=\mathrm{no}.\hfill \end{array}\right.$$

(4)

Candidates removed from $S_t$ are not reintroduced. If $|S_{t+1}| >1$, the loop continues with the reduced candidate state. If $|S_{t+1}| =0$, the trial fails because no target result remains. If $|S_{t+1}| =1$, the remaining candidate becomes the resolved target $\widehat{z}$.

3.4.3. EFE-Based Question Selection

The EFE-based selection module chooses which valid clarification question is shown to the user. It receives the current candidate state $S_t$, the posterior belief over candidates, and the valid proposal set ${\mathcal{Q}}_t$ from the structured VLM interface. It outputs one selected question $q_t$ for user presentation. At round t, the robot represents its posterior belief over the hidden target by $Q_t\left(z\right)$, supported on $S_t\subseteq \mathcal{C}$. The deployed policy initializes $Q_0\left(z\right)$ uniformly over the plausible candidates because no user-specific intention prior is available. With deterministic candidate elimination, the posterior remains uniform over the retained candidates. The target z is fixed within a trial; only the robot’s belief changes after the answer $r_t\in \{\mathrm{yes},\mathrm{no}\}$. The VLM proposes ${\mathcal{Q}}_t={\left\{q_k\right\}}_{k=1}^K$, where each question defines a one-step policy ${\pi }_k=\mathrm{ask}\left(q_k\right)$. For the yes/no branches in Equation (3), the deterministic observation model is

$$\begin{array}{cccc}\hfill P(r=\mathrm{yes}\mid z,{\pi }_k)& =\left\{\begin{array}{cc}1,\hfill & z\in Y\left({\pi }_k\right),\hfill \\ 0,\hfill & z\in N\left({\pi }_k\right),\hfill \end{array}\right.\hfill & \hfill P(r=\mathrm{no}\mid z,{\pi }_k)& =\left\{\begin{array}{cc}1,\hfill & z\in N\left({\pi }_k\right),\hfill \\ 0,\hfill & z\in Y\left({\pi }_k\right).\hfill \end{array}\right.\hfill \end{array}$$

(5)

After observing an answer, the robot updates its posterior using Bayes’ rule:

$$Q_{t+1}\left(z\right)={\displaystyle \frac{Q_t\left(z\right)P(r_t\mid z,{\pi }_k)}{{\sum }_{z^{\prime }\in S_t}{Q}_t\left(z^{\prime }\right)P(r_t\mid z^{\prime },{\pi }_k)}}.$$

(6)

The candidate state follows Equation (4). The predicted response masses are

$$\begin{array}{cc}\hfill p_Y\left({\pi }_k\right)& =\sum _{c_i\in Y\left({\pi }_k\right)}{Q}_t\left(c_i\right),\hfill \\ \hfill p_N\left({\pi }_k\right)& =\sum _{c_i\in N\left({\pi }_k\right)}{Q}_t\left(c_i\right).\hfill \end{array}$$

(7)

For each validated proposal ${\pi }_k$, the candidate identifiers assigned to its yes and no branches define $Y\left({\pi }_k\right)$ and $N\left({\pi }_k\right)$. Equation (7) sums the current belief mass within each branch to obtain the predicted probabilities of the two possible responses. These probabilities are determined by the candidate belief and the proposed partition. The same formulation supports a calibrated nonuniform intention prior: the equation then operates on branch probability mass rather than candidate cardinality, so the most informative partition need not divide the candidates into equally sized sets. The resulting branch masses are substituted into the EFE objective below, allowing all valid proposals in ${\mathcal{Q}}_t$ to be compared numerically before one question is selected.

ActivAsk uses EFE, motivated by active inference, as the criterion for selecting among the one-step clarification policies. For the predictive distribution $Q_t(r,z\mid {\pi }_k)=P(r\mid z,{\pi }_k)Q_t\left(z\right)$, the policy score is

$$G\left({\pi }_k\right)={\mathbb{E}}_{Q_t(r,z\mid {\pi }_k)}\left[\ln Q_t\left(z\right)-\ln Q_t(z\mid r,{\pi }_k)-\ln P^{★}\left(r\right)\right],$$

(8)

where $Q_t(z\mid r,{\pi }_k)$ is the answer-conditioned posterior, and $P^{★}\left(r\right)$ is the response preference distribution. The first two terms equal the negative mutual information $-I_t(z;r\mid {\pi }_k)$ and quantify epistemic value. In the present task, the robot values reduction of uncertainty over the intended target independently of answer polarity. Under a cooperative response channel and deterministic candidate partitions, “yes” and “no” answers are equally acceptable as long as they reduce target uncertainty. We therefore use a neutral response preference, $P^{★}\left(\mathrm{yes}\right)=P^{★}\left(\mathrm{no}\right)=1/2$. The neutral response preference does not assume that yes and no are equally likely for a given question; the predicted response masses $p_Y\left({\pi }_k\right)$ and $p_N\left({\pi }_k\right)$ are still determined by the candidate partition in Equation (7). Under this neutral preference, the pragmatic term is constant: because the predicted response probabilities sum to one, it contributes $\ln 2$ to every policy and therefore does not affect the $\arg \min$. Figure 3 summarizes this one-step comparison over several proposed candidate partitions.

Figure 3. Expected-free-energy (EFE)-based selection over candidate-grounded yes/no questions. The vision–language model (VLM) proposes several questions, each inducing yes/no branches, and the policy selects the proposal with the lowest deployed EFE score G. The superscript star denotes the selected policy/question. The user answer determines the updated candidate state.

The structured VLM output makes Equation (8) operational. Without a calibrated target prior, the deployed system uses the uniform belief

$$Q_t\left(c_i\right)=\left\{\begin{array}{cc}|S_t{|}^{-1},\hfill & c_i\in S_t,\hfill \\ 0,\hfill & c_i\notin S_t.\hfill \end{array}\right.$$

(9)

Substituting the uniform belief in Equation (9) into the general branch mass definition in Equation (7) gives

$$p_Y\left({\pi }_k\right)={\displaystyle \frac{\left|Y\right({\pi }_k\left)\right|}{|S_t|}},p_N\left({\pi }_k\right)={\displaystyle \frac{\left|N\right({\pi }_k\left)\right|}{|S_t|}}.$$

(10)

After omitting the constant pragmatic term, the deployed score is

$$G\left({\pi }_k\right)=p_Y\left({\pi }_k\right)\ln p_Y\left({\pi }_k\right)+p_N\left({\pi }_k\right)\ln p_N\left({\pi }_k\right),$$

(11)

with zero-mass terms omitted. The deployed score is equivalent to the negative entropy of the induced yes/no partition, matching the classical information gain view of experiment selection [37]. Under the uniform prior, deterministic binary partitions, and neutral response preference used here, EFE and simple information gain rank all valid questions identically. The current implementation is thus an EFE formulation of information-gain selection. The unified objective represents beliefs, observations, and preferences, and can incorporate pragmatic factors such as question cost, answer reliability, user specific preferences, and execution-related penalties without changing the candidate-grounded selection mechanism. Minimizing the deployed score favors balanced branch probabilities, and ActivAsk selects ${\pi }^{★}=\arg {\min }_{{\pi }_k}G\left({\pi }_k\right)$. The associated question is shown to the user, and the answer updates $S_t$ through Equation (4).

3.5. Execution: RGB-D Grasp Planning and Stretch 3 Control

The execution component receives a resolved target $\widehat{z}$ and the RGB-D observation, and it outputs a waypoint sequence for the Stretch 3 robot. Its role is to convert the resolved candidate into a physically executable top-down grasp. The implementation follows established RGB-D and point-cloud grasping practice [17,18,19].

The default execution path uses the resolved target’s detection box as an input to Segment Anything (SAM) [16], clips the mask to the local detection region, and uses the masked depth pixels as the target crop. The crop is back-projected through the RGB-D camera intrinsics and transformed into the robot planning frame; finite points inside the tabletop workspace and above the estimated table plane are retained for geometric estimation.

The planner estimates a robust vertical extent from height percentiles and extracts a center-height slice to reduce sensitivity to table points, rims, and missing depth. For compact objects, the slice is approximated by a planar cross-section; the grasp center is placed near the cross-section center, the gripper yaw is chosen from candidate horizontal directions, and the opening width is set from the estimated object diameter plus clearance. For slender objects such as forks and spoons, the planner switches to a principal component analysis (PCA)-based grasp: the major and minor planar axes of the slice determine the object orientation, and the fingers close across the narrow dimension.

The geometric planner outputs a grasp command $g=(x_{\mathrm{g}},y_{\mathrm{g}},z_{\mathrm{g}},{\psi }_{\mathrm{g}},w_{\mathrm{g}})$, where $(x_{\mathrm{g}},y_{\mathrm{g}},z_{\mathrm{g}})$ is the target grasp point, ${\psi }_{\mathrm{g}}$ is the wrist yaw or closing direction, and $w_{\mathrm{g}}$ is the gripper opening width. The Stretch execution layer solves inverse kinematics for a top-down pose within joint limits and executes ordered waypoints for pregrasp positioning, descent, gripper closure, lift, and arm retraction, following standard serial-manipulator kinematics practice [39]. After any large base repositioning, the system reobserves the scene and replans before final execution so that the grasp is not based on stale target coordinates.

4. Experimental Design

4.1. Experimental Goals, Overview, and Platform

The evaluation has two complementary roles. The offline experiment isolates target resolution and clarification efficiency without robot motion. The online experiment tests a representative policy subset with live sensing and grasp execution, where completion also depends on the downstream execution stages (segmentation, reachability, and contact).

Our experiments are organized around four research questions:

RQ1:

Does interactive clarification improve target resolution over noninteractive baselines?

RQ2:

Does EFE-minimizing selection reduce clarification burden while maintaining accuracy?

RQ3:

Do improved target decisions transfer to real robot grasping success?

RQ4:

Which failure modes remain in the complete system?

The main controlled difference is the target resolution policy. Candidate generation uses the same detector setting, interactive methods share the same candidate state representation and VLM interface, and online methods share the same RGB-D execution pipeline. In the implementation used for all interactive policies, the VLM was asked to propose up to four candidate questions at each clarification round. This fixed parameter provides proposal diversity for comparing selection rules while keeping the VLM interface, latency, and user-facing evaluation manageable. The main experiments did not vary this setting; Section 5.3 reports a supplementary analysis of proposal counts from one to five. To check whether the short policy-specific instructions altered proposal quality, the controlled protocol in Appendix B compares proposal informativeness across identical scene–instruction contexts. Proposal count is an implementation parameter of the evaluation.

Both experiments use cup-only, bottle-only, utensil-only, and mixed tabletop layouts with randomized placement, varied spacing, same category instances, and partial lighting variation. Scene-specific instruction pools reflect the visible objects and layout. Instruction types were assigned according to fixed criteria before the experiments. A clear instruction contains one or more modifiers that uniquely identify the intended object in the scene, such as a distinctive color, size, position, or combined description. A partial instruction contains a useful modifier, but that modifier is not unique in the current scene or may be spatially or contextually ambiguous, leaving multiple plausible targets. An ambiguous instruction provides little or no distinguishing modifier beyond the object category or context, such as “pick up my cup,” and needs extra contextual resolution. Figure 4 illustrates these three instruction types.

Figure 4. Representative instruction ambiguity examples used in the target resolution experiments. A clear instruction uniquely identifies one visible target; a partial instruction provides useful visual information but still leaves multiple plausible targets; and an ambiguous instruction contains few or no distinguishing modifiers, leaving several candidate objects compatible with the command and requiring additional clarification.

To make referential ambiguity concrete in the experimental scenes, each scene family contained repeated plausible alternatives. Cup-only scenes contained six cups in both experiments, while bottle-only scenes contained four bottles offline and three bottles online. Utensil-only scenes contained four to six objects: the online set included two scenes with four forks, two scenes with four spoons, and one scene containing six objects with four forks and two spoons; the offline set included three scenes with four forks, three scenes with four spoons, and three mixed utensil scenes containing six objects with fork to spoon ratios of 2:4, 3:3, and 4:2. Mixed category scenes contained eight objects, comprising two or three cups, two bottles, and two or three forks or spoons. Layouts with objects from the same category stress distinctions among visually similar instances, whereas the mixed layouts test whether target resolution remains reliable when category and attribute cues interact across a more diverse scene.

Each trial follows a predefined procedure to reduce operator-selection bias. For each trial, an instruction of the designated type is drawn from the predefined scene specific instruction pool. The operator then identifies the visible objects that correctly match the drawn instruction and randomly selects one of them as the intended target, without preference among the matching objects; if exactly one object matches, that object becomes the intended target. This target selection is performed outside the experimental system, and the intended target is never provided to the system. The same instruction pools and procedure are used across methods. During interactive trials, the operator answers according to the retained intended target and the visible scene, without revising the policy output.

The online platform is a Hello Robot Stretch 3 (Hello Robot Inc., Martinez, CA, USA) with a head-mounted Intel RealSense D435i RGB-D camera (Intel Corporation, Santa Clara, CA, USA) [40]. It captures aligned $1280\times 720$ color and depth at 15 Hz. Head pan and tilt are set to $-1.57$ and $-0.68$ rad so that the tabletop occupies a usable image region while limiting arm occlusion. A workstation equipped with four NVIDIA RTX 6000 Ada graphics processing units was used, with one graphics processing unit assigned to each run for detection, Xiaomi MiMo-VL inference, SAM segmentation, EFE scoring, and grasp planning; the robot executes the generated waypoint trajectory. Candidate construction uses $\tau =0.38$, and the default execution crop is the SAM mask conditioned on the box from Section 3.5. Before motion, the operator-controlled experimental verification is used for execution control and logging only; it does not revise the policy output.

4.2. Offline Target Resolution Experiment

The offline experiment evaluates perception-language target resolution and clarification without robot-motion variability. It contains 1512 trials across 36 real tabletop scenes, with 216 trials per method and near-equal coverage of clear, partial, and ambiguous instructions. This permits comparison of seven policies while preserving physical scenes and on-site human answers.

Table 2 reports additional characteristics computed from the trial-level data. The initial multi-candidate rate is defined as $\mathrm{Pr}\left(\right|S_0| >1)$.

Table 2. Additional characteristics of the offline evaluation trials.

Characteristic	Value
Unique instruction strings	189
Initial plausible candidates $|S_0|$, mean (range)	2.49 (0–8)
Initial multi-candidate trials, $\mathrm{Pr}\left(\right|S_0| >1)$	697/1512 (46.10%)

Each trial follows the procedure described in Section 4.1 and uses the real tabletop scene. The sampled instruction is provided to the assigned policy, while the operator retains the intended target for evaluation. Candidate construction initializes $S_0$, after which the assigned policy is applied. For interactive policies, the operator answers on site by observing the scene and reading the displayed question. No fixed clarification-round limit is imposed; interaction continues until a unique target is resolved or the pipeline fails to resolve a target. Target correctness, question count, latency, and failure category are then recorded.

The comparison uses equal method counts and matched coverage by scene and instruction type; exact trial-level instruction and intended-target pairings are not matched. All methods are evaluated on the same 36 scenes, with 216 trials per method and near-equal coverage of clear, partial, and ambiguous instructions. Instructions are drawn from the same scene specific pools, and the intended target is selected from the visible objects satisfying the instruction using the same procedure for every method. Because target selection and VLM proposal generation occur independently for each trial, the exact instruction string, intended target, and runtime proposal set can differ across methods. Candidate set size is included in the statistical model to account for variation in initial ambiguity. Clarification proposals are generated at runtime from the observation, instruction, candidate state, and dialogue history.

Table 3 lists the policies. Noninteractive baselines test direct target selection, interactive baselines test the value of clarification, and comparisons among interactive policies test whether EFE-based selection reduces question burden while maintaining accuracy.

Table 3. Target resolution policies compared in the offline and online experiments.

Policy	Clarifies	Offline	Online	Decision Rule
Top score detector	No	Yes	Yes	Highest detector confidence.
Random candidate	No	Yes	Yes	Uniform candidate sample.
Direct VLM decision	No	Yes	No	VLM selects target directly.
First VLM question	Yes	Yes	No	First valid VLM question.
Random VLM question	Yes	Yes	No	Uniform valid question sample.
VLM-selected question	Yes	Yes	Yes	VLM-preferred valid question.
ActivAsk	Yes	Yes	Yes	Lowest EFE over valid partitions.

The noninteractive policies differ only in how they choose a target from the same candidate set. All interactive policies enter the same clarification loop when $|S_0| >1$, use the same candidate state update after each yes/no answer, and differ only in how the question is selected from the valid proposals. Table 3 gives the corresponding decision rule for each policy.

The primary offline outcome is target resolution correctness: a trial succeeds when the selected target matches the user’s intended target, while wrong selections and failed target resolution outcomes are failures. The asked rate is the proportion of trials in which at least one clarification question is presented. Question count is reported over all trials and, separately, over asked trials only. Latency and failure category are secondary outcomes. Recorded latency is the end-to-end experimental duration and includes VLM generation and repair, system overhead, user response time, and logging, so it is not interpreted as pure conversational latency.

4.3. Online Real-Robot Grasping Experiment

The online experiment tests the complete pipeline with live RGB-D observations and real grasp execution. It uses 20 tabletop scenes and 228 balanced trials, with 57 trials per method and 19 trials per cell defined by method and instruction type. Cooperative human operators represent the user during the online experiment: they retain the intended target and provide the corresponding yes/no clarification answers. In an independent deployment, these answers would be provided by the actual user whose intended referent the robot is trying to identify.

The online experiment uses four representative policies: top score detector, random candidate, VLM-selected question, and ActivAsk. This subset preserves the comparison between noninteractive selection, VLM-selected clarification, and EFE-based selection while limiting robot wear, operator time, and hardware risk.

In the online setting, top score and random candidate are executed as direct-selection controls: they produce a single target without asking a clarification question. VLM-selected and ActivAsk both enter the same clarification loop when multiple candidates remain; the former relies on the VLM’s marked preferred proposal, whereas the latter applies the externally computed EFE score to the same type of valid proposal set. After a target is resolved, all four methods use the same operator-controlled experimental verification before entering the shared RGB-D grasp execution pipeline.

Each online trial follows the procedure described in Section 4.1 and begins with the physical tabletop scene and a live head-camera RGB-D observation. The sampled instruction is provided to the assigned policy, while the operator retains the intended target as the evaluation reference. For interactive methods, the operator answers each displayed question by observing the scene and interpreting the question text. The candidate state is updated after every answer until the policy resolves a target or fails to do so.

Before execution, the operator compares the resolved candidate with the retained experimental ground truth. This verification is used only to label the target resolution outcome and to avoid physically executing a known incorrect target during evaluation. It is not part of the ActivAsk policy, is not returned to the policy, and cannot revise its output. A wrong resolved target is recorded directly as a failed trial. Trials with no valid target result (unresolved trials) are also not executed but remain task failures, while targets that pass the verification proceed through the same RGB-D grasp pipeline in Section 3.5. The online dataset keeps all trials and does not filter to successful target decisions. The trial stores target correctness, execution status, grasp success, full task success, wrong object execution, question count, latency, and failure category before the scene is reset.

The online metrics report both stages and their composition. Target-selection accuracy measures whether the selected target matches the intended target. Grasp-attempt rate records whether execution occurred after the operator-controlled verification. Physical success given an attempt requires the robot to grasp, lift, and retain the selected object and is conditioned on the target having passed this verification. Full task success requires both correct target selection and successful grasping, and it is computed over all trials. Wrong object grasping refers to physical execution on an unintended object. Experimental non-execution counts unresolved outcomes or wrong resolved targets that were not physically executed after the operator-controlled verification; these outcomes remain target resolution and full task failures.

4.4. Statistical Analysis and Failure-Mode Evaluation

We now describe the statistical analyses used to interpret both experiments. Offline target resolution accuracy is binary, with wrong selections and failed target resolution outcomes treated as failures. Because trials from one scene share layout, object categories, camera view, and candidate distributions, planned offline comparisons use scene-clustered logistic generalized estimating equations (GEEs) with robust standard errors [41]. Results are reported as odds ratios, confidence intervals, raw p values, and Holm-adjusted p values [42]. This model estimates population-level method differences under the balanced scene and instruction-type design; it is not a trial-paired analysis of identical instruction and intended-target combinations.

Question count and latency are aggregated at the scene, method, and instruction type level. Because counts are discrete and latency is skewed, planned paired comparisons use nonparametric tests and bootstrap confidence intervals [43,44], with Holm correction within each comparison family.

The online experiment has a smaller sample size and includes physical robot noise, so the analysis emphasizes descriptive proportions with Wilson confidence intervals [45], paired scene-level effect sizes, sign-flip permutation tests where appropriate [46], and consistency with the larger offline target resolution experiment. Full model formulas, instruction-feature robustness checks, software versions, and bootstrap settings are reported in Appendix C. A supplementary answer-error analysis evaluates the deterministic update under independent accidental answer reversals using recorded interactive trials; Appendix C reports the analysis design and complete results.

Failures are manually reviewed and categorized according to the pipeline stage at which the failure occurred. Offline failure analysis focuses on candidate generation, unresolved or wrong singleton target resolution outcomes, inefficient or redundant clarification, and VLM-related clarification errors. Online failure analysis additionally includes operator-controlled non-execution, segmentation or depth failure, grasp-planning or reachability failure, and physical grasp failure such as a miss, slip, or unstable lift. Failure counts and representative examples are reserved for the Results and Discussion sections.

In addition, a retrospective human audit assessed semantic consistency for 572 displayed clarification questions from 290 trials for which candidate branches were recorded. The audit separately evaluated whether each question was visually consistent with the scene and whether its recorded yes/no branches matched the distinction expressed in the question text.

5. Results

5.1. Descriptive Overview and Data Balance

Table 4 summarizes the evaluation trials and their balance across conditions.

Table 4. Dataset balance summary for the offline and online experiments.

Experiment	Scale	Balance	Exec.	Purpose
Offline	1512 trials; 36 scenes; 7 methods; 3 types	216/method; 503 clear; 503 partial; 506 ambiguous	No motion	Policy comparison
Online	228 trials; 20 scenes; 4 methods; 3 types	57/method; 19/method/type	Operator verified	Full-system validation

The offline data contain 216 trials for each policy and 503 clear, 503 partial, and 506 ambiguous trials. Table 5 reports descriptive results with Wilson 95% confidence intervals (CIs). Noninteractive baselines achieved 53–54% target accuracy, whereas interactive policies reached about 90–92%. ActivAsk resolved 199 of 216 trials correctly (92.13%, 95% CI [87.76, 95.03]), matching VLM-selected clarification while asking fewer questions when clarification was used.

Table 5. Offline descriptive results by method. Latency is reported in seconds, and Wilson 95% confidence intervals (CIs) are shown for accuracy.

Method	N	Correct	Acc.	95% CI	Asked (%)	Mean Q All	Mean Q Asked	Mean Time	Median Time
Top score	216	116	53.70%	[47.05, 60.23]	0.00%	0.00	–	6.77	5.00
Random candidate	216	115	53.24%	[46.59, 59.78]	0.00%	0.00	–	7.77	4.23
Direct VLM	216	116	53.70%	[47.05, 60.23]	0.00%	0.00	–	40.59	9.75
First question	216	197	91.20%	[86.67, 94.30]	46.76%	1.11	2.37	86.02	29.55
Random question	216	196	90.74%	[86.13, 93.93]	46.76%	1.12	2.39	90.71	40.99
VLM-selected	216	199	92.13%	[87.76, 95.03]	47.22%	1.09	2.30	77.69	33.61
ActivAsk	216	199	92.13%	[87.76, 95.03]	48.15%	0.92	1.91	74.84	42.46

Note: Bold values mark the best result, and underlined values mark the second-best result for target accuracy and question-count metrics where a clear direction is defined. Tied best accuracies are both bolded.

At the instruction-type level, clear instructions were easiest, with 483 correct trials out of 503 (96.02%) and a question asked in only 1.79% of trials. Partial instructions were harder, with 349 correct trials out of 503 (69.38%), a 26.64% asked rate, and 1.30 questions on asked trials. Ambiguous instructions were most demanding, with 306 correct trials out of 506 (60.47%), a 52.37% asked rate, and 2.75 questions on asked trials. Instruction difficulty manifests as both lower target resolution accuracy and greater clarification burden. Scene-type descriptive checks were used only as context: 75.74% for bottle-only scenes, 74.02% for cup-only scenes, 80.58% for mixed scenes, and 71.39% for utensil-only scenes.

Figure 5 shows the same pattern by method and instruction type. Clear instructions were usually resolved correctly even without interaction. The main separation appears under partial and ambiguous instructions, where direct target selection often chooses an unintended candidate, but interactive clarification keeps multiple hypotheses active until the intended target can be resolved. Some interactive policies show slightly higher accuracy on ambiguous instructions than on partial instructions. In these data, broad ambiguous instructions more consistently triggered clarification, whereas partial instructions sometimes appeared specific enough to proceed or narrowed the candidate state toward an unintended object. The reversal reflects how the instruction types activated the clarification loop, not their intrinsic difficulty.

Figure 5. Offline target resolution accuracy by method and instruction type. Cell values are accuracy percentages, and the horizontal divider separates noninteractive target selection from clarification policies. Noninteractive policies remain strong on clear instructions but degrade sharply on partial and ambiguous instructions, whereas interactive clarification preserves high accuracy across instruction types.

5.2. Offline Target Resolution Accuracy: Interaction Versus Direct Selection

The planned offline comparisons answer RQ1. Table 6 reports scene-clustered logistic GEE estimates. ActivAsk significantly outperformed top score, random candidate, and direct VLM selection after Holm correction, with adjusted odds ratios of 15.52, 15.88, and 15.81, respectively. VLM-selected clarification also strongly outperformed direct VLM selection. These results indicate that the principal accuracy gain is driven by interaction, with smaller differences among the interactive selectors.

Table 6. Planned offline accuracy comparisons between interactive clarification and noninteractive selection. OR denotes odds ratio, and CI denotes confidence interval.

Comparison	Acc. A	Acc. B	Diff.	OR	95% OR CI	Raw p	Holm p	Result
ActivAsk vs. Top score	92.13%	53.70%	38.43 pp	15.52	[9.14, 26.37]	$3.49\times {10}^{-24}$	$1.39\times {10}^{-23}$	Significant
ActivAsk vs. Random candidate	92.13%	53.24%	38.89 pp	15.88	[9.30, 27.10]	$3.91\times {10}^{-24}$	$1.39\times {10}^{-23}$	Significant
ActivAsk vs. Direct VLM	92.13%	53.70%	38.43 pp	15.81	[8.57, 29.15]	$9.33\times {10}^{-19}$	$9.33\times {10}^{-19}$	Significant
VLM-selected vs. Direct VLM	92.13%	53.70%	38.43 pp	15.71	[8.70, 28.36]	$6.38\times {10}^{-20}$	$1.28\times {10}^{-19}$	Significant

Note: Accuracies and percentage-point differences are pooled summaries; odds ratios and confidence intervals come from the scene-clustered logistic GEE. Equal pooled accuracies can yield different odds ratios because the model uses trial-level clustering and covariates.

The main accuracy gain comes from interaction itself: both ActivAsk and VLM-selected clarification benefit from asking questions before action, whereas the noninteractive baselines must commit without observing the user’s intended referent. These results answer RQ1 positively: interactive clarification substantially improves target resolution under ambiguous instructions.

5.3. Clarification Efficiency: ActivAsk Reduces Questions While Maintaining Accuracy

RQ2 asks whether ActivAsk reduces clarification burden while maintaining target resolution accuracy. ActivAsk accuracy was statistically comparable to the other interactive baselines after Holm correction: first-question selection (OR 1.17, 95% CI [0.80, 1.69], Holm $p=0.832$), random-question selection (OR 1.22, 95% CI [0.86, 1.74], Holm $p=0.777$), and VLM-selected clarification (OR 1.01, 95% CI [0.71, 1.43], Holm $p=0.972$). ActivAsk preserves the high accuracy of interactive clarification; its measured advantage lies in question efficiency.

Figure 6 summarizes this accuracy–clarification trade-off among interactive policies. ActivAsk shows statistically comparable accuracy with fewer questions among trials in which clarification was requested. The VLM proposes candidate yes/no questions, and the EFE score evaluates their candidate-level partitions before one question is presented.

Figure 6. Offline accuracy–clarification trade-off among interactive policies. Each labeled point is one clarification policy, and the plot is restricted to interactive methods because RQ2 concerns question selection after the system has decided to ask. ActivAsk keeps the high target resolution accuracy of the interactive baselines while reducing the mean number of questions on trials where a question was asked.

Table 7 gives the planned question-count comparisons restricted to trials with at least one question. ActivAsk reduced the scene-level mean by 0.35–0.46 questions relative to interactive baselines, or 15.47–19.71% after Holm correction. Table 7 averages matched cells by scene and instruction type for each contrast, whereas Table 5 reports pooled means over asked trials. These aggregation levels produce the small variation in the ActivAsk mean across comparison partners. For ambiguous instructions, the mean fell from 2.85 to 2.20 versus first-question selection, 2.83 to 2.16 versus random selection, and 2.83 to 2.22 versus VLM-selected clarification; all three comparisons remained significant after correction.

Table 7. Question-count comparisons among trials with at least one clarification question. Values are scene-level means.

Context	Comparison	ActivAsk Q	Baseline Q	Diff.	Reduction	95% CI	Raw p	Holm p
Overall	ActivAsk vs. First question	1.90	2.36	−0.46	19.45%	[−0.73, −0.23]	0.000369	0.00111
Overall	ActivAsk vs. Random question	1.84	2.30	−0.45	19.71%	[−0.68, −0.23]	0.000655	0.00131
Overall	ActivAsk vs. VLM-selected	1.90	2.25	−0.35	15.47%	[−0.59, −0.10]	0.00422	0.00422
Ambiguous	ActivAsk vs. First question	2.20	2.85	−0.65	22.91%	[−0.95, −0.39]	$7.51\times {10}^{-5}$	0.000182
Ambiguous	ActivAsk vs. Random question	2.16	2.83	−0.68	23.88%	[−0.95, −0.42]	$6.07\times {10}^{-5}$	0.000182
Ambiguous	ActivAsk vs. VLM-selected	2.22	2.83	−0.61	21.43%	[−0.90, −0.33]	0.000555	0.000555

The design matched scene and instruction type, but not exact instruction–target combinations. We therefore examined the pooled set of 408 interactive trials in which at least one clarification question was displayed: 101 first-question, 101 random-question, 102 VLM-selected, and 104 ActivAsk trials. These trials comprised 394 correct resolutions, six wrong targets, and eight unresolved outcomes. All 408 trials, including the failures, contributed their observed question counts to the pooled burden. Among successful resolutions, the mean question counts were 2.38 for first-question selection, 2.46 for random-question selection, 2.33 for VLM-selected clarification, and 1.92 for ActivAsk. ActivAsk retained the lowest burden under this summary.

Latency provides a complementary view of this question burden. The recorded duration covers the full interaction, including VLM processing, system overhead, user response time, and logging. Among asked trials, the 90th-percentile latency decreased from 303.92 s for VLM-selected clarification to 243.62 s for ActivAsk. This difference was concentrated in slow, multi-round clarification trials.

Two supplementary analyses examined how proposal availability and initial ambiguity affected question efficiency. The first used 40 fixed contexts, each defined by a scene, instruction, and candidate state. For each context, the requested number of proposals was varied independently over $K\in \{1,2,3,4,5\}$, with three fresh VLM calls per condition, yielding 600 calls. Increasing K generally produced a slightly lower minimum available EFE, although the differences were modest. For ambiguous instructions, the mean minimum EFE changed from $-0.649$ at $K=1$ to $-0.669$ at $K=4$ and $-0.670$ at $K=5$. For partial instructions, the corresponding values were $-0.664$, $-0.675$, and $-0.681$. Mean latency increased from 3.49 s at $K=1$ to 10.42 s at $K=5$. Additional proposals modestly broadened the available choice set while increasing generation time.

The second analysis grouped the original interactive trials by initial candidate-set size, used here as an operational measure of clarification complexity. ActivAsk required mean question counts of 1.000, 1.727, 2.056, 2.333, and 2.700 for $|S_0| =2,\dots ,6$, respectively. These values were close to the minimum expected depth of an ideal binary decision tree under uniformly likely targets and unrestricted partitions. This idealized reference provides an exhaustive-partition efficiency benchmark while the deployed policy remains limited to semantically valid VLM proposals. Complete results are reported in Table A5 in Appendix B and Table A6 in Appendix C.

Fewer clarification rounds also reduce exposure to accidental answer errors. A supplementary sensitivity analysis modeled independent answer reversals on a balanced set of 392 recorded trials (98 per policy; Appendix C). At $ϵ=0$, ActivAsk and first-question selection were tied at 98.98%. ActivAsk achieved the highest expected accuracy at every evaluated nonzero answer-error level. At $ϵ=0.10$, the expected accuracies were 81.05% for ActivAsk, 77.75% for first-question selection, 76.96% for VLM-selected clarification, and 73.14% for random-question selection. The advantage increased with the assumed error probability because ActivAsk required fewer clarification answers.

These results support a positive answer to RQ2. ActivAsk preserves the target resolution accuracy of interactive clarification while reducing the number of user-facing questions, especially when the instruction is ambiguous.

5.4. Online Robot Results: Transfer from Target Resolution to Physical Grasping

The online real robot experiment evaluates whether ActivAsk target resolution behavior improves physical task success when the grasping pipeline is held fixed. Table 8 reports the online descriptive results with Wilson 95% CIs for the main online proportions. ActivAsk selected the correct target in 53 of 57 trials (92.98%, 95% CI [83.30, 97.24]) and completed the full correct object grasp task in 50 of 57 trials (87.72%, 95% CI [76.75, 93.92]). VLM-selected clarification also had high target accuracy, selecting the correct target in 52 of 57 trials (91.23%) and completing 46 of 57 tasks (80.70%). The noninteractive baselines selected the correct target in only 34–35 of 57 trials and completed about half of the full tasks. No wrong object grasps were physically executed: operator-controlled verification stopped wrong resolved targets before motion while retaining them as target resolution and full task failures.

Table 8. Online descriptive results by method.

Method	Target Acc.	Attempts	Grasp Success	Full Task Success	Non-executed	Mean Q
Top score	59.65% [46.70, 71.38]	34	88.24% [73.38, 95.33]	52.63% [39.92, 65.01]	23	–
Random candidate	61.40% [48.43, 72.94]	35	82.86% [67.32, 91.90]	50.88% [38.26, 63.38]	22	–
VLM-selected	91.23% [81.06, 96.19]	52	88.46% [77.03, 94.60]	80.70% [68.66, 88.87]	5	2.25
ActivAsk	92.98% [83.30, 97.24]	53	94.34% [84.63, 98.06]	87.72% [76.75, 93.92]	4	1.84

Note: $N=57$ trials per method. Values in brackets are Wilson 95% CIs. All trials contribute to target accuracy and full task success; grasp attempts occur only after operator-controlled experimental verification. Grasp success is physical success conditioned on an executed grasp attempt. Non-executed failures are unresolved outcomes or wrong resolved targets stopped before motion by this verification; they remain target resolution and full task failures. Bold values mark the best result, and underlined values mark the second-best result where a clear direction is defined.

Figure 7 separates the corresponding target selection, full-task, and conditional grasp outcomes. Because every attempt followed the operator-controlled experimental verification, physical success given an attempt conditionally evaluates the shared execution stage. Full task success reports the observed end-to-end outcome in each real scene. Reporting the observed chain avoids relying on a simple product of separately estimated target and grasp rates, which would assume independence and discard scene- and object-specific coupling between perception, reachability, and contact.

Figure 7. Online target selection accuracy, full task success, and physical success conditioned on a grasp attempt. All trials contribute to the first two outcomes, whereas a physical attempt occurs only after operator-controlled experimental verification. Target resolution improvements transfer to full correct object grasping because they produce more correct grasp attempts under the same RGB-D grasping pipeline.

Table 9 reports planned paired scene-level comparisons. Given the smaller sample and physical noise, our analysis emphasizes effect sizes and confidence intervals. Scene-level estimates need not equal pooled differences: ActivAsk versus VLM-selected target accuracy is +1.75 percentage points when pooled but −0.83 after averaging within-scene differences. The confidence interval crosses zero and the Holm-adjusted p-value is 1.000, indicating comparable target accuracy despite the small sign reversal. Versus the top score, the pooled and scene level target selection differences are +33.33 and +31.67 points, respectively. ActivAsk improved full task success over top score and random selection by 33.33 and 35.83 points, respectively, after Holm correction. The 5.00-point difference from VLM-selected clarification was positive but not significant. At this sample size, the gap corresponds to only a few trials per method and indicates comparable full task performance.

Table 9. Planned paired scene-level online comparisons.

Metric	Comparison	Diff.	95% CI	Raw p	Holm p	Result
Target selection	ActivAsk vs. Top score	31.67	[22.50, 40.83]	$6.10\times {10}^{-5}$	0.000183	Significant
Target selection	ActivAsk vs. Random candidate	30.83	[15.83, 45.00]	0.00115	0.00115	Significant
Target selection	ActivAsk vs. VLM-selected	−0.83	[−7.50, 5.00]	1.000	1.000	Comparable
Full task success	ActivAsk vs. Top score	33.33	[20.00, 46.69]	0.000549	0.00220	Significant
Full task success	ActivAsk vs. Random candidate	35.83	[20.00, 50.85]	0.000550	0.00220	Significant
Full task success	ActivAsk vs. VLM-selected	5.00	[−5.02, 14.17]	0.406	0.406	Positive but not significant

Note: Differences are paired scene-level means in percentage points, not pooled differences; each scene contributes one paired difference before averaging.

These results answer RQ3: improved target selection produces more valid correct object attempts and higher full task completion under the shared execution pipeline.

5.5. Failure-Mode Analysis

RQ4 asks which failure modes remain in the complete system. The offline failure review identified 348 failed trials with a specific stage label. The largest category was unresolved-ambiguity failure, with 200 cases (57.47% of categorized offline failures), mainly reflecting cases where a noninteractive policy had to guess among multiple plausible referents. The next largest category was candidate generation: in 79 failures (22.70%), GroundingDINO did not include the intended target in the candidate set. A further 65 failures (18.68%) were final wrong object selections, and only three failures (0.86%) were attributed to VLM-clarification errors. Two of these failures occurred because the generated question did not match the actual visual distinction or candidate partition, and one occurred because the question wording was unclear enough to produce a user-interpretation error. One failure (0.29%) was attributed to an incorrect detector label.

A broader retrospective human audit examined 572 displayed questions from 290 trials with recorded candidate branches. Six questions contained a partial mismatch between the question text and the visible scene (6/572, 1.05%), and ten contained a mismatch between the yes/no distinction expressed in the text and the recorded candidate branches (10/572, 1.75%). Among these 16 affected questions, 15 retained the intended target in the branch selected after the user’s response, and the trials containing those questions ultimately resolved the target correctly. Subsequent clarification continued from the retained candidate state and removed the remaining incorrect candidates. Only one affected question was associated with a final target resolution failure. Several mismatches arose from local detector label or visual attribute errors, but they did not change the final target when the intended candidate remained in the selected branch.

The instruction-level failure review explains why partial and ambiguous instructions are difficult in different ways. Ambiguous instructions were dominated by unresolved-ambiguity failures: 156 such failures occurred in ambiguous trials. Partial instructions had more candidate-generation and wrong-selection problems, with 51 missing-target failures and 52 wrong-object failures. As illustrated by the representative examples in Figure 8, these cases were common for utensil instructions and color-specific cup instructions, where the detector sometimes returned only one fork or one red cup even though several visually relevant objects were present. Color-sensitive errors also occurred under the room lighting; red, pink, and orange utensils could be confused by the detector or by the candidate labels. These failures are upstream of ActivAsk question selection.

Figure 8. Representative offline failure examples from manual review. Panels (a,b) show candidate sets that are too narrow for color specific cup and fork instructions, leaving no useful clarification path if the intended object is outside the returned set. Panels (c,d) show color and label confusion among utensils under the tabletop lighting. These examples illustrate that several remaining failures occur before or during candidate construction, outside ActivAsk question selection.

We additionally examined three perception-level recovery options on the recorded missing-target cases. For category-level ambiguous instructions, reducing the GroundingDINO box threshold from 0.38 to 0.20 retained all 21 relevant same-class objects, compared with 8 of 21 at the original threshold, but introduced five additional false detections. For partial instructions, falling back from the complete referring expression to the head category restored the intended target in all tested cases. Subsequent VLM-based filtering produced the exact desired candidate set in approximately 70% of cases; the remaining cases either retained additional candidates or removed valid ones. Finally, Grounding DINO 1.5 Pro [47], evaluated through its hosted API with the same images, queries, and post-processing settings, recovered the target in all 7 unique ambiguous and all 29 unique partial missing-target cases. The deployed pipeline uses the open-source GroundingDINO model; Grounding DINO 1.5 Pro was used only for this retrospective recovery check and requires paid API access beyond the provider’s trial quota. These checks show that perception-level recovery can substantially reduce the observed candidate-generation bottleneck, with different trade-offs in false-positive candidates, filtering reliability, latency, and computational cost.

The online experiment separates target resolution failures from physical grasp failures. Across 228 online trials, 54 trials (23.68%) failed because target resolution selected a wrong target or failed to produce an executable target and grasp execution was not attempted after operator-controlled verification, and 19 trials (8.33%) failed after selecting the correct target because the physical grasp did not succeed. Figure 9 shows the distribution by method. ActivAsk had four target resolution failures and three physical grasp failures, whereas the two noninteractive baselines had 22–23 target resolution failures each.

Figure 9. Online failure-stage distribution by method. The noninteractive baselines fail mainly before execution because target resolution selects a wrong target or fails to produce an executable target. After target resolution improves, remaining errors increasingly come from the shared physical grasp execution layer.

The failures localize to a small number of stages. Interaction mainly reduces unresolved and wrong singleton target resolution outcomes, while ActivAsk reduces inefficient clarification. Rare VLM errors motivate structured validation, clearer wording, and rule-based fallback; segmentation, depth, reachability, and contact failures remain after a target is correctly resolved.

Table 10 summarizes physical success by object category. It aggregates all 174 executed grasp attempts across the four online methods; method-specific attempt counts, such as the 53 ActivAsk attempts, are reported in Table 8. Cups had 57 successes in 59 attempts, while bottles, forks, and spoons showed lower success rates for different physical reasons.

Table 10. Physical grasp success by object category after a grasp attempt was made.

Object Category	Attempts	Successes	Failures	Success Rate	Main Observed Difficulty
Cup	59	57	2	96.61%	Occasional depth-crop offset or slip on smooth curved surfaces.
Bottle	61	52	9	85.25%	Depth-crop offset, height and clearance constraints, and slip on smooth cylindrical surfaces.
Fork	26	21	5	80.77%	Thin geometry; SAM mask or depth-boundary error; edge or tip contact.
Spoon	28	25	3	89.29%	Thin or curved geometry; grasp point shifted to edge or handle boundary.

The remaining physical failures were mainly associated with single-view RGB-D geometry uncertainty and contact instability. For cups and bottles, incomplete or noisy depth crops shifted the estimated grasp center, and smooth cylindrical surfaces occasionally slipped during closure or lifting. The bottles used in the online experiment were relatively smooth, which made stable contact more difficult than for cups even when the target was correctly selected. For forks and spoons, SAM or depth-boundary errors could shift the grasp toward an edge, tip, or handle boundary, leaving insufficient material inside the gripper. The fork–spoon difference appeared related to contact mechanics: forks expose gaps and flexible tine regions near the front, so closure near this region can make the object flex, rotate, or be pushed out of the grasp. Spoon bowls provide a more continuous and rigid contact surface, so front-biased grasps more often retained stable material between the fingers. These qualitative observations localize the remaining failures to geometry estimation, segmentation, contact placement, friction, and object compliance. Because slip, flexing, and small forward or backward offsets are difficult to interpret from a single static frame, we summarize them qualitatively here; execution videos in the public repository (see the Data Availability Statement) document these dynamic contact events.

6. Discussion

6.1. Interpretation of the Main Findings

The four research questions lead to a consistent interpretation: interactive clarification substantially improved target resolution compared with noninteractive selection; EFE-based question selection preserved the high accuracy of interactive clarification while reducing the number of user-facing questions; improved target decisions transferred to higher full correct object grasp success in the real robot experiment under the same RGB-D grasping pipeline; and the remaining failures concentrated in candidate generation, VLM question or partition quality, operator-controlled non-execution after target resolution failures, and downstream physical grasp execution.

The first result complements language-conditioned manipulation systems that can execute broad commands from language but often assume that the target or goal is sufficiently specified before action [2,3,4,5,6,7,8,9]. In same-category tabletop scenes, that assumption can fail even when open-vocabulary grounding works as intended: several cups, bottles, or utensils may all be plausible matches. ActivAsk treats grounding output as a candidate state before making a final target decision. This candidate preserving view is the reason that interaction provides such a large accuracy gain over top score, random, and direct VLM selection.

The second result refines the role of clarification. Prior work on help seeking and interactive disambiguation shows the value of asking under uncertainty [20,21,22,23,24,25,26,27,28,29,30]. Valid VLM proposals can induce informative, redundant, or highly unbalanced partitions, so asking alone does not determine interaction efficiency. In the response-neutral setting, EFE and information gain rank these partitions identically. ActivAsk and the other interactive policies achieve comparable target resolution accuracy because each can eliminate candidate hypotheses using the user’s responses; their main difference is question selection efficiency. On asked trials, ActivAsk reduced question count by 15.47–19.71% overall and by 21.43–23.88% for ambiguous instructions. This preserves the accuracy benefit of interaction with fewer interruptions, and the reduction can accumulate across repeated interactions or longer-horizon tasks with several ambiguous references.

The online metrics preserve the distinction between target resolution and physical execution without estimating completion from independent rates. Conditional grasp success describes the shared physical execution stage after operator-controlled experimental verification. This metric isolates grasp execution from target resolution errors under the experimental protocol; the evaluated policy itself contains no autonomous rejection stage. Full-task success includes both target resolution and execution over all trials. The online experiment tests the integrated sensing–selection–execution chain while reporting component-level outcomes separately. Better target resolution creates more valid correct object attempts, while final task completion still depends on segmentation, geometry, reachability, contact placement, and lifting.

The initial candidate state defines the hypothesis space available to the clarification policy. EFE can compare questions and reduce uncertainty only among candidates represented in that state; it cannot assign probability mass to an object that was never included. Candidate recall is therefore a prerequisite for successful clarification, whereas additional false-positive candidates mainly increase the number of hypotheses that must be eliminated through interaction. The recovery checks show that this bottleneck is modular: query relaxation, adaptive thresholding, or a higher-capacity hosted detector such as Grounding DINO 1.5 Pro can restore missing targets before the clarification loop resumes. Recovery must nevertheless balance recall against candidate-set inflation. Lower detection thresholds can introduce false-positive hypotheses, while aggressive VLM filtering after query fallback can remove valid candidates. A hosted detector also adds API cost and dependency. A practical recovery policy should prioritize target recall, retain uncertain candidates conservatively, and allow clarification to remove the remaining false positives through interaction.

Other failures arose from weak VLM partitions or unclear wording in the user-facing interaction. After the correct target has been selected, remaining grasp failures are mainly caused by single-view RGB-D geometry uncertainty, segmentation or depth-crop errors, contact placement, surface friction, and object compliance. These bottlenecks motivate improvements to candidate recovery, visual geometry estimation, response modeling, and physical manipulation.

6.2. Limitations and Future Work

The main system limitations concern candidate set coverage, structured VLM reliability, target verification, and physical execution. As noted in Section 3.1, clarification cannot recover an intended object that is absent from the candidate state. The recovery checks were retrospective and were not integrated into the deployed clarification loop. Future work will develop a candidate-completeness check and staged recovery policy that combines query fallback, conservative VLM validation, adaptive detector thresholds, alternative detectors, and re-observation before clarification resumes. This requires reliable recovery triggers and evaluation of the trade-offs among target recall, false-positive candidates, interaction burden, latency, and computational cost.

Target verification presents a separate limitation. The deployed policy has no independent signal for checking whether a resolved singleton matches the user’s intention; it treats the remaining candidate as the target. A final user confirmation or an additional VLM verification stage could provide this check. Unclear wording, visually inconsistent partitions, and single-view RGB-D geometry can still cause errors. Because proposal-level accepted, repaired, and fallback labels were not retained, their processing rates could not be reconstructed retrospectively. Better geometry estimation and contact planning could reduce execution failures.

The interaction model is evaluated with a uniform intention prior and cooperative yes/no responses, so the reported question-efficiency results apply to this protocol. The answer-error analysis models independent accidental reversals without recovery, but not systematic misunderstanding or an explicit uncertain answer. The current system does not support proactive information provision from the user beyond the requested yes/no response. Future work will extend the interface to accept optional descriptions or corrections and compare different question formulations and interaction styles. It will also evaluate noisy answer likelihoods, confirmation or correction actions, calibrated priors from user or task context, pragmatic EFE terms such as question cost and execution risk, and the effects of linguistic proposal type, candidate visual similarity, clutter, occlusion, and other scene–question interactions.

The evaluation covers common tabletop grasping conditions, while broader generalization remains to be established. The empirical evaluation used controlled tabletop scenes, four object categories, cooperative operators, and one Stretch 3 platform. ActivAsk is architecturally independent of a fixed task-specific category list or question library. Candidate construction uses the open-vocabulary GroundingDINO detector [15], which localizes objects specified by category names or referring expressions and supports zero-shot transfer to novel categories. Clarification proposals are generated at runtime from the observed image, instruction, and current candidate state. The pipeline can therefore be applied to additional categories, visible attributes, and finite candidate sets without retraining the EFE policy. Empirical reliability will still depend on detection and grounding quality for new objects, instructions, viewpoints, and scene configurations. The yes/no interface requires no user-specific model training, while multi-participant evaluation is needed to establish generalization across users. Future work should evaluate diverse unseen object categories, denser and less controlled scenes, more complex referring expressions and manipulation tasks, multiple users with different interaction preferences, and additional robot platforms.

7. Conclusions

This paper presented ActivAsk, a zero-shot active clarification framework that combines VLM-generated candidate partitions with EFE-based question selection. Offline results showed that interaction improved target accuracy from about 53–54% for noninteractive baselines to about 90–92% for interactive clarification, while ActivAsk preserved the best interactive accuracy and reduced the number of asked questions by 15.47–19.71% overall and by 21.43–23.88% for ambiguous instructions. Online real robot trials separately measured target selection, conditional grasp execution, and full task completion; ActivAsk achieved 92.98% target selection accuracy and 87.72% full correct object grasp success, showing that improved target decisions transfer to more correct object grasps under the same RGB-D grasping pipeline. The quality of the candidate set, VLM reliability, and physical execution remain the main limitations. Overall, the results support asking a focused clarification question when the intended referent remains ambiguous.

Appendix C.1. Question Efficiency by Initial Candidate-Set Size

The original interactive-trial records were summarized by initial candidate-set size $|S_0|$. In the reference calculation below, $n= |S_0|$ denotes the number of initial candidates. This quantity is used as an operational measure of clarification complexity because it directly represents the number of plausible target hypotheses to be distinguished.

For comparison, the minimum expected depth of an optimal binary decision tree under uniformly likely targets and unrestricted binary partitions is

$$L_{\mathrm{ideal}}\left(n\right)=k+2-{\displaystyle \frac{2^{k+1}}{n}},k=⌊{\log }_{2}n⌋.$$

(A7)

Here, $L_{\mathrm{ideal}}\left(n\right)$ is the ideal expected question count, and $k=\lfloor {\log }_{2}n\rfloor$ identifies the deepest complete binary-tree level for n targets. The reference averages over uniformly likely targets and assumes exhaustive access to all binary partitions, without requiring that a partition can be expressed as a valid visual question. It is a distribution-level reference, not a lower bound for every realized trial. A finite empirical mean can be smaller when the observed targets occur more often on shallow decision paths or when an early question directly isolates the intended target.

Table A6. Observed mean question counts by initial candidate-set size and differences from the ideal uniform binary-tree reference.

$|{\mathit{S}}_0|$	Ideal Reference	First Question	Random Question	VLM-Selected	ActivAsk
2	1.000	1.000 (+0.000)	1.053 (+0.053)	1.000 (+0.000)	1.000 (+0.000)
3	1.667	1.909 (+0.242)	1.909 (+0.242)	1.909 (+0.242)	1.727 (+0.061)
4	2.000	2.167 (+0.167)	2.444 (+0.444)	2.222 (+0.222)	2.056 (+0.056)
5	2.400	2.667 (+0.267)	3.000 (+0.600)	2.000 ($-0.400$)	2.333 ($-0.067$)
6	2.667	3.450 (+0.783)	3.158 (+0.491)	3.550 (+0.883)	2.700 (+0.033)

Note: Parentheses report the empirical difference from the ideal reference. The reference averages over uniformly likely targets and unrestricted binary partitions. Negative empirical differences indicate that the realized finite-sample targets followed shorter-than-average decision paths, including cases where an early question directly isolated the intended target.

Table A6. Observed mean question counts by initial candidate-set size and differences from the ideal uniform binary-tree reference.

$|{\mathit{S}}_0|$	Ideal Reference	First Question	Random Question	VLM-Selected	ActivAsk
2	1.000	1.000 (+0.000)	1.053 (+0.053)	1.000 (+0.000)	1.000 (+0.000)
3	1.667	1.909 (+0.242)	1.909 (+0.242)	1.909 (+0.242)	1.727 (+0.061)
4	2.000	2.167 (+0.167)	2.444 (+0.444)	2.222 (+0.222)	2.056 (+0.056)
5	2.400	2.667 (+0.267)	3.000 (+0.600)	2.000 ($-0.400$)	2.333 ($-0.067$)
6	2.667	3.450 (+0.783)	3.158 (+0.491)	3.550 (+0.883)	2.700 (+0.033)

Note: Parentheses report the empirical difference from the ideal reference. The reference averages over uniformly likely targets and unrestricted binary partitions. Negative empirical differences indicate that the realized finite-sample targets followed shorter-than-average decision paths, including cases where an early question directly isolated the intended target.

ActivAsk matched the ideal reference for two candidates and remained within 0.061, 0.056, and 0.033 questions of the reference for candidate-set sizes three, four, and six, respectively. For five candidates, its empirical mean was slightly below the uniform reference because the realized target paths were shorter than the uniform-tree average. Across the observed candidate-set sizes, ActivAsk remained close to the idealized binary-tree efficiency and generally required fewer questions than the other interactive policies.

Appendix C.2. Error Sensitivity Analysis

The deployed clarification loop uses a deterministic candidate state update. To evaluate sensitivity to accidental answer errors without rerunning the VLM, robot, or user experiments, we applied an independent per-answer error model to the recorded interactive trials.

Eligible trials were those in which the interactive policy asked at least one clarification question and the corresponding user answer was recorded. To provide equal sample sizes, 98 eligible trials were sampled for each policy, yielding a balanced analysis set of 392 trials.

Let $y_i\in \{0,1\}$ indicate whether trial i was originally resolved correctly, and let $m_i$ denote the number of answered clarification questions in that trial. Under an independent per-answer error probability $ϵ$, an originally correct trial preserves its original outcome only if all $m_i$ answers remain correct. Its expected success probability is therefore

$$y_i{(1-ϵ)}^{m_i}.$$

(A8)

For policy j, the expected accuracy $A_j\left(ϵ\right)$ is

$$A_j\left(ϵ\right)={\displaystyle \frac{1}{N_j}}{\displaystyle \sum _{i=1}^{N_j}}{y}_i{(1-ϵ)}^{m_i},$$

(A9)

where $N_j$ is the number of eligible trials for policy j. Originally wrong or unresolved trials remain failures. We evaluate per-answer error probabilities of 0, 0.05, 0.10, 0.15, and 0.20. This analysis quantifies cumulative exposure to accidental answer reversals under the deployed deterministic update. It does not model automatic recovery after an incorrect answer, question-dependent misunderstanding, or an explicit uncertain answer.

Table A7. Expected target resolution accuracy under independent per-answer errors.

Policy	N	$\mathit{ϵ}=0$	$\mathit{ϵ}=0.05$	$\mathit{ϵ}=0.10$	$\mathit{ϵ}=0.15$	$\mathit{ϵ}=0.20$
First-question selection	98	98.98%	87.78%	77.75%	68.78%	60.76%
Random-question selection	98	93.88%	82.92%	73.14%	64.39%	56.58%
VLM-selected clarification	98	96.94%	86.40%	76.96%	68.50%	60.90%
ActivAsk	98	98.98%	89.68%	81.05%	73.05%	65.62%

Note: Values are expected target resolution accuracies under an independent per-answer error probability $ϵ$. Originally wrong or unresolved trials remain failures. Bold indicates the highest value in each column; tied highest values are both bolded.

Table A7. Expected target resolution accuracy under independent per-answer errors.

Policy	N	$\mathit{ϵ}=0$	$\mathit{ϵ}=0.05$	$\mathit{ϵ}=0.10$	$\mathit{ϵ}=0.15$	$\mathit{ϵ}=0.20$
First-question selection	98	98.98%	87.78%	77.75%	68.78%	60.76%
Random-question selection	98	93.88%	82.92%	73.14%	64.39%	56.58%
VLM-selected clarification	98	96.94%	86.40%	76.96%	68.50%	60.90%
ActivAsk	98	98.98%	89.68%	81.05%	73.05%	65.62%

Note: Values are expected target resolution accuracies under an independent per-answer error probability $ϵ$. Originally wrong or unresolved trials remain failures. Bold indicates the highest value in each column; tied highest values are both bolded.

Expected accuracy decreases monotonically for all policies as the assumed answer-error probability increases. At $ϵ=0$, ActivAsk and first-question selection are tied at 98.98%. ActivAsk achieves the highest expected accuracy at every evaluated nonzero error level. At $ϵ=0.10$, ActivAsk exceeds first-question, random-question, and VLM-selected clarification by 3.30, 7.91, and 4.09 percentage points, respectively. At $ϵ=0.20$, the corresponding differences increase to 4.86, 9.04, and 4.72 percentage points. The widening advantage reflects the lower number of clarification answers required by ActivAsk and its lower cumulative exposure to accidental answer errors. The model holds an incorrect update fixed after a reversed answer and includes no recovery mechanism.