GPTArm: An Autonomous Task Planning Manipulator Grasping System Based on Vision–Language Models

Abstract

The integration of vision–language models (VLMs) with robotic systems represents a transformative advancement in autonomous task planning and execution. However, traditional robotic arms relying on pre-programmed instructions exhibit limited adaptability in dynamic environments and face semantic gaps between perception and execution, hindering their ability to handle complex task demands. This paper introduces GPTArm, an environment-aware robotic arm system driven by GPT-4V, designed to overcome these challenges through hierarchical task decomposition, closed-loop error recovery, and multimodal interaction. The proposed robotic task processing framework (RTPF) integrates real-time visual perception, contextual reasoning, and autonomous strategy planning, enabling robotic arms to interpret natural language commands, decompose user-defined tasks into executable subtasks, and dynamically recover from errors. Experimental evaluations across ten manipulation tasks demonstrate GPTArm’s superior performance, achieving a success rate of up to 91.4% in standardized benchmarks and robust generalization to unseen objects. Leveraging GPT-4V’s reasoning and YOLOv10’s precise small-object localization, the system surpasses existing methods in accuracy and adaptability. Furthermore, GPTArm supports flexible natural language interaction via voice and text, significantly enhancing user experience in human–robot collaboration.

1. Introduction

In recent years, enabling robotic arms to autonomously perform tasks based on predefined objectives has emerged as a critical research frontier, overcoming the limitations of traditional programming methods [1,2,3,4]. Researchers have explored a variety of strategies—from deep learning architectures [5,6,7] to human-centric interaction paradigms, such as facial recognition [8]. The rise of LLMs and VLMs has accelerated progress in this area [9,10], effectively bridging theoretical concepts with practical robotic applications. With their advanced proficiency in natural language processing and multimodal data integration, LLMs and VLMs have demonstrated transformative potential in human–robot collaboration [11,12] and cognitive task reasoning [13,14].

For robotic manipulation systems, accurately executing natural language instructions requires aligning linguistic specifications with kinematic constraints. Early studies [15,16] focused on syntactic alignment between language inputs and predefined action templates, while subsequent work [17,18] integrated AI models for spatial reasoning and command-to-action translation. Implementations such as that in [19] demonstrated a direct mapping from linguistic instructions to robotic trajectories. However, these systems suffered from weak error-handling strategies, relying heavily on command filtering and hard-coded motion primitives, which limited their operational robustness and practical applicability. Modern LLMs address these shortcomings through contextual language understanding and compositional task reasoning, establishing themselves as fundamental components for next-generation robotic intelligence [20]. The fusion of LLMs [21] with computer vision systems [22] has significantly advanced robotic competencies in complex, environment-aware planning [23], facilitating dynamic strategy generation via multimodal context interpretation [24,25,26] and high-precision action sequence formulation [27,28].

Recent advancements in robotic task processing frameworks have significantly enhanced autonomous systems by leveraging VLMs and LLMs. For example, RT-2 [29] showcases the ability to transfer web-derived knowledge to robotic control, enabling more sophisticated task execution, while interactive task planning [30] and ontological frameworks [31,32] enhance adaptability in dynamic, unstructured environments. These developments have improved robotic reasoning and decision-making, yet VLM-driven systems continue to encounter significant deployment challenges. Semantic disconnects between visual perception outputs and task-specific contexts, coupled with spatial localization inaccuracies, frequently result in flawed task planning and execution failures. Such issues often stem from the uncritical reliance on AI-generated hypotheses without sufficient environmental validation. Current solutions, such as hybrid architectures integrating VLMs [33,34,35] with verification modules [36,37,38] and 3D perception enhancements via stereo vision systems [39], have improved grasping precision and task accuracy, though they introduce high hardware costs and computational complexity. Approaches employing fixed monocular cameras [40] provide a cost-effective solution for environmental awareness [41] but suffer from accuracy degradation due to viewpoint dependency and limitations in adaptive planning for dynamic settings, making scalable VLM integration in robotics difficult. Recent frameworks [31,42] have advanced task processing architectures, yet persistent challenges in spatial reasoning and error recovery highlight the need for more robust approaches. Prior work [43] demonstrates motion plan refinement based on human–robot collaboration (HRC); however, it requires extensive training data and lacks cross-domain generalization.

Table 1. The advantages and disadvantages of current robotic task processing frameworks.

Reference (Year)	Advantage	Disadvantage
[14]-(2024)	Integrates segmentation and grasp synthesis to eliminate action model reliance.	Segmentation/synthesis errors may cascade.
[28]-(2024)	Proposes a Robot-GPT interaction framework with code generation and error correction.	Visual feedback absence restricts error comprehension.
[31]-(2024)	Combines internal/external error correction mechanisms.	Only grasps trained objects.
[43]-(2024)	Hierarchical planning decomposes long-term tasks.	Ineffective control of errors arising during execution.
[44]-(2024)	Conditional action sampling generates VLM-based targeted actions.	Predicted video–action mismatch degrades evaluation.

provides a comparative analysis of current robotic task processing frameworks, detailing their strengths and limitations.

To overcome the limitations in prior work—including extensive training data, poor cross-domain generalization, inadequate spatial reasoning, and error recovery—we present GPTArm, a VLM-driven robotic system that highlights autonomous task planning capabilities through our robotic task processing framework (RTPF). The RTPF introduces three key mechanisms: (1) autonomous task planning, which dynamically generates execution strategies based on user-defined objectives, enabling real-time adaptation to task and environmental changes; (2) dynamic error recovery, which self-corrects plans through closed-loop feedback in dynamic environments; and (3) multimodal context integration, which integrates vision, language, and spatial data to achieve context-aware precision. By directly addressing VLMs’ limitations in 3D spatial cognition and compositional reasoning, the RTPF achieves a 91.4% success rate in standardized benchmarks, thereby demonstrating its capability for autonomous task execution in real-world scenarios.

The key contributions of this work are as follows:

• Development of the RTPF: We propose a novel framework integrating GPT-4V with robotic arm control systems, leveraging environmental data from vision sensors and advanced reasoning capabilities. The RTPF implements a hierarchical decomposition paradigm that accomplishes the following:
  • Autonomously plans task execution based on user-defined objectives, generating context-aware subtasks in real time.
  • Selects optimal motion primitives from a predefined library to ensure precise and efficient operations.
  • Validates and corrects execution outcomes through real-time closed-loop feedback, ensuring robust performance in dynamic environments.
• Environment-Aware Adaptive Planning: We introduce a hybrid natural language processing pipeline that autonomously extracts semantic objectives from unstructured instructions and generates environment-adaptive strategies, supported by multimodal voice–text interaction.
• Cross-Model Performance Benchmarking: A comprehensive evaluation of state-of-the-art VLMs (Fuyu-8B, Qwen-vl-plus, GPT-4V) reveals that GPT-4V excels in task decomposition and execution accuracy under dynamic conditions. Experimental results highlight the framework’s exceptional versatility, achieving consistent performance across diverse tasks—from simple object grasping to complex, multi-step assembly operations.

2. Materials and Methods

GPTArm is an AI-integrated, versatile robotic arm system designed to overcome the limitations of conventional robotic arms. Although a wide variety of robotic arms are available on the market—each differing in type, model, and functionality—the development processes and design complexities vary significantly, thereby limiting their universality and interoperability. The primary objective of GPTArm is to address these challenges by enabling heterogeneous robotic arms to execute identical tasks through a unified task processing framework.

2.1. RTPF: A Hierarchical Framework for Environment-Aware Task Planning

The core of GPTArm is the RTPF, a novel hierarchical framework for autonomous task execution. Integrated with robotic arm control interfaces, it forms a closed-loop workflow through four specialized computational layers. By assigning complex task planning and multimodal data processing to GPT-4V and perception/execution to robotic hardware, the RTPF creates a human–robot intelligence collaboration paradigm.

The RTPF is primarily inspired by the hierarchical structure from [43] and the error correction mechanism from [31]. Through architectural improvements, it effectively addresses limitations such as the lack of dynamic error recovery and dependency on pre-trained datasets. Specifically designed as a cross-platform task processing framework, it aims to enhance universality across diverse robotic arm systems while maintaining hardware-specific motion capabilities. Figure 1 presents the detailed structure and operational workflow of the RTPF.

The multi-layer architecture of the RTPF operates through coordinated interactions among the following:

• Detection Layer: Implements real-time sensor fusion and environmental mapping.
• Strategy Layer: Employs GPT-4V for contextual task decomposition and constraint satisfaction.
• Execution Layer: Converts abstract plans into hardware-specific control signals.
• Evaluation Layer: Provides continuous performance monitoring and error compensation.

2.1.1. Detection Layer

The detection layer, consisting of a vision module and a processing unit, captures visual data from the robotic arm’s operating environment and transmits it to GPT-4V for subsequent processing and analysis. This ensures accurate environmental detection to support downstream task planning and execution.

In our system, the vision module uses a USB Video Class (UVC)-compliant camera mounted beneath the robotic arm’s gripper, moving with the arm, while fixed overhead cameras provide stable, undistorted views for simple grasp localization—they are less effective in scenarios involving small objects or constrained workspaces. In contrast, an integrated camera dynamically tracks target objects and adjusts its viewpoint in real time, making the system adaptable to a wide range of operational demands.

However, mounting the camera on the robotic arm introduces challenges. Tilted viewpoints can complicate object localization, especially with partial object occlusion. Additionally, arm-movement-induced viewpoint shifts may result in positioning inaccuracies. These issues are mitigated in the strategy layer, where GPT-4V intelligently corrects and optimizes the initial data, substantially enhancing grasping accuracy. Once the camera captures a raw image $x^t$ of the environment at time t, the system applies a series of preprocessing operations—collectively denoted by $f_{process}$—to yield a processed image $x_p^t$:

$$x_{p }^t=f_{process}\left(x^t\right)$$

(1)

These operations include basic image enhancement, noise reduction, and partitioning the image into a 50 × 50 grid to discretize the coordinate space for subsequent analysis. The processed data are then forwarded to GPT-4V for recognition and interpretation, thereby informing both the task planning in the strategy layer and the motion control in the execution layer. By organizing environmental sensing in this layered manner, the detection layer establishes a robust data foundation for subsequent task execution.

2.1.2. Strategy Layer

The strategy layer, built upon GPT-4V, serves as the core mechanism for task analysis and planning. Instead of just processing fixed instructions, it interprets user intentions, considers context, and designs efficient steps. Through well-configured prompts and formats, GPT-4V adapts flexibly to a wide variety of tasks and interaction requirements.

In the presented framework, GPT-4V typically receives two types of inputs. (1) Role definition specifies domain expertise and decision-making style (e.g., acting as an experienced designer for block arrangement tasks to evaluate block positions). (2) The standardized response format ensures consistent data exchange with the robotic arm via coordinate–action data. GPT-4V segments images into coordinate systems and returns precise positional data for translating strategies into control operations.

(1)

Task Decomposition.

Using processed visual data and high-level task goals, GPT-4V decomposes each task into manageable subtasks. For instance, in a block arrangement task, it identifies current block positions, determines target coordinates, and generates grasp–move–place actions to achieve the desired configuration. This decomposition converts abstract objectives into a structured set of subtasks.

(2)

Strategy Planning.

Once the task is decomposed, GPT-4V formulates a strategy represented as a sequence of action steps. Let $I$ denote the set of additional constraints or parameters (such as workspace boundaries or object properties). GPT-4V then generates a plan $p$:

$$p=GPT-4V\left(x_p^t, I\right)=\left\{s_1, s_2, \dots , s_n\right\}$$

(2)

Each step s_i specifies both the coordinates and the required action (for example, “grasp the object at (x_i, y_i)” or “place the object at (x_j, y_j)”). By comparing the current configuration of objects with the desired outcome, GPT-4V produces an ordered list of grasping and placing commands. If a target position is occupied, the plan adjusts to prevent interference, ensuring a coherent workflow with minimal unnecessary maneuvers. Finalized action sequences are translated into hardware-compatible control protocols, enabling accurate execution and enhancing task robustness and efficiency.

(3)

Secondary Verification.

Strategies generated by GPT-4V are not always correct. Incorrect ones may harm objects or even damage robots [31]. To address potential inaccuracies in object localization or suboptimal sequencing, the system incorporates a secondary verification. We define a function $F\left(p\right)$ that maps an initial strategy p to a final, validated strategy p_final as follows:

$$p_{final}=\left\{\begin{array}{ll} p\hfill & \mathrm{if} validate\_strategy\left(p\right)=\mathrm{True}\hfill \\ error\_correct\left(p\right)\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(3)

In this formulation, if the initial strategy p meets all feasibility criteria, it is retained; otherwise, the function error_correct(p) procedure adjusts and refines the plan to accommodate constraints. Experimental results indicate that this secondary verification stage significantly improves task success rates while maintaining practical response times (detailed results are presented in Section 3.2.1). Although additional iterations could further reduce residual errors, our experiments demonstrate that more than two verification loops yield only marginal improvements relative to the increased processing time.

By integrating task decomposition, coordinated planning, and a secondary verification mechanism, the strategy layer facilitates effective task execution. GPT-4V’s constraint-aware reasoning, combined with these design choices, enhances success rates and reduces end-user programming effort.

2.1.3. Execution Layer

The execution layer transforms the optimized instructions from the strategy layer into precise robotic arm movements. In the GPTArm system, this layer achieves accurate control by integrating a corrected motion library with robust control functions.

The motion library functions as a comprehensive database containing predefined motion functions, path planning algorithms, and execution routines tailored to various robotic arm models. It provides motion paths accounting for differences in motion control and hardware structure across different systems. These motion functions include inverse kinematics (IK) solvers [45], path planning modules, and interference correction routines that facilitate the conversion between joint space and Cartesian space [46], ensuring smooth and accurate execution of grasping and placement operations.

After task decomposition, GPT-4V forwards the segmented strategies to the computer. The computer then parses the current and desired object coordinates and transmits these parameters to the robotic arm control system. Based on this information, the robotic arm selects appropriate motion functions from the motion library, formally represented as:

$$code=extract\_motion\_code\left(p\right)$$

(4)

where extract_motion_code(p) is a function that maps the strategy p to the corresponding motion code. This motion code specifies the path planning algorithm and control parameters to be executed, thereby ensuring that the robotic arm performs the intended motion with high precision.

The execution process operates as a closed-loop control system, with continuous feedback from each step relayed to the evaluation layer for real-time verification of task completion and accuracy. This design ensures motion stability and precision while enabling rapid deviation correction.

GPT-4V’s versatility eliminates the need for object feature pre-training in conventional grasping tasks. However, predefined motion routines lack precision for low-profile items like small nuts, constrained by the robotic arm’s mechanical limits and the motion library’s resolution. To address this issue, we incorporated YOLOv10 for dynamic calibration. YOLOv10 demonstrates strengths in speed, accuracy, adaptability, and small-object detection, and it is widely used in object recognition for robotic arm grasping [47]. Compared to prior versions like YOLOv8, it brings a 2.6% improvement in the F1 score while cutting parameters by 37% [48]. When the robotic arm nears the target area, YOLOv10 is activated to detect the object in real time and provide precise center coordinates. Based on this visual feedback, the robotic arm continuously adjusts its posture and position, gradually aligning the end effector with the object’s center. This ensures the grasping operation is performed at the optimal angle and position, significantly improving the grasp success rate. The online correction mechanism, combining visual feedback and real-time adjustments, overcomes the limitations of traditional fixed-coordinate localization methods, enabling high-precision operation even in complex, dynamic environments. Detailed information is provided in Appendix A.

Overall, the design of the execution layer emphasizes the seamless integration of robotic arm control with intelligent planning from the strategy layer. This integration enables successful routine grasping and placement tasks and flexible adaptation to dynamic changes like target object displacement. By coupling efficient motion planning algorithms with diverse feedback control mechanisms, the execution layer achieves robust and efficient task execution across a range of operating conditions.

2.1.4. Evaluation Layer

Due to large-scale AI models’ reasoning and decision-making limitations, the instructions output by the strategy layer may contain errors affecting robotic arm operations. To mitigate this risk, we introduced an evaluation layer that performs real-time validation of the results at each execution step, reducing failure risks from erroneous commands.

The evaluation layer captures and assesses the instruction outcomes generated by the strategy layer after each task step. After robotic arm task completion, the vision module captures operational state images for GPT-4V analysis. By comparing the actual outcome with the expected task goals, GPT-4V evaluates the accuracy and completeness of the executed step. If the evaluation confirms that the task has met the expected criteria, the system proceeds to the next step. Otherwise, GPT-4V analyzes the error, regenerates the corresponding operation steps, and sends the adjusted instructions back to the execution layer for correction. This closed-loop feedback mechanism significantly enhances both the grasp success rate and the overall execution accuracy of the robotic arm.

Similar evaluation setups have been reported in recent research [31]. However, in multi-step workflows, errors occurring in earlier steps may lead to cumbersome error correction processes if relying solely on final outcome assessments. Additionally, this approach fails to provide an effective measure of the system’s performance during execution. Therefore, we introduced an “evaluation factor” to enable flexible adjustment of the evaluation frequency and stringency. Denoted by α (with 0 < α ≤ 1), this factor regulates evaluation frequency across operational steps. Higher α values trigger rigorous per-step evaluations for precision, while lower α values limit evaluations to post-task assessments to enhance efficiency. This parameter strikes a balance between task precision and overall efficiency, adapting to various operational needs.

To formalize this mechanism, let us define the following:

• N denotes the total number of task steps;
• α represents the evaluation factor;
• m is the number of steps designated for correction.

The number of steps requiring evaluation is computed as:

$$m=\lceil N\times \alpha \rceil$$

(5)

where ⌈.⌉ denotes the ceiling function, ensuring that mmm is an integer. The criterion for determining whether the current step n requires evaluation is defined by the function:

$$E\left(n, N, \alpha \right)=\left\{\begin{array}{ll}\mathrm{True}\hfill & \mathrm{if} n=N \mathrm{or} n=\lceil {\displaystyle \frac{n\times N}{m}}\rceil \hfill \\ \mathrm{False}\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(6)

In this formulation, the final step (n = N) is always evaluated, and the evaluation steps are uniformly distributed according to the computed m.

The evaluation layer introduces an adaptive correction mechanism, ensuring high operational precision and reliability even in complex dynamic environments. By combining visual feedback with intelligent analysis, it enhances task success rates and mitigates strategy-layer errors, improving system robustness and adaptability. The RTPF’s layered approach decouples perception, planning, and execution, forming a robust closed-loop system.

Pseudocode (see Algorithm 1) is provided to illustrate the overall process, where x^t is the raw image captured by the camera at time t.

Algorithm 1: RTPF

1: Initialize camera C, task M, natural language instruction I, evaluation factor α 2: while task not completed do 3:       Capture raw image xt = C(t) 4: $\mathrm{Preprocess} \mathrm{image} x_{p }^t=f_{process}\left(x^t\right)$    // Noise reduction, grid partitioning 5:       p = GPT-4V(xt, I)                              // Generate initial plan 6:       if validate_strategy(p) == False then 7:           pfinal = error_correct(p)                // secondary verification 8:        else 9:           pfinal = p 10:     end if 11:     code = extract_motion_code(p, M)    //Map to motion functions 12:     Execute code, update gripper pose 13:     xnew = C(tnew) 14:     for k = 1 to N do                              //N: total steps in pfinal 15:         if E(k, N, α) = True then 16:             result = GPT-4V(xnew, pfinal)   //Apply evaluation feedback 17:             if result = False then 18:               Trigger feedback loop: send xnew to strategy layer for re-planning 19:             end if 20:         end if 21:     end for 22: end while 23: return Success

2.1.5. Pre-Instructions of RTPF Layers

Figure 2 illustrates the mapping between the predefined requirements and the pre-instructions of each RTPF layer.

In the detection layer, the role-defining and task-specification pre-instructions are crucial. They assign GPT-4V the role of a mathematician well-versed in spatial geometry, enabling precise image coordinate processing by partitioning the image into a 50 × 50 coordinate system and specifying object position output formats. The ultimate objective is to empower GPT-4V to accurately represent object positions and occupied coordinate ranges.

The strategy layer encompasses three types of pre-instructions: general, secondary, and task-specific fine-tuning pre-instructions. The general pre-instructions guide overall coordinate system partitioning and object-arrangement formats. The secondary pre-instructions evaluate strategy feasibility and prevent overlaps and collisions. The task-specific fine-tuning pre-instructions are customizable for specific tasks, like prompting GPT-4V to move blocks with coordinate conflicts to unoccupied areas in a building block placement task, aiming to generate accurate and viable task-execution strategies.

The execution layer acts as a bridge, receiving instructions from the strategy layer and translating them into physical movements of the robotic arm. By integrating a motion library and control functions, it converts optimized instructions into smooth and accurate grasping and placement operations, ensuring compliance with the strategy layer’s directives.

Lastly, the evolution layer is equipped with holistic evaluation and adjustment-oriented pre-instructions. These pre-instructions direct GPT-4V to assess the overall task based on the visual input from the image, monitor the progress, and determine whether the final positions of the objects match the expected standards. In the case of any discrepancies, GPT-4V adjusts the strategy at minimal cost to rectify task-execution deviations and achieve the desired outcome.

The hierarchical design of the RTPF offers two critical advantages. First, it enables automatic strategy planning by allowing GPT-4V to decompose complex tasks into executable sub-steps using real-time environmental inputs. Second, it incorporates a verification process supported by closed-loop feedback that detects and corrects errors in the initial plan. Together, these mechanisms enhance the system’s adaptability, precision, and scalability for diverse robotic manipulation tasks.

The RTPF employs a modular architecture to ensure compatibility with diverse robotic systems rather than being restricted to our specific robotic arm. It can be seamlessly integrated into any task processing system meeting three core requirements: (1) a camera (any type) for real-time visual feedback; (2) a motion library enabling precise navigation to coordinates within the camera’s field of view; and (3) network connectivity (low-cost implementations use devices like ESP32 microcontrollers instead of computers, reducing system costs by up to 90%).

While supporting robotic arms across cost ranges, the coordinate-based operation limits deployment to continuous planar workspaces directly in front of the robotic arm. This constraint arises from the system’s reliance on 2D Cartesian coordinate mapping for motion planning and object localization.

2.2. Intelligent Human–Robot Interaction via Integrated Voice and Text Command

In the context of human–robot interaction, we integrated LLMs with advanced visual processing technologies to significantly enhance the flexibility and accuracy of robotic arms during task execution. Robotic arms equipped with the RTPF demonstrate clear advantages in human–robot interaction and target task completion compared to traditional systems without AI integration (see Figure 3 for an overview of the architecture).

Traditional robotic arms face two primary challenges in instruction processing and object recognition:

• Strict Dependence on Precision in Instructions: Conventional systems can only respond to explicit, precise operational commands and are incapable of handling ambiguous or vague natural language expressions.
• Limited Recognition Capability: Traditional robotic arms rely on pre-trained image recognition models and can only recognize objects included in their training datasets, rendering them ineffective at identifying new or unseen objects.

To overcome these limitations, we integrated the Qwen-audio-asr model from the Tongyi Qianwen platform to develop a comprehensive speech input–output system supporting multi-language speech input. Research demonstrates that Qwen-Audio surpasses models like SALMONN in speech recognition. Integrating with LLMs [49], it supports multiple tasks, including automatic speech recognition (ASR), speech translation, and audio question answering [50]. Speech commands in various languages are first converted into English text and then processed by the GPT model. When faced with ambiguous instructions, GPT synthesizes contextual information to generate specific execution decisions. For example, in response to a vague command such as “I feel a bit unwell”, GPT can infer appropriate task steps based on the surrounding context. After processing, GPT’s textual output is converted back into speech for real-time voice feedback, ensuring a seamless bidirectional communication channel.

The voice output module supports real-time human–robot communication by providing natural language feedback on execution states. For instance, upon completing a grasping task, the system announces “Task completed”, while failure to detect a target object prompts the message “No target object detected within the field of view”. It also offers customizable responses and simple daily conversations, enhancing the bidirectional user experience.

A notable feature of our design is zero-shot object grasping. Due to the inherent recognition capabilities of GPT-4V, the system reliably identifies and locates novel objects by analyzing the coordinate positions of objects captured in images. Consequently, it generates a corresponding grasping strategy that is transmitted to the robotic arm’s control unit, which then selects an appropriate motion function from the motion library to execute the grasp. This approach enables the robust performance of the system to handle un-trained objects.

By combining the natural language processing capabilities of large language models with high-precision object detection and multimodal voice–text interaction, our system not only interprets and executes ambiguous commands but also improves task accuracy through real-time visual and auditory feedback. This integrated approach significantly enhances the robotic arm’s adaptability and flexibility in complex environments, laying a robust foundation for future advancements in human–robot collaboration.

3. Results

3.1. Experiment Setup and Evaluation

3.1.1. System Setup

All experiments in this study were conducted using a six-degrees-of-freedom (DoF) LeArm robotic arm (one rotating base, four movable joints, and a gripper). This integrated robotic platform, combined with various VLMs, is termed GPTArm. Our primary VLM is GPT-4V (provided by OpenAI), with Qwen-vl-plus (from the Tongyi Qianwen platform) and Fuyu-8B (accessed through the Baidu Smart Cloud) used for supplementary comparisons. Each VLM is incorporated via dedicated API endpoints, enabling GPTArm to process visual or textual inputs and generate robotic commands.

A standard 2D camera mounted beneath the gripper captures real-time workspace images. These visual data are processed by the RTPF to segment scenes and map them to a grid-based coordinate system as needed. The experiment area itself is a flat work surface where both target objects and distractor items are placed in varied arrangements. Trials start with the arm in a neutral pose and end upon task completion or failure under constraints.

3.1.2. Evaluation Metrics

To assess the system’s effectiveness under various conditions and difficulty levels, we adopt the following metrics [27].

• Success Rate (SR).
  The proportion of tasks in which all intended goal conditions are met by the end of the robotic arm’s operation. For tasks comprising multiple sub-steps, a single failure in any crucial step renders the entire task unsuccessful.
• Executability (EXEC).
  A measure of how many planned action steps can be physically executed without error. Formally,

  $$EXEC={\displaystyle \frac{N_{correct}}{N_{total}}}$$

  (7)

  where N_correct represents the number of correctly executed steps and N_total means the total steps in the plan.
• Goal Condition Recall (GCR).
  An indicator of how many final goal conditions are satisfied relative to the total conditions specified. GCR is computed as follows:

  $$GCR=1-{\displaystyle \frac{UGC}{TGC}}$$

  (8)

  where TGC is the total number of goal conditions (e.g., precise object positions/orientations) and UGC is the number of these conditions still unmet after execution. A GCR value of 1 implies that all intended goal states were realized, whereas lower values reflect partial fulfillment or missed targets.

3.2. RTPF Validation: Task-Specific Performance Analysis

To evaluate GPTArm’s performance across a range of manipulation challenges, we designed ten distinct tasks, each focusing on grasping and arranging common objects (e.g., medicine bottles, nuts, erasers, blocks). These tasks were categorized into four difficulty levels, reflecting both the complexity of the actions involved and the precision demands of sequential operations.

• B (basic) tasks entail direct single-object grasps under relatively clear conditions.
• I (intermediate) tasks incorporate either two-step sequential operations or target objects located in less favorable viewing conditions.
• A (advanced) tasks involve multi-step planning or partial re-planning upon receiving updated instructions
• HA (highly advanced) tasks increase the spatial or sequential complexity further, often requiring stacking operations and dynamic corrections.

Table 2. Task classification and complexity levels.

No	Task	Difficulty	Description
Task 1	Medicine Bottle Grasping	B	Pick up a single bottle from the workspace.
Task 2	Nut Grasping		Pick up a small nut from the workspace.
Task 3	Block Grasping		Pick up a block from the workspace.
Task 4	Circular Grasping	I	Circularly grasp all the small nuts from the workspace, where there may be slight occlusion or a complex environment.
Task 5	Edge-of-View Object Grasping		Requires careful localization when objects are only partially visible.
Task 6	T-Shaped Block Arrangement	A	Requires placing multiple blocks in a specific geometric configuration; demands accurate positioning and orientation handling.
Task 7	Command-Driven Regrasping Following Misinterpretation		Simulates an instruction correction: pick up one object (e.g., a bottle) then replace it upon revised instructions (e.g., “move the charger instead”). Tests dynamic re-planning.
Task 8	Partial T-Shaped Rearrangement		Involves moving or removing some blocks from a T-configuration and re-stacking them according to new instructions.
Task 9	T-Shaped Construction with Additional Block Stacking	HA	Combines arrangement (forming the T) with vertical stacking of new blocks. Requires stable, multi-step sequencing with minimal positional error.
Task 10	Multiple Block Stacking		Requires sequentially stacking three or more blocks with distinct target positions, potentially re-planning if partial occlusions or collisions occur.

provides an overview of the tasks, their difficulty classification, and a brief description of key requirements. For the video materials related to the experiment, please refer to Supplementary Video S1.

We evaluated the RTPF on four representative tasks of increasing difficulty: (B) task 1, medicine bottle grasping, (I) task 4, circular grasping, (A) task 6, T-shaped block arrangement, and (HA) task 10, multiple block stacking. Each time GPT-4V produces a new strategy, it encapsulates the object’s current and intended coordinates in a standardized transmission format, which is then relayed to the experimental apparatus. This approach ensures that the execution layer can systematically drive the robotic arm and enables human operators to reliably assess the plan’s correctness.

3.2.1. Strategy Layer Performance Evaluation

At the beginning of each trial, GPT-4V receives the current camera image and a concise task description (e.g., “Grasp the Medicine Bottle” or “Arrange Blocks to Form a T-Shape”) and then produces a textual strategy of the required coordinate mappings. To address potential self-detection limitations, the framework automatically performs a secondary verification in which GPT-4V re-examines the proposed strategy. If inconsistencies (e.g., conflicting coordinates or missing steps) are detected, a revised strategy is output; otherwise, the original plan is validated. Human reviewers monitor these strategies to gather accurate statistics but do not intervene in execution decisions.

To gauge the reliability of this strategy generation and revision process across tasks of varying complexity, we performed 210 trials per task (B, I, A, HA) 210 times. For each trial, we recorded whether GPT-4V’s secondary verification triggered revision and whether those revisions resolved the original inconsistencies. This procedure yielded a comprehensive dataset reflecting both first-pass success rates and the frequency and effectiveness of strategy revisions, as summarized in

Table 3. Strategy layer outcomes with secondary verification.

Task	Initial Strategy SR	Strategy Revisions Initiated	Successful Revisions	Finial Strategy SR
Task 1	0.748	50	44	0.957
Task 4	0.710	58	44	0.919
Task 6	0.652	69	49	0.886
Task 10	0.657	62	47	0.881

.

As shown in Table 3, the initial strategy yielded moderate success rates (65–75%). However, significant improvements emerged once GPT-4V was prompted for a revision when mistakes were flagged. For task 1 (“medicine bottle grasping”, B), the final SR rose from 0.748 to 0.957, illustrating how a single correction could resolve common oversights (such as slight coordinate misalignments). Similarly, the most challenging task 10 (“Stack Multiple Blocks”, HA) demonstrated a notable gain, climbing from 0.657 to 0.881. This underscores the importance of rechecking in scenarios involving intricate object arrangements. Although the secondary verification does not guarantee perfect corrections every time, the data confirm that it considerably enhances overall strategy reliability for all tasks.

3.2.2. Evaluation Layer Performance Evaluation

Once the strategy layer has finalized its plan, whether it has been revised or not, the execution layer carries out each motion step. Parallel to this, an evaluation layer monitors correctness at periodic step intervals, governed by an evaluation factor α that determines how frequently each step is checked. In our implementation, α took values of 0, 0.5, and 1.

• α = 0, checks only when the entire task finishes;
• α = 0.5, checks approximately half of the steps;
• α = 1, checks every step in real time.

We ran each task under all three α settings (70 trials each) and recorded the final success rates, execution durations, and error correction statistics.

Table 4. Execution accuracy and duration under different evaluation layer configurations.

Task	α	Error Step Count	Corrected Step Count	Final SR	Avg. Time (s)
Task 1	0	1	1	0.886	34
	0.5	2	1	0.871	35
	1	1	0	0.914	33
Task 4	0	3	1	0.771	35
	0.5	6	2	0.857	56
	1	8	3	0.843	53
Task 6	0	4	2	0.714	59
	0.5	9	2	0.757	82
	1	12	5	0.800	110
Task 10	0	5	1	0.686	69
	0.5	11	4	0.714	88
	1	10	4	0.771	106

illustrates representative results. Each row corresponds to the same tasks from Table 3, tested with various evaluation factor settings.

Although detailed step-by-step verification (α = 1) often yields a slightly higher success rate, it can significantly prolong execution time, as seen with the “T-shaped block arrangement” (A) task (59 s vs. 110 s). By contrast, sparse checking (α = 0) is faster but may fail to detect mid-task anomalies until completion, leading to more rework or unrecognized failures. In essence, selecting α involves balancing speed and thoroughness. For relatively simpler tasks (B and I), continuous validation does not drastically alter success rates, whereas advanced tasks (A and HA) benefit more from frequent checks but suffer in efficiency.

3.2.3. Comparison of the RTPF with State-of-the-Art Frameworks and Different VLMs

The proposed GPTArm framework advances robotic task planning and execution in dynamic and unstructured environments.

Table 5. Comparative performance of VLM-driven robotic grasping systems in sparse single-object grasping.

Reference (Year)	Model	SR	Generalization	Instruction Input
GPTArm (This Work)	GPT-4V	0.900	Unseen	Image, Speech, and Text
[28]-(2024)	Robot-GPT	0.800	Unseen	Text
[44]-(2024)	Qwen-vl	0.767	Unseen	Image and Text
[29]-(2023)	PaLM-E	0.740	Unseen	Image and Text
[43]-(2024)	GPT-4	0.913	Seen	Text
[31]-(2024)	GPT-4V	0.900	Seen	Image and Text
[14]-(2024)	GPT-4V	0.833	Seen	Image and Text

compares the SR of VLM-integrated robotic arms in sparse environments for single-object grasping tasks.

As highlighted in Table 5, GPTArm achieves an SR of 0.900, outperforming recent benchmarks such as Robot-GPT (0.800) [27], Qwen-vl (0.767) [42], and PaLM-E (0.740) [43]. Notably, this performance superiority is particularly evident in handling unseen objects, where GPTArm leverages GPT-4V’s zero-shot reasoning and adaptive verification mechanisms. Two key innovations contribute to this success: (1) the hierarchical RTPF architecture, which decouples perception, planning, and execution while enabling iterative refinement through closed-loop feedback, and (2) GPT-4V’s ability to interpret user intent and generate context-aware strategies, even under partial occlusions or ambiguous instructions.

To evaluate how well the RTPF accommodates different VLMs, we performed repeated tasks with Fuyu-8B, Qwen-vl-plus, and GPT-4V. Each task targeted a distinct level of complexity, ranging from straightforward single-object manipulation to more elaborate multi-step arrangements. Figure 4 presents the bar charts for SR, GCR, and EXEC, enabling direct comparisons among the three models under identical conditions.

The bar charts indicate that GPT-4V maintains a notable performance advantage over the other two models, particularly in complex tasks such as 6 and 10. For example, in Experiment 6 (additional objects/repositioning), GPT-4V achieves an SR of 0.800, a GCR of 0.932, and an EXEC of 0.935, while Fuyu-8B records 0.514, 0.696, and 0.708, respectively, and Qwen-vl-plus lies in between (0.557, 0.789, 0.848). Even in simple tasks (e.g., Experiment 1), GPT-4V shows an SR of 0.914 compared to Fuyu-8B at 0.743 and Qwen-vl-plus at 0.814, demonstrating its consistent superiority across varying task complexities.

While GPT-4V’s results appeared strongest, Fuyu-8B and Qwen-vl-plus also showed acceptable performance, indicating the RTPF’s flexibility with different VLM architectures. This adaptability suggests the potential for extending the framework to other VLMs, enabling broader application scenarios.

3.3. The Performance Evaluation of GPTArm Across Multiple Tasks

3.3.1. Overview of Completed Task Set

To investigate the practical performance of GPTArm, we conducted ten distinct experiments using GPT-4V as the underlying VLM. Introduced in Section 3.1, these experiments encompass a range of manipulation tasks, each requiring varying degrees of object recognition, grasp precision, and multi-step planning. To test GPTArm’s capacity for generalization, we grouped the ten tasks into two categories:

• Seen (Tasks 1, 2, 4, 5, and 7): The target objects for these tasks were present in the YOLOv10 training set, so the system had prior visual knowledge of each item.
• Unseen (Tasks 3, 6, 8, 9, and 10): The target objects for these tasks never appeared in the YOLOv10 training data, posing a more substantial challenge that requires GPTArm to adapt to novel shapes, colors, or configurations.

All ten experiments used α = 0 in the evaluation layer.

Table 6. Performance of GPTArm (using GPT-4V) across ten real-world tasks.

Task	Number of Successes/Total Attempts	SR ± Uncertainty
Task 1	64/70	91.4 ± 3.35%
Task 2	61/70	87.1 ± 4.01%
Task 3	63/70	90.0 ± 3.58%
Task 4	59/70	84.3 ± 4.35%
Task 5	62/70	88.6 ± 3.80%
Task 6	56/70	80.0 ± 4.78%
Task 7	59/70	84.3 ± 4.35%
Task 8	52/70	74.3 ± 5.22%
Task 9	55/70	78.6 ± 4.90%
Task 10	54/70	77.1 ± 5.02%

presents success rates for each task based on 70 repeated trials. Despite varying task complexity and object layouts, GPTArm maintains robust performance. The highest success rates appear in tasks where the target objects were already “seen” by YOLOv10, yet the system also shows strong potential with “unseen” items. This result highlights GPTArm’s adaptability when confronted with shapes or configurations outside of the training dataset.

Table 5 presents the success rates (SRs) of GPTArm across ten real-world tasks, ranging from 74.3% (Task 8) to 91.4% (Task 1). GPTArm demonstrates notable strengths, particularly in tasks such as Task 1 (91.4%), Task 3 (90.0%), and Task 5 (88.6%), where it achieves consistently high performance. These results underscore the system’s capability to autonomously plan and execute tasks with precision, especially in scenarios with clear objectives and moderate complexity. This feasibility in autonomous task processing highlights GPTArm’s potential as a reliable framework for real-world robotic applications.

Despite these strengths, limitations surface in tasks with lower SRs, such as Task 8 (74.3%) and Task 10 (77.1%), reflecting challenges with instruction ambiguity and spatial complexity. These shortcomings indicate that GPTArm’s performance can degrade when faced with nuanced task demands or complex contexts, revealing areas where its current configuration falls short of optimal reliability.

To address these issues, we propose targeted enhancements drawing on recent insights. First, crafting more precise task prompts with structured language could mitigate misinterpretations, as Liu et al. [43] showed improved model accuracy with refined instructions. Second, enhancing spatial analysis via motion library calibration, precision manipulator upgrades, and depth camera adoption could improve spatial performance [51]. Third, adjusting GPT-4V’s training to emphasize diverse spatial scenarios may strengthen its adaptability, a principle supported by Brohan et al. [29]. These specific refinements aim to elevate GPTArm’s consistency while building on its established strengths.

3.3.2. Illustrative Example of Autonomous Task Planning

To showcase GPTArm’s autonomous planning capabilities, we focus on Task 9, an unseen scenario. The robotic arm must assemble five blocks into a T-shaped structure and place extra blocks on top of the T’s center. The principal challenge lies in autonomously generating a multi-step plan that positions multiple blocks in the correct arrangement without manual intervention.

We introduced several task-specific fine-tuning instructions to reduce collisions, overlapping, and related issues. The system also accounts for the gripper’s size and object footprints. When a target position is already occupied, the plan relocates the interfering block to a different location before placing the new one.

In our experiments, both voice and text inputs were tested for issuing commands. Figure 5 illustrates a voice-driven run in which the system interprets the spoken request to assemble the T-shape. The strategy first moves a yellow block out of the central workspace. It then detects that the pink block’s target coordinates overlap with a green block, prompting a temporary shift of the green block. Once that space is free, the pink block is placed at the T’s center, followed by the remaining two green blocks and a final blue block on top.

Although the operation succeeds, GPT-4V does not minimize the total steps. Alternate strategies, such as swapping two blocks earlier, could reduce repositioning. Repeated trials indicate that GPTArm often clears peripheral areas first, possibly due to caution about clustering objects near the center. With more refined prompt engineering and enhanced mechanical precision, one might expect more efficient action sequences. Nonetheless, the system’s ability to manage a multi-step routine and reassign objects underscores its strength in autonomous task planning.

3.3.3. Non-Customized Interaction and Dynamic Re-Planning

For ambiguous instruction scenarios, we highlight Task 7, where the system reorganizes objects based on vague user commands. GPT-4V parses natural-language directives that often lack direct item references (for example, “I feel a bit unwell”, which the system interprets as a need for medication). By analyzing these colloquial cues semantically, GPTArm can execute the required grasp action even without explicit item names.

Initially, the system places the requested item (a medicine bottle in this task) at the center of the workspace. If the user revises the request, such as “I actually need the charger”, the GPTArm recognizes a changed goal. The strategy first puts the bottle back in its original position, then positions the charger in its place. It effectively re-plans the sequence without requiring a special script for each revised request. This layered structure, combined with contextual understanding, enables GPTArm to adapt fluidly to altered instructions. The execution process of this task is presented in Figure 6.

Repeated trials show that even with ambiguous or colloquial directives, GPTArm can interpret user intent and redirect actions as needed, although occasional misidentifications may arise when multiple objects share similar features. Overall, the system’s ability to interpret non-custom instructions, accommodate evolving user demands, and adjust its motion plan on the fly suggests a level of adaptability beyond that of a purely preprogrammed mechanism.

These real-world experiments underscore GPTArm’s strengths in autonomous strategy generation, multi-step sequencing, and responsive user interaction under uncertain conditions. Whether encountering previously unseen objects or adapting to sudden goal shifts, GPT-4V’s integration with the RTPF planning structure provides a robust foundation for sophisticated manipulative tasks.

4. Discussion

The development of GPTArm and its underpinning robotic task processing framework (RTPF) marks a significant advancement in autonomous robotic manipulation, particularly in bridging the semantic and operational gaps that have long challenged traditional robotic systems. The experimental results across ten diverse tasks (Section 3.3, Table 6) demonstrate GPTArm’s efficacy, with success rates reaching up to 91.4% in standardized benchmarks (Task 1). This performance underscores the system’s proficiency in autonomous task planning, dynamic error recovery, and multimodal interaction—core innovations of the RTPF. By leveraging GPT-4V’s reasoning capabilities alongside YOLOv10’s precise object localization, GPTArm achieves robust adaptability, excelling in both seen and unseen object scenarios (e.g., Tasks 3 and 6, with SRs of 90.0% and 80.0%, respectively). These outcomes validate the RTPF’s hierarchical approach, which decouples perception, planning, and execution while integrating real-time feedback loops to ensure resilience in dynamic environments.

When compared to state-of-the-art frameworks, GPTArm offers distinct advantages. For instance, Robot-GPT [28] achieves an SR of 0.800 for single-object grasping but lacks the closed-loop error recovery mechanisms that enable GPTArm to self-correct mid-task (Section 3.2.2). Similarly, PaLM-E [29] (SR 0.740) and Qwen-vl [44] (SR 0.767) demonstrate versatility in multimodal inputs but fall short in handling unseen objects with the same consistency as GPTArm (Section 3.2.3, Table 5). The RTPF’s hierarchical decomposition and secondary verification process (Section 2.1.2) address limitations in prior works, such as the absence of dynamic re-planning in [43] or the reliance on pre-trained datasets in [31]. This positions GPTArm as a forward step in VLM-driven robotics, particularly for tasks requiring contextual reasoning and zero-shot generalization.

GPTArm’s natural language interface democratizes robotic control, enabling non-experts to use complex systems for applications like assistive healthcare (e.g., fetching medication) and smart home automation (e.g., object organization). Its modular, software-centric design (Section 2.1.5) ensures scalability across diverse robotic platforms, lowering costs compared to hardware-intensive solutions like stereo vision systems [39], and enhances industrial automation efficiency in dynamic settings.

However, the system is not without constraints. The reliance on GPT-4V’s API introduces latency, averaging 33–110 s per task (Section 3.2.2, Table 4), which may limit real-time applicability in time-critical scenarios. Additionally, while effective for rigid objects, GPTArm struggles with irregularly shaped or deformable items (e.g., Task 10, SR 77.1%), where geometric reasoning remains suboptimal due to the 2D coordinate-based operation (Section 2.1.5). These limitations indicate the RTPF’s current design falls short of fully addressing the complexities of unstructured environments. Future improvements include reducing API latency via local deployment or streamlined protocols for real-time use, enhancing zero-shot recognition of irregular objects with few-shot learning or physics-informed models [34], and extending the RTPF to multi-arm coordination and deformable object manipulation using federated learning and physics-based planning to broaden its real-world applicability.

5. Conclusions

This study presents GPTArm, an AI-driven robotic arm system that integrates GPT-4V’s multimodal reasoning with adaptive task planning to address real-world manipulation challenges. The RTPF’s hierarchical architecture enables autonomous task decomposition, closed-loop error recovery, and environment-aware execution, overcoming the rigidity of traditional pre-programmed systems. Empirical validation across ten tasks demonstrates the framework’s efficacy, achieving a peak success rate of 91.4% in standardized scenarios and robust generalization to unseen objects. The system’s natural language interface enables non-experts to intuitively control complex operations through voice or text commands, bridging the gap between human intent and robotic execution.

By advancing the synergy of VLMs and robotic control, this work provides a scalable solution for dynamic environments, with potential applications in industrial automation, assistive healthcare, and smart home systems. Future research will focus on latency optimization for real-time deployment and enhanced zero-shot recognition for irregular objects, further pushing the boundaries of autonomous robotic systems.