Integrating Vision–Language–Action Models and RGB-D Sensing for Robotic Waste Sorting on KUKA LBR iiwa

Abstract

Robotic waste sorting presents significant challenges, including object variability, cluttered environments, and the predominant reliance on deep learning and traditional computer vision techniques, which typically demand extensive datasets and task-specific training. This paper introduces a robotic waste sorting system that integrates the Gemini Vision–Language–Action (VLA) model with a KUKA LBR iiwa collaborative robot and an RGB-D camera. Our approach leverages the advanced reasoning capabilities of large, pre-trained VLA models to perform waste sorting, without requiring explicit training or dataset collection. Key contributions include the development of effective prompt engineering strategies for waste object identification, the assessment of the VLA’s performance in terms of inference time and accuracy, and the development of different grasping strategies for operation in cluttered scenarios. Our experimental tests demonstrated that the system’s inference time is between 2 and 4 s, which is suitable for collaborative robotic applications, and the system achieved a high overall classification accuracy of 89.64%. Crucially, we demonstrated that integration of RGB-D sensing enhanced the model’s ability to perceive object heights, resolve occlusions, and make informed grasping decisions in realistic, three-dimensional settings. We further validated multiple real-world grasping strategies, demonstrating tradeoffs between system efficiency and safety in heavily cluttered scenarios. This work establishes a practical and adaptable framework for deploying VLA-driven intelligence on commercial robotic platforms, highlighting the potential of VLAs for complex manipulation tasks beyond waste sorting.

1. Introduction

Waste sorting represents a key benchmark task for robotic applications, due to the inherent challenges it presents. The core difficulties of this task arise from the variety of waste items, which range widely in material, shape, size, and condition, and the unstructured, often cluttered, scenarios in which they are typically encountered [1]. These difficulties make waste sorting an ideal testbed for evaluating advanced robotic intelligence and dexterous manipulation skills [2].

While various sensor modalities can be used for waste sorting, computer-based vision remains the most prevalent approach. Traditional computer vision algorithms, however, struggle with the great variability in waste item characteristics. Consequently, deep learning-based neural networks have become the standard approach for this task, employing diverse model architectures to achieve robust classification [3,4]. Despite their successes, these models introduce their own set of significant challenges. In particular, they require large labeled datasets, which are time-consuming to collect. Existing publicly available datasets [5] often prove inadequate, either being collected in controlled, idealized environments that poorly reflect real-world clutter and object degradation, or lacking sufficient diversity and scale. Furthermore, the training of these complex neural networks is computationally intensive, requiring substantial hardware resources (e.g., high-performance GPUs) and extensive experimentation to fine-tune hyperparameters, thereby limiting their accessibility and rapid deployment [6].

Beyond the critical task of visual classification, the real-world deployment of robotic waste sorting systems faces additional engineering challenges [7]. Crucially, the robot must not only identify waste items but also effectively manipulate them. This involves selecting appropriate grippers capable of handling objects with highly variable shapes, sizes, and weights [8]. Furthermore, waste items are frequently found in heavily cluttered scenes, piled on top of each other. Picking these items necessitates complex spatial reasoning to determine optimal grasp points, prioritize picking sequences for overlapping objects, and avoid collisions. Accurate estimation of object height, often unknown a priori, becomes indispensable, typically requiring the integration of depth-sensing cameras to perceive the three-dimensional environment effectively [9,10].

The combination of these challenges, including data scarcity, model generalization, real-time integration into robotic platforms, and complex physical interaction in unstructured and cluttered environments, highlights a significant research gap. Current approaches often address these issues separately, focusing on either improved classification or specific manipulation, but rarely offer a comprehensive and adaptable solution for real-world deployment in dynamic robotic waste sorting scenarios.

Recently, Vision–Language–Action (VLA) models have emerged as a transformative technology in the field of robotics. Unlike traditional neural networks, VLAs are pre-trained on vast amounts of internet-scale data, often encompassing diverse modalities including text, images, and actual robotic teleoperation data [11]. This extensive pre-training provides VLAs with robust reasoning capabilities and the ability to generalize to novel tasks and environments without requiring explicit task-specific training or large custom datasets. For robotic waste sorting, VLAs offer a compelling paradigm shift [12]. They potentally enable the rapid setup of a robotic sorting system in cluttered environments with zero training, leveraging their inherent understanding of objects, actions, and spatial relationships. This eliminates the complex process of dataset collection and model training, and significantly reduces the need for expert knowledge in neural network architectures and hyperparameter tuning. However, despite their promising reasoning and generalization capabilities, the practical integration and comprehensive performance evaluation of VLA models, specifically in industrial robotic systems for complex tasks like waste sorting, remains largely unexplored in real-world settings.

In this paper, we integrate the advanced reasoning capabilities of the Gemini VLA model [13] with a KUKA LBR iiwa collaborative robot and an RGB-D camera to showcase and assess the potential of this new technology in a complex real-world scenario such as waste sorting. The key contributions of this work are as follows:

• Development and evaluation of effective prompt engineering strategies for waste object identification, advanced spatial reasoning (e.g., optimal grasp point detection, grasping priority, and graspability), and robust handling of cluttered scenes;
• Quantitative assessment of the system’s inference time, demonstrating its suitability for collaborative robotic applications, and achievement of high overall classification accuracy, demonstrating its robustness;
• Validation of RGB-D sensing integration, showcasing its enhancement of the VLA model’s ability to perceive object heights, resolve occlusions, and make informed grasping decisions in realistic, three-dimensional settings;
• Testing and demonstration of multiple real-world grasping strategies for cluttered environments, highlighting the trade-offs between system efficiency (i.e., number of API calls) and safety.

2. Related Works

Robotic waste sorting presents a challenging and critical task for autonomous systems, specifically requiring complex capabilities in object identification, grasp planning, and dexterous manipulation within highly variable and unstructured environments [2]. Historically, computer vision has been central to waste classification. Early approaches relied on traditional image processing techniques, but their limitations in handling the vast heterogeneity of waste items quickly led to the adoption of deep learning. Convolutional Neural Networks (CNNs) and advanced object detection models (e.g., YOLO, Mask R-CNN) have become dominant for robust waste material classification [14,15,16]. While these methods demonstrate high accuracy in controlled settings, their performance is heavily dependent on extensive labeled datasets, and they often struggle with generalization to novel items or real-world variations not present in their training data.

Collecting and classifying large datasets for deep learning models is a time-consuming and labor-intensive process: for this reason, researchers often rely on existing publicly available resources rather than creating new ones. While several datasets exist, they commonly suffer from significant drawbacks: many feature single objects on idealized or synthetic backgrounds, failing to represent the realistic clutter and occlusions inherent in real-world waste sorting scenarios (e.g., TrashNet [5], Sekar’s [17]). Others, like TACO [18], include multiple objects but often present highly diverse, mobile-robot-centric backgrounds (e.g., street scenes) that are not directly applicable to stationary industrial sorting. Datasets such as Washington RGB-D [19] were often collected for general object recognition rather than specific waste sorting, and synthetic datasets like RESORT-IT [20] may lack the full fidelity of real-world object appearance. Private datasets (e.g., WasteRL [21], Domestic Garbage Dataset [22]) exist but are generally inaccessible, further worsening the data scarcity problem for researchers. These limitations underscore the difficulty in developing generalizable waste sorting robots using conventional supervised learning.

Beyond visual classification, successful robotic waste sorting demands robust physical interaction. Grasping irregularly shaped, often deformable, and sometimes damaged waste objects presents significant challenges for traditional robotic grippers and grasp planning algorithms [23]. Many studies have explored specialized end-effectors developed for waste handling, ranging from vacuum grippers to various types of compliant or underactuated mechanical grippers designed to adapt to diverse object geometries [8,24,25,26]. Furthermore, waste items are typically found in heavily cluttered scenes, often piled and occluded. This requires advanced spatial reasoning, including accurate object height estimation and effective picking sequence determination, which is typically addressed through the integration of 3D perception sensors like RGB-D cameras [27,28].

In recent years, Vision–Language Models (VLMs) and, more specifically, Vision–Language–Action (VLAs) models have emerged as a transformative paradigm in robotics [29,30]. These models, such as Gemini Robotics [13], are pre-trained on vast internet-scale datasets that include diverse modalities, such as text, images, and robotic teleoperation data. This extensive pre-training provides VLAs with powerful reasoning capabilities and the ability to generalize to novel tasks and environments, reducing the need for explicit task-specific training or large custom datasets. VLAs have shown promising results in various general robotic manipulation tasks, from instruction following to complex pick-and-place operations in unstructured environments [31,32]. However, to the best of the authors’ knowledge, no prior work has integrated a VLA model into a commercial robot for robotic waste sorting and comprehensively assessed its performance, particularly with the integration of depth sensing. This work aims to bridge this gap by comprehensively evaluating the system’s performance in terms of inference time, classification accuracy, and the trade-offs between number of API calls and ability to deal with different levels of clutter.

3. Materials and Methods

3.1. Vision–Language–Action (VLA) Framework

Vision–Language–Action (VLA) models enable camera-equipped robots to perform complex reasoning and visual recognition tasks without the need for task-specific training data. Because VLAs communicate primarily through text-based Application Programming Interfaces (APIs), integrating the AI’s high-level reasoning with the robot’s low-level execution requires carefully engineered system prompts. Both the instructions provided to the model and the data returned by it are entirely text-based.

To utilize this architecture for robotic waste sorting, it is essential to design a prompt that constrains the AI’s output into a machine-readable format. The information workflow between the system prompt, the input image, and the VLA is depicted in Figure 1. In this study, we utilize Gemini Robotics ER 1.5, a model specifically fine-tuned to process spatial and robotic data (though the proposed architecture is adaptable to other multimodal large language models).

The prompt begins by defining a specific system role for the model, explicitly instructing it to act as a vision and grasp planning module for a robotic manipulation system. Following this role assignment, the AI is given a specific image recognition task: for example, to identify all waste items in the provided image, calculate their geometric centers (or optimal grasping points), and classify them into one of six predefined categories (metal, paper, glass, plastic, organic, and unsorted waste).

To enable real-time integration with the robot’s control software, the prompt requires the output to be formatted exclusively as a JSON (JavaScript Object Notation) string. When the API call is executed using standard programming languages such as Python, the textual prompt and the visual data (an RGB image, potentially combined with a depth image, as further discussed in Section 3.3.4) are sent to the model. The model processes the visual information and returns a structured JSON array. This text-based JSON response contains the recognized classes, their corresponding 2D pixel coordinates, and any requested grasping priorities. This string is then parsed by the local control system to generate a visualized output (Figure 1, right) and to extract the numerical coordinates required to move the robot to the grasping points.

3.2. Robotic Workcell and System Integration

The experimental setup consists of a 7 DOF collaborative robot (LBR iiwa 14 R820, KUKA, Augsburg, Germany), equipped with a pneumatic vacuum gripper and a depth camera (Intel RealSense D345, Intel Corporation, Santa Clara, CA, USA). This depth camera provides a depth map, i.e., a z-value (depth) for each pixel. As waste items are typically cluttered and stacked, this depth information is essential to avoid collisions, plan a successful grasping order and, extract the z-value for a given $(x,y)$ grasping point. Crucially, with the camera mounted directly on the robot’s flange, the z-value provided by the depth map inherently represents the distance between the object and the robot’s flange. This distance is then transformed into the robot’s world coordinates via a transformation matrix.

The robot controller (KUKA Sunrise Cabinet, KUKA, Germany) is natively designed to be programmed in Java. To facilitate integration with the camera and the AI model, the robot is controlled from an external computer using Python (version 3.14). This is achieved through the iiwaPy3 library [33], which serves as a Python wrapper for the KUKA Sunrise Toolbox [34]. This setup allows the user to control the manipulator directly from the external computer without programming in Java [35]. All data processing, including camera control via the pyrealsense2 and OpenCV libraries, is executed in Python. Once the JSON response is received from the model, the pixel coordinates of the identified waste objects are transformed into the robot coordinate system using a homography matrix. When multiple overlapping items are present, the system processes the grasping order to ensure safe picking. Finally, based on the material identified in the JSON string, the system selects a specific place position that corresponds to the appropriate sorting bin.The complete system setup and information workflow are illustrated in Figure 2.

3.3. Experimental Design

We designed a series of experiments to evaluate the proposed robotic waste sorting system. Specifically, we aimed to answer the following research questions:

• What prompts are most effective for robotic waste sorting, and how can they be designed to move beyond simple image classification to identify optimal grasping points and picking sequences for overlapping waste items?
• What is the system inference time, and how does it change based on prompt complexity, the number of waste items, and the level of visual clutter?
• What is the overall classification accuracy, and how does this accuracy vary across different material categories?
• Can depth images be included as inputs to the cloud API, and if so, does this improve object identification in cluttered environments and affect the processing time?
• How can the outputs from the language model and the depth camera be combined to execute successful physical picking strategies on a commercial robot?

To address these questions, we designed five distinct sets of experiments, which are detailed in the following subsections.

3.3.1. Prompt Engineering and Selection

To evaluate the potential of the VLA model, we designed and tested various system instructions. We ultimately selected six benchmark prompts that are particularly effective for robotic waste sorting and capable of handling scenarios with varying degrees of visual clutter.

3.3.2. Inference Time Analysis

To evaluate the computational efficiency of the system, we analyzed how inference time varies with prompt complexity and the number of items captured in the input image. We defined three scenarios of increasing complexity: simple (1–3 objects, not overlapped), medium (4–8 objects, partially occluded), and cluttered (8+ objects, heavily occluded). For each scenario, 10 distinct images were processed across the 6 different prompt configurations. This approach ensures that the recorded inference times account for both network latency and the cognitive load of the model across diverse visual inputs, avoiding potential artifacts from server caching. Finally, we computed the mean inference time and the standard deviation for each scenario and prompt combination.

3.3.3. Classification Accuracy Analysis

To evaluate the classification accuracy of the system, we selected a set of images representing mixed scenarios, including both distinct and cluttered objects, totaling approximately 400 waste items. We calculated the overall classification accuracy, the specific accuracy for each material class, and the confusion matrix to identify misclassification trends across different categories. Additionally, we analyzed the data to determine whether the classification errors were systematic or purely random.

3.3.4. Depth Image Integration

Some of the selected prompts aimed to determine a successful grasping order or identify the top object to pick in cluttered scenarios. However, relying solely on a 2D RGB image is limiting because the AI lacks information about object heights, which can lead to sequence errors. Therefore, we evaluated how much the system performance improves when providing a 3D depth image alongside the standard RGB input.

In addition, we assessed whether processing this additional depth data significantly increased the system inference time. For this reason, we selected 10 pairs of RGB and depth images representing cluttered waste sorting scenarios. We then compared the outputs and processing times when feeding the AI with only the RGB image versus both the RGB and depth images. This test was repeated across the specific prompts designed to handle occluded environments.

3.3.5. Real-World Grasping Strategies

To evaluate the practical application of the system, we developed different picking strategies for the KUKA manipulator based on the specific task goals and the level of clutter among the waste items. Depending on whether the objects are separated or overlapping, we designed specific physical routines to achieve a successful grasp. These strategies vary in efficiency and execution: for instance, some strategies rely on a single API call to quickly pick multiple separated items, while others require multiple API calls and more complex prompts to ensure safe picking in highly occluded environments.

4. Results

4.1. Prompt Engineering and Selection

Developing effective prompts for robotic waste sorting requires addressing both visual recognition and physical manipulation. Unlike standard identification tasks where finding the object is the final goal, a robotic system must also determine how to safely grasp an item and move it to a target location. This physical interaction introduces significant complexity because waste objects typically pile up, occlude one another, and vary greatly in height. Therefore, the instructions provided to the model must explicitly account for these spatial constraints to generate a successful picking strategy. To evaluate how well the system handles these physical realities, we defined six benchmark prompts:

• Prompt 1: Detect waste objects and their geometric centers. This serves as a baseline instruction that limits the task to identifying and classifying objects while providing a single coordinate point. When objects are physically separated, this simple approach is highly effective and forms the basis for the isolated picking method described in Section 4.5.1.
• Prompt 2: Detect waste objects and locate their optimal grasp points. Because irregular waste items often have unstable surfaces, this advanced instruction directs the model to find a secure gripping location rather than just returning the mathematical center.
• Prompt 3: Detect waste objects and generate their two dimensional bounding boxes. Knowing the physical boundaries of an object allows the system to estimate the required gripper opening and evaluate the object orientation, which helps prevent collisions during the approach phase.
• Prompt 4: Detect the geometric center of the topmost object to be grasped first. This instruction targets cluttered environments where overlapping items must be removed sequentially. As detailed in Section 4.5.2, the robotic system can this prompt to iteratively pick the highest object and then capture a new image to find the next graspable object.
• Prompt 5: Detect waste objects, their geometric centers, and their grasping priority. Instead of relying on multiple separate requests, this instruction allows the system to determine the correct picking sequence for multiple objects in the same image, ensuring that items on top of a pile are removed first (as detailed in Section 4.5.3).
• Prompt 6: Detect waste objects, their geometric centers, and whether they can be grasped. This prompt is used to identify which items are currently graspable and allows the robot to pick only those specific items, and then repeat the process with a new image and API call (as detailed in Section 4.5.4).

The extended versions of these benchmark instructions are provided in Appendix A.1. As an example, Figure 3 illustrates the visualized output generated by the model for all six benchmark prompts given the same input image, with the corresponding inference time recorded in the bottom left corner of each image. As can be observed from Figure 3, the classification accuracy remains consistent regardless of the prompt complexity. The model reliably assigns the correct material category to each object across all six tests. In addition, the computational processing time increases with the increase in prompt complexity, a trend that is quantified and discussed further in Section 4.2.

Regarding more basic prompts, prompt 1 successfully calculates the object centers with sufficient precision for robotic picking. Although the coordinates identified for the plastic spray bottle and the paperboard packaging are not perfectly central, they still enable a successful vacuum grasp. The outputs generated by prompt 1 and prompt 2 are nearly identical, with only the grasping point on the paperboard packaging showing a slight shift. Because the difference between the geometric center and the optimal grasping point proved to be minimal in practice, we opted to use the geometric center for all subsequent prompts. Finally, prompt 3 correctly identifies all bounding boxes.

Regarding the more advanced prompts specifically designed for cluttered environments, prompt 4 identifies the metal can as the topmost object to be grasped. While the metal can is a valid target because it rests on top of the plastic spray bottle, it is not the highest object in the image. In fact, the object with the greatest physical height is the green paper box. This spatial miscalculation occurs because the two dimensional input image lacks the depth data required to evaluate physical height, an issue that is addressed by integrating a depth camera in Section 4.4. Prompt 5 establishes a feasible grasping priority sequence. The model assigns the first priority to the metal can, followed by the polystyrene piece, the paper towel, the plastic bottle, the plastic spray bottle, the green paper box, and finally the cardboard packaging. This order is feasible because objects resting on other items are scheduled to be picked before the items beneath them. However, the ideal strategy would prioritize the objects with the highest absolute elevation, as picking a lower item while a taller object remains in the workspace increases the risk of collisions between the robot end-effector and the taller objects. Finally, prompt 6 successfully distinguishes between graspable and ungraspable items in most cases. However, it makes two errors: it incorrectly classifies the plastic spray bottle as graspable despite the metal can resting on it, and it misclassifies the paper towel as graspable even though the polystyrene piece is on top of it.

4.2. Inference Time Analysis

To evaluate the computational efficiency of the system across the six benchmark prompts, we analyzed the inference time required for different levels of visual complexity. The results are summarized in Figure 4, which presents the mean inference time and standard deviation over 10 images for each of the three defined scenarios, alongside a representative example of the visual clutter involved.

Starting with the simplest case (scenario A, 1–3 objects, not overlapped), the data clearly shows that both the inference time and its standard deviation increase as the prompt becomes more complex. Specifically, the inference time rises significantly from from $1.79\pm 0.31 \mathrm{s}$ for prompt 1 to $2.61\pm 0.94 \mathrm{s}$ for prompt 6. This absolute difference of approximately $0.8 \mathrm{s}$ translates to a 45% increase in inference time when moving from basic identification to more advanced reasoning tasks.

A similar trend is observed in scenario B (4–8 objects, partially occluded), where inference time consistently scales with prompt complexity, with the exception of prompt 4. This deviation is reasonable because prompt 4 only requires the model to identify a single topmost object, whereas the other prompts demand data for all items in the image. Interestingly, while transitioning from scenario A to scenario B increases the processing time for most prompts due to the added visual clutter, prompt 4 actually becomes faster. This suggests that the model identifies a clear topmost object more easily in a stacked pile than when evaluating several separated items. Excluding this exception, the inference time in scenario B grows from $2.31\pm 0.36 \mathrm{s}$ for prompt 1 to $3.09\pm 0.72 \mathrm{s}$ for prompt 6, maintaining the same absolute increase of roughly $0.8 \mathrm{s}$ seen in the simpler scenario.

Scenario C (8+ objects, heavily occluded) exhibits much of the same behavior, as the inference time continues to increase with prompt complexity, again excepting the single-object identification required by prompt 4. However, it is worth noting that in this highly cluttered environment, the longest processing time shifts to prompt 3, as accurately defining bounding boxes around heavily occluded objects is far more computationally demanding than simply locating their geometric centers. Consequently, for scenario C, prompt 4 yields the lowest inference time at $2.62\pm 0.81 \mathrm{s}$, while prompt 3 yields the highest inference time at $4.35\pm 0.86 \mathrm{s}$. If we exclude these two outliers and compare the baseline prompt 1 ($3.04\pm 0.26 \mathrm{s}$) to the advanced prompt 6 ($3.91\pm 0.56 \mathrm{s}$), the absolute difference remains consistent at approximately $0.85 \mathrm{s}$.

To summarize, the results in Figure 4 demonstrate that both prompt complexity and image clutter significantly influence the system inference time. For a given visual scenario, moving from a basic identification task to a more complex reasoning task results in a consistent increase in inference time of approximately $0.8 \mathrm{s}$, corresponding to a 30–40% relative increase. Similarly, applying the same prompt to increasingly cluttered images also increases the required processing time. Across all tested scenarios and prompts, the overall inference time for a single API call generally ranges between 2–4 s. While more complex prompts require a longer inference time per request, they ultimately improve overall system efficiency by allowing the AI to identify multiple objects and establish a picking priority simultaneously, which significantly reduces the total number of API calls required.

4.3. Classification Accuracy Analysis

The classification accuracy of the proposed system, both overall and across individual material categories, is detailed in

Table 1. Overall and class-specific classification accuracy of the vision–language–action model.

	Overall	Metal	Paper	Glass	Plastic	Organic	Unsorted Waste
Accuracy	89.64%	96.22%	95.38%	96.36%	94.69%	97.87%	60%

. The model achieved a high overall accuracy of 89.64%. More in detail, the system maintained highly reliable classification for distinct material classes, exceeding 94% accuracy for metal, paper, glass, plastic, and organic waste, with organic waste peaking at 97.87%. However, the system’s performance was significantly hindered by the unsorted waste category, which yielded an accuracy of only 60%, thereby lowering the overall performance metric. The confusion matrix, illustrated in Figure 5, provides deeper insight into these classification trends: the data clearly indicates that the system’s errors are concentrated within the unsorted waste category, which the model frequently misclassified as either paper or plastic.

A qualitative analysis of these errors revealed that the misclassifications stem primarily from the prompt design rather than a failure in the model’s visual recognition. For instance, the AI consistently identified soiled tissues as paper and oil-contaminated packaging as plastic. While materially correct, both items actually belong in unsorted waste due to their condition. As shown in Appendix A.1, our baseline prompts instructed the model to classify items solely by their primary material. Because real-world waste sorting often depends on an object’s state (such as its level of contamination) simple material identification can be insufficient. Expanding the prompts to include explicit sorting rules for contaminated items or regional guidelines would likely resolve these ambiguities and further improve system accuracy.

4.4. Depth Image Integration

Integrating depth data into the system prompt is particularly advantageous in cluttered environments where waste items overlap or are stacked. Consequently, this analysis focused on prompts 4, 5, and 6, which require the model to reason about object height, stacking order, and graspability. To accommodate this, the prompts were modified to instruct the model on interpreting a color-mapped depth image alongside the standard visual input (an example of this modified workflow is detailed in Appendix A.2).

To evaluate the impact of this additional data across various cluttered scenarios, the VLA model was fed either a standard 2D RGB image or a combined RGB-D input. As a representative example, Figure 6 illustrates a single highly cluttered scene processed using prompts 4, 5, and 6, comparing the inputs and their corresponding visual outputs. The addition of depth data significantly improved the model’s spatial awareness and decision-making across all three tasks. For prompt 4 (detecting the topmost object), using only the 2D RGB image led the model to incorrectly identify the plastic bottle as the highest item. While the bottle is graspable, the depth map clearly reveals that the white paper box is actually the topmost object. When provided with the combined RGB-D input, the model successfully corrected this error and targeted the white paper box. Similarly, for prompt 5 (determining grasping priority), the 2D-only input generated a feasible but suboptimal picking sequence, prioritizing the lower plastic bottle and metal can over the physically higher paper and cardboard boxes. With the RGB-D input, the VLA model updated the grasping sequence to reflect true physical elevations. It correctly assigned the highest priorities to the white paper box and large cardboard box, followed by the metal can, the plastic bottle, and finally the smaller cardboard box. This spatially accurate sequencing ensures that no collisions occur between the end-effector and surrounding objects. Finally, for prompt 6 (determining graspability), the 2D-only input resulted in the metal can being incorrectly classified as not graspable. The inclusion of depth data resolved this visual ambiguity, allowing the model to correctly classify all overlapping items in the scene.

After establishing that including a depth image yields superior performance in cluttered environments, we evaluated whether processing this additional data significantly impacted the system’s inference time. To quantify this, we selected 10 distinct sets of paired 2D RGB and 3D depth images. Each set was processed using prompts 4, 5, and 6 under two conditions: using only the 2D image, and using the combined 2D and depth images. The comparative mean inference times and corresponding standard deviations are presented in Figure 7.

For prompt 4, the inference time remained nearly identical. This observation can be explained by the fact that the color-coded depth map provides clear spatial cues that allow the model to rapidly identify the highest object. For prompts 5 and 6, which require more complex reasoning to determine grasping priority and graspability, the inference time showed only a slight increase. Ultimately, this minor addition to the processing time can be an acceptable trade-off, considering the improvements in operational safety and task reliability.

4.5. Real-World Grasping Strategies

4.5.1. Strategy 1: Non-Cluttered Waste Items, Single API Call with Prompt 1 (Basic Classification)

When waste items are well-separated and non-overlapping, the basic classification provided by Prompt 1 is sufficient. The KUKA robot is first moved to a home position, where both a 2D RGB image and a depth map are acquired simultaneously. The RGB image is then used, along with prompt 1, as the input for a single API call to identify all objects without assigning a grasping priority. While the depth map is not provided to the VLA model, its data is essential for the local control system to execute the physical grasp. For each object listed in the JSON response, the system extracts the $(x,y)$ pixel coordinates and the corresponding z-value from the acquired depth map, and commands the robot to pick and place the item into the appropriate sorting bin. This process is repeated for all identified items. The primary advantage of this strategy is its high efficiency, as it requires only one image acquisition and one API call to process multiple objects. Its main drawback, however, is that it is only applicable to simple, non-cluttered scenarios. A video demonstration is available in the Data Availability Statement part, and the corresponding action-information flow is illustrated as a Gantt chart in Figure 8.

4.5.2. Strategy 2: Cluttered Waste Items, Multiple API Calls with Prompt 4 (Topmost Object First)

For environments with multiple overlapping items, strategy 2 uses prompt 4 to iteratively target the topmost object. The process begins with the robot at a home position, where it captures both a 2D RGB image and a depth map. Both images are then used as input for an API call with Prompt 4, which instructs the model to identify only the single highest object. Upon receiving the JSON response, the system extracts the object’s $(x,y)$ coordinates and the corresponding z-height from the depth map, and executes the pick-and-place sequence. This cycle of image acquisition, API call, and grasping is repeated iteratively, with the robot removing one object at a time until the workspace is clear. The primary advantage of this iterative approach is its safety and reliability, as it eliminates the risk of collision in heavily cluttered scenes. However, the need for a new image acquisition and API call for every single item makes the strategy computationally inefficient and substantially reduces the overall sorting throughput. A video demonstration is provided in the Data Availability Statement part, and the workflow is detailed in Figure 9.

4.5.3. Strategy 3: Cluttered Waste Items, Single API Call with Prompt 5 (Grasping Priority)

Strategy 3 aims to improve efficiency by using prompt 5 to establish a complete grasping sequence from a single API call. In this approach, the VLA model processes both the 2D RGB and depth images to return a prioritized list of all recognized objects. The local control system then extracts the z-height of each recognized item before initiating the multi-part pick-and-place sequence. The primary advantage of this method is its high efficiency, allowing the robot to sort an entire cluttered scene with just one initial API call.

However, this strategy has a key limitation when objects are perfectly stacked. If a lower object’s geometric center is covered, the system will incorrectly extract the surface height of the upper object. As a result, the robot stores the wrong z-value for the lower item and will fail to grasp it after the top object is removed. This makes the strategy highly effective for partial overlap but unreliable for perfectly aligned stacks of objects. The workflow for this strategy is illustrated in Figure 10, with a corresponding video demonstration available in the Data Availability Statement part.

4.5.4. Strategy 4: Cluttered Waste Items, Multiple API Calls with Prompt 6 (Graspable vs. Non Graspable)

The previous strategies present a trade-off in cluttered environments: strategy 2 is safe but inefficient due to its reliance on multiple API calls for single objects, while strategy 3 is more efficient but can fail when objects are fully occluded. To address this limitation, we developed strategy 4, which offers a robust and safe method for handling even the most complex, heavily overlapping scenarios.

This strategy utilizes prompt 6, which instructs the VLA to distinguish between graspable and not graspable items based on occlusion. The system makes an API call using both the RGB and depth images as input. The VLA’s response identifies all currently accessible objects, the system extracts their z-height and the robot then proceeds to pick all items identified as graspable. Once these objects are cleared, a new image is captured, and the process is repeated. This iterative cycle continues until all objects have been sorted.

The primary advantage of this approach is its ability to resolve the trade-off between safety and efficiency. It ensures robust performance in densely cluttered scenes by iteratively re-evaluating the workspace, while significantly improving on the inefficiency of Strategy 2. By identifying and clearing all currently graspable items in a single cycle, it minimizes the number of API calls and image acquisitions required to clear a complex pile. A video demonstration of this strategy is available in the Data Availability Statement part. The workflow, illustrating the iterative nature of the process, is depicted in Figure 11.

5. Discussion

In this work, we successfully integrated a Gemini VLA model with a depth camera and a collaborative robot for robotic waste sorting. Our results demonstrate that this approach is highly effective, enabling the system to perform complex visual recognition and physical manipulation tasks without any task-specific training or dataset collection. This study therefore validates the feasibility of leveraging large, general-purpose AI models to control commercial robotic hardware for complex, real-world applications.

Firstly, our experiments highlight the key role of prompt engineering in defining the system’s capabilities. Simple instructions, such as those in prompts 1–3, enabled the VLA to function effectively as a replacement for traditional computer vision models, accurately identifying object centers, optimal grasp points, and bounding boxes with sufficient precision for successful robotic manipulation. However, the true potential of the VLA was unlocked through more complex prompts (4–6): these instructions successfully prompted the model to perform higher-level reasoning, such as establishing a grasp priority sequence or determining object graspability based on occlusion. This transition from basic perception to strategic reasoning is a key advantage of the VLA architecture.

Our analysis reveals that the system’s inference time for a single API call typically ranges between 2 and 4 s. As expected, latency is directly influenced by two main factors: prompt complexity and scene clutter. We quantified that moving from a basic object identification prompt to one requiring more complex reasoning, such as determining a grasping priority, consistently increased the processing time by approximately 45%. However, this increase in per-call latency is balanced by a significant gain in overall system efficiency. By enabling the identification and prioritization of multiple objects in a single request, complex prompts drastically reduce the total number of API calls needed to complete the sorting task. While a 2–4 s latency does not meet the requirements for high-speed, real-time industrial applications, it is entirely feasible for the intended context of collaborative robotics, where operational speeds are inherently slower and cycle times are less critical.

The system demonstrated high classification accuracy, achieving an overall accuracy of 89.64%, which is higher than most state-of-the-art robotic waste sorting system based on standard deep learning models [2]. Metal, paper, glass, plastic and organic categories were identified with high individual accuracy exceeding 94%. However, the unsorted waste category proved to be the primary source of classification errors, exhibiting an accuracy of only 60% and significantly impacting the overall performance. The confusion matrix further revealed that items belonging to unsorted waste were frequently misclassified as either paper or plastic. This trend was observed in cases where, for instance, used paper tissues were categorized as paper and oiled plastic packages as plastic, despite both fitting typical criteria for unsorted waste. This suggests that the initial, basic prompts for classification might benefit from more specific instructions, explicitly defining waste sorting rules for ambiguous items. Customizing prompts to account for different regional or country-specific sorting guidelines could further enhance the system’s accuracy and practical utility.

The integration of RGB-D data proved to be crucial for robust performance in cluttered environments where objects are often stacked and overlapping. Our experiments with Prompts 4, 5, and 6, which demand spatial awareness, clearly showed that incorporating depth images significantly enhanced the system’s ability to accurately identify the topmost object, establish correct grasping priorities, and distinguish between graspable and non-graspable items, while only slightly increasing the inference time. This confirms depth sensing as an indispensable component for waste sorting in realistic, three-dimensional settings.

The practical application of the VLA model on the KUKA LBR iiwa allowed us to demonstrate diverse real-world grasping strategies designed for different levels of environmental clutter. For non-cluttered scenes, strategy 1, utilizing a single API call with prompt 1 (basic classification), proved efficient for picking multiple separated objects. In cluttered environments, we explored three different approaches. Strategy 2 (prompt 4, iterative single-object picking) prioritized safety in complex stacks but was inefficient, requiring an API call for each waste object. Strategy 3 (prompt 5, single API call with grasping priority) offered greater efficiency but exhibited limitations, leading to potential errors when objects were perfectly overlapping. To address these challenges, strategy 4 (prompt 6, iterative identification of graspable objects) was determined to be the most robust approach. This strategy identifies all the graspable objects in a scene and proceeds to pick all of them before repeating the API call, which is more efficient than strategy 2 and safer than strategy 3. These demonstrations collectively validate the feasibility of integrating VLA-driven reasoning with commercial robotic platforms for complex manipulation tasks in real-world waste sorting.

Despite the promising capabilities demonstrated, our approach presents some limitations that merit consideration for future development. Firstly, the reliance on a cloud-based API, particularly its free-tier call rate limits, requires a paid subscription for robust industrial deployment. Secondly, while the observed inference time of 2–4 s is suitable for collaborative robotic applications, it does not meet hard real-time requirements for high-speed industrial processes or integration with moving conveyors. Thirdly, the classification prompts employed were intentionally generic regarding material types. Integrating more specific, country- or state-specific waste sorting guidelines into prompt engineering could further enhance accuracy for ambiguous items. Lastly, our system currently extracts $(x,y)$ coordinates via the VLA model and z-height from the depth camera. This information is sufficient for our current setup, which utilizes a pneumatic vacuum gripper and is designed to manipulate objects positioned parallel or nearly parallel to the robot’s base. However, for more complex scenarios involving heavily cluttered environments or the use of different, more versatile end-effectors, identifying the full 6-DOF pose of each item (including its orientation around the x, y, and z axes) would be essential. This capability could potentially be achieved either by refining VLA prompts to include specific gripper rotation instructions or by integrating additional computer vision techniques for precise pose estimation. Developing such robust orientation detection and manipulation remains an important area for future investigation, particularly for applications requiring more dexterous manipulation or handling of irregularly shaped items that cannot be grasped with simple vertical suction.

In summary, this work highlights the significant potential of VLA models like Gemini in the field of robotics. We demonstrated that sophisticated robotic waste sorting, including complex object identification, spatial reasoning, and dynamic grasping strategies, can be achieved without prior task-specific training, extensive dataset collection, or deep computer vision expertise. This approach, where a robot leverages large, pre-trained AI for intelligence, substantially lowers the barrier to deploying advanced robotic solutions. Although our focus was on waste sorting, the methodologies and insights gained here suggest that VLAs can be effectively used to empower robots to perform a diverse range of complex manipulation tasks across various industrial and service domains, contributing to the development of more adaptable and intelligent robotic systems.

6. Conclusions

In this paper, we presented a robotic waste sorting system that integrates the Gemini Vision–Language–Action (VLA) model with a KUKA LBR iiwa collaborative robot and an RGB-D camera. Our work successfully demonstrated the feasibility of leveraging large, pre-trained AI models for complex physical manipulation tasks without requiring extensive task-specific training, dataset collection, or computer vision expertise.

Key contributions include the development of effective prompt engineering strategies that enabled advanced spatial reasoning and grasping, moving beyond the limitations of traditional image classification. We quantified the system’s inference time, confirming its suitability for collaborative robotic applications, and achieved high classification accuracy across diverse waste categories, noting areas for improvement in the unsorted waste category. The integration of RGB-D sensing proved indispensable for robust performance in cluttered environments, significantly enhancing the model’s ability to handle object occlusions and determine graspable items. Furthermore, we validated multiple real-world grasping strategies, identifying an iterative approach (strategy 4) that balances efficiency and safety for highly cluttered scenes.

This study represents a significant step forward in robotic autonomy. It establishes a practical and adaptable framework for deploying VLA-driven intelligence on commercial robotic platforms. The principles and methodologies demonstrated here extend beyond waste sorting, highlighting the potential of VLAs to empower robots for a wide range of complex manipulation tasks, thereby contributing to the development of more adaptable and intelligent robotic systems across various industrial and service sectors.