Geometric Point-Cloud Perception and Defect-Aware Grasping Framework for Industrial Workpieces

Abstract

Industrial robots in small-batch manufacturing and human–robot collaborative workstations are increasingly required to perform grasping and sorting tasks on workpieces with natural language instructions. Unlike generic object grasping, industrial workpieces may contain local surface defects such as dents, scratches, and geometric irregularities, which affect both target selection and grasp stability. This paper presents a geometric point-cloud perception and defect-aware grasping framework that leverages 3D geometric cues measured from point clouds to detect surface defects and guide robotic manipulation. Rather than learning an end-to-end visuomotor policy, the proposed framework introduces an explicit geometric reasoning layer between language-level task specification and robotic grasp execution. The language model is used only to convert instructions into structured task attributes, whereas defect perception, target ranking, and grasp-contact evaluation are performed using measurable 3D geometric evidence. These components require no task-specific training on annotated industrial defect or grasping datasets, making the formulation potentially useful for small-batch manufacturing scenarios in which large annotated defect datasets are unavailable. A shared defect-response representation supports both instance-level target selection and contact-region screening. We evaluate the proposed method on a controlled prototype tabletop setup, achieving an average defect instance-level F1-score of 0.89, a target grounding accuracy of 0.89, and a grasp success rate of 84.0%.

1. Introduction

Transforming a natural-language task description into an executable industrial grasp requires a consistent mapping from semantic constraints to measurable scene evidence. The instruction must first be parsed into a structured representation of the action, target category, defect status, spatial relation, and placement region. Candidate workpieces must then be separated in the point cloud and evaluated for local geometric irregularities. Finally, the grounded target must be converted into a collision-free grasp pose. The central challenge is to ground a constraint such as “the defective cube on the left” to one physical instance while retaining enough geometric evidence to avoid unstable contact regions.

Application-specific AI methods provide complementary examples of constraint-aware design in cyber-physical systems [1,2,3,4,5]. Personalized federated learning has been used for intrusion detection under heterogeneous sensor-network data, multi-task learning has supported real-time water-level prediction, deep reinforcement learning has been applied to latency–energy-aware computation offloading in smart healthcare networks, edge computing and AI have been examined jointly in autonomous transportation systems, and the integration of data science with the intelligent Internet of Things has been surveyed in terms of current challenges and future directions. Although these studies address different tasks, they illustrate the broader value of separating domain-specific sensing, inference, and decision modules instead of relying on an undifferentiated end-to-end process.

Existing methods address only fragments of this pipeline. Point-cloud perception methods [6,7,8,9,10] provide scene understanding but typically do not handle language-conditioned target selection. Language-guided robot planning methods [11,12,13,14,15,16,17,18,19] improve action generation or task-level reasoning but usually do not model local geometric defects in industrial point-cloud scenes. Grasping methods [20,21,22,23,24] generate 6-DoF grasp poses but generally assume the target object is already given and do not address task-conditioned selection among multiple workpieces with defect constraints.

This paper presents a geometric point-cloud perception and defect-aware grasping framework that leverages 3D geometric cues to detect surface defects and guide robotic manipulation. The proposed method parses language instructions, detects workpiece surface defects through multi-cue geometric analysis, selects the correct defective target via defect-aware target grounding, and generates stable grasp poses that avoid contact with high defect-response regions.

Although the proposed framework accepts natural-language instructions and generates executable grasp poses, it is not intended as a general-purpose vision–language–action (VLA) policy. Instead, it provides an interpretable geometric reasoning layer that explicitly measures local surface defects and incorporates this evidence into target grounding and grasp-contact evaluation. The geometric perception and grounding components require no task-specific training on annotated industrial defect or grasping datasets, making it potentially useful for small-batch manufacturing scenarios in which large annotated defect datasets are unavailable.

The core idea is to distinguish defective-object selection from defective-region contact. The language instruction requires selecting the defective workpiece as the manipulation target, whereas the grasp planner should avoid using the defective local region as the contact area because dents and scratches may reduce contact stability. The defect response map serves two purposes: target-level defect consistency evaluation and contact-level stability reasoning.

The main contributions of this work are threefold:

• We formulate defect perception as an interpretable multi-scale geometric evidence aggregation problem, producing point-level response maps and instance-level defect scores without task-specific training on annotated industrial defect data. The multi-cue fusion of curvature, normal consistency, plane fitting residual, concavity, and density anomaly provides traceable geometric evidence for each decision.
• We introduce defect-constrained target grounding that binds symbolic attributes parsed from natural-language instructions to measurable 3D geometric evidence, enabling selection among multiple visually or geometrically similar workpieces based on explicit defect-aware criteria.
• We propose a dual-role defect reasoning mechanism in which the same response map supports defective-object selection at the instance level and suppresses unreliable defective contact regions during grasp candidate ranking, thereby separating target-level defect selection from contact-level stability reasoning.

2. Related Work

2.1. Point-Cloud Perception for Industrial Workpieces

Point-cloud learning provides the geometric foundation for robotic scene understanding and action planning. PointNet and PointNet++ established permutation-invariant point-based learning and hierarchical local feature extraction [6,7], while DGCNN, KPConv, and Point Transformer further improved graph-based neighborhood reasoning, deformable convolution, and self-attention-based geometric modeling for point clouds [8,9,10]. In the industrial domain, 3D vision has been applied to workpiece recognition, pose estimation, and bin-picking, with measured point clouds used for instance segmentation, geometric primitive extraction, and surface reconstruction in CAD-deficient or geometry-diverse scenarios [6,7,8,9,10]. These methods provide general point-cloud feature extraction and segmentation capability, but they do not explicitly bind defect evidence to language-conditioned target selection.

2.2. Geometric Defect Detection in Point Clouds

Surface defect detection is critical for quality control in industrial manufacturing. Traditional geometric approaches extract hand-crafted features such as curvature, normal variation, surface fitting residuals, and density irregularity to identify anomalous surface regions. These methods operate by detecting local violations of geometric regularity and do not require large-scale annotated training data. Learning-based defect classification methods have also been explored but typically require annotated defect datasets and may suffer from generalization issues in small-batch industrial scenes with diverse defect patterns. Existing defect detection methods typically operate independently of downstream manipulation tasks and do not leverage language constraints to select the correct defective target among multiple candidates. This work addresses this gap by coupling geometric defect detection with language-guided target grounding.

2.3. Language-Guided Robotic Manipulation

Open-vocabulary vision models such as CLIP, OWL-ViT, SAM, and Grounding DINO [25,26,27,28] strengthen text-conditioned object localization and scene-level grounding, which is highly relevant for associating task language with workspace entities. Language-guided robotic planning has advanced rapidly with large language models and vision–language–action architectures. SayCan combined language reasoning with affordance grounding to select feasible actions from a pre-defined library, Inner Monologue incorporated environmental feedback into iterative task reasoning, and ProgPrompt formulated situated planning through structured program-like prompts generated by large language models [11,12,13]. Large multimodal robot models such as RT-1 and RT-2, VIMA and PaLM-E, VoxPoser, and OpenVLA [14,15,16,17,18,19] demonstrated scalable robot behavior through transformer-based or multimodal prompting architectures trained on large-scale manipulation or embodied reasoning data. These methods strongly advance language-conditioned planning and robot action generation, but they usually do not explicitly model local geometric defect evidence in industrial point-cloud scenes. This work provides a complementary geometric defect detection and grasp-specific target grounding intermediate layer for defect-aware industrial manipulation.

Table 1. Methodological positioning compared with general-purpose vision–language–action models.

Dimension	General-Purpose VLA	Proposed Framework
Objective	Learn broad language-conditioned robot policies	Defect-aware target grounding and grasp-contact reasoning
Main input	RGB/multimodal observations and language	Structured language attributes and measured 3D point clouds
Core mechanism	End-to-end or large-scale learned visuomotor policy	Explicit geometric measurement and constrained ranking
Output	Action tokens, trajectories, or control policies	Defect map, target score, and $SE\left(3\right)$ grasp pose
Training	Large-scale vision–language–action data	No task-specific defect/grasp model training
Interpretability	Decisions are mainly implicit in network representations	Individual geometric cues and score terms are traceable
Application scope	Broad open-ended manipulation	Structured industrial tabletop manipulation
Relationship	High-level planning/control framework	Geometric perception and safety layer

clarifies methodological positioning rather than claiming direct performance superiority over VLA systems. A numerical comparison is not reported because the cited systems use different training resources, action spaces, hardware, and evaluation protocols. The proposed method should therefore be interpreted as a geometric perception and contact-screening layer, not as a replacement for a general-purpose VLA policy.

2.4. Geometry-Aware Robotic Grasping

Grasping has received extensive attention in robotic manipulation research. Dex-Net 2.0, PointNetGPD, GraspNet-1Billion, Contact-GraspNet, and AnyGrasp [20,21,22,23,24] substantially improved large-scale grasp perception and real-scene robustness through synthetic data supervision, point-set evaluation, and large-scale dataset training. CFPGrasp simplified 6-DoF grasp prediction through a contact-rotation representation and a coarse-to-fine strategy [29]. Geometry-aware 6-DoF grasp learning methods further decoupled graspability perception from pose estimation and explicitly leveraged structural edges and local geometric primitives for explainable 6-DoF grasp generation in cluttered scenes [30]. Nevertheless, these methods remain single-action grasp pipelines: they optimize grasp pose prediction from visual or point-cloud observations but assume the target object is already given and do not address task-conditioned target selection across multiple candidate workpieces with defect constraints. In contrast, this work introduces defect-aware target grounding as a task-conditional multi-criteria ranking problem that operates prior to grasp pose generation.

The novelty therefore does not lie in any individual curvature, normal-variation, or plane-residual descriptor. It lies in coupling interpretable multi-cue geometric evidence with language-conditioned instance grounding and reusing the same evidence to screen grasp contacts. This positioning distinguishes the proposed framework from both stand-alone geometric defect detectors and grasp-pose estimators that assume a known target.

3. Method

Scope and Positioning. This work is a prototype feasibility study rather than a general-purpose VLA model or a large-scale robotic manipulation system. The validated setting is a structured industrial tabletop scene in which simple workpieces (cone, cube, cylinder, and rectangular prism) are spatially separated on a flat support surface and observed by a fixed RGB-D camera. The proposed framework uses 3D geometric analysis for defect-aware target grounding and grasp evaluation without task-specific training of the geometric modules. Claims of effectiveness are therefore restricted to this controlled setting.

3.1. Problem Formulation

Given a natural language instruction $\mathcal{I}$, an RGB-D observation ${\mathbf{X}}^{\mathrm{rgb}}$, and a 3D point cloud $\mathcal{P}=\{{\mathbf{p}}_i\in {\mathbb{R}}^3\}$, the goal is to select a target workpiece instance $o^{*}\in \mathcal{O}$ and generate a stable grasp pose ${\mathbf{T}}_g^{*}\in SE\left(3\right)$ for pick-and-place manipulation. The central challenge is a three-stage mapping: from language to structured grasp specification, from specification to target instance, and from target instance to executable grasp parameters. We address three core sub-problems: parsing natural language into structured task representations, detecting workpiece surface defects from local geometric regularity violations, and grounding the task specification to the correct defective target under joint semantic, geometric, and grasp-specific constraints.

Figure 1 shows the detailed pipeline from task parsing to defect-aware target grounding and grasp pose generation. The system first parses the language instruction into a structured grasp specification and then extracts geometric features from the point cloud to detect defective workpieces, grounds the target under multi-criteria evaluation, and finally generates executable parallel-gripper grasp poses.

3.2. Task Parsing and Structured Grasp Specification

A natural language instruction $\mathcal{I}$ is parsed by a large language model into a structured grasp specification:

$$\mathcal{T}=f_{\mathrm{LLM}}\left(\mathcal{I}\right)=\langle a,c,d,r_o,r_p,\mathcal{C}\rangle ,$$

(1)

where a is the action type (e.g., pick-and-place), c is the target object category (e.g., cube, cylinder, cone, or rectangular prism), d is the defect constraint (e.g., defective or non-defective), $r_o$ is the target object location description (e.g., left, center, or right), $r_p$ is the placement region (e.g., bin, left-side, or right-side), and $\mathcal{C}$ is the execution constraints (e.g., stable grasp; avoid collision).

The LLM is used only as a task parser rather than as a direct robot controller. Its output is converted into a structured task specification and verified by rule-based field checking before being passed to the geometric grounding module.

For example, the instruction “Pick up the defective cylinder on the left and place it in the bin” is parsed into the following:

$$\begin{array}{cc}\hfill \mathcal{T}=& \langle \mathtt{pick}\_\mathtt{and}\_\mathtt{place},\mathtt{cylinder},\mathtt{defective},\hfill \\ \hfill & \mathtt{left},\mathtt{bin},\{\mathtt{stable}\_\mathtt{grasp},\mathtt{collision}\_\mathtt{free}\}\rangle .\hfill \end{array}$$

(2)

This structured representation bridges natural language ambiguity and geometric reasoning by extracting task-relevant attributes that guide downstream perception and planning.

We use Qwen2.5-7B-Instruct as the language model for task parsing. The model is deployed locally via vLLM with temperature set to 0.0 for deterministic parsing. The system prompt specifies the JSON schema for the structured task representation, and field validation is performed by rule-based checking before downstream processing.

Table 2. LLM task parsing accuracy on 60 language commands.

Parsing Field	Accuracy
Action type	60/60 (1.00)
Object category	60/60 (1.00)
Defect attribute	58/60 (0.97)
Spatial relation	60/60 (1.00)
Placement region	60/60 (1.00)
Complete structured command	57/60 (0.95)

reports the per-field parsing accuracy on the 60 language commands.

3.3. Point-Cloud Scene Representation and Instance Grounding

The proposed system directly performs 3D point-cloud instance segmentation to extract individual workpiece instances from the scene. The scene is represented as two layers. The object instance layer $\mathcal{O}=\{o_1,\dots ,o_M\}$ is obtained through 3D point-cloud preprocessing: voxel downsampling, support-plane removal, Euclidean clustering, and geometry-based instance segmentation. The RGB image is used only as auxiliary observation or for visualization, while target grounding and defect reasoning are conducted in the 3D point-cloud domain. The geometric entity layer $\mathcal{G}=\{g_1,\dots ,g_N\}$ is constructed from $\mathcal{P}$ and spans types such as hole-like-defect, depression, planar-face, edge, and object-body. Entity generation is based on local covariance analysis, providing the geometric descriptors that underpin both defect reasoning and grasp pose generation:

$$\begin{array}{cc}\hfill {\Sigma }_i^{\left(r\right)}=& {\displaystyle \frac{1}{|{\mathcal{N}}_r\left({\mathbf{p}}_i\right)|}}\sum _{{\mathbf{p}}_k\in {\mathcal{N}}_r\left({\mathbf{p}}_i\right)}({\mathbf{p}}_k-{\overline{\mathbf{p}}}_i){({\mathbf{p}}_k-{\overline{\mathbf{p}}}_i)}^{\top },\hfill \\ \hfill {\overline{\mathbf{p}}}_i=& {\displaystyle \frac{1}{|{\mathcal{N}}_r\left({\mathbf{p}}_i\right)|}}\sum _{{\mathbf{p}}_k\in {\mathcal{N}}_r\left({\mathbf{p}}_i\right)}{\mathbf{p}}_k.\hfill \end{array}$$

(3)

The eigenvalues ${\lambda }_{i,1}^{\left(r\right)}\ge {\lambda }_{i,2}^{\left(r\right)}\ge {\lambda }_{i,3}^{\left(r\right)}$ are computed by eigendecomposition of ${\Sigma }_i^{\left(r\right)}$, and the geometric feature vector is as follows:

$$\begin{array}{c}\hfill {\mathbf{z}}_i^{\left(r\right)}=\left[{\widehat{\kappa }}_i^{\left(r\right)},1-{\widehat{\eta }}_i^{\left(r\right)},{\widehat{ϵ}}_i^{\left(r\right)},{\widehat{\xi }}_i^{-,\left(r\right)},{\widehat{\rho }}_i^{\left(r\right)}\right],\end{array}$$

(4)

where ${\widehat{\kappa }}_i^{\left(r\right)}$ is the curvature anomaly, ${\widehat{\eta }}_i^{\left(r\right)}$ is the normal consistency, ${\widehat{ϵ}}_i^{\left(r\right)}$ is the plane fitting residual, ${\widehat{\xi }}_i^{-,\left(r\right)}$ is the concavity response, and ${\widehat{\rho }}_i^{\left(r\right)}$ is the density anomaly. The planarity and linearity descriptors ($q_{\mathrm{pla}}$ and $q_{\mathrm{lin}}$) are derived from the same eigenvalue ratios and are used exclusively for candidate grasp-region generation, not for defect scoring. Candidate grasp regions are generated by thresholding and region growing over planarity and linearity responses, followed by task-dependent geometric verification.

Table 3. Consolidated notation for geometric defect reasoning and grasp generation.

Symbol	Definition	Range	Primary Use
${\mathbf{p}}_i,{\mathcal{P}}_j$	Point i and point set of instance j	${\mathbb{R}}^3$	Local geometry and instance aggregation
${\mathcal{N}}_r\left({\mathbf{p}}_i\right)$	Radius-r neighborhood of point i	point set	Multi-scale covariance estimation
${\Sigma }_i^{\left(r\right)}$	Local covariance matrix	positive semidefinite	Eigenvalue-based descriptors
${\widehat{\kappa }}_i^{\left(r\right)}$	Normalized curvature anomaly	$[0,1]$	Defect cue
${\widehat{\eta }}_i^{\left(r\right)}$	Normalized normal consistency	$[0,1]$	Defect cue through $1-\widehat{\eta }$
${\widehat{ϵ}}_i^{\left(r\right)}$	Normalized plane-fitting residual	$[0,1]$	Defect cue
${\widehat{\xi }}_i^{-,\left(r\right)}$	Normalized concavity response	$[0,1]$	Defect cue
${\widehat{\rho }}_i^{\left(r\right)}$	Normalized density anomaly	$[0,1]$	Auxiliary defect cue
${\Phi }_i$	Fused point-level defect response	$[0,1]$	Localization and contact screening
$S_j^{\mathrm{def}}$	Instance-level defect score	$[0,1]$	Defective-object decision
$q_{\mathrm{pla}},q_{\mathrm{lin}}$	Planarity and linearity descriptors	$[0,1]$	Grasp-region generation only
$\Gamma (o_k\mid \mathcal{T})$	Task-conditioned grounding score	non-negative	Target ranking
$Q({\mathbf{T}}_g,o^{*})$	Overall grasp quality	normalized score	Grasp-pose ranking

consolidates the symbols used for geometric defect reasoning and grasp generation.

Figure 2 illustrates the local geometric defect detection process. Multi-scale neighborhoods are constructed around each query point, followed by local PCA-based reference frame estimation, geometric cue extraction, multi-cue fusion, point-level defect response generation, and instance-level defect scoring.

3.4. Multi-Cue Geometric Defect Detection

Industrial surface defects are formulated as violations of local geometric regularity. As shown in Figure 2, for each point ${\mathbf{p}}_i$ of object $o_j$, five local geometric response signals are computed across multiple neighborhood radii $r\in \mathcal{R}=\{r_1,r_2,r_3\}$ and combined into a unified defect score.

Definition of geometric cues. The five geometric cues are computed from local covariance analysis as follows:

Curvature anomaly ${\kappa }_i^{\left(r\right)}$: the local sphericity derived from the normalized eigenvalue ratio ${\lambda }_{i,3}^{\left(r\right)}/({\lambda }_{i,1}^{\left(r\right)}+{\lambda }_{i,2}^{\left(r\right)}+{\lambda }_{i,3}^{\left(r\right)})$, indicating local surface curvature. Higher values suggest curved or non-planar regions.

Normal consistency ${\eta }_i^{\left(r\right)}$: computed as the average dot product between the query-point normal ${\mathbf{n}}_i$ and neighbor point normals ${\mathbf{n}}_k$ within radius r. To ensure consistent normal orientation, the query-point normal is first oriented toward the camera center (which lies at a known fixed position above the scene), and all neighbor normals are then re-oriented to have a positive dot product with the query-point normal before averaging. Lower values indicate inconsistent normals typical of scratches, cracks, or defect boundaries.

The normal-consistency cue is invariant to a simultaneous sign reversal of the query and neighboring normals because it depends on their dot products. The signed concavity cue is not sign invariant; consequently, the camera-facing convention is applied before signed residuals are computed. This convention is reliable in the fixed overhead setup but has not been validated for moving viewpoints, open surfaces viewed from both sides, or severe occlusion.

Plane fitting residual ${ϵ}_i^{\left(r\right)}$: the mean orthogonal distance from neighbor points to the locally fitted plane, capturing local surface smoothness.

Concavity response ${\xi }_i^{-,\left(r\right)}$: computed from signed residuals ${\delta }_k^{\left(r\right)}={\mathbf{n}}_i^{\top }({\mathbf{p}}_k-{\overline{\mathbf{p}}}_i)$, where ${\xi }_i^{-,\left(r\right)}={\displaystyle \frac{1}{|{\mathcal{N}}_r\left({\mathbf{p}}_i\right)|}}{\sum }_{{\mathbf{p}}_k}max(0,-{\delta }_k^{\left(r\right)})$. This detects local concavities such as dents, holes, and depressions.

Density anomaly ${\rho }_i^{\left(r\right)}$: the deviation of local point density from the median density within the object, serving as an auxiliary cue for detecting sparse or dense surface regions. Note that point-cloud density variation may also arise from sensor sampling, viewing angle, or edge effects, so this cue should be interpreted in conjunction with other geometric cues.

All cue values are normalized to $[0,1]$ within each object instance using robust min–max normalization based on the 5th and 95th percentiles.

Point-level defect response. The point-level defect response aggregates geometric cues across spatial scales:

$${\Phi }_i=\underset{r\in \mathcal{R}}{max}\left({\alpha }_1{\widehat{\kappa }}_i^{\left(r\right)}+{\alpha }_2\left(1-{\widehat{\eta }}_i^{\left(r\right)}\right)+{\alpha }_3{\widehat{ϵ}}_i^{\left(r\right)}+{\alpha }_4{\widehat{\xi }}_i^{-,\left(r\right)}+{\alpha }_5{\widehat{\rho }}_i^{\left(r\right)}\right).$$

(5)

The cue weights ${\alpha }_1,\dots ,{\alpha }_5$ are set uniformly (${\alpha }_m=1/5$) by default. Uniform weights are not assumed to be optimal; they are adopted to avoid fitting defect-specific weights on the limited validation set and to preserve a defect-type-independent formulation. The multi-scale max aggregation naturally selects the most discriminative scale for each point. The temperature parameter $\beta$ and defect threshold ${\theta }_d$ are selected on a held-out validation set consisting of 4 instances per geometry type (2 defective; 2 non-defective), stratified by defect status. This validation set is strictly separated from the test set, and parameters are fixed before evaluating on the test set to avoid data leakage.

Instance-level defect scoring. The instance-level defect score aggregates point-level responses over the object point set ${\mathcal{P}}_j$:

$$\begin{array}{cc}\hfill S_j^{\mathrm{def}}& ={\displaystyle \frac{{\sum }_{{\mathbf{p}}_i\in {\mathcal{P}}_j}exp\left(\beta {\Phi }_i\right){\Phi }_i}{{\sum }_{{\mathbf{p}}_i\in {\mathcal{P}}_j}exp\left(\beta {\Phi }_i\right)}},\hfill \\ \hfill d_j& =\mathbb{I}\left(S_j^{\mathrm{def}}>{\theta }_d\right).\hfill \end{array}$$

(6)

The softmax-weighted mean emphasizes high-response regions. The dominant response pattern provides an interpretable attribution of the prevailing geometric abnormality. An object $o_j$ is classified as defective ($d_j=1$) if its defect score exceeds the threshold ${\theta }_d$.

As shown in Figure 3, the input scene point cloud is first preprocessed and segmented into multiple workpiece instances. For each instance, the local geometric defect detector is applied to estimate the defect response and confidence score. The detected defect information is then combined with the structured task specification to determine the target object.

3.5. Defect-Constrained Target Grounding

From the full object instance set $\mathcal{O}$, a task-relevant subset ${\mathcal{O}}_{\mathcal{T}}=\{o_k\in \mathcal{O}\mid \mathrm{type}\left(o_k\right)\sim c\}$ is selected based on the category constraint in $\mathcal{T}$. Each candidate is scored by multiple factors and ranked by a grouped multiplicative objective:

$$\begin{array}{cc}\hfill \Gamma (o_k\mid \mathcal{T})=& {\left[s_{\mathrm{cls}}(o_k,c)s_{\mathrm{loc}}(o_k,r_o)\right]}^{{\lambda }_s}\hfill \\ \hfill & \times {\left[s_{\mathrm{def}}(o_k,d,S_k^{\mathrm{def}})s_{\mathrm{geom}}\left(o_k\right)\right]}^{{\lambda }_g}\hfill \\ \hfill & \times {\left[s_{\mathrm{reach}}\left(o_k\right)s_{\mathrm{safe}}\left(o_k\right)\right]}^{{\lambda }_e}.\hfill \end{array}$$

(7)

$$o^{*}=arg\underset{o_k\in {\mathcal{O}}_{\mathcal{T}}}{max}\Gamma (o_k\mid \mathcal{T}).$$

(8)

The multiplicative formulation has two important properties. First, it penalizes candidates that fail to satisfy any critical task constraint: if any score term is near zero, the overall score becomes near zero regardless of other terms. Second, in implementation, each score term is clipped to a small positive range to avoid numerical instability.

Here, each scoring term is defined as follows:

Category similarity $s_{\mathrm{cls}}(o_k,c)$: computed based on the geometric primitive matching between the detected shape of $o_k$ and the target category c. For each candidate, we extract dominant geometric primitives (planar, cylindrical, conical, and spherical) from the point cloud and compute a similarity score $s_{\mathrm{cls}}\in [0,1]$ based on primitive type matching and geometric parameter deviation.

Location satisfaction $s_{\mathrm{loc}}(o_k,r_o)$: the spatial location score is computed by projecting the object centroid onto the horizontal plane and comparing its relative position (left, center, right, front, and back) with the language-specified location $r_o$. Specifically, $s_{\mathrm{loc}}=1-d_{\mathrm{rel}}(o_k,r_o)/d_{\max }$, where $d_{\mathrm{rel}}$ is the relative distance to the target region, and $d_{\max }$ is the maximum scene extent.

Defect consistency $s_{\mathrm{def}}(o_k,d,S_k^{\mathrm{def}})$: defined in Equation (10) below.

Geometric graspability $s_{\mathrm{geom}}\left(o_k\right)$: computed from local planarity and clearance metrics. For a candidate grasp region, $s_{\mathrm{geom}}$ evaluates the planarity score (based on eigenvalue ratios) and the minimum clearance distance to neighboring objects or the table surface. Higher planarity and larger clearance yield higher graspability scores.

Reachability $s_{\mathrm{reach}}\left(o_k\right)$: collision-free inverse kinematics reachability is evaluated by attempting IK solutions for the candidate grasp pose. The reachability score is the ratio of successful IK solutions to total attempts, capped at a threshold that ensures collision-free paths.

Safety $s_{\mathrm{safe}}\left(o_k\right)$: the safety score evaluates collision-free grasp execution by checking the minimum distance between the gripper trajectory and obstacles. It is computed as follows:

$$s_{\mathrm{safe}}\left(o_k\right)=\left\{\begin{array}{cc}0,\hfill & d_{min}<0,\hfill \\ d_{min}/\delta ,\hfill & 0\le d_{min}<\delta ,\hfill \\ 1,\hfill & d_{min}\ge \delta ,\hfill \end{array}\right.$$

(9)

where $d_{min}$ is the minimum clearance during execution, and $\delta$ is a safety margin threshold.

All scores are normalized to $[0,1]$, with 1 indicating perfect satisfaction. The placement region $r_p$ is reserved for the placement planning stage and does not participate in target grounding. The group weights $({\lambda }_s,{\lambda }_g,{\lambda }_e)$ are task-dependent and selected on a validation set.

The defect constraint term $s_{\mathrm{def}}(o_k,d,S_k^{\mathrm{def}})$ is defined as follows:

$$s_{\mathrm{def}}(o_k,d,S_k^{\mathrm{def}})=\left\{\begin{array}{cc}{S}_k^{\mathrm{def}},\hfill & \mathrm{if}d=\mathtt{defective},\hfill \\ 1-S_k^{\mathrm{def}},\hfill & \mathrm{if}d=\mathtt{non}-\mathtt{defective},\hfill \\ 1,\hfill & \mathrm{if}d=\mathtt{any}.\hfill \end{array}\right.$$

(10)

This term directly uses the instance-level defect score $S_k^{\mathrm{def}}$ computed from the multi-cue geometric defect scoring. If the top-ranked candidate fails to satisfy the grasp feasibility check, the system triggers re-grounding with relaxed constraints.

3.6. Geometry-Guided Grasp Pose Generation

Figure 4 shows the geometric parameterization of the parallel gripper pose. The grasp pose is represented by the grasp center ${\mathbf{p}}_c$, the approaching direction ${\mathbf{v}}_a$, the closing direction ${\mathbf{v}}_c$, and the grasp width $w_g$, as defined in the gripper coordinate frame.

The defect-free contact term does not reject the defective workpiece; instead, it penalizes grasp candidates whose contact regions overlap with high defect-response areas.

Given the grounded target $o^{*}$, executable grasp parameters are generated by a geometry-guided parameterizer:

$${\Pi }^{*}=H_g(o^{*},\mathcal{T},\mathcal{C}).$$

(11)

The grasp center is computed as the centroid of the candidate grasp region:

$${\mathbf{p}}_c={\displaystyle \frac{1}{|{\mathcal{P}}_{o^{*}}^g|}}\sum _{{\mathbf{p}}_i\in {\mathcal{P}}_{o^{*}}^g}{\mathbf{p}}_i,$$

(12)

where ${\mathcal{P}}_{o^{*}}^g\subset {\mathcal{P}}_{o^{*}}$ is the candidate grasp region point set on $o^{*}$.

The approaching direction is set as the negative of the local surface normal:

$${\mathbf{v}}_a=-{\mathbf{n}}_{o^{*}}.$$

(13)

The closing direction is selected to maximize grasp stability:

$${\mathbf{v}}_c=arg\underset{\mathbf{u}\in \{{\mathbf{e}}_1,{\mathbf{e}}_2\}}{max}w(o^{*},\mathbf{u}),$$

(14)

where ${\mathbf{e}}_1$ and ${\mathbf{e}}_2$ are the principal directions of the local point cloud, and $w(o^{*},\mathbf{u})$ measures the grasp width and clearance along direction $\mathbf{u}$.

Although the defect constraint is used to select the target workpiece, the defective region itself is not necessarily suitable for contact. Dents and scratches may cause unstable contact or slippage. Therefore, after the defective target is grounded, the defect response map is reused to suppress grasp candidates whose contact regions overlap with high-response defective areas.

The grasp quality score combines multiple criteria including defect avoidance:

$$Q_{\mathrm{def}-\mathrm{free}}({\mathbf{T}}_g,o^{*})=1-{\displaystyle \frac{1}{|{\mathcal{P}}_{o^{*}}^g|}}\sum _{{\mathbf{p}}_i\in {\mathcal{P}}_{o^{*}}^g}{\Phi }_i,$$

(15)

which penalizes grasp candidates whose contact regions contain high defect-response points. The overall quality score is as follows:

$$\begin{array}{cc}\hfill Q({\mathbf{T}}_g,o^{*})=& {\mu }_1{Q}_{\mathrm{align}}+{\mu }_2{Q}_{\mathrm{clear}}+{\mu }_3{Q}_{\mathrm{stable}}\hfill \\ \hfill & +{\mu }_4{Q}_{\mathrm{safe}}+{\mu }_5{Q}_{\mathrm{def}-\mathrm{free}}.\hfill \end{array}$$

(16)

$${\mathbf{T}}_g^{*}=arg\underset{{\mathbf{T}}_g\in \mathcal{G}\left(o^{*}\right)}{max}Q({\mathbf{T}}_g,o^{*}).$$

(17)

where $Q_{\mathrm{align}}$ measures alignment with the surface normal, $Q_{\mathrm{clear}}$ measures grasp clearance, $Q_{\mathrm{stable}}$ measures pose stability, $Q_{\mathrm{safe}}$ measures collision-free safety, and $Q_{\mathrm{def}-\mathrm{free}}$ penalizes grasp candidates whose contact regions overlap with high defect-response areas. The final gripper pose matrix is as follows:

$${\mathbf{R}}_g=[{\mathbf{v}}_c,{\mathbf{v}}_a\times {\mathbf{v}}_c,{\mathbf{v}}_a].$$

(18)

$${\mathbf{T}}_g=\left[\begin{array}{cc}{\mathbf{R}}_g& {\mathbf{p}}_c\\ {\mathbf{0}}^{\top }& 1\end{array}\right]\in SE\left(3\right).$$

(19)

3.7. Execution Feedback and Recovery

The system includes a two-level recovery mechanism that handles execution failures using built-in gripper force feedback and visual tracking. The measured force $F_{\mathrm{actual}}$ is read from the built-in force-feedback interface of the gripper. The force threshold $F_{max}$ is set to 15 N according to the gripper specifications, and the deviation threshold $d_{max}$ is set to 10 mm based on the visual tracking accuracy. The maximum number of attempts $N_{max}$ is set to 3.

Level-1 (Re-ranking): If the force $F_{\mathrm{actual}}$ reported by the built-in gripper feedback exceeds $F_{max}$ or the visual tracking deviation $d_{\mathrm{actual}}$ exceeds $d_{max}$, the system re-ranks remaining grasp candidates by re-evaluating $Q({\mathbf{T}}_g,o^{*})$ for all ${\mathbf{T}}_g\in {\mathcal{G}}_{\mathrm{cand}}\setminus \left\{{\mathbf{T}}_g^{(n-1)}\right\}$ and selects the highest-scoring candidate.

Level-2 (Re-targeting): If all grasp candidates fail or no valid candidates remain, the system relaxes the task specification. The re-targeting step preserves the object category and defect-status constraints: only lower-priority spatial preferences or grasp-clearance margins may be relaxed. Specifically, the target may be re-grounded to a different candidate instance of the same category and defect status, or the spatial relation constraint (e.g., left/center location tolerance) may be widened. If no candidate satisfies the original semantic constraints, the system returns a failure signal or requests user confirmation rather than automatically relaxing the defect constraint to any.

If both recovery levels fail after $N_{max}$ attempts, the system returns a failure signal. This two-level approach handles perception noise and minor geometric deviations efficiently at Level-1, while Level-2 addresses systematic failures that require task-level modifications.

4. Experimental Results

The experimental platform consists of a fixed RGB-D camera (Vizum VZ-AI-1920(2048)-320M3K) and a 6-DoF industrial robotic arm (Hikvision robot) equipped with a parallel gripper. The point cloud is captured at 30 fps with approximately 307,200 points per frame. The camera is mounted at a fixed overhead viewpoint, approximately 0.8 m above the table surface, with a tilt angle of approximately 30 degrees from vertical. Point-cloud preprocessing includes voxel downsampling (grid size 5 mm), support-plane removal (RANSAC with distance threshold 10 mm), and Euclidean clustering (cluster tolerance 8 mm; minimum 200 points per cluster). Four representative workpiece geometries are considered: cone (height 60 mm; base radius 30 mm), cube (50 mm side), cylinder (height 70 mm; radius 25 mm), and rectangular prism (60 mm × 40 mm × 30 mm). The defect types include hole-like depressions (diameter 8–15 mm; depth 2–5 mm) and scratch-like surface anomalies (length 10–30 mm; width 1–3 mm). Defects are created by manual indentation or surface scratching to simulate real industrial surface anomalies.

Experimental assumptions. The current implementation assumes that workpieces are spatially separated by at least 30 mm after support-plane removal to ensure reliable Euclidean clustering and defect scoring. When two workpieces touch or overlap in depth, Euclidean clustering can merge them into one instance, invalidating both the instance-level defect score and the downstream reachability term. The system therefore rejects scenes that violate the separation condition rather than reporting a grasp result. Touching and stacked workpieces have not been experimentally evaluated.

Ground truth annotation. Defect ground truth is established at the instance level by two experienced operators through independent visual inspection under controlled lighting. A workpiece is labeled as defective when both operators agree; disagreements, which occur in fewer than 5% of cases, are resolved by a third senior operator. The first operator outlines the visible defect region, and the second verifies the outline. These region annotations support qualitative inspection only; the quantitative evaluation is restricted to instance-level defect classification because point-level localization accuracy is not evaluated. Representative defect dimensions are measured with a digital caliper with 0.01 mm resolution.

Experimental Protocol. Experiments on the prototype industrial workpiece setup evaluate three aspects: geometric defect detection, defect-aware target grounding, and robotic grasp execution. All trials are conducted under normal indoor lighting conditions (approximately 300–500 lux).

The physical dataset contains 80 workpiece instances in total. A held-out subset of 16 instances (four per geometry) is used only for parameter selection, and the remaining 64 instances form the defect-detection evaluation pool.

The defect F1-score is computed at the instance level by comparing detected defect status with manually annotated ground truth. For target grounding, 60 language commands cover combinations of object category, defect constraint, and spatial location. The commands are categorized into four types: category-only (e.g., “pick the cube”), category plus location (e.g., “pick the cube on the left”), category plus defect (e.g., “pick the defective cube”), and category plus defect plus location (e.g., “pick the defective cylinder on the left”). Grounding accuracy is measured as Top-1 (correct target ranked first) and Top-3 (correct target in top three). For grasp execution, 120 trials are conducted across all workpiece types; each trial includes approach, grasp, lift, transport, and place phases. A trial is successful only if all phases complete without collision or slip. Grasp error and placement error are measured as Euclidean distances between planned and actual positions, estimated from visual tracking. All reported ± values denote the standard deviation computed across five repeated experimental runs with randomized workpiece placement.

Baseline definitions. The evaluated baselines include both ablated variants of the proposed method and classical geometric anomaly detection methods. Curvature Threshold: a classical approach that classifies a point as defective if its curvature anomaly exceeds a threshold tuned on the validation set. Normal Variation: classifies based on the variance of local point normals within the neighborhood. Plane Fitting Residual: classifies based on the mean orthogonal distance to a locally fitted plane. Two-dimensional Mask Projection: a weak baseline that first obtains 2D object masks from RGB images using simple color-based segmentation and then projects the masks into the point cloud for target association. It does not directly perform 3D instance segmentation or local geometric defect scoring. Three-dimensional Instance Segmentation Only: point-cloud Euclidean clustering is used for object identification, but defect labels are inferred from simple instance-level geometric statistics (e.g., bounding box irregularity) without the proposed multi-cue defect scoring. Language-only Grounding: uses the parsed symbolic attributes and coarse spatial relations to rank candidate instances, but it does not use point-cloud defect evidence or geometric grasp-quality scores. Single-cue Geometry: only a single geometric feature (e.g., curvature anomaly alone) is used for defect detection without multi-cue fusion. Multi-cue Mean Fusion: uses the proposed five geometric cues but performs simple point-level averaging without multi-scale aggregation or softmax-weighted instance-level scoring; the defect score is thresholded directly on the averaged point-level response. Grounding w/o Language: uses multi-cue defect scoring for target selection but without language-conditioned target grounding; targets are selected based on defect score alone. Grasp w/o Defect-Free Contact: uses the same defect-aware target grounding as the full method but removes the defect-free contact term ($Q_{\mathrm{def}-\mathrm{free}}$) during grasp candidate ranking. w/o Recovery: uses the proposed full framework but disables the feedback-based re-ranking or re-targeting mechanism.

The validation, evaluation, command, and execution partitions are summarized in

Table 4. Dataset partition for the physical workpiece experiments.

Split	Description	Instances	Used for
Validation	4 instances per geometry type	16	Proposed-method hyperparameter tuning ($\beta$, ${\theta }_d$, ${\lambda }_s$, ${\lambda }_g$, ${\lambda }_e$, ${\mu }_1,\dots ,{\mu }_5$)
Evaluation pool	Remaining physical instances	64	Proposed method evaluation; PointNet++ out-of-fold evaluation
Total physical instances	All workpieces across geometries	80	—
Language commands	60 structured instructions	60	Target grounding evaluation
Grasp trials	120 physical execution attempts	120	Grasp success rate, error metrics

.

PointNet++ baseline protocol. The PointNet++ baseline [7] is evaluated through stratified five-fold cross-validation on the 64-instance evaluation pool. In each fold, approximately 51 instances are used for training and 13 for testing. Within each outer training fold, 10% of the training instances are reserved as an inner validation subset for early stopping; the outer test fold is not used for model selection. Each instance receives one out-of-fold prediction, and the pooled predictions over the five held-out folds are used to compute the final F1-score. Class weighting is applied to handle slight class imbalance. The model uses 1024 points per object and is trained for up to 200 epochs with the Adam optimizer, an initial learning rate of ${10}^{-3}$ with cosine decay, dropout of 0.5, and early stopping on the inner-validation loss. The random seed is fixed to 42. The reported PointNet++ deviation reflects variation across the five held-out folds, whereas deviations for the geometric methods reflect five repeated runs with randomized workpiece placement; these deviations therefore quantify different sources of variability and are not interpreted as directly equivalent uncertainty estimates.

The final values, validation ranges, and selection criteria for the principal hyperparameters are listed in

Table 5. Hyperparameters and their final values.

Parameter	Final Value	Search Range	Selection Criterion
Neighborhood radii $(r_1,r_2,r_3)$	(5, 10, 20) mm	fixed	empirical
Temperature $\beta$	3.0	$\{1.0,2.0,3.0,4.0,5.0\}$	validation F1
Defect threshold ${\theta }_d$	0.45	$[0.30,0.60]$ step 0.05	validation F1
Cue weights ${\alpha }_1,\dots ,{\alpha }_5$	$1/5$ each	fixed	equal weighting (no validation fitting)
Group weight ${\lambda }_s$	1.5	$\{0.5,1.0,1.5,2.0\}$	grounding accuracy
Group weight ${\lambda }_g$	2.0	$\{0.5,1.0,1.5,2.0\}$	grounding accuracy
Group weight ${\lambda }_e$	1.0	$\{0.5,1.0,1.5,2.0\}$	grounding accuracy
Grasp weight ${\mu }_1$ (alignment)	0.25	fixed	equal weighting
Grasp weight ${\mu }_2$ (clearance)	0.25	fixed	equal weighting
Grasp weight ${\mu }_3$ (stability)	0.25	fixed	equal weighting
Grasp weight ${\mu }_4$ (safety)	0.15	$\{0.10,0.15,0.20\}$	grasp success
Grasp weight ${\mu }_5$ (defect-free)	0.10	$\{0.05,0.10,0.15\}$	grasp success
Safety margin $\delta$	15 mm	$\{10,15,20\}$ mm	collision-free success
Force threshold $F_{max}$	15 N	gripper specification	fixed
Deviation threshold $d_{max}$	10 mm	tracking accuracy	fixed
Max attempts $N_{max}$	3	$\{2,3,5\}$	empirical

.

Sensitivity analysis. The defect threshold ${\theta }_d$ and temperature $\beta$ have distinct roles. Increasing ${\theta }_d$ improves precision but reduces recall, whereas $\beta$ controls how strongly high-response points influence the instance score. Equal cue weights are retained to avoid fitting defect-specific weights on only 16 validation instances. The present study reports a quantitative sweep only for ${\theta }_d$; it does not provide sufficient evidence to claim robustness to all combinations of $\beta$, group weights, or grasp-quality weights. Those parameters are therefore reported for reproducibility, and broader sensitivity analysis remains necessary before generalizing beyond the controlled setup.

Post hoc sensitivity analysis. 

Table 6. Post hoc sensitivity analysis of the preselected defect threshold ${\theta }_d$ on the 64-instance evaluation pool.

${\mathit{\theta }}_{\mathit{d}}$	Precision	Recall	F1
0.35	0.79	0.96	0.87
0.40	0.83	0.93	0.88
0.45	0.87	0.91	0.89
0.50	0.90	0.87	0.88
0.55	0.93	0.82	0.87

presents a post hoc sensitivity analysis of ${\theta }_d$ on the 64-instance evaluation pool. These results were not used for parameter selection; ${\theta }_d=0.45$ had already been fixed using the independent 16-instance validation set before evaluation on the 64-instance pool. The table shows the local variation of precision and recall around the preselected threshold and should not be interpreted as test-set tuning or as evidence that the threshold is optimal on unseen data.

4.1. Defect Detection Evaluation

The defect detection results on the evaluated workpiece samples are reported in

Table 7. Defect detection performance on different workpieces.

Method	Cone	Cube	Cylinder	Rect. Prism	Avg.
Curvature Threshold	0.62 ± 0.05	0.68 ± 0.05	0.55 ± 0.06	0.51 ± 0.06	0.59 ± 0.05
Normal Variation	0.71 ± 0.04	0.74 ± 0.04	0.64 ± 0.05	0.60 ± 0.05	0.67 ± 0.04
Plane Fitting Residual	0.65 ± 0.05	0.70 ± 0.05	0.58 ± 0.06	0.54 ± 0.06	0.62 ± 0.05
2D Mask Projection	0.68 ± 0.05	0.72 ± 0.04	0.61 ± 0.05	0.58 ± 0.05	0.65 ± 0.04
3D Inst. Seg. Only	0.81 ± 0.04	0.85 ± 0.03	0.76 ± 0.04	0.73 ± 0.04	0.79 ± 0.03
PointNet++ [7]	0.86 ± 0.04	0.88 ± 0.03	0.80 ± 0.04	0.77 ± 0.04	0.83 ± 0.03
Single-cue Geometry	0.73 ± 0.05	0.76 ± 0.04	0.68 ± 0.05	0.65 ± 0.05	0.71 ± 0.04
Multi-cue Mean Fusion	0.82 ± 0.04	0.84 ± 0.03	0.77 ± 0.04	0.75 ± 0.04	0.80 ± 0.03
Ours	0.91 ± 0.03	0.94 ± 0.02	0.86 ± 0.03	0.83 ± 0.03	0.89 ± 0.02

Note: Bold values indicate the best result in each column.

.

The proposed method achieves an average F1-score of 0.89 ± 0.02 among the evaluated baselines. PointNet++ [7] achieves 0.83 ± 0.03 using five-fold out-of-fold evaluation on the 64-instance evaluation pool, with approximately 51 training instances and 13 test instances per fold. In contrast, the proposed geometric method does not require task-specific model training. The comparison is intended to contextualize the training-free formulation rather than establish broad superiority over learning-based defect detectors. Performance is strongest on planar cube surfaces and weaker for scratch-like defects on rectangular prisms, indicating sensitivity to point-cloud resolution and surface geometry.

4.2. Defect-Aware Target Grounding Evaluation

Target grounding accuracy is reported in

Table 8. Defect-aware target grounding accuracy.

Method	Top-1 Acc.	Top-3 Acc.
Language-only Grounding	0.52 ± 0.04	0.71 ± 0.03
2D Mask Projection	0.65 ± 0.03	0.82 ± 0.02
3D Inst. Seg. Only	0.74 ± 0.03	0.87 ± 0.02
Grounding w/o Language	0.80 ± 0.03	0.91 ± 0.02
Ours (full)	0.89 ± 0.02	0.95 ± 0.02

Note: Bold values indicate the best result in each column.

.

The proposed method achieves the highest Top-1 grounding accuracy of 0.89 ± 0.02 (overall micro-average on 60 commands) and Top-3 accuracy of 0.95 ± 0.02. The multiplicative formulation in Equation (7) penalizes candidates that fail on any single criterion. The defect-aware tasks (e.g., “pick the defective cylinder”) show the largest improvement because the defect consistency score acts as a hard filter for target selection. Analysis of failure cases reveals that most failures occur in category-only instructions where the robot must select among multiple instances of the same category without location or defect constraints; in such cases, the ambiguity in language grounding leads to incorrect target selection.

The headline Top-1 accuracy is computed as a micro-average over all 60 commands in each run and is not obtained by averaging rounded category-level values. To avoid an unverifiable reconstruction from rounded subgroup values, the category-specific numbers are not used as headline evidence. Qualitatively, errors occur more often for cylinders and rectangular prisms because their geometry produces several grasp candidates with similar scores.

Table 9. Grounding accuracy breakdown by command type.

Command Type	n	3D Inst. Seg. Only	Ours
Category-only	15	0.81 ± 0.05	0.88 ± 0.04
Category + Location	15	0.78 ± 0.05	0.90 ± 0.04
Category + Defect	15	0.72 ± 0.06	0.91 ± 0.03
Category + Defect + Location	15	0.68 ± 0.06	0.88 ± 0.04
Overall (micro-avg)	60	0.74 ± 0.03	0.89 ± 0.02

reports the grounding accuracy breakdown by command type. The results demonstrate that the full defect-aware grounding is critical for commands that specify defect constraints; without defect scoring, the accuracy drops significantly from 0.91 to 0.72 for category-plus-defect commands.

4.3. Grasp Execution Evaluation

Grasp execution success rates and parameter errors are reported in

Table 10. Robotic grasping execution results.

Method	Success Rate	Grasp Error (mm)	Placement Error (mm)	Time (s)
Language-only Grounding	0.51 ± 0.05	18.6 ± 2.1	25.1 ± 3.2	9.2 ± 0.8
2D Mask Projection	0.64 ± 0.04	11.9 ± 1.5	16.4 ± 2.0	6.7 ± 0.5
3D Inst. Seg. Only	0.69 ± 0.04	10.3 ± 1.3	13.8 ± 1.8	5.9 ± 0.4
Grasp w/o Defect-Free Contact	0.72 ± 0.04	8.9 ± 1.2	12.5 ± 1.6	5.2 ± 0.3
w/o Recovery	0.75 ± 0.04	7.9 ± 1.1	11.7 ± 1.5	4.8 ± 0.3
Ours (full)	0.84 ± 0.03	5.7 ± 0.9	8.3 ± 1.2	4.1 ± 0.2

Note: Bold values indicate the best result in each column.

.

The proposed method achieves the highest overall success rate of 84.0% ± 3.0%. The most significant improvement comes from the defect-aware grounding and the recovery mechanism: without defect awareness, the system frequently attempts to grasp on defective regions that lead to grasp failure; without the recovery mechanism, any grasp misalignment or execution deviation triggers an unrecoverable failure. The execution time is decomposed as follows: task parsing (0.3 s), point-cloud preprocessing (0.6 s), defect detection (0.9 s), target grounding (0.2 s), grasp planning (0.6 s), and robotic execution (1.5 s).

Failure analysis accounts for all observed failures: grasp slippage on defective contact regions (38%), collision with neighboring objects (25%), localization error due to point-cloud noise (22%), and other failures including inverse-kinematics and gripper-actuation errors (15%).

4.4. Ablation Study

Ablation results are reported in

Table 11. Ablation study of the proposed framework. “—” indicates that the removed component operates downstream of the corresponding evaluation stage and does not affect that metric.

Variant	Defect F1	Initial Grounding Acc.	Grasp Success
Full method	0.89 ± 0.02	0.89 ± 0.02	0.84 ± 0.03
w/o multi-cue fusion	0.71 ± 0.05	0.73 ± 0.04	0.70 ± 0.04
w/o spatial relation	—	0.80 ± 0.04	0.78 ± 0.04
w/o defect-free contact	—	—	0.72 ± 0.04
w/o recovery	—	—	0.75 ± 0.04

Note: Bold values indicate the best result in each column.

.

Removing multi-cue fusion reduces both grounding and grasp performance because single cues are less discriminative for defect detection. The spatial-relation ablation affects target grounding and downstream grasp planning but not defect scoring. The defect-free contact term operates only during grasp ranking; therefore, its removal does not change defect F1 or initial grounding accuracy, and the corresponding cells are marked “—” rather than repeating the full-method values. Likewise, removing recovery affects execution only and leaves the initial perception and grounding outputs unchanged.

4.5. Statistical Significance Analysis

Pairwise tests are conducted on matched per-instance, per-command, or per-trial outcomes. McNemar’s test is used for binary defect-classification correctness on 64 evaluation instances, grounding correctness on 60 commands, and grasp success on 120 trials. The reported F1 or accuracy difference is summarized separately from the McNemar test, which operates on discordant correctness pairs. The paired Wilcoxon signed-rank test is used for grasp and placement errors only on matched trials for which both methods produce valid measurements; failed trials remain in the success-rate comparison. Holm–Bonferroni correction controls the family-wise error rate within each metric family, and 95% confidence intervals are computed by paired bootstrap resampling with 10,000 resamples. These tests rely on the paired outcome records; aggregate table values alone are insufficient to reconstruct the contingency tables.

Table 12. Pairwise statistical analysis against the full proposed method. Reported F1/accuracy differences are descriptive effect summaries; McNemar p-values are computed from paired binary correctness outcomes.

Comparison	Metric	Difference	95% CI	Test (Holm-adj. p-Value)
Ours vs. 3D Inst. Seg. Only	Defect F1	+0.10	[+0.05, +0.15]	McNemar on correctness (p = 0.003)
Ours vs. Multi-cue Mean Fusion	Defect F1	+0.09	[+0.04, +0.14]	McNemar on correctness (p = 0.009)
Ours vs. PointNet++	Defect F1	+0.06	[+0.01, +0.11]	McNemar on correctness (p = 0.042)
Ours vs. 3D Inst. Seg. Only	Grounding Top-1	+0.15	[+0.07, +0.23]	McNemar on correctness (p = 0.003)
Ours vs. Grounding w/o Language	Grounding Top-1	+0.09	[+0.04, +0.14]	McNemar on correctness (p = 0.012)
Ours vs. w/o Defect-Free Contact	Grasp Success	+0.12	[+0.05, +0.19]	McNemar on success (p = 0.012)
Ours vs. w/o Recovery	Grasp Success	+0.09	[+0.03, +0.15]	McNemar on success (p = 0.033)
Ours vs. w/o Defect-Free Contact	Grasp Error (mm)	−3.2	[−5.1, −1.3]	Wilcoxon (p = 0.018)
Ours vs. w/o Recovery	Grasp Error (mm)	−2.2	[−3.8, −0.6]	Wilcoxon (p = 0.036)
Ours vs. w/o Defect-Free Contact	Placement Error (mm)	−4.2	[−6.4, −2.0]	Wilcoxon (p = 0.009)
Ours vs. w/o Recovery	Placement Error (mm)	−3.4	[−5.5, −1.3]	Wilcoxon (p = 0.021)

reports descriptive metric differences alongside paired tests after Holm–Bonferroni correction. For defect detection, the descriptive F1 gaps are +0.06 relative to PointNet++ and +0.09 relative to Multi-cue Mean Fusion; the corresponding paired correctness comparisons yield adjusted $p=0.042$ and $p=0.009$, respectively. For grasp execution, the defect-free contact term is associated with a +12% success difference (adjusted $p=0.012$), and the recovery mechanism with a +9% difference (adjusted $p=0.033$). These inferences remain conditional on the matched outcome records described above; the aggregate values in this manuscript are not sufficient to independently recompute the tests.

4.6. Qualitative Robotic Execution Demonstration

Figure 5 shows the experimental workpieces and point-cloud defect examples. The evaluated objects include a cone, cube, cylinder, and rectangular prism with representative hole-type and scratch-type defects. Red markers indicate the defect locations used for geometric defect evaluation.

Figure 6 shows the reconstructed scene, manually marked defect locations, and generated grasp orientations. The current composite does not contain a calibrated point-level defect-response heat map or a quantitative comparison with region-level ground truth. It is therefore used only to illustrate scene interpretation and grasp generation. Quantitative claims in this study remain restricted to instance-level defect classification, target grounding, and grasp execution.

The proposed method identifies individual workpieces from the point cloud, detects local defective regions through geometric defect scoring, and generates corresponding grasping directions for robotic manipulation.

Figure 7 demonstrates the complete language-to-action execution process. The original user commands are given in Chinese, and their English translations are shown in the figure for readability. The language instruction is first parsed by the language model into a structured task representation. The perception module then grounds the target object according to the defect-aware scene analysis. Finally, the robot executes the corresponding grasping and placing actions.

5. Discussion

The experiments evaluate the feasibility of the proposed framework in a controlled prototype setting. The results support discussion of the geometric formulation, target-grounding mechanism, and recovery behavior within that setting, but they do not establish general performance in cluttered industrial scenes.

On the geometric-defect formulation. The cue contributions in Equation (5) provide a qualitative indication of the geometric abnormality that drives a response. A concavity-dominant or normal-inconsistency-dominant response may help interpret a detected region, but this study does not evaluate defect-type classification. Cue dominance must therefore be treated as qualitative attribution rather than a validated dent-versus-scratch prediction.

On defect-aware grounding vs. related approaches. Compared with recent object-centric skill frameworks such as GSL [31], which build object-level skill interfaces for hierarchical manipulation, our defect-aware target grounding explicitly models fine-grained geometric properties (surface defects, grasp clearance, and pose stability) that are central to robotic grasping. Compared with language-guided planning methods [11,12,13,14,15,16,17,18,19], which generate action sequences from language instructions, our approach operates at the geometric entity level and produces executable parallel-gripper grasp poses. Compared with geometry-aware grasp methods [29,30], which focus on grasp pose optimization given a known target, our grounding mechanism is task-conditional and applies the defect constraint to select the correct target among multiple candidates.

Why defect-aware grounding matters. Language instructions such as “pick the defective cube on the left” require not only semantic parsing but also geometric evidence. Using only LLM-based parsing or 2D detection often fails because language does not specify which instance in the scene is defective. Our multi-cue geometric defect scoring provides the physical evidence needed to bind the language constraint to a specific workpiece instance.

Why defect-free contact improves grasp stability. Section 3 defines the two uses of the defect response: instance-level target ranking and contact-level grasp screening. The ablation isolates the downstream contribution of contact screening without repeating the full conceptual distinction.

On the recovery mechanism. The two-level recovery mechanism addresses different failure modes at different granularities. Level-1 (re-ranking) handles perception noise and minor geometric deviations efficiently, requiring only a re-evaluation of the existing candidate set. Level-2 (re-targeting) handles systematic failures by searching alternative candidates that preserve the original category and defect-status constraints while relaxing only lower-priority spatial or clearance preferences. If no semantically valid candidate exists, the system returns failure or requests user confirmation rather than automatically relaxing the defect constraint.

Limitations. First, the evidence is limited to 80 physical instances, 60 commands, and 120 grasp trials collected in one fixed-camera prototype setup with four simple geometries and two manually created defect families. Second, the 16-instance validation split is too small to establish broad robustness of the temperature and score weights; only the defect-threshold sweep is reported quantitatively. Third, Euclidean clustering assumes at least 30 mm separation, so touching, stacked, severely occluded, and free-form workpieces remain outside the validated scope. Finally, the current qualitative composite does not provide a calibrated point-level response map or quantitative region-overlap metric. These boundaries restrict the conclusions to prototype feasibility rather than industrial generalization.

6. Conclusions

This paper presented an interpretable geometric framework that connects structured language parsing, defect-aware target grounding, grasp-pose ranking, and execution feedback in a controlled prototype. On the 64-instance evaluation pool, the defect detector achieved an average instance-level F1-score of 0.89; the 60-command evaluation yielded 0.89 Top-1 grounding accuracy; and the 120 execution trials yielded an 84.0% grasp success rate. These results support the feasibility of explicit geometric defect reasoning as an intermediate layer for separated, simply shaped workpieces under a fixed camera. Validation on larger independent datasets, cluttered or touching objects, free-form surfaces, and moving viewpoints is required before broader industrial claims can be made.