- Article
24 Pages
Language-conditioned tabletop manipulation requires a robot to ground a natural language command in a visual scene and execute the resulting operation. This study presents a modular framework for placing a specified object in one of four planar spatial relations to a reference object. The system comprises three modules for object detection, target selection, and operation-parameter generation: a Faster R-CNN detector provides candidate boxes and local features; a matching network combines these features with global scene context and a GRU instruction representation to select the movable and reference objects; and a ResNet34 regressor estimates grasp position and gripper angle, with a clearance-maximizing search determining placement. A programmatically generated dataset provides 220,000 scene images and over 1.5 million template-derived instructions. Evaluation in simulation and on a UR5 platform yields detection mean average precision (mAP) of 100.00% and 92.37% at an intersection-over-union (IoU) threshold of 0.7, joint target pair accuracies of 90.13% and 84.75%, and full-pipeline success rates of 87.35% over 2000 simulated trials and 78.75% over 400 real trials. Ablations support the benefits of global scene context, training time domain randomization, and the grasp input and placement designs. Oracle comparisons and failure analysis identify scene perception and target selection as major remaining failure sources. These findings highlight the value of a modular architecture for translating language-specified goals into executable robot actions while enabling systematic analysis of component performance and simulation-to-real transfer.
Electronics
17 October 2026












