Abstract
Construction safety inspections require identifying violated rules and localizing the associated workers or hazardous areas. Although multimodal large language models can describe hazards, supervision based solely on structured answers does not explicitly constrain the correspondence between rule records and visual targets. This study proposes RTR-Align, a Rule Token-to-Region Alignment method for construction safety violation recognition and localization. The method links the specific occurrence position of the Rule Token in each violation record to the corresponding target region, imposing separate spatial constraints on different instances associated with the same rule. It jointly learns rule assessment and target localization during LoRA adaptation of Qwen3-VL-8B-Instruct. The study uses the ConstructionSite10k dataset, which contains 10,013 construction site images with violation question-answering and localization annotations, and produces instance-level annotations for training and evaluation by separating grouped instances, adjusting bounding-box boundaries, and adding annotations for missed targets. In the main run, RTR-Align achieved an F1 score of 62.72% and a total intersection over union (Total IoU) of 39.40% on the 3004 test images, exceeding the base model by 48.11 and 32.55 percentage points, respectively. RTR-Align produced fewer false positives than the base model, but its recall was 1.19 percentage points lower. Analyses across rules and scenarios, along with parameter and qualitative analyses, revealed performance differences and remaining challenges. These results support joint learning of violation recognition and rule-associated localization, enabling the model to identify relevant targets with fewer false positives and provide spatial evidence for manual verification.