Application of Vision Language Models in the Shoe Industry ^†

Abstract

The confluence of computer vision and natural language processing has yielded powerful vision language models (VLMs) capable of multimodal understanding. We applied state-of-the-art VLMs for quality monitoring of the shoe assembly industry. By leveraging the ability of VLMs to jointly process visual and textual data, we developed a system for automated defect detection and contextualized feedback generation to enhance the efficiency and consistency of quality assurance processes. We conducted an empirical evaluation by evaluating the effectiveness of the developed VLM system in identifying standard procedures for assembly, using the video data from a shoe assembly line. The experimental results validated the potential of the VLM system in detecting the quality of footwear assembly, highlighting the feasibility of future practical deployment in industrial quality control scenarios.

1. Introduction

The need for the footwear manufacturing industry to adopt Industry 4.0 stems from a confluence of pressures and opportunities requiring a paradigm shift from traditional practices. To improve efficiency and productivity amid rising labor costs and potential shortages, it has become critical to integrate automation, robotics, the Internet of Things (IoT), and artificial intelligence vision systems into labor-intensive tasks in order to optimize workflows [1]. IoT provides the connectivity and data foundation necessary for intelligent manufacturing. Robotics delivers automation, precision, and flexibility to manufacturing processes. AI acts as the “brain” of the intelligent factory.

The application of AI in this industry is categorized into four distinct stages (Figure 1). The initial phase, pattern recognition, focuses on identifying and classifying pre-defined patterns within industrial data [2]. The second stage with machine learning (ML) enables systems to learn from data and make predictions or classifications without explicit programming [3]. This is followed by the third stage, dominated by deep learning (DL), which leverages complex neural networks to automatically extract intricate features from large datasets, significantly enhancing performance in image recognition and anomaly detection. Currently, industrial AI is in its fourth stage, characterized by the integration of large language models (LLMs) into generative AI, with new capabilities that can be applied to the footwear manufacturing industry. This study aims to develop a large-scale language model to explore new developments in AI.

2. Literature Review

2.1. DL and LLMs

The past two decades have witnessed a monumental surge in the fields of DL and the nascent development of LLMs. DL emerged as a dominant force, fueled by advancements in computational power, especially graphics processing units. After that, the foundations for LLMs were laid. While early natural language processing relied on statistical methods and simpler ML models, the late 2010s marked a turning point with the introduction of the transformer in 2017. This architecture, with its self-attention mechanisms, proved highly effective in capturing long-range dependencies in text, paving the way for the first generation of LLMs. Models such as Word2Vec [4] for word embeddings and the initial versions of the generative pre-trained transformer (GPR) and bidirectional encoder representations from transformers [5,6] demonstrated a significant leap in language understanding and generation capabilities.

2.2. Multimodal LLMs (MLLMs)

MLLMs represent a significant evolution in artificial intelligence, extending the capabilities of traditional LLMs beyond text to process and generate information across multiple modalities such as images, audio, and video (Figure 2) [7]. A key feature in their development is the combination of powerful architectures including transformers, which excel at sequence modeling and have become the backbone of modern LLMs, such as convolutional neural networks, which are predominantly used in visual feature extraction. These visual encoders, often with transformers, are crucial for processing image and video data. Mechanisms to fuse visual and textual features enable MLLMs to perform image captioning, visual question answering, generating images from text, and even more complex reasoning involving multiple modalities. While still rapidly developing, MLLMs hold immense promise for creating versatile and human-like AI systems capable of understanding and interacting with the world in a richer, more comprehensive way.

3. Methodology

In this study, we utilized the available VLM hosted on Hugging Face spaces that accept video input, including VideoLLaMA 2 [8] and Qwen2.5-VL [9]. These platforms ensure the capabilities of state-of-the-art VLMs in understanding and processing video content. By leveraging these pre-existing spaces, we tested and evaluated the models’ performance on tasks involving video analysis and interpretation, taking advantage of their pre-trained abilities to process both visual and textual information extracted from the video inputs.

3.1. VideoLLaMA 2

VideoLLaMA 2 is developed to enhance spatial-temporal modeling and audio understanding in video and audio-related tasks [8]. Building upon its predecessor, VideoLLaMA, version 2 incorporates a tailored spatial-temporal convolution (STC) connector to effectively capture the complex spatial and temporal dynamics within video data. It integrates an Audio Branch through joint training, allowing the model to seamlessly incorporate audio cues and enrich its multimodal comprehension abilities.

3.2. Qwen2.5-VL

Qwen2.5-VL was developed by the Qwen team at Alibaba Cloud [9]. Building upon the foundations of previous Qwen models, Qwen2.5-VL demonstrates significant advancements in understanding and processing visual content alongside textual information. Its key improvements include enhanced general image recognition across a wider range of categories, more precise object grounding capabilities with diverse output formats like bounding boxes and points, and substantially improved text recognition and understanding within images, including multi-lingual and multi-oriented text.

3.3. Zero-Shot Learning and In-Context Learning for VLMs

Zero-shot learning [10] for VLMs refers to the capability of these models to perform tasks or recognize concepts for which they have not been trained. Instead of requiring task-specific labeled data, VLMs leverage their pre-existing knowledge acquired from training on massive datasets of paired images and text to generalize to new, unseen scenarios. This is achieved by learning a shared embedding space where visual and textual representations are aligned. When presented with a new task or concept described in texts, VLMs use this alignment to understand the relevant visual features, even if it has never seen an example of that specific task or concept before. In-context learning [11] for VLMs is useful when the VLM learns to perform new tasks or follow specific instructions simply provided with a few input-output examples within the prompt, without any explicit fine-tuning or gradient updates. VLMs enable task demonstrations, figure out the underlying pattern, and apply it to new, unseen inputs.

4. Results and Discussion

In this study, we collected two types of video clips: videos showing that a worker inspected two tanker trucks in a simulated workplace security scenario (Figure 3), and real shoe assembly workshops (Figure 4). We submitted questions as text prompts along with the videos to a VLM.

4.1. VLM-Based Surveillance Video Analysis with Zero-Shot Learning

Two VLMs were evaluated using workplace security videos for their ability to understand and interpret complex visual scenes. A set of ten targeted questions was designed to test the models’ reasoning from various aspects of human activities, including counting (“How many workers are there?”), safety compliance (“Is the worker wearing a helmet?”), behavior detection (“Is the worker smoking?” “Did the worker talk on the phone?”), and object recognition (“Is this vehicle a bus?”) (

Table 1. Questions on surveillance assessed by VLM.

Questions	VideoLLaMA 2		Qwen2.5-VL
	Video (a)	Video (b)	Video (a)	Video (b)
How many workers are there?	X	X	V	V
How many vehicles are there?	V	V	V	V
Is the worker wearing a helmet?	V	V	V	V
Is the worker smoking?	X	X	V	V
Do workers wear masks?	V	V	X	X
Did the worker fall?	V	V	V	V
Did the workers take notes?	X	X	V	V
Did the worker talk on the phone?	X	X	V	V
Is the vehicle moving?	X	V	V	V
Is this vehicle a bus?	X	V	V	V

X for right answers and V for wrong answers.

). The questions covered a range of tasks, including activity recognition, attribute detection, and object identification, which were relevant to workplace safety and surveillance. By analyzing the accuracy and relevance of the models’ responses, their capabilities in real-world video understanding tasks were evaluated.

4.2. VLM-Based Assembly Quality Monitoring

We evaluated the performance of VLMs on shoe assembly workshop videos, focusing on their ability to understand and interpret complex visual scenes relevant to footwear manufacturing. The questions in

Table 2. Questions on shoe assembly assessed by VLM.

Questions	VideoLLaMA 2		Qwen2.5-VL
	Video (a)	Video (b)	Video (a)	Video (b)
Is the worker applying glue to the soles of the shoes?	V	X	X	V
Is the worker’s action assembling the sole and upper?	X	V	X	X
Is the worker wearing a glove on his right hand?	V	V	V	V
Is the worker wearing a glove on his left hand?	V	V	V	V
Do the workers wear shoes?	X	X	V	X

X for right answers and V for wrong answers.

were presented to VLMs to assess their comprehension of specific actions, object identification, and worker attributes: “Is the worker applying glue to the soles of the shoes? Is the worker’s action assembling the sole and upper? Is the worker wearing a glove on his right hand? Is the worker wearing a glove on his left hand? Do the workers wear shoes?” The models’ responses to these questions were analyzed to evaluate their capacity for detailed visual understanding within an industrial context.

4.3. In-Context Learning for Assembly Quality Monitoring

The Qwen2.5-VL model outperformed VideoLLaMA 2 in the tests. The final evaluation was conducted using responses to the text prompts for in-context learning. Figure 5 shows the questions and responses. This shows the potential of VLMs for industrial applications.

5. Conclusions

We evaluated the capabilities of VLMs for monitoring quality in the shoe assembly industry. The results indicated that the Qwen2.5-VL model demonstrated strong reasoning capabilities in various industrial scenarios. VLMs hold significant potential for optimizing efficiency, improving quality control, and enabling more intelligent automation throughout the footwear manufacturing process.