Human Event and Action Analysis Using Transformer-Based Multimodal AI ^†

Abstract

With the increasing demand for enhanced security and surveillance, the integration of multimodal AI has shown significant promise. We developed and fine-tuned a transformer-based model, the Large Language and Vision Assistant–OneVision, tailored for human event and action recognition. By utilizing a multimodal approach, we identified specific human actions, including eating, running, fighting, sitting, and sleeping, within diverse real-world settings. Through knowledge distillation and Low-Rank Adaptation, the model’s performance was optimized in demonstrating substantial improvements in context-aware recognition and response generation. Evaluation results showed recall-oriented understudy for obtaining evaluation (ROUGE)-1 score of 0.6844, ROUGE-2 score of 0.5751, ROUGE-L score of 0.6520, and the bilingual evaluation understudy score of 68.20, demonstrating significant gains in accuracy and interpretability. The model’s success highlights its potential for real-time applications in surveillance, healthcare, and interactive AI systems, providing reliable, efficient, and context-sensitive human action detection.

1. Introduction

With the rapid advancement of AI and machine learning, image and video analytics have become a key to security and surveillance. Traditional methods require manual inspection of footage, which is time-consuming and prone to human error [1]. Switching to an AI-enhanced or assisted system streamlines the surveillance system.

The Large Language-and-Vision Assistant OneVision (LLaVA-OneVision) model is one of many multimodal models tailored to enhance and streamline surveillance systems. Building on the shift towards AI-driven surveillance, accurate human event and action recognition has become increasingly critical for real-time analysis of footage. Traditional methods, such as convolutional neural networks and recurrent neural networks, struggle with handling complex multimodal inputs and adapting to diverse environments [2]. In contrast, transformer-based architectures, which are known for their success in natural language processing (NLP) and image analysis, offer a robust alternative [2,3]. In this study, we fine-tuned the LLaVA-OneVision model, leveraging its multimodal capabilities for the improved recognition of specific human actions. This approach aims to enhance action detection accuracy while ensuring context-aware and user-friendly outputs for practical surveillance applications. Generally, the broader community benefits from enhanced safety and security measures that can be implemented more effectively in this system.

To enhance transformer-based multimodal AI for precise and context-aware human action recognition with human-readable outputs and interactions, a LLaVA-OneVision model is improved to accurately identify eating, running, fighting, sitting, and sleeping by leveraging multimodal dataset inputs. The model is trained to incorporate and adapt to contextual variables such as environment, lighting conditions, and viewing angles, ensuring reliable performance across various real-world scenarios. The model’s response generation is refined to provide direct and contextually relevant answers, improving its interpretability and usability for applications requiring quick and precise feedback.

2. Literature Review

2.1. LLaVA-OneVision

LLaVA-OneVision produces text based on one or more pictures or videos fed into it. It is the first single model that single-handedly pushes the limits of open large multimodal model performance in three crucial computer vision scenarios: video, multi-image, and single-image scenarios [4,5]. This model enables the system to perform automated activity detection that also transcribes descriptions based on what is seen in the security surveillance system.

2.2. Multimodal AI

Multimodal AI processes different types of data more efficiently and effectively. It integrates several data kinds, or modes, to produce more accurate judgments, more accurate predictions, or more insightful conclusions regarding real-world issues [6]. To produce reliable results and a thorough understanding, multimodal AI integrates and evaluates several data sources compared to traditional AI models, which process one data source [7]. This type of model AI helps the system achieve the objectives the proponents intend to achieve.

2.3. Human Action Dataset

Human beings accomplish tasks with different actions or activities to go through life. Human activity refers to the different acts, behaviors, and customs that people and civilizations engage in that influence the social and physical environment [8]. The model utilizes datasets to function as intended. The final stage of a series of earlier tasks, including image acquisition, segmentation, tracking, identification, and classification, is the recognition of human actions [9]. The datasets are essential for the training of the model that the proponents intend to create. In security, AI is used as an intruder countermeasure, including a surveillance system. AI gathers, codes, stores, and processes data extracted from people’s daily lives and contributes to the strength of a security system [10]. Studies were conducted on system integration with sensor networks, drones, and edge computing [11]. In this study, we used a multimodal AI to process datasets.

3. Methodology

3.1. Dataset Collection and Curation

The Human Action Recognition Dataset is retrieved, covering 15 categories of human actions: calling, clapping, cycling, dancing, drinking, eating, fighting, hugging, laughing, listening to music, running, sitting, sleeping, texting, and using a laptop computer. From this dataset, five action categories were selected for training: eating, running, fighting, sitting, and sleeping, with 100 images per category, resulting in a curated subset of 500 training images, while 50 images per category were collected to populate a curated subset of 250 test images.

3.2. Knowledge Distillation on Dataset

The 750 curated images were processed through Gemini Pro 1.5 to leverage its advanced vision-language capabilities (Figure 1). Each of the 750 images was paired with a generated response from the AI model, forming a dataset for knowledge distillation. This method adopts the existing model’s expertise as a foundation for fine-tuning the much smaller model.

Figure 1. Sample output.

To obtain detailed contextual descriptions, we prompted Gemini Pro with: “In one brief but concise sentence, describe what you see, the viewing angle, the environment, and the lighting conditions (Figure 2). Each response was saved alongside its corresponding image in JSON format, resulting in a dataset that combines visual data with precise textual descriptions of key contextual variables.

Figure 2. JSON code for datasets.

3.3. Prompt Optimization for Training

To adapt the model’s standard response generation style to the objective, the prompt in the training dataset was simplified to focus on primary visual elements. For training and testing, the prompt was modified to: “What do you see in the image?” Using this method, the AI model’s standard response was refined to produce concise yet context-aware responses when prompted with minimal guidance.

3.4. Model Fine-Tuning Using Low-Rank Adaptation of Large Language Models (LoRA)

To fine-tune the model, we used the open-source codebase from the GitHub repository ‘lmms-finetune’ (accessed on 8 November 2024; commit 86895101a7f794c47cb3acc1061d0a148bc0b1df) [12]. It was modified to accommodate specific requirements, parameters, and workflows (Figure 3). Given that the base model is fully quantized, the fine-tuning process relies on LoRA, a training technique designed to attach new model weights on top of the base model while reducing the number of trainable parameters. This architecture allows for lightweight and efficient adaptation, making it particularly suitable for fine-tuning models such as LLaVA-OneVision.

Figure 3. Formulas and diagrams of LoRA.

For this fine-tuning session, the LoRA rank was set to 2, the number of epochs to 10, the gradient accumulation step to 4, and the learning rate to 10⁻⁵. These parameters were chosen in response to the number of images in the dataset, preventing overfitting, as well as to balance computational efficiency with optimal performance gains, ensuring that the model adapts effectively to the pursued objectives. The source code and dataset were provided, and the Google Colab notebook used was accessible to all for a more interactive and hands-on experience with the model [13].

4. Results and Discussion

After completing the fine-tuning process, the model’s performance was evaluated using training metrics and standard performance evaluation metrics. The results were used to validate the training efficiency and the effectiveness of our fine-tuned model for action recognition.

4.1. Training Performance

The final evaluation loss achieved was 1.608, indicating the model’s improved ability to generalize to unseen data after fine-tuning. In comparison, the final training loss of 1.404 reflects effective pattern learning, with the close alignment between training and evaluation losses suggesting minimal overfitting. Regarding runtime efficiency, the model required 57.77 s for evaluation and 3460.75 s for training. With an evaluation throughput of 4.327 samples per second and a training throughput of 1.445 samples per second, these metrics demonstrate that the LoRA fine-tuning process enabled efficient learning and adaptation, even with quantized parameters. The model was trained for 9.82 epochs with a gradient accumulation step of 4, balancing between computational efficiency and performance, as evidenced by the consistent sample processing rate (Figure 4).

Figure 4. Model training performance: (a) Training epochs; (b) Training gradient norm; (c) Training learning rate; (d) Evaluation loss; (e) Training loss.

4.2. Performance Metrics: Recall-Oriented Understudy for Gisting Evaluation (ROGUE) and Bilingual Evaluation Understudy (BLEU) Scores

To evaluate the quality of the model’s output, ROUGE F1 and BLEU scores were calculated using the following equations, which are the standard methods of performance gauging in NLP models for measuring the similarity between generated and reference texts.

$$\mathrm{R}\mathrm{O}\mathrm{U}\mathrm{G}\mathrm{E}-\mathrm{N}={\displaystyle \frac{{\sum }_{S\in \left\{\mathrm{R}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{S}\mathrm{u}\mathrm{m}\mathrm{m}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{e}\mathrm{s}\right\}}{\sum }_{\mathrm{g}\mathrm{r}\mathrm{a}{\mathrm{m}}_n\in S}{\mathrm{C}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{t}}_{\mathrm{m}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}}\left({\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{m}}_n\right)}{{\sum }_{S\in \left\{\mathrm{R}\mathrm{e}\mathrm{f}\mathrm{e}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{c}\mathrm{e}\mathrm{S}\mathrm{u}\mathrm{m}\mathrm{m}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{e}\mathrm{s}\right\}}{\sum }_{\mathrm{g}\mathrm{r}\mathrm{a}{\mathrm{m}}_n\in S}\mathrm{C}\mathrm{o}\mathrm{u}\mathrm{n}\mathrm{t}\left({\mathrm{g}\mathrm{r}\mathrm{a}\mathrm{m}}_n\right)}}$$

(1)

$$\mathrm{B}\mathrm{L}\mathrm{E}\mathrm{U}=\min \left(1,{\displaystyle \frac{output-length}{reference-length}}\right){\left(\prod _{i=1}^{4}precisio{n}_i\right)}^{{\displaystyle \frac{1}{4}}}$$

(2)

In (2), output-length is the number of tokens in the candidate translation, while reference-length is the number of tokens in the reference translation(s). The term ${\mathit{precision}}_i$ denotes the modified n-gram precision for $i=\mathrm{1,2},\mathrm{3,4}$. The expression $\min \left(1,{\displaystyle \frac{\mathit{output}-\mathit{length}}{\mathit{reference}-\mathit{length}}}\right)$ represents the brevity penalty applied to overly short outputs.

The ROUGE score was used to evaluate the overlap of n-grams, such as unigrams, bigrams, and longest common subsequences, between the model’s generated descriptions and the reference descriptions. As seen in Table 1, the original base model achieved ROUGE-1, ROUGE-2, and ROUGE-L F1 scores of 0.4745, 0.2832, and 0.4165, respectively. After fine-tuning, these scores improved to 0.6844 (ROUGE-1 F1), 0.5751 (ROUGE-2 F1), and 0.6520 (ROUGE-L F1), reflecting more accurate and contextually relevant outputs compared to the base model (Table 1).

Table 1. ROUGE F1 scores.

Score	Base Model	Fine-Tuned Model
Average ROUGE-1 F1	0.4745	0.6844
Average ROUGE-2 F1	0.2832	0.5751
Average ROUGE-L F1	0.4165	0.6520

The BLEU score was used to measure n-gram overlap between generated text and reference texts to assess translation accuracy. The original base model showed an average BLEU score of 42.24, as seen in Table 2 below, whereas the fine-tuned model reached 68.2. This substantial improvement demonstrates a higher degree of similarity between the fine-tuned model’s outputs and the reference descriptions, confirming enhanced action recognition, context-aware generation, and sensitivity.

Table 2. BLEU Score.

Score	Base Model	Fine-Tuned Model
Average BLEU Score	42.24	68.20

4.3. Discussion

The improvements in both ROUGE and BLEU scores between the base and the finetuned model imply the effectiveness of the fine-tuning process. The fine-tuned model showed substantial gains in accurately recognizing the five specified human actions and capturing essential contextual variables, such as viewing angle and lighting conditions (Figure 5).

Figure 5. Sample image.

In text generation, the model produced responses that were more concise and focused, aligning to generate direct and actionable descriptions. These results highlight the potential of transformer-based multimodal artificial intelligence models, such as LLaVA-OneVision, to deliver highly contextual and accurate human action recognition when fine-tuned for specific real-world tasks (Figure 6).

Figure 6. Sample prompt and models comparison.

5. Conclusions and Recommendations

We fine-tuned a transformer-based multimodal AI model to enhance human action recognition, concentrating on five specific actions: eating, running, fighting, sitting, and sleeping. By leveraging LoRA for efficient parameter adjustment, the model achieved significant improvements in accuracy and contextual relevance, as demonstrated by substantial gains in ROUGE and BLEU scores compared with the original base model. The fine-tuned model effectively recognized actions across diverse environments and maintained robustness against variables such as viewing angles and lighting conditions, making it well-suited for real-world applications in surveillance, healthcare, and interactive AI systems. The results underscore the effectiveness of combining knowledge distillation with prompt optimization to encourage concise, context-aware responses from transformer-based models.

It is necessary to expand the set of action categories to improve versatility for applications such as workplace safety and autonomous vehicles and incorporate additional contextual variables, including time of day and object interactions, to enhance real-world adaptability. Runtime efficiency and edge computing solutions must be enhanced to enable faster real-time action recognition. The model’s performance must be evaluated on diverse datasets to strengthen generalizability in complex, multi-action scenarios.

The scope of this study was limited to recognizing eating, running, fighting, sitting, and sleeping due to the availability of labeled data. The use of LoRA was also constrained, as its performance gains diminish with larger datasets. The LLaVA-OneVision model size was restricted to 0.5 billion parameters because of graphics processing unit and CPU limitations, which reduce its ability to capture more complex patterns. These limitations are acknowledged as areas for improvement in future work, particularly with expanded datasets and enhanced computational resources.