Integrating Segment Anything Model 2 and Multimodal Generative Models for a Lightweight Projection Mapping Framework ^†

Abstract

A lightweight framework integrating Segment Anything Model 2 and multimodal generative AI is developed in this study to automate projection mapping. By leveraging Google Vertex AI cloud application programming interfaces (APIs) and a Python-TouchDesigner bridge, the developed framework bypasses local hardware constraints and replaces costly physical sensors with algorithms for automated masking and content synthesis. Implementation using FastAPI and stable video diffusion establishes a stable asynchronous pipeline. Experts evaluated the framework as significantly reducing labor time and equipment costs, providing an intuitive, high-efficiency workflow for creators to achieve precise alignment and visual generation without high-end graphics processing unit requirements.

1. Introduction

Projection mapping technology has become popular in artistic curation and commercial exhibitions [1]. In this research, we created scenarios for high spatial flexibility and frequent visual content adjustments, such as virtual prototyping in retail spaces and agile curation [2]. Creators need to conduct diverse visual tests and alignments for different objects or spatial configurations within a short timeframe. Although generative AI (GenAI) can produce high-quality assets [3], creators still expend significant effort on tedious manual calibration and mask drawing [4]. Therefore, this study aims to explore how AI algorithms can simplify the calibration process, liberating creators from repetitive technical labor to focus on creative content development [5].

Current mainstream alignment technologies face multiple challenges. First, active alignment relies heavily on depth-sensing hardware such as Kinect or Red-Green-Blue (RGB)-depth sensors [4], which increases costs and limits the flexibility of on-site installation [1]. While GenAI has matured, solutions such as stable diffusion or stable video diffusion often involve complex environment deployment and node-based logic [6,7], presenting a high entry barrier and learning curve for both non-technical artists and professional engineers. Finally, the hardware requirements of mainstream solutions make it difficult for creators to perform real-time prototype development on lightweight devices [8].

To address these issues, we constructed a lightweight framework integrating Segment Anything Model 2 (SAM2) image segmentation and cloud-based GenAI models. The framework adopts the SAM2 model to achieve automated object segmentation, replacing the time-consuming process of traditional manual point-pulling and enhancing efficiency through its transformer-based streaming memory architecture [9]. Simultaneously, a text-to-content-driven workflow was developed, leveraging multimodal handling interfaces [10] to simplify software parameter adjustments and shorten the production cycle. The feasibility of the framework in replacing high-end local graphics processing units (GPUs) with cloud computing using Fast application programming interfaces (APIs) and ngrok [11] was evaluated, demonstrating that precise projection mapping alignment and content generation can be fluently executed without high-end hardware or physical sensors [8].

2. Materials and Methods

2.1. Framework Architecture

The developed framework employs a hybrid local-cloud architecture that minimizes hardware requirements while maintaining high alignment precision. As shown in Figure 1, the workflow is categorized into three primary layers.

Figure 1. The proposed framework architecture that integrates SAM2, TouchDesigner, and cloud-based generative AI services.

• Edge detection layer (Top 1): To achieve sensorless alignment, the framework utilizes SAM2 in a local Python environment. This layer extracts geometric masks directly from live video feeds, which are then transmitted to the integration environment as alpha channels for precise projection mapping;
• Framework integration layer (Top 2): Utilizing TouchDesigner (v2023.11600, Derivative, Toronto, ON, Canada) as the central hub, this layer manages bidirectional communication between the local interface and remote AI services. It orchestrates semantic prompts for static content and coordinates the timing for dynamic narrative generation;
• Cloud services layer: This layer offloads heavy computational tasks to remote infrastructure to bypass local GPU limitations.
  • Static generation: Semantic prompts are sent to Google Vertex AI (Imagen 3) using the REST API to receive high-fidelity backgrounds.
  • Dynamic synthesis: For dynamic content, an asynchronous pipeline is established with Google Colab (Google LLC, Mountain View, CA, USA) via FastAPI (v0.110.0) and ngrok(v3.8.0). This setup hosts the stable video diffusion (SVD) model, utilizing the static Seed Image as a reference to ensure visual consistency.

2.2. Implementation

The framework balances local computational efficiency and the expansive capabilities of cloud-based generative models. The implementation was conducted as follows.

2.2.1. Computing Environment and Framework Integration

The framework was executed on a workstation (Intel Core i7-9750H, Intel Corporation, Santa Clara, CA, USA; 16.0 GB RAM) utilizing a Docker-containerized environment [12] to optimize SAM2 inference. The framework is orchestrated within TouchDesigner, where a Python 3.9 environment facilitates a live notation approach [13]. While traditional setups require specialized tracking hardware [14], such an integration enables descriptive prompts to act as dynamic instructions that are instantly mapped onto physical targets, significantly shortening the cycle from conceptualization to realization.

2.2.2. Software and Development Environment

To provide transparency in the intermediate processing stages, the framework interface within TouchDesigner visualizes the internal data flow. This allows the creator to monitor the transition from raw input to generated content.

• Interactive segmentation (Figure 2): After the user defines a target, the SAM2 model generates a high-fidelity mask, visualized as a blue overlay for real-time verification. A status message with a confidence score (0.982) is displayed to ensure the mask accurately fits the object contours before content generation begins.
  

  Figure 2. Framework interface for real-time mask extraction and verification.
• API monitoring and control: To manage the cloud-based generative tasks, the framework integrates a custom-built interface (designated as Google Gemini Bridge) that operates via a Python-based console. This interface serves as the primary terminal for executing structured prompt protocols and monitoring the lifecycle of each API request. As shown in Figure 3, the window provides real-time feedback from the remote nodes, displaying critical information such as the active prompt and the total inference time. This integrated console allows the creator to track the generation progress without interrupting the main TouchDesigner workflow.This integrated console allows the creator to track the generation progress without interrupting the main TouchDesigner workflow. As illustrated, the synthesized base image is displayed on the left, which is handled by a custom ‘Image_base’ operator component within TouchDesigner alongside the ‘Google Gemini Bridge’ monitoring window.
  

  Figure 3. Custom command interface for monitoring prompt delivery and model execution status within TouchDesigner.
• Alignment pre-visualization: Before the final projection output, the framework provides a virtual mapping stage to verify spatial consistency. As shown in Figure 4, the synthesized texture is automatically fitted into the extracted geometry’s ultraviolet space within the TouchDesigner environment. This pre-visualization allows the creator to inspect the edge alignment and visual fit, ensuring that the AI-generated content perfectly matches the physical target’s constraints before it is sent to the projector.
  

  Figure 4. Real-time preview within the interface for checking the alignment before final projection.

2.2.3. Communication Protocols and Asynchronous Pipeline

The framework utilizes Open Sound Control and API requests via a secure tunnel established by FastAPI and ngrok. To ensure pipeline integrity, API key authentication is implemented through the cloud gateway [15]. This distributed architecture effectively prevents local hardware overloads during high-load synthesis, ensuring that the primary creative interface remains responsive for spatial alignment even on mid-range hardware. As shown in Figure 5, the remote node (Google Colab) handles the AI generation tasks in the background. This ensures that the local TouchDesigner environment stays stable and lag-free throughout the entire production process.

Figure 5. Monitoring the remote Google Colab node while it processes AI generation tasks in the background.

2.3. Experiment

To validate the technical feasibility and spatial adaptability of the developed framework, an experiment was conducted as follows. The Nail Polish Billboard was developed using a printed geometric nail polish artwork as the primary subject [Figure 6a]. To ensure robust mask extraction by the local SAM2 engine, the subject was positioned within the input frame, maintaining high visual contrast against its immediate environment. This setup was specifically designed to minimize algorithmic errors during edge detection and to optimize the focal consistency of the segmentation process, enabling precise mask generation [Figure 6b].

Figure 6. Visual synthesis pipeline of the nail polish billboard prototype: (a) original input image; (b) segmentation mask generated by SAM2; (c) final high-fidelity texture synthesized by Imagen 3 using the structured prompt protocol.

Content was generated through the developed framework, ensuring visual coherence with the subject’s detected geometry. Creators followed a structured prompt hierarchy, including the following (Table 1):

Table 1. Prompt specification for nail polish billboard.

Category	Function	Parameter
Subject and constraint	Boundary Definition	Deep-sea opal and refractive fluid; strictly constrained within mask contour; background: absolute void black.
Inner mask	Physical Simulation	Material: Viscous translucent jelly-liquid with black-opal core Logic: High-tension rounded fluid surface.
Aesthetic details	Visual Fidelity	Suspended dichroic foil flakes (emerald, electric blue); layered crystalline depth; mirror-like wet finish.
Lighting and style	Rendering Output	Light: Internal cinematic glow Style: 8k macro detail View: Pure 1:1 orthographic view
Negative prompt	Noise Reduction	Literal galaxy shapes, stars, furniture, hands, blurry edges, knitted texture, and light leaking.

• Subject and spatial constraints: Defines the core visual theme and enforces strict boundary adherence. This ensures the background remains an absolute void-black to prevent light leakage in non-target areas;
• Inner mask materiality: Specifies the physical properties and surface logic of the content to ensure the generated texture naturally conforms to the mask edges;
• Aesthetic and crystalline details: Defines micro-level features and depth layers, such as suspended flakes and refractive finishes, to enhance the visual density of the projected content;
• Lighting and rendering style: Controls the cinematic quality and technical perspective, utilizing a 1:1 orthographic view to ensure exact spatial fitting onto the physical subject;
• Negative constraints: Explicitly excludes common GenAI artifacts and environmental noise to maintain the integrity of the target’s silhouette and prevent unwanted visual interference.

The generated visual assets from Google Vertex AI (Imagen 3) demonstrated fidelity and precise spatial alignment. By enforcing absolute void-black backgrounds within the structured prompt protocol, the model effectively produced high-definition textures with sharp, high-contrast boundaries that directly correspond to the SAM2-extracted masks. As illustrated in Figure 6, the Nail Polish Billboard prototype confirmed the framework’s capability to generate visually coherent content tailored to complex geometric silhouettes. The results showed that precise fitting can be achieved through prompt-driven boundary constraints, effectively eliminating the need for traditional manual spatial calibration or labor-intensive vertex manipulation.

3. Results

The evaluation follows the Creative Support Index (CSI) [5], using task-oriented walkthroughs and semi-structured interviews with two industry experts (Table 2). The assessment focuses on five core dimensions: productivity enhancement (D1), lowering barriers (D2), hardware lightweighting (D3), visual quality (D4), and commercial viability (D5).

Table 2. Scores in five dimensions.

Dimension	D1	D2	D3	D4	D5
Expert A	6	5	7	5	4
Expert B	5	6	7	5	5
Average	5.5	5.5	7	5	4.5

3.1. Evaluation Results

The framework was rated on a 7-point Likert scale (1: strongly disagree, 7: strongly agree). The quantitative results are summarized in Table 2 and Figure 7.

Figure 7. Radar chart of the results.

Hardware lightweighting (D3) showed a perfect mean score of 7.0, validating the framework’s ability to offload heavy computation to the cloud and bypass local GPU constraints. Productivity enhancement (D1) showed a score of 5.5, confirming that the SAM2-based automated masking successfully replaces the time-consuming manual calibration process. Visual performance (D4) showed a mean score of 5.0. While experts agreed that the current output was sufficient for demonstrative purposes and rapid prototyping, they expressed higher expectations for GenAI-based video synthesis.

3.2. Qualitative Analysis

Semi-structured interviews were conducted to synthesize expert perspectives into the following three key themes.

3.2.1. Efficiency in Alignment Automation

The experts unanimously affirmed that SAM2-driven automated edge extraction is significantly more efficient than traditional manual point-pulling methods. While the necessity of physical sensors remains context-dependent, the sensorless characteristic of the proposed framework was recognized as visionary for commercial demonstrations. This feature allows for rapid on-site setup and enhances deployment flexibility in temporary exhibition environments.

3.2.2. Technical Barriers

In addressing the reduction of technical barriers, experts identified distinct benefits across user demographics. For technical creators, the encapsulation of complex AI environments within a Python–open sound control bridge significantly reduces technical friction. Experts noted that further modularization could enable developers to integrate external plugins with greater ease.

3.2.3. Quality and Stability Assessment

The experts agreed that the static image quality (Imagen 3) and its spatial fitting were excellent for demonstrative purposes. However, they stated that SVD maintained temporal coherence, while blurriness tended to occur in longer sequences, indicating a need for model evolution or manual post-processing. Furthermore, to mitigate the risks associated with cloud API versioning and stability, the experts recommended a hybrid infrastructure: hosting generative models on an on-premises studio server, enabling mobile laptops to connect remotely for lightweight, stable deployment.

4. Conclusions

We developed an AI-assisted projection mapping framework leveraging SAM2 and multimodal GAI. The framework significantly optimizes production workflows and achieves hardware lightweighting through its sensorless architecture. By simplifying complex technical environments into a prompt-driven pipeline, the framework effectively lowers the entry barriers for creators and shifts the focus toward creative iteration. Although the framework excels in rapid prototyping, the current reliance on cloud-based API stability remains a limitation, as performance is subject to network latency and service availability. Therefore, it is necessary to evaluate higher-fidelity generative models and enhance the framework’s architectural robustness to ensure consistent performance in diverse commercial environments.