Editable 3D Vehicle Generation from Four-View 2D Sketches Using Generative AI ^†

Abstract

We investigate the feasibility of interactive 3D vehicle modeling using multi-view 2D inputs and generative AI. Specifically, we examine whether a generative model (Meshy AI) can produce consistent 3D models from four-view vehicle images and whether localized edits applied to these images are reflected in the generated 3D geometry. Using a small set of vehicle sketches, we conduct experiments focusing on reconstruction capability and edit consistency. The results demonstrate that plausible 3D models can be generated from limited inputs and that region-based modifications in 2D views are partially reflected in the resulting 3D models, indicating the feasibility of the proposed interaction paradigm.

1. Introduction

Recent advances in generative AI have significantly improved the ability to generate 3D models from visual inputs. Technologies such as deep learning and neural rendering have enabled the reconstruction of complex shapes from images, making 3D content creation more accessible for applications including virtual reality, augmented reality, and digital design.

Despite these advances, most existing 3D generation approaches follow a one-directional pipeline: users provide input data, and a 3D model is generated as output. Once the model is created, modifying specific parts of the geometry often requires manual editing using specialized 3D modeling software. This process is not intuitive and can be time-consuming, especially for users without expertise in 3D design. To address usability challenges, sketch-based modeling has been explored as an alternative approach. By allowing users to interact with 3D models through 2D representations, such as drawings or images, these methods aim to provide a more natural and intuitive interface. However, sketch-based approaches often suffer from ambiguity, as limited 2D input does not always uniquely determine the corresponding 3D structure.

Another important challenge to address is the lack of controllability in generative 3D modeling. While recent models can produce visually plausible results, users have limited ability to modify specific regions of the generated model in a precise and intuitive manner. This gap between generative capability and user control remains a key issue in current systems.

In this study, we develop an interactive 2D–3D modeling framework that enables users to edit 3D vehicle models through structured multi-view 2D representations. By linking 2D edits with 3D generation, the developed framework presents intuitive interaction and consistent geometric modification. The feasibility of the interactive modeling paradigm is validated using a generative 3D model (Meshy AI). We also evaluate whether 3D models can be generated from four-view vehicle images and whether local modifications in 2D inputs are reflected in the generated 3D geometry.

2. Related Works

2.1. 3D Reconstruction from Images

Deep learning has significantly improved 3D reconstruction from images. Pixel-to-Voxel Generation with Context-Aware Fusion and Multi-Scale Generation [1] aggregates multi-view features using context-aware fusion, while Pixel to Mesh: 3D Mesh Model Generation via Progressive Deformation [2] employs graph convolutional networks to deform meshes from image features. These methods achieve high-quality reconstruction but still depend on input consistency and viewpoint coverage.

Traditional reconstruction techniques, including structure-from-motion and multi-view stereo, have also been widely studied. Reference [3] summarizes both classical and deep learning-based approaches, including neural radiance fields. However, these methods typically require multiple images and are not suitable for interactive editing scenarios.

2.2. Sketch-Based 3D Modeling

Sketch-based modeling provides an intuitive interface for 3D content creation. Rapid 3D Modeling of Objects from Single Freehand Sketches [4] introduces a framework that generates 3D objects from a single sketch while estimating viewpoint and structure, achieving state-of-the-art performance. The method also improves user experience and significantly reduces modeling time compared to conventional approaches. Contour-based approaches [5] leverage edge information to reconstruct geometry, and CAD-oriented methods [6,7] integrate geometric constraints and templates to improve accuracy and usability.

Sketch-based 3D modeling methods are categorized on the basis of input type, viewpoint handling, and modeling techniques such as neural networks, diffusion models, and implicit representations. The survey by Tono et al. [8] highlights a trend toward using minimal input (e.g., single sketches) combined with additional information such as text or multi-view constraints.

2.3. Generative and Learning-Based Approaches

Generative models have recently been applied to 3D content creation. DreamCar [9] leverages generative priors for vehicle reconstruction, enabling realistic shape generation even from limited input. In addition, reinforcement learning-based methods [10] model the reconstruction process as a sequence of actions, providing interpretable and flexible generation pipelines. These approaches demonstrate the potential of combining generative AI with structured modeling.

2.4. Objectives of This Study

Existing studies have advanced both image-based reconstruction and sketch-based modeling. However, reconstruction accuracy and user controllability must be traded off. Image-based methods achieve high-quality results but offer limited intuitive control, while sketch-based approaches provide user-friendly interaction but suffer from ambiguity and inconsistency.

We introduced the developed framework that combines structured multi-view 2D representations with generative 3D modeling. By leveraging multiple orthographic views, the proposed approach reduces ambiguity while maintaining intuitive interaction. The framework enables region-based editing by directly modifying specific areas in 2D inputs and reflecting these changes in the generated 3D model. This provides a practical solution for achieving both controllability and consistency in interactive 3D modeling.

3. Method

The objective of this study is to evaluate the feasibility of interactive 3D vehicle modeling using multi-view 2D inputs and a generative model. Instead of developing a new reconstruction model, we utilize an existing generative AI system and investigate its behavior under controlled input conditions.

The overall process of the developed framework is shown in Figure 1. The framework presents four-view images and a corresponding 3D model to the user. If the generated result does not meet the user’s expectations, the user modifies the input images, and a new 3D model is generated based on the edited inputs. This process can be repeated iteratively until a satisfactory result is obtained. The following aspects are examined in this study: whether 3D models can be generated from four-view vehicle images and whether local edits in 2D inputs are reflected in the generated 3D models.

Figure 1. Overview of the proposed interactive 3D vehicle modeling process. The system first presents four-view images and an initial 3D model to the user. If the generated result does not meet the user’s expectations, the user modifies the 2D images, and a new 3D model is generated based on the edited inputs. This process is iteratively repeated until a satisfactory result is obtained. Blue and green labels distinguish system operations and user editing, respectively. Red highlights the modified regions and the iterative feedback process.

Input Design and Generation Process

The input to the system consists of four orthographic images of a vehicle: front, side, top, and rear views. These images are manually prepared to ensure consistent alignment across views, allowing corresponding regions to represent the same semantic parts of the vehicle. The use of multiple orthographic views is motivated by their ability to provide complementary geometric constraints. Each view captures different aspects of the object: the side view represents the overall profile and length, the front and rear views constrain width and symmetry, and the top view provides information about layout and proportions. By integrating these views, the framework reduces ambiguity and infers a more consistent 3D structure compared to single-view inputs. To focus on structural information, line-drawing style images are used as inputs. This allows us to evaluate whether geometric structure alone is sufficient for 3D reconstruction, without relying on texture or shading cues.

The prepared images are provided to Meshy AI, which generates a 3D vehicle model. In this study, Meshy AI is treated as a black-box generative model, and no modifications are made to its internal architecture. The generated models are evaluated qualitatively based on their plausibility and consistency with the input views.

To examine editability, localized modifications are applied to specific regions in the input images. For example, a door region is manually altered by adding color or changing its shape. The modified images are then used as input to generate a new 3D model. By comparing the generated models before and after the modifications, we analyze whether the changes in the 2D inputs are reflected in the corresponding regions of the 3D model. The focus of this study is shown in Figure 2.

Figure 2. Evaluation process for reconstruction capability and edit consistency. A 3D model is first generated from the original four-view images. After applying local modifications to specific regions in the 2D images, a new 3D model is generated. The consistency between the edited regions in the 2D inputs and the corresponding regions in the 3D output is visually evaluated. Gray arrows indicate the workflow and comparison between the original and updated 3D models. Red highlights the user-modified regions, while the orange border emphasizes the 3D generation stage.

The feasibility of the proposed interaction paradigm is evaluated on the basis of reconstruction capability (whether plausible 3D models can be generated from limited multi-view inputs) and edit consistency (whether local modifications in 2D images are reflected in the generated 3D outputs).

4. Results

4.1. Experimental Setup

To evaluate the feasibility of the proposed interaction paradigm, we conducted an evaluation experiment using a single set of vehicle images. The dataset consists of four orthographic views (front, side, top, and rear), manually prepared as line drawings to ensure structural consistency across views.

First, the original four-view images were provided to Meshy AI to generate an initial 3D model. Next, to evaluate editability, a localized modification was applied to a specific region of the input images. In this experiment, the door region was manually colored in the four-view images. The modified images were then used as input to generate a new 3D model. The results were evaluated qualitatively by comparing the generated models before and after the modification.

4.2. 3D Reconstruction

The reconstruction capability of the framework was evaluated using the original four-view images shown in Figure 3a. The generated 3D model exhibits a coherent global structure and captures the overall shape characteristics of a vehicle, despite the absence of texture and shading information in the input (Figure 3b).

Figure 3. 3D reconstruction result from four-view drawings: (a) Input four-view images (front, side, top, rear) represented as line drawings; (b) Generated 3D model reconstructed from the input images using Meshy AI. The result demonstrates that a plausible 3D structure can be inferred from limited structural information without texture cues.

More specifically, major structural components such as the vehicle body, roof, and wheel positions are reasonably reconstructed, indicating that the model can infer volumetric structure from line-based representations when consistent multi-view constraints are provided. This suggests that the use of four orthographic views effectively reduces the ambiguity inherent in sketch-based inputs. However, geometric inaccuracies and smoothing effects are observed in finer details, which are attributed to the limited input information and the black-box nature of the generative model. These results indicate that while the proposed approach is capable of producing plausible coarse structures, further improvements are required for precise geometric control. The preliminary results show that multi-view line drawings provide sufficient structural cues for generating a reasonable 3D model, supporting the feasibility of the proposed interaction paradigm.

4.3. Edit Consistency

Next, we evaluate whether local modifications in the input images are reflected in the generated 3D model. As shown in Figure 4a, the door region in the four-view images is manually colored to introduce a localized change. When the 3D model is regenerated using the edited inputs, the corresponding region of the vehicle is also colored, as illustrated in Figure 4b. This indicates that the generative model is able to preserve a certain level of correspondence between the 2D input space and the generated 3D output, particularly for semantically meaningful regions.

Figure 4. Evaluation of edit consistency in multi-view-based 3D generation: (a) Input four-view images with a localized modification, where the door region is manually colored; (b) 3D model regenerated from the edited inputs. The color modification is reflected in the corresponding region of the generated model, indicating that local edits in 2D inputs can influence the resulting 3D geometry. Red indicates the user-modified door region in both the 2D input images and the regenerated 3D model.

However, the reflection of the edit is not perfectly precise. In some cases, the colored region appears slightly diffused or extends beyond the intended boundary, suggesting that the model does not enforce strict geometric alignment between views. This behavior may be attributed to the implicit nature of the generative process and the lack of explicit region constraints. Despite these limitations, the results indicate that localized edits in structured multi-view inputs can influence specific regions of the generated 3D model. This supports the feasibility of using 2D-based interaction as a means of controlling 3D generation intuitively.

4.4. Discussion

The experimental results demonstrate the feasibility of the proposed interaction paradigm from two key perspectives. First, in terms of reconstruction capability, a plausible 3D model can be generated from a single set of four-view images, even when only structural information is provided. This result suggests that multi-view representations effectively constrain the reconstruction process and significantly reduce the ambiguity inherent in sketch-based inputs. By combining complementary geometric information from different viewpoints, the system is able to infer a consistent global structure despite the absence of texture or shading cues. Second, in terms of edit consistency, local modifications applied to the 2D images are reflected in the generated 3D model. This indicates that the generative model preserves a certain level of semantic correspondence between the input space and the output geometry. Although the experiment is limited to a simple color modification, the results suggest that structured multi-view inputs can serve as an intuitive interface for controlling specific regions of a generated 3D model.

From a practical perspective, the developed framework has potential applications in interactive design systems. Compared with conventional 3D modeling workflows that require expert knowledge and manual manipulation, the ability to iteratively refine 3D models through simple 2D edits provides a more accessible and user-friendly approach. This is particularly beneficial for applications such as game asset creation, concept design, and rapid prototyping. However, limitations must be overcome.

First, the current study is based on a single example and qualitative evaluation, which limits the generality of the conclusions. Second, the use of a black-box generative model restricts precise control over the generated geometry, resulting in potential inconsistencies such as diffusion or misalignment of edited regions. Finally, the lack of explicit constraints across views may lead to ambiguity in fine-grained structures. Therefore, the results of this study must be interpreted as a proof-of-concept rather than a comprehensive evaluation. The performance evaluation must be conducted using a larger dataset, introducing quantitative metrics for reconstruction accuracy and edit consistency, and incorporating mechanisms for improving controllability and alignment across multi-view inputs.

5. Conclusions

In this study, we investigated the feasibility of interactive 3D vehicle modeling using multi-view 2D inputs and a generative AI model. By utilizing four-view images, we examined whether a plausible 3D model can be generated from limited structural information and whether local modifications in the input images are reflected in the resulting 3D model. The experimental results show that a coherent 3D vehicle model can be generated from a single set of four-view images, even when only line-drawing representations are used. Furthermore, localized color modifications applied to the input images were reflected in the generated 3D model, indicating the feasibility of region-based editing through multi-view inputs.

By combining structured multi-view representations with generative models, intuitive and controllable 3D modeling is conducted. However, the framework application is limited to a single example and qualitative evaluation. Future work will focus on extending the evaluation to a larger dataset and introducing quantitative metrics to assess reconstruction accuracy and edit consistency.