High-Fidelity Synthetic Data Generation Framework for Unique Objects Detection

Abstract

One of the key barriers to neural network adoption is the lack of computational resources and high-quality training data—particularly for unique objects without existing datasets. This research explores methods for generating realistic synthetic images that preserve the visual properties of target objects, ensuring their similarity to real-world appearance. We propose a flexible approach for synthetic data generation, focusing on improved accuracy and adaptability. Unlike many existing methods that rely heavily on specific generative models and require retraining with each new version, our method remains compatible with state-of-the-art models without high computational overhead. It is especially suited for user-defined objects, leveraging a 3D representation to preserve fine details and support integration into diverse environments. The approach also addresses resolution limitations by ensuring consistent object placement within high-quality scenes.

1. Introduction

Object recognition is one of the most popular practical applications of computer vision. Solving the problem of visual detection opens opportunities for optimizing many processes. In manufacturing, systems often identify and sort objects without human intervention [1]. The existence of autonomous vehicles is impossible without automatic object localization, which allows for the detection of obstacles and route planning. Object recognition systems help monitor important events or suspicious activities in the security sector, significantly enhancing monitoring efficiency. In healthcare, such systems assist in diagnostics. Optimizing the model development process for recognition will significantly save resources [2].

Synthetic data for training object detection models has attracted growing interest due to its scalability and flexibility. However, existing methods often suffer from adaptability, precision, or generalizability limitations.

Traditional simulation tools [3,4] offer scalable data generation but require complex scene setups and are limited to generic object categories. Copy-paste techniques [5,6,7] are simpler and produce pure consistency with background and appearance in terms of positioning, interaction with other objects, and lightning. GAN-based methods [8,9,10] provide realism but often require complicated training; hence, it is difficult to adapt them to new objects. Recent diffusion models [11,12,13,14,15] achieve high visual quality but still struggle with precise spatial control and object-specific generation, limiting their effectiveness for domain-specific tasks.

In contrast, the proposed method introduces a framework that leverages the strengths of current generative models—particularly diffusion models—while addressing their limitations in spatial control and fidelity. Our architecture integrates a 3D representation to preserve object-specific geometric features, allowing for consistent and accurate placement within complex scenes. After relevant views are sampled, context is generated using generative vision models.

By avoiding heavy retraining or architecture-specific modifications, our method remains compatible with new generative model versions as they become available, making it significantly more adaptable and resource-efficient than prior work.

The proposed solution holds practical value for developing object recognition models, as it provides a flexible and scalable way to obtain training data applicable in various fields—from industry to medicine and life safety. These changes will enhance production capacities, reduce energy resource consumption and material costs, and improve product quality [16].

On the other hand, it presents new challenges for society, such as improving workplace safety and adapting the workforce to new working conditions. It is also important to consider new technologies’ social and environmental aspects, such as their impact on employee health and well-being, environmental preservation, ethical issues related to artificial intelligence, and other concerns that require a balanced approach and careful investigation [17].

The relevance of this research is determined by the need to develop strategies to implement technological innovations effectively. These strategies must consider economic, social, and environmental aspects to ensure sustainable development and an improved quality of life.

This research aims to investigate existing image generation methods and propose a holistic approach for building recognition models based on synthetic training data. The contributions of this work include:

• Comprehensive analysis of current synthetic data generation methods for automated dataset creation.
• Introduces a diffusion-based pipeline that supports robust object localization with minimal real data, enhancing efficiency and reducing computational cost.
• This work validates the algorithm’s effectiveness on real-world data.

2. Related Works

Object recognition generally involves two main stages: object detection—locating all objects within a predefined set—and classification—assigning each detected object its correct label. Traditional approaches such as SIFT [18] and HOG [19] rely on handcrafted features; SIFT detects scale and rotation invariant key points, while HOG computes gradient histograms over subregions. These methods, however, require significant manual effort and often fall short when dealing with complex object variations. Early classification techniques like SVM [20] and Adaboost [21] built on these features but were limited by their simplistic representations. The advent of Convolutional Neural Networks [22] (CNNs) transformed object recognition by automatically learning hierarchical features. Starting from AlexNet [23] and evolving through architectures such as VGG [24] and ResNet [25], CNNs have dramatically improved both accuracy and efficiency. Modern object detection frameworks now fall into two categories: two-stage detectors (e.g., Faster R-CNN [26]) that first propose candidate regions before detailed classification and single-stage detectors (e.g., YOLO [27], SSD [28]) that perform localization and classification simultaneously, offering a practical balance between speed and precision. However, due to the empirical nature of these methods, high-quality training data are essential to achieve robust performance.

2.1. Synthetic Data Generation for Object Detection

Simulation-based approaches such as Unity Perception [4] offer scalable pipelines for generating large amounts of synthetic data through traditional 3D rendering. These methods leverage 3D scene setups to produce realistic visualizations; however, they require complex scene design and calibration, and they are typically limited to broad, general-purpose datasets featuring everyday objects like vehicles or pedestrians. As demonstrated in [3], the performance gains from synthetic datasets often scale non-linearly—especially when the generated samples lack the necessary realism or exhibit insufficient spatial variability.

Copy-paste techniques provide an alternative augmentation strategy by compositing object instances from one scene into another. Methods like those presented in [5,6] have shown measurable improvements in low-data regimes. Despite their simplicity and efficiency, these techniques do not synchronize background and object appearance. This difference leads to unrealistic object placements and reduced generalization, particularly when dealing with occlusions or non-standard viewpoints.

GAN-based approaches—including [8,9,10]—aim to generate synthetic training images via adversarial training. Although these models show promise in producing detailed images, they often struggle to maintain spatial consistency and preserve object fidelity in complex or industrial scenes. Their training processes are computationally intensive and require significant re-adaptation when extending to new object categories or domains.

Recent diffusion-based methods, such as DiffusionEngine [11], ODGEN [12], Gen2Det [13], and InstaGen [29], represent a noteworthy advancement in text-to-image synthesis by achieving high image quality. However, these models typically lack the precise spatial control to dictate object placement, scale, or orientation. Even advanced systems like GEODIFFUSION [14] and The Big Data Myth [15] rely on coarse structural guidance or necessitate domain-specific fine-tuning with real data, which limits their scalability in practical applications.

For product placement scenarios, approaches like [30,31] have explored reference-based inpainting. While these methods can insert products under generated masks, they are generally confined to advertising applications and do not support arbitrary, user-defined spatial control. In contrast, techniques such as [32,33]’s framework offer more straightforward yet effective strategies for maintaining object identity during image generation.

Overall, each synthetic data generation method category presents its advantages and limitations regarding scalability, realism, and spatial control. Our work addresses these challenges by proposing a novel, model-agnostic framework that integrates 3D reconstruction with controlled image generation. This framework ensures high visual fidelity, maintains precise spatial representation, and is well-suited for generating synthetic data for unique or custom object detection tasks.

2.2. Methods of 3D Object Reconstruction

One of Computer Vision’s most popular research directions is 3D object reconstruction. Modern 3D reconstruction methods allow for highly precise digitalization of objects and environments. This ensures their wide application in various fields, such as game development and industry. We will examine the main approaches to 3D reconstruction.Structure from Motion (SfM) [34] is a widely used method for reconstructing 3D geometry from 2D images taken from multiple angles. Instead of generating a complete 3D model, SfM creates a point cloud. Its simplicity and low computational requirements make it accessible, even with a basic camera or smartphone. However, the quality of the point cloud is highly dependent on the consistency and quality of the input images; poor image quality or inconsistent lighting can lead to errors in keypoint matching and, consequently, a lower-quality reconstruction. Figure 1 shows the basic idea of Structure from Motion (SfM) and other 3D reconstruction methods is to use multiple images taken from different angles.

NeRF (Neural Radiance Fields) [35] is a modern 3D reconstruction method that implicitly trains a compact neural network to encode scene geometry from multiple 2D images. The model predicts pixel values using a ray-based sampling process by specifying a camera position. NeRF excels at reconstructing complex, reflective, or refractive surfaces from limited input data, making it ideal for applications in virtual reality and digital art. However, its high computational demands and slow rendering prevent real-time visualization and require significant GPU resources.

The 3D Gaussian Splatting method [36] is a recent 3D reconstruction technique that offers high-quality, real-time rendering through a compact scene representation. Unlike NeRF, which relies on large neural networks and extensive training, Gaussian Splatting uses a set of 3D Gaussians—allowing for similar visual fidelity without heavy computational requirements or a GPU for visualization. The process starts with a Structure-from-Motion (SfM) reconstruction, which generates an initial point cloud that forms the basis for placing the 3D Gaussians. These Gaussians are then iteratively optimized using a differentiable rendering process similar to NeRF, where gradient backpropagation minimizes the difference between the predicted and original images. Each Gaussian is defined by its position, radii along three axes, color parameters, and transparency value [36]. Overall, the Gaussian Splatting method delivers exceptional detail and performance, making it an attractive option for computer vision tasks in complex real-world scenarios, including virtual and augmented reality applications.

3. Method

The method for generating synthetic data begins with the stage of 3D reconstruction of the object, which allows for the creation of a detailed three-dimensional model of the object. The previously mentioned SfM reconstruction method is used for this step. This approach produces a digital model that can generate various perspectives of the object. As a result, the obtained three-dimensional models can represent the object from different viewpoints, enabling it to learn to recognize the object from any angle, simulating various scenarios of its appearance in the real world.

3.1. 3D Reconstruction

A comparative analysis was conducted between Structure from Motion (SfM) and Gaussian Splatting. The objective was to determine the method that most effectively generates a 3D model of a selected object using a fixed set of input images.

The 3D reconstruction process employed in this study utilizes COLMAP’s incremental Structure-from-Motion (SfM) pipeline, followed by dense reconstruction and mesh generation. Sparse reconstruction produces point clouds of 10,000 to 100,000 points, depending on the object’s complexity and image quality. In this experiment, 50 images with a resolution of 1024 × 1024 were used, resulting in nearly 20,000 points produced after the first stage.

The official implementation of Gaussian Splating was used. No additional filtering was applied. Reconstruction was done on the same image set as in SfM.

Table 1. Objects for comparing 3D reconstruction methods.

	SfM	GSP3d
View 1
View 2
View 3
View 4

summarizes four representative reconstruction views. Gaussian Splatting successfully reproduced the object from training-set viewpoints 1 and 2. However, it introduced significant artifacts when rendering unseen angles, such as the sole or rear of the sneaker, as shown in views 3 and 4.

In contrast, SfM delivered less realistic images but maintained consistent reconstruction across all viewpoints. Given this consistency and acceptable quality, SfM is preferred for future reconstruction tasks where stability across views is critical.

3.2. Rendering View Sampling

Each training sample requires a camera view, but not all views are equally valid or meaningful. Objects appear more frequently from certain angles—such as front or top-down views—while others, like views from underneath or extreme side angles, are less plausible or useful. A semi-automated method for view sampling was introduced. Algorithm 1 demonstrates constrained view sampling on the unit sphere. The user sets sampling constraints in spherical coordinates. This allows for reproducible and flexible sampling of plausible viewpoints while maintaining a uniform distribution within the desired range.

Algorithm 1: Constrained View Sampling on the Unit Sphere

Input:  1. Number of views to sample: N  2. Azimuth angle range: θ ∈ [θ_min, θ_max]  3. Elevation angle range: φ ∈ [φ_min, φ_max] (measured from the positive z-axis) Output:  1. List of 3D unit vectors (x, y, z) representing sampled view directions Procedure:  For each of the N views, do:   Sample azimuth angle θ uniformly from [θ_min, θ_max]   Sample a uniform random number u ∈ [0, 1], then compute:     φ = arccos(cos(φ_min) − u * (cos(φ_min) − cos(φ_max)))   Convert spherical coordinates (θ, φ) to Cartesian coordinates on the unit sphere:     x = sin(φ) * cos(θ)     y = sin(φ) * sin(θ)     z = cos(φ)   Store the vector (x, y, z) as a sampled camera direction

3.3. Prompt Generation

A structured prompt generation method that uses a LLM (Large Language Model) was developed to enable the creation of contextually rich and precise scene descriptions. In our case, we applied OpenAI’s GPT-4o model. The guide prompt begins with a clear objective statement that specifies the central target object and the overall narrative goal, followed by guidelines outlining the scene context—such as location, time of day, and weather. This approach ensures that the resulting description is comprehensive. Finally, style constraints and example prompts are provided to guide the LLM toward producing coherent and context-rich narratives. The LLM is not forced to generate a target object-centric prompt because it will cause bias in target object locations. In contrast, all remaining objects are described in detail, and additional object categories may appear in the foreground to more closely emulate real-world scenarios.

3.4. Object Locations Sampling

In object detection tasks, the precise location of the target object within training samples is critical because many detection models are not translation invariant. To address this issue, we estimate the spatial distribution of the target object through a multi-step process. First, 1000 training samples were generated using our text-to-image Flux.1-dev model, guided by the prompt generation procedure. Figure 2 demonstrates the resulting spatial distribution of the generated samples.

Next, a text-based object detector OWL-ViT [37] was applied to these samples to provide rough localization of the target object. Images were then filtered based on a model confidence score to filter out generations without target objects. In instances where multiple objects were detected, all detected instances were kept and incorporated into the statistical analysis. Figure 3 presents a heatmap showing the distribution of shoe bounding boxes across the image frame.

3.5. Proposed Generation Method

The environment is generated after the object’s visualization is obtained, and the position is sampled from the estimated spatial distribution. This is done using a two-step generation process that uses FLUX.1 fill [38]; a diffusion model was used as the base model.

In the first step, a local context around the object is generated. The target object may rarely appear in the training data of the generative model, so it requires considerable “attention” for accurate generation. The generation quality is primarily limited by the object’s size in latent space. Therefore, in the first generation step, the target object occupies almost the entire image. This approach allows for the precise and accurate generation of the local context because of the object size in the model latent space.

Next, we place the generated image in a sampled location on the global context image with a sampled scale from the estimated spatial distribution. At this stage, the global context is generated. Here, the object is positioned in a larger environment corresponding to its natural surroundings. The generation of this step is performed using outpainting via FLUX.1 fill [38]. Figure 4 demonstrates a visualization of the proposed generation method. This method allows for gradually expanding the image around the target object and its local context, adding new context and details that match the style and texture of the selected environment.

The combination of these technologies—3D reconstruction and generative outpainting—creates high-quality synthetic data that feature details and a variety of perspectives. Including a wide range of contexts significantly improves the quality of model training, making them more adaptive and capable of achieving high accuracy in real-world conditions.

4. Experiments

4.1. Implementation Details

All components were implemented using Python 3.10 and the PyTorch 2.5.1 library, which are widely adopted as the current industry standard for deep learning and computer vision tasks. Considering the significant computational resources required for image generation using diffusion models, this research was conducted using an NVIDIA A40 graphics card. This GPU provides substantial computational power, enabling work with high resolutions of 1024 × 1024 pixels and supporting significant volumes of concurrent calculations necessary for complex image generation. Using a powerful graphics card significantly accelerates the generation process, reducing the execution time of each iteration and enhancing overall productivity.

Additionally, to flexibly manage computational resources and optimize development costs, the platform vast.ai [39] was selected. This platform allows for renting the necessary graphics cards under flexible conditions, enabling adaptation to the workload and minimizing expenses on cloud computing. By utilizing vast.ai [25], we could quickly scale computational power for testing and refining the model without compromising the quality or performance.

The Diffusers library is used for the generation process, which is a powerful tool for creating generative image models. This library offers ready-made modules for working with diffusion models, allowing specialists to adapt the model to the project’s specific needs without the necessity of building all components from scratch.

4.2. Synthetic Data Generation Methods for Comparison

We considered various approaches to generating synthetic data. Methods that require training an additional generative model were dismissed due to their resource intensity in time and computation. Training additional models such as GANs is unstable due to the need to balance the generator and discriminator depending on the complexity of the target data.

We utilized the classical copy-paste approach from [6] as a baseline. This method is highly efficient and does not require GPU acceleration, which enables rapid dataset generation at a minimal cost. In addition, the Gen2Det [13] framework was used for comparison. The original Gen2Det implementation employs an inpainting model that generates objects under a mask based solely on text descriptions, thereby lacking high-precision control over object placement. To align it more closely with our approach, we extended Gen2Det with the Paint-by-Example [40] method, which leverages a reference image of the target object for inpainting. We also evaluated another setup that uses the FLUX.1 fill model combined with an IP adapter [41] for enhanced visual control. Importantly, none of these methods require pretraining, and each is designed to preserve the identity of the target object. Moreover, precise bounding box extraction is a critical component of our pipeline, as all these approaches provide explicit control over the region in which the target object is generated, resulting in highly accurate labels.

Figure 5 shows comparison of synthetic training data generation method.

4.3. Results of Training the Detector on Synthetic Data

A comparative analysis of the training results of the models on datasets generated using the proposed method versus conventional approaches was conducted. The primary objective of the experiment was to assess the impact of the proposed methodology on the quality of the models in terms of various accuracy and adaptability metrics, as well as to investigate the dependence of these metrics on the volume of data used.

The test dataset was collected manually. Example images from the test set are shown in Figure 6. We had access to the target object, allowing us to capture it in various locations. In total, 200 photos of the sneakers were gathered in different environments. These data are sufficient for a quality assessment of the models’ accuracy within a realistic time investment.

Table 2. Synthetic data generation method comparison.

Metric	Copy-Paste	Gen2Det + Paint-by-Example	Gen2Det + FLUX IP Adapter	Proposed Method
Precision	88.7	47.2	80.3	93.5
Recall	85.4	71.9	75.9	90.3
F1 Score	87.0	56.9	78.0	91.8

demonstrates a synthetic data generation method comparison.

Three metrics were chosen to evaluate the performance of the models. Precision describes the percentage of the detected objects that are target objects. Recall indicates how many of the available objects were found. The F1 score is the harmonic mean of the precision and recall, providing a single number that reflects the model’s quality. The threshold value for IoU was selected manually and fixed at 0.8. This choice of metrics allows for a standardized and detailed assessment of the recognition model’s performance. All metrics are in percent from 0 to 100. These metrics are measured on our custom test set.

The effectiveness of the proposed method was tested and confirmed through an experiment. In this experiment, the model was trained on four datasets. A standard YOLO v5 [42] model architecture and identical training parameters for both datasets were used. This allowed for more objective results, which are presented in Table 2. The analysis of the results showed that the dataset generated using the proposed method achieves higher scores across all metrics (Table 2).

The reason for such performance is the nature of testing data. Often, an object appears under various lightning, as shown in Figure 6. The copy-paste approach does not change the lightning of the object; hence, it tends to fail when the object is under lightning, which significantly differs from the initial photo set used to generate training data.

The Gen2Det-based approaches face significant challenges in preserving the identity of the target object, particularly in the variant utilizing Paint-by-Example, as illustrated in Figure 5. This limitation negatively impacts performance, as the object detector converges less effectively when trained on noisy or inconsistent data.

This supports the hypothesis that the proposed methodology’s additional context and coherence enhance the model’s overall quality and ability to operate in real-world conditions.

4.4. Data Volume Influence

For a deeper analysis, we conducted an experiment to determine the relationship between accuracy and other metrics and the amount of data used. This experiment involved training models on different data volumes: 10%, 25%, 50%, 75%, and 100% of the initial amount for both dataset generation methods (

Table 3. Analysis of data volume influence.

Volume of the Dataset	Copy-Paste [F1 Score]	Our [F1 Score]
25%	81.7	85.9
50%	84.0	88.1
75%	85.5	90.7
100%	87.0	91.8

). Moreover, 100% corresponds to 3000 samples. The results showed that even with 50% of the data, the model trained on the dataset generated by the new method achieved similar or better metrics than the model trained on the full volume of data created using the traditional method. This indicates the effectiveness of the proposed method. While the traditional method demonstrated results close to this level with the full dataset, our findings confirm that the proposed method allows for competitive performance, even with reduced datasets on selected test sets. The reason is that the proposed generation method more effectively covered selected real-world test cases.

Because of the lower sim to the real gap between training and testing data, the proposed approach enables better detector convergence and reduces the need for training data on the proposed test set; other works also investigate the impact of the quality of training data [43,44]. This intuition applies in synthetic data because the data generation process aims to create samples that are as close as possible to real-world inputs. Hence, better similarity increases metrics and reduces the need for large datasets, which works because the increase in size adds samples that improve the overall coverage of real situations.

4.5. Generation Speed and Memory Usage

Memory usage and generation time are highly dependent on generative model selection. In our implementation, 48 GB of VRAM is required.

To estimate the spatial distribution of the target object, we generated 1000 samples using a diffusion model at 768 × 768 resolution with 20 sampling steps per image. This process took approximately 6 s per sample on an NVIDIA A40 GPU. Subsequently, we applied the OWL-ViT model to detect bounding boxes, which required about 0.5 s per image.

In the final synthetic image generation stage, the rasterization of object masks and scene elements took approximately 100–150 milliseconds per image. The inpainting process, performed using a guided diffusion model at 1024 × 1024 resolution with 30 sampling steps, required around 15 s per image on the NVIDIA A40. This inpainting stage is the most computationally intensive part of the pipeline and scales approximately linearly with the number of sampling steps.

5. Conclusions

This study presented a novel pipeline for generating high-fidelity synthetic datasets to train object recognition models, particularly when real data are limited or unavailable. The proposed method combines Structure from Motion (SfM) for robust 3D reconstruction with a two-stage diffusion-based generative model that includes local and global context generation. This hybrid approach ensures high visual realism, spatial control, and object identity preservation—key aspects for effectively training object detection models.

The experimental evaluation demonstrated that models trained on data generated by our method outperformed those trained using traditional techniques such as copy-paste and Gen2Det-based inpainting. Specifically, the proposed method achieved the highest F1 score of 91.8, surpassing both copy-paste (87.0) and the best Gen2Det variant (78.0). Additionally, the method showed strong performance even when trained on only 50% of the data, indicating its efficiency and potential for reducing labeling and data collection costs.

One of the limitations of the current pipeline lies in its reliance on large GPU memory for the inpainting stage, which may restrict its accessibility for some users. Also, further two-stage generation can be replaced with only one stage when the latent space resolution of generative models increases and will be able to preserve all object details while spending less time on generation.

As a next step, we plan to develop an approach that enables the generation of high-quality 3D object models from only a few input images [45,46,47,48,49,50,51,52]. This direction builds on recent advances in mesh and texture reconstruction, such as the Hunyuan3D framework recently released by Tencent. Incorporating such techniques will significantly simplify and accelerate the data generation process while preserving the geometric and visual consistency of the target object across all synthetic samples [53,54,55,56,57]. This improvement is expected to further enhance the efficiency of training datasets, especially for applications where obtaining high-quality image collections for reconstruction is impractical.