3D Human Reconstruction from Monocular Vision Based on Neural Fields and Explicit Mesh Optimization

Abstract

Three-dimensional Human Reconstruction from Monocular Vision is a key technology in Virtual Reality and digital humans. It aims to recover the 3D structure and pose of the human body from 2D images or video. Current methods for dynamic 3D reconstruction of the human body, which are based on monocular views, have low accuracy and remain a challenging problem. This paper proposes a fast reconstruction method based on Instant Human Model (IHM) generation, which achieves highly realistic 3D reconstruction of the human body in arbitrary poses. First, the efficient dynamic human body reconstruction method, InstantAvatar, is utilized to learn the shape and appearance of the human body in different poses. However, due to its direct use of low-resolution voxels as canonical spatial human representations, it is not possible to achieve satisfactory reconstruction results on a wide range of datasets. Next, a voxel occupancy grid is initialized in the A-pose, and a voxel attention mechanism module is constructed to enhance the reconstruction effect. Finally, the Instant Human Model (IHM) method is employed to define continuous fields on the surface, enabling highly realistic dynamic 3D human reconstruction. Experimental results show that, compared to the representative InstantAvatar method, IHM achieves a 0.1% improvement in SSIM and a 2% improvement in PSNR on the PeopleSnapshot benchmark dataset, demonstrating improvements in both reconstruction quality and detail. Specifically, IHM, through voxel attention mechanisms and Mesh adaptive iterative optimization, achieves highly realistic 3D mesh models of human bodies in various poses while ensuring efficiency.

1. Introduction

With the rapid development of Virtual Reality (VR) technology [1,2,3,4], 3D human reconstruction holds significant application value in the digital industries such as model assets, film production, military, and entertainment. Three-dimensional human reconstruction [5,6,7,8] is a crucial task in the fields of computer vision and graphics, with important research value in areas such as human behavior recognition, pose and capture, facial expression rendering, and clothing simulation.

Significant progress has been made in the field of 3D human reconstruction. As proposed by [9], the POP3D framework effectively addresses the longstanding challenges of generalization and fidelity in reconstructing from a single RGB image. However, the computation time associated with 3D model training increases significantly with the number of camera parameters. Currently, the framework requires approximately 7 h to reconstruct a single object using a single 3090 RTX GPU. There is also a global optimization method [10] that jointly learns skinning fields and surface normals in a canonical representation. This method effectively recovers surface details and supports the animation of human avatars in novel, unseen poses. However, to achieve high-quality textured 3D human model reconstruction, it relies on supervision from depth maps or scanned model assets.

Another novel approach is HumanNeRF [11], which delivers state-of-the-art results for free-viewpoint rendering of moving humans from monocular videos. By accurately modeling body poses and motion alongside a canonical optimization process, the method achieves high-fidelity results in this challenging scenario. Nevertheless, HumanNeRF models human shape and appearance in a pose-agnostic canonical space. To reconstruct models from images of humans in different poses, it incorporates human prior templates (e.g., animation skinning) and differentiable deformation and rendering algorithms to map the model to the posed space.

Among them, InstantAvatar [12] still needs improvement in terms of detail capture and reconstruction accuracy in complex scenes. Human as Points represents the human body as explicit point clouds and reconstructs 3D human shapes from single-view RGB images. While point-based representation offers great flexibility, it may fall short in achieving the fine detail resolution attainable by other methods, thus affecting the realism and fidelity of the reconstruction. Additionally, PIFu [13] and its extension, PIFuHD [14], are restricted in their widespread application due to high computational costs and input data quality requirements. HaMeR [15] introduces a self-supervised video framework that reduces hand-reconstruction error to 0.6 mm without requiring calibrated multi-view rigs, demonstrating that voxel-attention-style occupancy refinement can be directly ported to finger-level detail. MultiPly [16] jointly optimizes multi-person SMPL-X poses and shared neural radiance fields, enabling unconstrained outdoor scenes but at the cost of 4× inference time; extending IHM’s voxel grid to instance-encoded channels offers a lightweight route to the same multi-subject capability. Vid2Avatar-Pro [17] delivers high-fidelity avatars but needs 30 min and a heavy pretrained prior; StruGauAvatar [18] renders 60 fps with structured Gaussians yet yields bumpy, non-manifold surfaces; DressRecon [19] handles loose garments in 4D yet requires eight views; Surfel-GIR [20] offers relightable surfels but no watertight mesh. IHM keeps single-view input and outputs animation-ready meshes in 10 min, without universal prior pretraining.

From the analysis of these methods, we can see that although the combination of 3D models and images continues to improve in terms of accuracy and robustness, many methods still require high-performance hardware, and current technologies still perform inadequately in complex scenes or heavily occluded conditions. To reconstruct highly realistic 3D human models with arbitrary poses from monocular video, including realistic texture details such as clothing wrinkles in different poses, this paper proposes a novel Instant Human Model (IHM) generation method, combining the powerful flexibility of Neural Radiance Fields (NeRF) with the intuitive and controllable nature of explicit mesh modeling.

First, to more efficiently and accurately reconstruct and represent the 3D shape and texture of dynamic humans, the InstantAvatar method is used to learn the human body’s shape and appearance in the canonical space. Then, to optimize the original method of maintaining a voxel occupancy grid to skip sampling points in empty spaces, this paper introduces a voxel attention mechanism to enhance the importance of voxels during the reconstruction process, thereby improving reconstruction accuracy.

Next, the Marching Cubes algorithm [21] is used to extract a rough human mesh model from the enhanced voxel representation. At the same time, rendered images of the human body from different viewpoints are used to align the implicit and mesh models through model view projection (MVP) and mesh translation fields, further achieving adaptive iterative mesh optimization and learning high-quality texture maps. Given a canonical shape represented by an implicit function, we use IHM to define continuous fields on the surface, enabling detailed modeling of dynamic humans. By controlling SMPL parameters, arbitrary human pose meshes are quickly reconstructed, and specular and texture maps of the corresponding mesh model are learned.

Compared to methods relying solely on single images or depth map sequences, IHM combines the flexibility of NeRF in shape representation with the advantage of explicit mesh modeling in detail expression, achieving high-quality textured 3D human model reconstruction without relying on depth maps or scanning model assets for supervision. IHM is targeted at niche AR/VR scenarios where “30 s mobile phone selfies can drive Avatars within 10 min”. The reduction in user waiting time brought by a 36× speedup ratio is more valuable than a 0.1% increase in SSIM.

Experimental results show that IHM outperforms baseline methods across multiple datasets, particularly in applications such as shape reconstruction, animation generation, and texture transfer. The following sections will discuss these findings in four parts.

2. Materials and Methods

Neural Radiance Fields (NeRF) [22] is a method for representing the 3D geometry and appearance of scenes using neural networks. By inputting multiple 2D images from different viewpoints, NeRF can generate images from new perspectives. Training NeRF typically requires a large amount of paired data, including multi-view images and their corresponding camera parameters [23,24]. Research has shown that data diversity is crucial for model learning; a lack of sufficient training samples can lead to poor reconstruction results, especially in complex scenes. Recent works aimed at improving NeRF [25,26,27,28,29]. For example, NeRF++ attempts to address some challenges posed by complex environments (like outdoor scenes) but still operates under the assumption that multiple high-quality views of the scene are available.

With the development of Virtual Reality (VR) and Augmented Reality (AR) technologies, the demand for high-quality 3D human models has been increasing. Traditional 3D human reconstruction techniques [30] typically rely on explicit geometric modeling or multi-view stereo methods to reconstruct 3D models through images from multiple viewpoints. For example, photometric stereo methods [31] infer an object’s 3D structure by analyzing variations in illumination. While these methods excel at reconstructing static objects, they often require very precise camera parameters and complex post-processing for dynamic targets like humans.

In recent years, deep learning-based methods have gained widespread application in the 3D reconstruction field [32,33], including techniques based on Convolutional Neural Networks(CNNs) [34] and Generative Adversarial Networks (GANs) [35]. These methods typically learn an end-to-end mapping from images to models, generating 3D shapes directly from 2D images without the need for explicit geometric modeling. However, they often require a large amount of labeled data for supervised training and may produce 3D models that are blurry or low in detail.

Compared to traditional methods, a significant advantage of our proposed Instantaneous Human Model Generation Method is that it can generate 3D models from 2D video data without supervision. This flexibility allows it to operate effectively without a large amount of paired data, adapting to the difficulties of data collection in real-world scenarios. Our proposed instantaneous human model generation method makes significant improvements over existing work, particularly in terms of being unsupervised and utilizing multi-view implicit rendering, providing a more flexible and efficient solution for 3D human reconstruction. This method also shows a significant improvement in rendering quality, especially in handling dynamic poses and generating high-quality textured mesh models.

A flowchart for the Instant Human Rendering Method is shown in Figure 1. Given a monocular camera video sequence frame and the corresponding SMPL model parameters, IHM uses linear blend skinning in the first stage to transform sampled points from the canonical space to the deformed space. Then, these sampled points are input into a neural radiance field based on multi-resolution hash encoding to learn the volume density and color of the sampled points. The volume density is further input into a voxel attention mechanism to optimize the density grid in the canonical space, achieving accelerated convergence and rendering effects. In the second stage, a coarse mesh model in a specified pose is exported using the Marching Cubes method. By employing model-view projection and introducing a translation vector, the rendered image set and the mesh model in the corresponding pose are aligned to obtain the mesh point cloud under a specified view. This point cloud is then input into the neural radiance field based on multi-resolution hash encoding to render the projected image. The difference between the corresponding rendered image pixels in the first stage and the current rendered pixels is minimized. By translating the mesh grid vertices and eliminating or subdividing triangular facets, the projection error of the mesh model is optimized, making the mesh grid model more refined during the iteration process. After convergence, a high-fidelity texture map is generated.

2.1. Effective Dynamic Neural Radiance Fields

In the first stage, the human body shape and appearance in canonical space are modeled using a radiance field $F_{\sigma ,c}$, which predicts the density $\sigma$ and color c for each 3D sampling point in canonical space:

$$F_{\sigma ,c}\left(\mathbf{x}\right)\to \sigma ,c$$

(1)

here, Instant-NGP is used to parameterize $F_{\sigma ,c}$, achieving fast training and inference speeds by storing feature grids of different coarse scales using a hash table. To predict the texture and geometric properties of a query point in space, they read features at adjacent grid points and perform trilinear interpolation, then concatenate interpolated features from different levels. The concatenated features are finally decoded using a shallow MLP. To create animations and learn from posed images using articulation radiance fields, a deformed radiance field $F_{\sigma ,c}^{\prime }$ needs to be generated in the target pose:

$${F^{\prime }}_{\sigma ,c}\left({\mathbf{x}}^{\prime }\right)\to \sigma ,c$$

(2)

It outputs the color and density for each point in pose space. In this chapter, the skinning weight field $w=\left\{w_1,\dots ,w_{nb}\right\}$ is used in canonical space to model articulations, where nb is the number of bones in the skeleton. To effectively reduce computational costs [36,37], a low-resolution voxel grid is used to represent the skinning weight field. The value of each grid point is determined as the skinning weight of its nearest vertex on the SMPL model. With this canonical skinning weight field and the target bone transformation matrices $B=\left\{B_1,\dots ,B_{nb}\right\}$, a point x in canonical space is transformed into deformed space $x^{\prime }$ through linear blend skinning, as follows:

$$x^{\prime }={\sum }_{i=1}^{n_b}{w}_i{B}_{i}x$$

(3)

the corresponding canonical space point for a point $x^{\prime }$ in pose space is defined by the inverse linear skinning equation, establishing a mapping from a point $x^{\prime }$ in pose space to its corresponding point in canonical space ${F^{\prime }}_{\sigma ,c}\left(x^{\prime }\right)=F_{\sigma ,c}\left(x^{*}\right)$

2.2. Voxel Attention Mechanism

Unlike InstantAvatar, which directly uses the voxel density grid, this method introduces a voxel attention mechanism to enhance the optimization effect of the SMPL voxel density grid in standard space. Referring to the empty space skipping strategy [12], during training, the cost of constructing an occupied grid in each training iteration is no longer negligible. To avoid this overhead, a single occupied grid is constructed for the entire sequence by recording the union of occupied areas in each individual frame. Specifically, at the beginning of training, a $64\times 64\times 64$ voxel occupancy grid in Apose pose is initialized in normalized space, and its density is queried from the pose radiation field, which is then input into the voxel attention module to obtain new voxels, updated once per iteration by taking the moving average of the current occupancy value and the density queried from the pose radiation field in the current iteration. In this space, the global orientation and translation of the voxel occupancy grid are decomposed so that the union of occupied spaces is as tight as possible, further reducing unnecessary queries.

$$v_p^{\prime }=V_{att}\left(F_{\sigma ,c}^{\prime }\left(v_p\right)\right)$$

(4)

the pose radiation field $F^{\prime }\sigma$, cpredicts the sampled point density and color, which are rendered into a new view through volume rendering. Given a pixel, a ray $\mathrm{r}=\mathrm{o}+\mathrm{td}$ is cast, where o is the camera center and d is the ray direction. Points are sampled along the ray based on N intervals ${\left\{{\mathrm{x}}^{\prime }\mathrm{i}\right\}}^N$ between the near and far planes. By mapping ${\left\{{\mathrm{x}}^{\prime }\mathrm{i}\right\}}^N$ back to canonical space and querying from the corresponding NeRF model $F_{\sigma ,c}$, the color and density of each point are queried from the pose radiation field $F_{\sigma ,c}^{\prime }$. Then, the brightness and density of the queried points are accumulated along the ray to obtain the pixel color C,

$$C={\sum }_{i=1}^N{\prod }_{j<i}\left(1-a_i\right)c_j,with{a}_i=1-exp\left({\sigma }_i{\delta }_i\right)$$

(5)

where $\delta \mathrm{i}=∥{\mathrm{x}}^{\prime }\mathrm{i}+1-{\mathrm{x}}^{\prime }\mathrm{i}∥$ is the distance between sampled points. Each attention weight ${\alpha }_v$ is computed by a 2-layer MLP that takes three inputs: (i) the current voxel density ${\sigma }_v$; (ii) the Euclidean distance from the voxel center to the nearest SMPL surface; (iii) the frequency of occupancy across the temporal window. The MLP outputs a scalar in $[0,1]$, which is used to update the voxel occupancy via an exponential moving average with decay $0.95$ and learning rate $1\times {10}^{-4}$ for 5000 iterations.

2.3. Mesh Adaptive Optimization and Texture Generation

In the Figure 2, to render an image, the mesh is rasterized, 3D positions are properly interpolated into image space, and joint optimization is continued for pixel color loss. The coarse mesh extracted from the density field by Marching Cubes often has defects. These defects include inaccurate vertices and densely and uniformly distributed triangular facets, leading to huge disk storage and slow rendering speed. The goal of this chapter is to recover a fine mesh similar to an artificial mesh by refining vertex positions and facet density.

Given an initial coarse mesh $M_{\mathrm{coarse}}=\mathcal{V},\mathcal{F}$, a trainable offset $\Delta v_i$ is assigned to each vertex $v_i\in \mathcal{V}$. We use differentiable rendering [38] to optimize these offsets through backpropagation [39] of the image space loss gradient. Additionally, during experimentation, it was found that there is a certain displacement deviation between the implicit NeRF model and the mesh model. Therefore, we propose an overall mesh offset $\Delta \mathcal{V}=\Delta {\overline{v}}_i$ to align the implicit model with the explicit model, where $\Delta {\overline{v}}_i$ represents the average offset of all vertices.

During the training process, we reproject the 2D pixel rendering errors onto the corresponding mesh faces and accumulate triangular face errors. After a certain number of iterations, all triangular face errors $E_{face}$ are sorted to determine two thresholds:

$$\begin{array}{c}{e}_{\mathrm{subdivide}}=\mathrm{percentile}\left(E_{\mathrm{face}},95\right)\hfill \\ e_{\mathrm{decmiate}}=\mathrm{percentile}\left(E_{\mathrm{face}},50\right)\hfill \end{array}$$

(6)

triangle faces with errors above $e_{\mathrm{subdivide}}$ undergo midpoint subdivision [40] to increase triangle density, while those with errors below $e_{\mathrm{dicimate}}$ are decimated and remeshed to reduce triangle density. After updating the mesh, vertex offsets and triangular face errors are reinitialized, and training continues. This process is repeated multiple times until the second phase is complete.

To export a textured surface mesh compatible with commonly used 3D hardware and software, a refined surface mesh $M_{fine}$ is obtained after the second phase of training, but the appearance is still encoded in the 3D color mesh. To extract the appearance as texture images, UV coordinates for $M_{fine}$ are first unfolded using XAtlas [41]. Subsequently, the diffuse color $cd$ and specular features $f_s$ of the surface are baked into separate images: a diffuse map $I_d$ (RGB texture) and a specular map $I_s$ [42], as shown in Figure 2. The two MLPs are consistent with those in Instant-NGP.

2.4. Training Loss

This term simply minimizes the pixel-wise color error between the rendered and the ground-truth image. We train the current model by minimizing the error between the rendered image pixels C and the corresponding real image pixels $C_{gt}$. Here, the more robust Huber loss $\rho$ is chosen:

$$L_{rgb}=\rho \left(∥C-C_{gt}∥\right)$$

(7)

to suppress floating semi-transparent artifacts, we penalize any deviation of the rendered silhouette from the foreground mask. Additionally, to reduce floating artifacts in space, we calculate the loss between the transparency $\alpha$ value of the rendered 2D image and the transparency ${\alpha }_{gt}$ of the human body mask in the dataset.

$$L_{\mathrm{alpha}}=∥a-a_{gt}∥$$

(8)

we encourage the density field to converge to a binary inside/outside decision, thereby eliminating ghost voxels. Following LOLNeRF [43], we also add further regularization to encourage the NeRF model to predict solid surfaces:

$$L_{\mathrm{hard}}=-log\left({exp}^{-\left|a\right|}+{exp}^{-|a-1|}\right)+\mathrm{const}$$

(9)

where const is a constant to ensure that the loss value is non-negative. Encouraging solid surfaces helps speed up rendering by terminating rays early when the cumulative opacity reaches 1.

Previous methods used to learn human avatars [38,39] typically encouraged models to use the SMPL body model as a regularizer to predict zero density for points outside the surface and solid density for points inside the surface, reducing artifacts near the body surface. However, this regularization makes strict assumptions about body shape and does not generalize well to loose clothing. Voxels outside the SMPL hull are forced to zero density, preventing shape leakage. We use a voxel attention mechanism to optimize the prediction of the occupancy grid, which is beneficial for better estimating human and clothing shapes, and defines an additional loss $L_{reg}$ that encourages zero density within blank cells of the occupancy grid:

$$L_{\mathrm{density}}=\left\{\begin{array}{c}\left|\sigma \right(\mathrm{x}\left)\right|\mathrm{x}\mathrm{is}\mathrm{located}\mathrm{in}\mathrm{a}\mathrm{blank}\mathrm{area}\hfill \\ 0\mathrm{others}\hfill \end{array}\right.$$

(10)

to prevent geometric shape mutations, Laplacian [44] smoothing loss $L_{smooth}$ is applied to the mesh during the second stage of training in this chapter. Finally, we apply a standard Laplacian term to keep the mesh surface smooth. Additionally, we regularize the overall mesh offset and independent vertex offsets using L2 loss:

$$\begin{array}{c}{L}_{\mathrm{offset}1}={(\Delta V)}^2\\ L_{\mathrm{offset}2}=\sum _i{\left(\Delta v_i\right)}^2\end{array}$$

(11)

where $L_{offset1}$ effectively aligns the NeRF model with the Mesh grid, and $L_{offset2}$ ensures that Mesh vertices do not deviate excessively from their original positions.

3. Results

To validate the effectiveness of the proposed method, we conducted experiments on multiple datasets, comparing the synthesis effects of new perspectives and poses of the human body. Experimental results show that our model outperforms baseline models on most tasks.

3.1. Datasets and Baseline Methods

PeopleSnapshot We conducted experiments on the PeopleSnapshot [45] dataset, which contains videos of humans rotating in front of a camera. This section follows the evaluation protocol defined in Anim-NeRF [46]. Since the pose parameters provided in this dataset are not perfect and do not always align with the images, while Anim-NeRF optimizes the human poses of training and test frames, we use the optimized human pose parameters to train the model and freeze them during training for a fair comparison.

NeuMan Due to the limited pose variations (self-rotation) of the human body during movement in the PeopleSnapshot dataset, to evaluate performance on more challenging test poses, this section also uses the Neuman dataset [47] in real-world scenarios for training. NeuMan provides large pose sequences in real scenarios, which can fully expose the influence of SMPL pose errors on reconstruction, and thus is used to evaluate the robustness of IHM. During the training process, we unify the partitioning strategy of the Neuman dataset to ensure a fair comparison between the current method and baseline methods.

Our method also supports creating custom datasets [47] for human reconstruction by following these steps: First, convert the captured real videos into image frame collections, then use detectron2 for image segmentation to obtain masks. Next, perform sparse 3D scene reconstruction based on colmap to obtain sparse point clouds. Establish dense correspondences between pixels and 3D object geometry using densepose_ rcnn. Use openpose for human keypoint detection. Perform monocular depth estimation with BoostingMonocularDepth. Use ROMP for the SMPL human body model parameter encoding, SMPL alignment, and contour optimization.

Meanwhile, the following methods are considered as baselines:

Anim-NeRF [46] This baseline uses an MLP-based NeRF to model human shape and appearance in canonical space. Given a pose, they first generate an SMPL body in the target pose. Then, for each query point in deformation space, its corresponding skinning weight is defined as the weighted average of the skinning weights of its K nearest vertices on the SMPL pose mesh. Finally, through skinning weights, query points can be transformed into canonical space according to inverse LBS.

Neural body [48] This baseline learns a set of latent codes anchored to a deformable SMPL mesh. These latent codes deform with the SMPL mesh and are decoded into radiance fields for different poses.

InstantAvatar [12] This baseline uses a multi-resolution hash-encoded neural radiance field to learn human appearance, combined with an SMPL voxelized density grid, and employs an empty space skipping strategy to achieve rendering acceleration.

3.2. Experimental Setup and Training Details

The proposed method was trained on a single NVIDIA A6000 GPU. To ensure fair comparisons, the data processing scripts provided by InstantAvatar were used to preprocess the original PeopleSnapshot and Neuman monocular 2D video datasets for training. The same parameters were applied across different methods to set up the training and testing datasets. Due to variations in video lengths across different sub-datasets, the division parameters also differ, as detailed in the

Table 1. The division parameters.

	Train			Test
	Start	End	Skip	Start	End	Skip
Male-3-casual	0	455	4	456	675	4
Male-4-casual	0	659	6	660	872	6
Female-3-casual	0	445	4	446	647	4
Female-4-casual	0	335	4	335	523	4
bike	0	104	1	103	104	4
seattle	0	37	1	36	37	4
lab	0	102	1	101	102	4
citron	0	103	1	102	103	4
jogging	0	42	1	41	42	4
parkinglot	0	41	1	40	41	4

.

3.3. Model Evaluation Metrics

To evaluate the appearance quality of the reconstructed avatar, we animate and render the reconstructed model using test frame poses from the PeopleSnapshot and Neuman datasets, respectively. We calculate the Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity Index Measure (SSIM) between the rendered images and real images to quantify the reconstruction effect. As shown in

Table 2. Comparison of results on the representative PeopleSnapshot sub-dataset.

	Male-3-Casual		Male-4-Casual		Female-3-Casual		Female-4-Casual
	PSNR↑	SSIM↑	PSNR↑	SSIM↑	PSNR↑	SSIM↑	PSNR↑	SSIM↑
Neural Body	24.94	0.9428	24.71	0.9469	23.87	0.9504	24.37	0.9451
Anim-NeRF	29.37	0.9703	28.37	0.9605	28.91	0.9743	28.9	0.9678
InstantAvatar	29.65	0.973	27.97	0.9649	27.9	0.9722	28.92	0.9692
Our	29.9	0.974	28.19	0.9661	28.7	0.9737	29.57	0.9715

and

Table 3. Comparison of results on the Neuman dataset.

	InstantAvatar		Ours
	PSNR↑	SSIM↑	PSNR↑	SSIM↑
bike	24.18	0.9498	24.78	0.9515
seattle	26.57	0.9674	26.74	0.9685
lab	27.34	0.9731	27.41	0.9732
citron	24.95	0.9491	25.01	0.9526
jogging	23.4	0.9338	23.74	0.9368
parkinglot	24.14	0.952	24.13	0.9547

, the images generated in this section are significantly better than those of Neural Body [48] and achieve comparable quality to InstantAvatar.

The training duration and computational resource cost of our method are comparable to those of InstantAvatar. High-quality rendering results can be generated on a single A6000 in just 1 min, significantly faster than the multi-hour training requirements of other methods like Anim-NeRF [46] and Neural Body [48]. Meanwhile, our method can export high-fidelity textured mesh models in arbitrary poses within 30 min. Anim-NeRF achieves a marginally higher SSIM (0.0006) on Female-3-casual. However, IHM maintains an advantage for real-time applications due to its significantly lower inference time (5 s/frame with poses frozen) compared to Anim-NeRF’s 180 s/frame, which requires test-time pose optimization. Due to the lack of real images and mesh data for new poses, this section only provides a qualitative evaluation of new pose synthesis and mesh generation. Figure 3 illustrates the process from inputting a monocular 2D video to implicitly rendering and generating a textured, high-quality human mesh model.

To probe robustness beyond the clean studio sequences, we additionally evaluated IHM on three challenging scenarios from the NeuMan dataset: (i) severe occlusion (parking lot scene with pedestrians); (ii) extreme pose (jogging with 90° hip flexion); (iii) night-time complex lighting (bike scene with flickering street-lamp). While IHM preserves global body shape, we observe slight texture blurring under low-light and minor finger artifacts under heavy occlusion, indicating room for future improvement.

Figure 4 demonstrates a comparison of the rendering effects between the current method and InstantAvatar. Compared to InstantAvatar, our method can effectively reduce floating artifacts and improve detail performance, such as the details of the palm in Figure 4. This benefit is attributed to the further application of the voxel attention mechanism, which optimizes the SMPL voxel occupancy grid in canonical space and improves the accuracy of the transformation from canonical space to pose space, thereby enhancing image quality and strengthening robustness and rendering effects in complex real-world scenarios.

3.4. Ablation Experiment

This section conducts an ablation study on the geometric optimization stage. Specifically, we compare the complete model with variations that exclude mesh translation regularization or the iterative mesh refinement process. The results indicate that:

After removing mesh translation regularization, due to significant translational errors between the mesh projection and the image under the supervised camera pose, the resulting surface mesh exhibits poor rendering quality. Irregularities and severe vertex deviations make the model very rough. Additionally, because the iterative mesh refinement process cannot adequately handle such irregular surfaces, the mesh size increases. These irregular surfaces also lead to chaotic textures.

When iterative mesh refinement is removed, the triangle density tends to be uniform, resulting in increased mesh size and slightly poorer rendering quality. This is because the facet density cannot adaptively adjust based on the reprojection rendering error.

However, by utilizing translation vectors, our current method can learn a smooth and closed mesh model.

Table 4. Statistics of the mesh grid for Male-3-Casual before and after optimization.

	Triangular Patch	Vertex	Disk Usage
Before mesh optimization	785,896	400,355	14.3 MB
after mesh optimization	21,035	63,105	2.07 MB

and Figure 5 demonstrate the effects of our method before and after mesh optimization. There is a significant reduction in mesh vertices and triangles, along with a notable decrease in disk memory usage.

4. Discussion

4.1. Metrics

This paper proposes a novel 3D human reconstruction method called the Instant Human Model (IHM) generation method. By integrating 2D video data into the canonical space, it generates a rough, pose-controllable 3D human model without any supervision. The method then aligns the rough mesh using multi-view implicit rendering images and learns texture maps while adaptively iterating to optimize mesh vertices and triangles, enabling the export of human models in arbitrary poses. Compared to baseline methods, this algorithm not only enhances rendering quality but also achieves high-quality textured mesh model generation.

4.2. Limitations

Table 5. Pros and cons of the proposed Instant Human Model (IHM).

Module	Advantage	Disadvantage
Voxel Attention	Captures finger-gap/cloth wrinkles	GPU memory +15%
Mesh Iteration	70% fewer faces, 10× storage save	May produce non-manifold edge
Depth-free	Selfie video only, no extra sensor	Loose skirt can stick to legs
Hash-NGP backbone	5 s/frame inference on RTX 4090	SSIM gain marginal (+0.1%)

summarizes the principal strengths and limitations of the Instant Human Model (IHM) as observed in our experiments. Each row pairs a key component with its main benefit and its most relevant drawback. Overall, the table shows that IHM is best suited for scenarios where fast, single-view acquisition and compact, animation-ready meshes are required, and where the listed drawbacks (e.g., extra GPU memory or occasional topological artifacts) are acceptable.

5. Conclusions

In contrast to traditional photography methods that rely on heavy equipment, the technique described here reduces equipment dependency and lowers the cost and complexity of data acquisition. This advantage allows the method to quickly construct personalized 3D models under low-cost conditions, potentially broadening the application scope of this technology significantly.