Abstract
Dynamic geometries acquisition of human is one of the most popular topics in the fields of computer vision and computer graphics. This paper presents the related techniques on rapidly generation of human model sequences and makes skinning animation using single Kinect. In specified applied conditions, it has good animation effects and interaction functions. As the comparatively low request on resolution by animation, using the Kinect can greatly save human body animation production time and reduce production cost.
1. Introduction
With the development of the technology of Virtual Reality and the network technology, the research and application of realistic 3D human body model have been more and more important in the virtual reality system. At present, there are mainly two methods to obtain high quality 3D human body models [1]: 1) draw 3D human body models by using some 3D modeling software, such as Rhinoceros 3D, Poser, Maya, 3D Max; 2) obtain geometric data from the human body surface by using the 3D scanner. It is beautiful to draw models by using software. But it is complex and difficult to operate for the common users in a short time. The human body model obtained by using 3D scanning equipment has more comprehensive data and rich texture. But it needs the specialized equipment to scan, and also the equipment not only is expensive but also has slower scanning speed.
The appearance of the Kinect accelerates the process of the rapidly established 3D human body model and also increases the popularity of the application of 3D human body model. The Kinect reduces the price of producing high quality model and greatly shortens the development cycle of the products. So it can be quickly applied to the real-time virtual reality systems. Human animation is one of the most important parts of the computer animation and interactive design [2,3]. The capturing technology can produce the real-time animation, which solves the complex adjustment process of software on human body animation of 3D human animation. However, most of the capturing equipments have quite expensive price [4].
In this paper, our work is based on the completion of the rapid modeling use one single Kinect to capture data of human bones, using mature skin technology to drive human model and human body output skin animation. The framework consists of four phases: Firstly, human geometries are extracted from the given color, depth and skeleton data captured by the Kinect; Secondly, borderline removing, noise smoothing, point cloud matching and merging are performed on the human geometries and a mesh model is reconstructed based on RBF fitting; Again, the human mesh is further refined; Finally, double quaternion skinning is employed to interactively generate sequences.
2. Related Work
In computer graphics, there are already some researches on Kinect to obtain the human body model. Tong et al. [5] used 3 Kinects to obtain the data of human body model and propose a human body reconstruction method based on Kinect. Zheng et al. [6] proposed a method that uses the Kinect to obtain the 3D human posture to be fitted to get the human body model. Zhou et al. [7] used the iterative closest point algorithm and variable model to match by using 2 Kinects and fit the point cloud data scanned to generate 3D human body model. This method has some defects when it is used for noise processing so that the quality of 3D human body model will be affected. Song et al. [8] used Kinect to scan the human body data and aimed at gaining the human depth data so that they can further construct the point cloud data on the surface of human body.
Recently, KinectFusion which is an open source project developed by Richard [9] and Microsoft is being very popular for using Kinect to construct model rapidly. It uses mobile scanning mode to scan the objects by using the handheld camera equipment (such as Kinect). At present, the KinectFusion project can derive .stl and .obj format data by human body modeling. However, these two kinds of human body model is relatively rough and has noise, moreover, the generated human body model has many non-endemic area composed of discontinuous small grids. Cui et al. [10] used the data scanned by single Kinect to make the reconstruction of human body model. Since this method needs relatively long time to post process the model, Wang et al. [11] adopted Kinect to scan the whole human body and aligned the scanning point cloud data to decompose into the corresponding region. They matched the human body database by using the posture to generate 3D human body model. Zwicker et al. [12] also used the method of global alignment to solve the larger deformation problems of scanning data, but only had a better effect on the jagged registration. Alexander et al. [13] developed SCAPE model library [14], capturing the contour of the human body by using Kinect to estimate the parameter of the human body model. The disadvantage of this method is that the scanned human needs to wear tight clothes.
These methods by using Kinect to build human body model still needs to be mature. Presently, the algorithms of human body modeling by using Kinect are still required to rotate. The efficiency of generating the model is relatively slow which leads to the applications in real-time virtual reality systems are still inconvenient. Besides, the scanning also needs time of a certain length to guarantee the adequacy of data. Thus, it is very necessary to develop a fast human body modeling algorithm and make the animation to be convenient.
3. Rapid Human Body Modeling
The process of using Kinect to obtain the raw data is the first step to establish the model. The method of the acquisition of the raw data directly affects the complexity of the algorithm.
3.1. Human Data Processing
From the camera of Kinect, the color data and depth data in the scene, and also the user’s ID and the data of human bones can be fetched. Further, the human body geometry, texture data and other information can be obtained from them. The scanning data of the entire human body can be completed usually in 6 10 seconds. The extraction of the human body data is mainly that the human body depth data is segmented from the complex 3D scenes. The depth image data of Kinect has two formats. Different users have different IDs and the high 13 bits represent the distance from the pixel to the origin coordinate of Kinect and the low 3 bits record the ID of the pixel which belongs to a human. When the users ID is zero, it is said that this ID is “non-user”. The effective distance of Kinect is 0.8 m–4.0 m.
In this paper, we extract the contour data of the human body by identifying the user’s ID. The 3D data of the contour inside is the data needed and should be saved for post processing of the rapid modeling. Figure 1 shows the human body data which is segmented according to the depth information and the information of the user’s ID. After the contour of the human body is obtained, the depth data of the human body model can be segmented by the aligning the depth data and the colorful data of human body.
Figure 1.
Human body depth data extraction.
Figure 2.
Model segmentation.
The raw data obtained from the Kinect has about 180 frames, including the depth information of human body and texture information. Because of the complexity of the scene, there are many isolated graphic areas in the obtained data. As Figure 3 shows, the data segmentation algorithm of the previous section can be used to extract the human body data, greatly reducing the complexity of dealing with complex background. But if the objects having the shape of the human body or having movement interference appear in the scene, the data cann’t be segmented, causing noise region or geometric debris existed in graphics. The main reason is that the result is caused by the objects similar with the human body or actions, not the background factors.
Figure 3.
Front and back of the point cloud data map.
Therefore, in order to reduce the complexity of the model construction and the time of the processing, the noise data of the obtained data needs to be removed before the model is constructed to reduce the amount of data of the model and to simplify model processing. By using the OpenMesh [15] database, we use the connected region to compute grid data and the human body data region of section to match and get the data region of the human body model.
3.2. Human Body Model Alignment
In order to quickly generate models, we need the data for each frame of fast registration and alignment. In addition, the data on the obverse and reverse should be quickly matched. With the development of relatively inexpensive technique for three-dimensional capture devices, such as Kinect and Asus Xtion Live, alignment algorithms have become a popular research tool. Align includes three-dimensional space on a two-dimensional image and point cloud data. For modeling based on human data, we focus on the alignment algorithm for three-dimensional space, namely the multi-frame scanning data space Transformation or deformation processing to reach the corresponding data points matching each other. In order to keep the consistency of the model after alignment, we should minimize the energy after alignment, thus to obtain the high accuracy of the matching. The energy equation is described as Ereg = Em + Ep. Where Em represents the distance from the target point to source point, Ep represents the target point and the source point by the transformation or deformation energy after the match. Since the current alignment algorithms are relatively mature, the energy computation algorithm in [16] is adopted to scan the rigid alignment part, and the non- rigid part is calculated by using the energy formula in [17].
3.3. Model Merging
One innovation in this work is to just use the Kinect to scan the front and back of human data, and the body does not need to do complex movements, therefore, we need to merge the data on the obverse and reverse. However, the merging will usually inevitable appearing the incomplete match part, so that it will appear incomplete mesh like holes and cracks. Thus, we must check the integrity of the model, patching holes and cracks and other algorithm design and implementation.
The data of human body obtained through Kinect are reversed as the steps: First, the acquired data are preprocessed, shown in Figure 4. Then the rough model after pretreatment is meshed, shown in Figure 4. Finally, OpenMesh library is used to mark merge area, as shown in Figure 5 (the red area labeled as part of the merger area). The merge area is in grid data without the side edge corresponding to the half of the opposite direction.
Figure 4.
Merging of the human body.
Figure 5.
System design flowchart.
By the preprocessing above, we can obtain simple human model after merging. Through the alignment and grain processing on a series of simple models, it is able to depict the texture details of the real model. Due to there are some holes inevitably arose during the scanning, and also the location of these holes appear mostly around the human body model, we need to handle of stitching holes and repair zigzag linear place in the model.
Repairing algorithm mainly include surface oriented algorithms and volumetric algorithms. In the former one, direct input mesh data is detected for the presence of holes, and the topology information is used to repair the patch mesh. In the later one, the input model is indirectly expressed as a volume element, and then the holes is detected and repaired. The repair algorithm for the human body model in this paper is based on the usage of surface RBF algorithm [18], which can quickly realize holes repair. After model merging, the OpenMesh library is applied to test the holes information on the mesh and then the RBF Algorithms works to quickly repair the holes of the human body model.
4. Skinning Animation
Skinning technology is through the control handle to control the movement or deformation of the body skin. Currently, there are some common types of body control, such as point control, bone control, cage deformation control. Controlling the correspondence between the handle and skin mesh determines the mesh to follow the movement of the control handle to generate animation. This section analyzes the skinning animation by controlling deformation of the skin mesh and the bones which are generally placed inside the skin mesh. Through the corresponding between the bones and skin on the mesh position can determine the deformation degree of the skin. In literature, the bone skinning algorithms are categorized as Liner Blending Skinning algorithm (LBS), quaternion algorithm, dual quaternion linear blending algorithm.
Since the LBS algorithm may have volume lost and depression when deformation is large, and also the computation cost of dual quaternion linear blending algorithm and LBS algorithm are the almost same, in this paper we use Kinect to obtain the data of bone, and quickly build human model driven by Dual quaternion linear blending algorithm to generate the animation.
An important factor in skinning based on the skeletal data from Kinect is the proportion of captured skeleton data is similar to human body model. Thus the skinning effect is much better than manual establishment of a skeletal model. In this issue, we adopted Microsoft SDK libraries to obtain skeleton data, further recorded the bone date with hierarchical structure and also each frame skeleton data. Totally, we obtained 20 bones based on BHV nodes in the hierarchy of storage. The data transformation between two adjacent frames is set as the transformation matrix, which drives the rapid generation of human body model animation.
5. System Design
Based on the algorithms above, the implementation system includes three parts: rapid mannequin modeling, skeletal data processing and skinning animation. Firstly, quick mannequin modeling is based on the raw data gained from Kinect, then is combined with the standard model under parameterization method to quickly generate realistic mannequins. Secondly, skeletal data processing is used to divide the human body and drive the function of the model. Thirdly, skinning animation works to generate skeletal data and drive the model to genenrate skinning animation. The system design flowchart is shown in Figure 5.
This module aims to obtain a series of raw data through the Kinect then handles the data to get rapid modeling of human. Our experimental scene is in natural environment without any special process (as shown in Figure 2). The subject stood away from the Kinect with a fit distance in the marking area. According to the guideline, the front and back body surface is scanned, and the entire scanning process can be completed within 6-10 seconds. The design flowchart of this part is shown in Figure 6.
Figure 6.
Flowchart of rapid mannequin modeling module.
In order to facilitate model rendering and implementation of the skinning animation with the OGRE graphics engine, the format of the model file is set as the file structure of .mesh. Figure 7 shows the respective process of the front and back sides of the body. The processing involves the human body data segmentation, point cloud noise removal, meshing, smoothing model and a series of operations. Finally through merging the front and back models, the whole human body model is generated. Skeletal data processing module functions to extract the bone hierarchy of nodes and record the bone data, and then obtain the human body model driven by the skinning technique. In this paper, we use Microsoft’s Kinect SDK as Kinect development libraries to obtain human body’s 20 bone nodes.
Figure 7.
Human body modeling from front and back sides.
Skinning animation module is to use the skeletal driver obtained from Kinect to quickly build the human body model. We achieved the skinning deformation animations by dual quaternion. The examples of animation of real-time model on arms and legs are shown in Figure 8 and Figure 9.
Figure 8.
Skinning animation sequence of arms.
Figure 9.
Skinning animation sequence of legs.
6. Comparison of Experimental Results
This developed system for rapid human body modeling can generate human models with relatively abundant surface texture. When combining the captured skeletal data, the skinning techniques can be applied and generate the real-time skinning animation. The rapid modeling framework proposed in this work can extract the human body data from the complex scenes, the system would not affected by the environment, and has few effect by the background, thus it has widely potential applications.
Considering the scanning models and computer hardware configuration are not the same (the system hardware environment of our work: Intel (R) Core (TM) i3-2100 CPU @ 3.10 Hz, GeForce GTS 450 graphics card with 1G memory, 4G memory, win7 system), we just made rough comparison with the results in the work of Tong [5] and Cui [11]. The respective final results of the human body models are shown in Figure 10. It can see the quality of the final models produced is similar. The detail of the leg by our work is treated better than that in [11]. As to the model scanning time, the time in [5] and [11] is about 30 s, while the scanning time in our work is about 6-10 s. The time for model generation in [5] and [11] is bout 359 s and 826 s, while the modeling time in our work is about 330 s. Furthermore, the scanning is more natural in our work since the human body need not to rotate. At the skinning phase, when the body is in the effective visual distance of Kinect, the real-time animation can be output. In the this system, there still are some shortcomings and needs to be improved in our work. For example, it’s not easy to separate the part of human body area through the depth segmentation method when there are more than on one model.
Figure 10.
The generated models comparison.
7. Conclusion
The rapid 3D human body modeling and skinning animation proposed in this paper is achieved by single Kinect, which comes from the actual demand in the applications [20]. We realized the framework and algorithm of fast human body modeling based on the analysis of the existing algorithms. And in the part of skinning animation, we discussed the skinning techniques used for making human body animation. It’s a promising solution to achieve fast human body modeling by using portable scanning devices with low cost. But there are still many challenges in this field when we hope to develop 3D human models just like taking a photo.
References
- He, Kunlin; Xu, Yong; Mao, Aihua; et al. Rapid 3D human body modeling based on Kinect technology. Comput. Aided Draft. Des. Manuf. (CADDM) 2013, 23(1), 7–13. [Google Scholar]
- Tevs; Berner, A.; Wand, M.; et al. Animation cartography-intrinsic reconstruction of shape and motion. ACM Trans. Graph. (TOG) 2012, 31(2), 12. [Google Scholar] [CrossRef] [Scilit]
- Wampler, K.; Andersen, E.; Herbst, E.; et al. Character animation in two-player adversarial games. ACM Trans. Graph. (TOG) 2010, 29(3), 26–35. [Google Scholar] [CrossRef] [Scilit]
- Parent, R.; Ebert, D. S.; Gould, D.; et al. Computer Animation Complete: All-in-one: Learn Capture, Characteristic, Point-based, and Maya Winning Techniques; Elsevire: Burlington, MA, USA, 2009. [Google Scholar]
- Tong, Jing; Zhou, Jin; et al. Scanning 3d full human bodies using Kinects. Vis. Comput. Graph. 2012, 18(4), 643–650. [Google Scholar] [CrossRef] [Scilit]
- Zheng, Xiao; Fu, Mengyin; et al. 3D human postures recognition using Kinect. International Conference on Intelligent Human-Machine Systems and Cybernetics, Nanchang, China, 2012; pp. 344–347. [Google Scholar]
- Zhou, Jin; Pan, Jianjiang; Tong, Jing; et al. Fast 3D human body reconstruction from Kinects. J. Comput.-Aided Des. Comput. Graph. 2013, 25(6), 873–879. [Google Scholar]
- Song, Shichao; Yu, Suping; Xu, Wujun. Study on 3D body scanning, reconstruction and measurement techniques based on Kinect. J. Tianjin Polytech. Univ. 2012, 31(5), 34–37. [Google Scholar]
- Newcombe, R. A.; Davison, A. J.; Izadi, S.; et al. KinectFusion: Real-time dense surface mapping and tracking. 2011 10th IEEE International Symposium on the Mixed and Augmented Reality (ISMAR), Basel, Switzerland, 2011; pp. 127–136. [Google Scholar]
- Cui, Y.D. Stricker, 3D shape scanning with a Kinect. ACM SIGGRAPH Posters (2011), Hong Kong.
- Wang, R.; Choi, J.; Medioni, G. Accurate full body scanning from a single fixed 3D camera. In 3DIMPVT ’12 Proceedings of the 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visualization and Transmission, Washington, USA, 2012; pp. 432–439. [Google Scholar]
- Chang, W.; Zwicker, M. Global registration of dynamic range scans for articulated model reconstruction. ACM Trans. Graph. (TOG) 2011, 30(3), 26. [Google Scholar]
- Weiss; D. Hirshberg, M. J. Black, Home 3D body scans from noisy image and range data. In ICCV’11 Proceedings of the 2011 International Conference on Computer Vision, Washington, USA, 2011; pp. 1951–1958. [Google Scholar]
- Anguelov, D.; Srinivasan, P.; Koller, D.; et al. Scape: Shape completion and animation of people. In ACM SIGGRAPH 2005 Papers; Gross, Markus, Ed.; ACM: New York, 2005; pp. 408–416. [Google Scholar]
- Mirza, M. J.; Boyer, K. L. Performance evaluation of a class of M-estimators for surface parameter estimation in noisy range data. IEEE Trans. Graph. 1993, 9(1), 75–85. [Google Scholar]
- Bouaziz, S.; Pauly, M. Dynamic 2D/3D registration for the kinect. ACM SIGGRAPH 2013 Courses 2013, 21–31. [Google Scholar]
- Du, Ji; Zhang, Liyan; Wang, Hongtao; et al. Hole repairing in triangular meshes based on radial basis function. J. Comput.-Aided Des. Comput. Graph. 2005, 17(9), 1976–1982. [Google Scholar]
- Mao, Aihua; Li, Yi; Wang, Ruo Mei. A multi-disciplinary strategy for computer-aided clothing thermal engineering design. Comput.-Aided Des. 2011, 43(12), 1854–1869. [Google Scholar] [CrossRef] [Scilit]
© 2015 Textile Bioengineering and Informatics Society. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).









