MaterialSeg3D++: Large-Scale Material Prediction for 3D Assets from 2D Priors

Abstract

Recent image diffusion models have enabled automatic 3D object creation from text or image guidance, but their 2D generative priors often bake illumination and shadow into textures, making relighting and physically based rendering (PBR) difficult. To address this issue, we propose MaterialSeg3D, a framework that predicts surface materials for 3D assets by leveraging 2D material semantics. Given a mesh and its albedo UV map, MaterialSeg3D renders multi-view images, performs material segmentation using a 2D prior model, projects the predictions back to UV space, and fuses them through weighted voting and region unification to obtain coherent material maps. To train the prior model, we construct MIO++, a large-scale single-object material segmentation dataset containing 115,542 images, 12 object themes, and 30 fine-grained material categories, substantially extending the previous MIO dataset. Each MIO++ material category is associated with an independent pair of roughness and metallic values for PBR assignment. We further observe that poor topology in AI-generated assets can degrade PBR quality even when plausible materials are assigned, and introduce a plane-simplification strategy as an auxiliary preprocessing step for such meshes. Experiments show that MIO++ improves material segmentation and that the resulting material maps support more consistent relightable renderings for both human-crafted and AI-generated 3D assets.

1. Introduction

3D asset creation is an important topic in computer graphics, with applications in virtual reality, augmented reality, games, and film production. A central objective is to construct assets that are visually plausible, artistically controllable, and compatible with different environments. Physically based rendering (PBR) is particularly important for this purpose because it models how light interacts with surface properties such as roughness and metallicity. Creating assets that satisfy PBR requirements remains labor-intensive in traditional production pipelines. Artists must not only construct suitable geometry and textures, but also define material properties that behave consistently under changing illumination. With the recent development of diffusion models, many methods have been proposed for efficient 3D asset generation, including approaches based on point clouds [1,2,3,4,5], implicit functions [6,7], and meshes [8,9,10,11,12,13,14]. These methods can generate 3D objects from textual or visual guidance, and many of them rely on powerful 2D generative priors. However, illumination and shadows can be baked into the generated appearance, so an RGB texture alone does not necessarily provide a disentangled material representation. Without explicit PBR material information, relighting in novel scenes remains difficult. Directly learning PBR material generation is also constrained by data availability. Many existing 3D assets provide geometry and appearance textures but lack complete roughness and metallic maps, while material information represented in UV space is expensive to annotate densely. This motivates us to transfer material reasoning to the better-supported 2D perception space and then project the predictions back to the 3D asset.

To address this problem, we build a large-scale material dataset covering common material types in 3D asset creation and propose a 3D material prediction workflow based on 2D priors. We follow the Disney principled bidirectional reflectance distribution function (BRDF) and use roughness and metalness as the primary physical material properties. These properties modulate the BRDF terms and enable a surface to respond differently under changing illumination. To design a dataset that is useful for PBR while remaining practical to annotate, we draw on the observation that material semantics can often be inferred from object appearance. Given reference images of an object, a modeler may infer, for example, that silver chair legs are metallic while a dark padded seat is likely leather or fabric. This motivates the use of 2D material segmentation as a source of material prior knowledge for 3D assets. Existing material-related segmentation datasets such as DMS [15] or MINC [16] only provide material labels for open scenes including multiple instances, which are less reliable in dealing with single-object component segmentation. More importantly, images in these datasets are taken from very limited camera poses, which makes it hard for models to learn the material prediction from all angles. With the motivation of establishing a database to construct 2D material prior knowledge for individual objects, we collect Materialized Individual Objects (MIO), a novel 2D single-object segmentation dataset consisting of dense material-semantic annotations of objects with intricate semantic classes and captured camera angles. Images are (a) collected from both real-world captures and 3D asset renderings, augmenting the prior knowledge from reality and easing the domain gap; (b) sampled with various camera angles including but not limited to top and side views; and (c) annotated and supervised by professional annotators. For each material class, we associate representative roughness and metalness values selected with guidance from experienced modelers. The original MIO dataset contains 23,062 images, five object themes, and 14 material categories. We substantially extend it to MIO++, which contains 115,542 images, 12 object themes, and 30 fine-grained material categories. In MIO++, materials with different PBR behavior are separated into finer categories, such as smooth metal, matte metal, and rusty metal. Importantly, the 30 categories are retained in the actual MaterialSeg3D pipeline and each category has its own roughness and metallic assignment.

Using MIO++, we develop MaterialSeg3D, a workflow for assigning surface materials to 3D objects. Given a geometry mesh and an albedo UV map, the method renders the asset from multiple camera poses, predicts material labels in each view, projects the predictions back to temporary UV maps, and combines them through weighted voting and region unification. The resulting material-label UV map is converted to roughness and metallic maps using the class-specific assignments defined for MIO++. As shown in Figure 1, this process transfers 2D material priors to a 3D asset and provides explicit PBR material maps that can be used for relighting.

We also observe that poor topology in AI-generated assets can reduce PBR quality even when a plausible material has been assigned. Small geometric irregularities on surfaces that should be planar may produce unstable normals and visually inconsistent reflections. For AI-generated assets, we therefore include an auxiliary plane-detection and simplification stage before applying the material workflow. This stage is intended as practical geometry preprocessing rather than as a general-purpose mesh reconstruction method.

This article is a revised and substantially expanded version of the conference paper entitled “MaterialSeg3D: Segmenting Dense Materials from 2D Priors for 3D Assets”, which was presented at the 32nd ACM International Conference on Multimedia (MM ’24), Melbourne, VIC, Australia, 28 October–1 November 2024 [17]. Compared with the conference version, this article introduces MaterialSeg3D++, a substantially expanded study with several major extensions. First, we expand the original MIO dataset from 23,062 images, five object themes, and 14 material classes to MIO++ with 115,542 images, 12 object themes, and 30 fine-grained material categories, yielding about a five-fold increase in dataset scale and a more than two-fold expansion in label space. Second, MIO++ introduces more fine-grained material categories based on roughness and metalness, and training on MIO++ improves material prediction performance, increasing mIoU from 83.92% to 86.75% after mapping fine-grained subclasses to the original meta-classes. Third, we extend the original workflow to AI-generated assets by adding a plane-simplification strategy to improve physically based rendering on poor-quality generated meshes. Finally, we provide additional quantitative and qualitative evaluations, including detailed validation on MIO++, comparisons with image-to-3D and texture generation methods, and more visual analyses under different lighting conditions.

For clarity,

Table 1. Summary of the main extensions from the ACM MM 2024 conference version to this journal version.

Aspect	ACM MM 2024 Version	This Journal Version
Dataset	MIO: 23,062 images	MIO++: 115,542 images
Object themes	5	12
Material-label space	14 categories	30 fine-grained categories
PBR assignment	14 class-level settings	30 independent class-level settings
AI-generated meshes	Original material workflow	Added plane-oriented preprocessing
Evaluation	Original MIO evaluation	MIO++ analysis and expanded 3D evaluations

summarizes these extensions side by side.

To summarize, the contributions of this paper are as follows:

• We formulate surface material assignment for 3D assets as a 2D material prediction and multi-view fusion problem, exploiting the relationship between visual semantics and material categories.
• We construct the MIO++ dataset, containing 115,542 single-object images from diverse camera angles, 12 object themes, and 30 fine-grained material categories with class-specific PBR assignments.
• We introduce MaterialSeg3D, which aggregates multi-view material predictions in UV space through weighted voting and region unification and converts the resulting labels to roughness and metallic maps.
• We analyze how irregular topology in AI-generated assets can affect PBR appearance and introduce an auxiliary plane-detection and simplification stage to improve planar surface quality before material assignment.

2. Related Works

2.1. 3D Asset Generation

Early methods in 3D asset generation adapted convolutional neural networks (CNNs) and generative adversarial networks (GANs) to generate 3D voxel grids [18,19,20,21,22,23]. Although straightforward, voxel representations can incur substantial memory and computational cost at high resolutions.

Subsequent research explored point-cloud representations [1,2,3,4,5] and implicit functions [6,7]. Gaussian-based reconstruction methods have further improved large-scale rendering and geometric accuracy [24,25], while controllable procedural frameworks have explored large-scale 3D scene generation [26]. Mesh-based generative models [8,9,10,11,12,13,14] are particularly convenient for standard graphics pipelines because their outputs can be directly rendered and edited in conventional engines.

Recent 3D generation methods also use text or image guidance together with Neural Radiance Fields (NeRFs) [27], CLIP-based alignment [28,29,30], or diffusion priors. DreamFusion [31] introduced score distillation sampling to optimize 3D representations from a diffusion model, while Magic3D [32] adopted a coarse-to-fine strategy. Other methods [33,34,35,36,37,38,39,40] combine learned 3D representations with powerful 2D generative priors. These advances improve geometry and appearance generation, but explicit PBR material assignment remains a distinct problem.

2.2. Surface Material Generation

PBR properties such as metalness and roughness determine how a 3D surface interacts with incident light and are therefore important for realistic relighting and reflection behavior.

Traditional material-acquisition methods often estimate physically based materials under controlled lighting and may require multi-view [41] or polarization [42] equipment. Learning-based approaches use synthetic data to train single-view Spatially Varying Bidirectional Reflectance Distribution Function (SVBRDF) prediction networks [43], sometimes combined with real single-view data [44,45] or specialized training strategies [46,47,48]. These methods address material capture directly, whereas our setting assumes an existing 3D asset and focuses on semantic material assignment over its surface.

Recent work has also investigated material segmentation and controllable generation of SVBRDF maps [49,50,51,52]. Fantasia3D [53] decouples geometry and appearance modeling and uses a BRDF-based representation for high-quality 3D content. PhotoScene [54] uses procedural material graphs, while DiffMat [55] and Material Palette [56] address material generation or extraction in related settings. MatAtlas [57] generates relightable textures and material assignments for 3D models from text guidance. Our work differs in that it predicts discrete material semantics from rendered views and fuses these predictions in the UV domain. More broadly, recent dense-perception methods have explored bidirectional feature aggregation and scene-adaptive perception to improve robustness across visual conditions [58]; these ideas are complementary to our view-level fusion setting, although they address a different task domain.

2.3. Existing 3D and 2D Datasets

Learning material priors requires sufficiently diverse training data. Several large-scale 3D datasets have recently been released [59]; representative examples include Objaverse-1.0 [60] and Objaverse-XL [61], with approximately 800,000 and more than 10 million assets, respectively. However, many Objaverse assets do not contain complete material information suitable for supervised PBR learning. Other datasets such as KIT [62], YCB [63], BigBIRD [64], and Pix3D [65] provide calibrated object models but are much smaller. Photorealistic object collections [66,67] and CAD datasets [68,69,70] also do not consistently provide the albedo and PBR material maps required by our setting.

Because 2D images are easier to collect, material recognition has access to larger 2D datasets. DMS [15] contains 44,560 indoor and outdoor images with annotations for 3.2 million dense segments. OpenSurfaces [71] provides annotations for 37 material categories on 19,000 residential images, and MINC [16] contains millions of annotated material samples. These datasets are valuable for general material recognition, but they are dominated by multi-object scene imagery and conventional viewpoints. Our application instead requires dense predictions on a single object rendered from many surrounding and top/oblique viewpoints.

3. Significance of Material for PBR

Creating high-quality materials in computer graphics is a demanding task that requires substantial expertise. Correct PBR material properties allow the same 3D asset to respond plausibly under different lighting conditions, whereas missing or inappropriate material parameters can produce unrealistic reflection and relighting behavior. Figure 2 illustrates this difference under a common lighting setup.

To assess whether existing 3D collections could directly support material learning, we examined more than 270,000 assets from Objaverse [61]. Only about 3k of the examined assets contained material information that was sufficiently complete for our initial experiments. This limited subset was not adequate for learning the material-semantic distribution needed by our pipeline.

Recent 3D generation and material generation methods have made progress toward relightable appearance [50,51,53,57]. In our experiments with generated assets, we nevertheless observe two practical issues that motivate the following analysis.

One issue is that a uniform or weakly differentiated PBR setting can ignore semantic differences between object parts. As shown in Figure 3a, the hammer handle and head receive the same PBR values even though they correspond to visibly different materials. A second issue is semantic inconsistency within a continuous object region. In Figure 3b, parts of the chair back are assigned metal even though the region is visually more consistent with a continuous material such as fabric or nylon.

These observations motivate the use of semantic material priors. As an exploratory check of this motivation, we surveyed 100 participants and showed each participant 10 images drawn from indoor and outdoor scenes, giving 1000 image–participant observations in total. Participants were asked to record the material types that they considered likely to occur in each image, and the reported frequencies were aggregated as shown in

Table 2. Aggregated material selection counts from an exploratory survey of 100 participants, each evaluating 10 indoor/outdoor scene images (1000 image–participant observations in total).

Material Label	Number	Material Label	Number
metal	935	brick	186
wood	842	porcelain	163
plastic	768	clay terracotta	154
glass	712	concrete	152
paint	626	nylon	75
rubber	524	rusty metal	53
leather	437	stone	46
fabric	391	bone	25
fruit&leaf	273	bamboo	22
flower	252	others	181

. The survey was exploratory rather than demographically stratified, and detailed recruitment or demographic metadata were not retained. It is used only to support the general observation that people can associate visible object regions with material categories, not to determine the numerical roughness or metallic values assigned to individual material classes; those PBR assignments were established separately with experienced 3D asset modelers, as described in Section 4.2.

4. MIO++ Dataset

4.1. Consideration for Dataset Design

In an early attempt to solve the material assignment problem, we explored material priors from existing 3D asset datasets such as Objaverse-1.0 and Objaverse-XL. We collected objects with available material UV maps and applied k-means clustering with the elbow method to group materials according to their roughness and metalness values. However, only a small fraction of the examined assets contained sufficiently complete and usable PBR information (about 3k out of 200k examined assets), and the resulting data were not adequate for training a robust material recognition model. We therefore constructed a dedicated material dataset instead of relying on these existing 3D assets.

Our pilot study suggests that 2D appearance provides useful prior information for material categorization. Rather than annotating material labels directly in UV space, we therefore formulate material prediction as pixel-wise segmentation in rendered or captured 2D views and later project those predictions back to the UV map. Different roughness and metallic properties can lead to substantially different appearances under the same illumination conditions, as illustrated in Figure 4.

To determine the material vocabulary, we first examined more than 1000 material presets used in common asset-authoring tools. Many presets share similar roughness and metalness while mainly differing in albedo or texture details. With input from art experts, the original MIO dataset grouped these presets into 14 common material categories. MIO++ extends this vocabulary to 30 categories so that materials with visibly different PBR behavior can be represented more explicitly, for example, by separating smooth, matte, and rusty metal. The detailed category statistics of MIO and MIO++ are reported in

Table 3. Occurrence statistics for material categories in MIO and MIO++. In MIO, ceramic (No.18) is named porcelain. The ↑ symbol indicates MIO++ subclasses whose counts are combined when mapped back to an original MIO meta-material.

Label Id.	Material Label	MIO	MIO++
1	rubber	5324	7244
2	smooth metal	8946	18,047
3	matte metal	↑	15,429
4	rusty metal	↑	1560
5	paint	3496	4003
6	glass	5802	19,637
7	leather	3417	3496
8	smooth wood	7088	11,691
9	rough wood	↑	6156
10	rough fabric	3373	12,270
11	smooth fabric	↑	1283
12	plush fabric	↑	2199
13	smooth plastic	6928	11,807
14	rough plastic	↑	10,939
15	brick	1017	2689
16	concrete	794	4306
17	clay	910	2290
18	ceramic	921	6604
19	flower	1677	2577
20	fruit	1742	4758
21	leaf	↑	3020
22	meat	-	3345
23	gel	-	2438
24	cream	-	1957
25	biscuit	-	5287
26	shell	-	756
27	water	-	1596
28	jade	-	519
29	pearl	-	212
30	plaster	-	2014

. Each of the 30 MIO++ categories has an independent roughness/metallic pair; the complete values are provided in the Supplementary Materials.

4.2. Data Collection and Annotation

To reduce the domain gap between ordinary 2D images and renderings of 3D assets, we collect and construct image samples under three guidelines: (a) each image contains one dominant foreground object; (b) the collection includes both real-world images and 3D asset renderings; and (c) the images cover diverse camera angles, including less common top and lower-side views. These guidelines make the training distribution closer to the multi-view renderings used by MaterialSeg3D at inference time.

The sources of the collected images are freely accessible public datasets [72,73] and 2D image renderings from 3D objects in website photo libraries [74]. In addition, we also procured some well-designed 3D assets that are used for game development and expanded the data collection by rendering multi-view images of these high-quality assets.

An important feature of MIO++ is the explicit association between semantic material labels and PBR parameters. The material vocabulary and the corresponding roughness/metallic assignments were discussed with professional 3D asset modelers. The modelers examined candidate material spheres drawn from more than 1000 presets in public material libraries and asset-authoring tools, including Adobe Substance 3D Painter 2023, and used their modeling experience to select representative PBR settings. The original MIO label space contains 14 categories. MIO++ expands this space to 30 fine-grained categories, and each MIO++ category has its own independent roughness and metallic values. The complete 30-class parameter table is provided in the Supplementary Materials. The mapping from these 30 categories back to the original 14 MIO categories is used only for the backward-compatible evaluation against the original MIO label space; it is not used in the actual MaterialSeg3D material generation pipeline.

After defining the label space, a specialized annotation team performed pixel-wise dense annotation of the collected images. Only the dominant foreground object was assigned material labels, while the background was treated as a single background class. Annotation was initialized with a Segment Anything [49]-assisted tool and then manually refined. The annotated samples were subsequently passed to other annotators for multiple rounds of checking and correction rather than being accepted from the first pass. Ambiguous boundaries or material regions were handled within this manual review process; the released annotation format does not contain a separate ambiguity label. The annotation process was deliberately paced at approximately 50 images per annotator per day to allow manual boundary and material-label inspection. A numerical inter-annotator agreement score is not reported in the present study; annotation quality was controlled through the SAM-assisted initialization, manual refinement, and repeated checking and correction by different annotators. Figure 5 shows an example of material annotation and PBR alignment, and Figure 6 illustrates the range of object types and viewpoints in MIO++.

4.3. Dataset Distribution

The original Materialized Individual Objects (MIO) dataset contains 23,062 single-object images captured or rendered from diverse viewpoints, with 14 material classes and five object themes: furniture, cars, buildings, musical instruments, and plants. The real/rendered image statistics are reported in

Table 4. Number of rendered images and real images of every meta-class in the MIO dataset.

Classname (Abbr.)	Rendered Image	Real Image	Total
Furniture (fur.)	4152	5455	9607
Cars (car)	1935	4117	6052
Buildings (bui.)	418	1752	2170
Musical Instrument (ins.)	627	1637	2264
Plants (pla.)	552	2417	2969

. Approximately 4000 top-view images are included to provide viewpoints that are uncommon in conventional scene datasets; examples from the car theme are shown in Figure 7. MIO++ expands the collection to 115,542 images, 12 object themes, and 30 material types, as summarized in

Table 5. Number of images in each object theme in the MIO dataset and the number of images newly added in the MIO++ dataset.

Classname (Abbr.)	MIO	New in MIO++	Total
Furniture (fur.)	9607	2499	12,106
Car (car)	6052	5558	11,610
Building (bui.)	2170	6488	8658
Musical Instrument (ins.)	2264	8337	10,601
Statue (sta.)	-	2400	2400
Household appliance (hou.)	-	11,991	11,991
Kitchen ware (kit.)	-	11,992	11,992
Weapon and tool (wea.)	-	11,891	11,891
Clothing (clo.)	-	13,906	13,906
Food (foo.)	-	14,970	14,970
Jewelry (jew.)	-	2808	2808
Plants (pla.)	2969	-	2969
Total	23,062	92,840	115,542

. Several broad MIO categories are split according to PBR-relevant appearance. For example, metal is divided into smooth metal, matte metal, and rusty metal; wood, fabric, and plastic are also separated into finer subclasses, while new categories such as jade and plaster are added. For the segmentation experiments in this paper, model selection and the detailed results in

Table 6. Per-class IoU (%) on the MIO++ validation set, stratified by object theme. “–” denotes a material–theme combination for which no valid IoU is reported in this stratified evaluation. The last row reports the theme-wise mIoU. Class occurrence frequencies in the full MIO++ collection are given in Table 3.

	bui.	car	clo.	hou.	foo.	jew.	kit.	mus.	sta.	fur.	wea.	pla.
rubber	–	73.57	73.84	–	–	–	–	8.46	–	–	59.98	–
smooth metal	–	55.78	41.81	41.67	11.06	70.78	52.73	76.67	23.41	19.91	42.02	–
matte metal	49.78	–	42.85	45.11	–	21.54	45.4	43.57	52.45	36.61	84.63	–
rusty metal	49.91	–	–	19.73	–	–	1.17	–	31.33	–	75.35	–
paint	–	79.93	–	–	–	–	–	–	–	–	–	–
glass	76.3	84.97	57.82	80.06	18.63	69.28	59.89	–	–	81.82	–	–
leather	–	52.25	66.27	45.24	–	–	–	61.63	–	62.26	26.74	–
smooth wood	–	–	–	51.35	–	–	73.86	74.27	43.55	66.08	48.76	–
rough wood	71.24	–	57.3	–	–	–	55.94	48.96	22.49	61.13	54.18	78.73
rough fabric	64.22	–	88.97	44.29	–	43.66	24.36	–	–	84.18	67.95	–
smooth fabric	–	–	29.48	–	–	–	–	–	–	–	–	–
plush fabric	–	–	47.57	–	–	22.34	–	–	–	–	–	–
smooth plastic	–	61.07	29.31	39.33	–	–	23.24	66.43	–	43.06	51.84	–
rough plastic	–	36.56	42.32	71.2	–	–	63.81	42.45	–	52.76	24.39	–
brick	76.73	–	–	–	–	–	–	–	–	–	–	–
concrete	80.27	–	–	–	–	–	–	–	–	–	–	–
clay	64.59	–	–	–	–	–	32.57	–	38.81	–	–	80.18
ceramic	–	–	–	17.84	45.72	–	77.25	–	–	47.35	–	86.48
flower	–	–	–	–	–	–	27.41	–	–	–	–	94.19
fruit	–	–	–	–	87.26	–	69.08	–	–	–	–	–
leaf	76.12	–	–	–	39.04	–	57.29	55.63	–	36.86	–	92.30
meat	–	–	–	–	67.25	–	–	–	–	–	–	–
gel	–	–	–	–	53.82	–	–	–	–	–	–	–
cream	–	–	–	–	72.54	–	–	–	–	–	–	–
biscuit	–	–	–	55.3	77.04	–	44.71	–	–	–	–	–
shell	–	–	–	–	75.57	–	–	–	–	–	–	–
water	–	–	–	–	43.21	–	37.18	–	–	–	–	–
jade	–	–	–	–	–	61.16	–	–	–	–	–	–
pearl	–	–	–	–	–	54.77	–	–	–	–	–	–
plaster	–	–	–	–	–	–	–	–	89.56	–	–	–
mIoU	67.68	62.59	52.50	46.47	53.74	49.08	46.62	53.12	43.09	53.82	53.584	86.38

are reported on a held-out MIO++ validation partition. For rendered multi-view samples, all views generated from the same 3D asset are grouped before partitioning and assigned to the same split; therefore, different views of one 3D asset do not cross the training and validation partitions. The independent 3D evaluation is conducted separately on Objaverse-1.0 assets.

5. Method

5.1. Material Segmentation

We formulate 2D material prediction as semantic segmentation tailored to single-object images observed from diverse camera poses. Given an image I, let $x_i$ denote the RGB value and $y_i$ denote the annotated material label at pixel i. The network encodes the image and predicts a per-pixel probability vector $P_i=(p^{(i,1)},p^{(i,2)},\dots ,p^{(i,n)})$ over n classes. The final label is obtained by taking the class with maximum predicted probability.

We notice that the Segment Anything Model (SAM) [49] has shown its ability to handle semantic region segmentation on single-object images in previous work [75]. Thus, we formulate the material segmentation network with a modified ViT [76] backbone using pretrained segmentation weights from the SAM-B model. The decode head follows the UPerNet [77] setting with cross-entropy loss as supervision. To prevent possible long-tail problems caused by imbalanced training data, we adopt a class-balanced sampling strategy [78] to enhance the robustness and generalization ability of the model. During training, the cross-entropy loss is calculated as

$$L=-{\displaystyle \frac{1}{HW}}{\displaystyle \sum _{i=1}^{HW}}{\displaystyle \sum _{c=0}^{n-1}}{y}^{(i,c)}log\left(p^{(i,c)}\right),$$

(1)

where H and W denote the height and width of the input image, n is the number of classes, and $y^{(i,c)}$ and $p^{(i,c)}$ denote the ground-truth indicator and predicted probability, respectively, for class c at pixel i.

5.2. MaterialSeg3D

In this section, we introduce MaterialSeg3D, a workflow for generating material information for 3D assets. As shown in Figure 8, the workflow contains three main components: multi-view rendering, material prediction, and material UV generation. First, the input asset is rendered from camera poses covering the object surface. Second, the material segmentation model trained on MIO++ predicts a material label for each visible pixel in every view. Third, the view-level predictions are projected into UV space and combined by weighted voting and region unification. The final material-label UV map is then converted to roughness and metallic maps using the class-specific PBR assignments defined for the 30 MIO++ categories.

Multi-View Rendering. To provide material predictions over the object surface, the rendering cameras are distributed around the full $360°$ azimuth range. We first define five high-value views with elevation/azimuth pairs $(90°,0°)$, $(15°,0°)$, $(15°,90°)$, $(15°,180°)$, and $(15°,270°)$. These views provide one top view and four common inspection views. We then divide the full azimuth range into 12 directions. At each direction, three elevation settings are used: $0°$ and two additional elevations randomly sampled within the positive and negative $30°$ ranges, respectively. An example of the multi-view rendering process is shown in Figure 9. This camera set provides complementary observations over the asset, including top and oblique surfaces. The five predefined views receive a larger contribution during UV voting, as described below. These camera and weighting settings are fixed empirical implementation choices in the current pipeline rather than learned parameters.

Material Prediction. Following Section 5.1, the trained segmentation model predicts a material-label map for each rendered view. These view-level label maps are subsequently projected to UV space.

Material UV Generation. For each rendered view, the predicted material labels are projected to the corresponding texels of the albedo UV map using the known camera and mesh geometry, producing a set of temporary label maps $M_{view}=\{M_1,\dots ,M_n\}$. A UV texel may be observed by multiple views, so its final label is determined from all valid projected labels rather than by sequentially overwriting the map.

Instead of sequentially updating the material-label UV [79], we use a weighted majority vote. Let $m_i\left(u\right)$ denote the valid material label projected from view i to texel u, and let $\mathbb{1}[·]$ denote the indicator function. The vote score for material class c is

$$S_c\left(u\right)={\displaystyle \sum _{i=1}^n}{w}_i\mathbb{1}[m_i\left(u\right)=c],M_{material}\left(u\right)=\underset{c}{argmax} S_c\left(u\right),$$

(2)

where $w_i=\alpha$ for the five predefined high-value views and $w_i=1$ for the remaining views. We set $\alpha =2$ in all experiments; in implementation, this is equivalent to counting each valid prediction from a high-value view twice and each prediction from another view once. Labels outside the valid material-label range are ignored. If no rendered view observes a texel, it remains label 0 (unassigned/background). If two or more valid classes have exactly the same maximum vote count, the implementation selects the first maximum in label-index order, i.e., the smallest material-label ID among the tied classes. The value $\alpha =2$ is an empirical setting used consistently throughout the experiments rather than a theoretically optimized parameter.

Region Unification. Some fine-grained labels can fluctuate locally after voting, especially between visually similar subclasses such as smooth and matte metal. To suppress small isolated label changes, we apply a region-level post-processing step. A $2\times 2$ erosion is first used to break weak connections, after which connected non-zero regions are identified. Within a connected region, the frequency of material labels is counted. For subclasses belonging to the same meta-material, the majority subclass is used to unify inconsistent local predictions; small minority label regions occupying less than $5\%$ of the connected component are also replaced by the dominant label. The $5\%$ threshold, like the view weight, is a fixed empirical setting in this study.

After obtaining the final material-label UV map, each labeled texel is assigned the roughness and metallic values associated with its MIO++ class. The complete 30-class mapping is reported in the Supplementary Materials.

5.3. Plane Simplification on AI-Generated Contents

In addition to human-crafted assets, we apply MaterialSeg3D to assets produced by single-image-to-3D methods [12,13,14]. These generated assets provide a geometry mesh and albedo UV map, but locally irregular triangles and unstable normals can produce unwanted reflection changes on surfaces that should be approximately planar, as illustrated in Figure 10. We therefore use plane-oriented simplification as an auxiliary preprocessing step before material assignment. Related work has explicitly studied structured planar reconstruction for improving geometric consistency and mesh topology in planar regions [80]; our component is instead intended only as lightweight preprocessing for reducing obvious planar artifacts before material assignment. It is not presented as a general mesh-quality benchmark or as a replacement for dedicated geometry-reconstruction methods.

Specifically, as shown in Figure 8, we use four rendered views of the target 3D object (the four high-value side views, excluding the top view) and apply an off-the-shelf 2D surface-normal estimator [81]. The RGB renderings and predicted normal maps are concatenated as cues for K-means clustering of pixels that are likely to belong to the same planar region. The clustered pixels are projected back to 3D vertices through the known camera parameters, after which RANSAC [82] removes spatial outliers. The resulting vertex groups provide plane candidates that guide a mesh-simplification tool. Figure 11 visualizes the stages of this auxiliary preprocessing pipeline. In the present work, its role is assessed qualitatively through the resulting surface and rendering examples; a separate geometry benchmark is discussed as future work in Section 6.4.

6. Experiments

6.1. Implementations and Evaluations

Learning reliable 2D material priors is the first stage of the MaterialSeg3D pipeline. We use a ViT backbone initialized from SAM-B [49] segmentation weights. The optimizer is AdamW [83], with learning rate $6\times {10}^{-5}$ and weight decay $1\times {10}^{-2}$. The batch size is 8, training lasts 80k iterations, and images are resized to $1024\times 1024$. All segmentation experiments are implemented with MMSegmentation [84] and run on four 80GB NVIDIA A100 GPUs. Detailed class- and theme-wise results on the held-out MIO++ validation partition are reported in Table 6. Class-balanced sampling and SAM initialization are kept fixed throughout the reported experiments; they are implementation components rather than separately optimized variables in this study.

Table 6 is intended to expose the class-wise behavior within each object theme rather than only reporting a single aggregate score. The lower values are not confined to one theme: they occur frequently for fine-grained subclasses whose visual difference is subtle, such as smooth/matte/rusty metal, and in material–theme combinations with relatively limited support. This pattern is consistent with the long-tailed occurrence statistics in Table 3; for example, some newly added categories occur much less frequently than common materials such as metal or glass. The theme-wise mIoU in the final row is therefore reported together with the individual class IoUs. A full pixel-level confusion-matrix analysis would provide a more detailed account of subclass confusions, but it is outside the evaluation retained for the current experiments and is left for a dedicated diagnostic study.

6.2. Compared with Previous Work

Material Segmentation. To evaluate the 2D material segmentation component described in Section 5.1, we compare it with five commonly used semantic segmentation backbones on the MIO label space.

Table 7. Quantitative results of semantic segmentation methods on the MIO validation label space. Ours(MIO) denotes the model trained on MIO. Ours(MIO++) denotes the model trained on MIO++; only for this backward-compatible comparison, its 30 fine-grained predictions are mapped to the corresponding 14 MIO meta-material categories. The actual MaterialSeg3D pipeline retains all 30 MIO++ categories and their independent PBR assignments.

Method	MIO Dataset (%)
	car	fur.	bui.	ins.	pla.	mIoU
ConvNeXt [85]	71.03	74.85	69.33	72.40	76.72	72.87
HRNet [86]	75.71	79.94	76.37	80.14	81.35	78.70
ViT [76]	73.96	77.67	75.53	79.45	78.66	77.05
Swin-T [87]	75.09	79.04	78.45	80.92	81.40	78.98
MAE [88]	76.42	82.06	77.59	82.74	85.92	80.95
Ours(MIO)	81.83	85.22	81.76	84.39	86.38	83.92
Ours(MIO++)	85.14	87.35	84.49	88.82	87.97	86.75

Bold values indicate the best performance among all compared methods.

reports mIoU on the MIO validation setting. We also evaluate the model trained on MIO++ under the same 14-category label space. For this comparison only, the 30 MIO++ predictions are mapped back to their MIO meta-materials (for example, smooth, matte, and rusty metal are counted as metal; smooth and rough wood are counted as wood). This evaluation-only mapping makes the MIO and MIO++ models directly comparable and yields an mIoU increase from 83.92% to 86.75%. In the subsequent 3D material generation pipeline, no such collapse is performed: all 30 MIO++ categories and their independent PBR values are retained.

Overall Performance. Figure 12 provides qualitative comparisons in three settings: complete single-image-to-3D generation, texture/material generation on a given mesh, and public human-crafted 3D assets. Wonder3D [12], TripoSR [13], and OpenLRM [14] reconstruct both geometry and appearance from a reference image, whereas MaterialSeg3D assumes that a mesh and albedo UV map are already available. These are therefore different input settings; the single-image-to-3D results are included as end-to-end contextual references and are not intended as a controlled material-only comparison. For methods that can operate on a provided mesh, we additionally show results from Fantasia3D [53], Text2Tex [79], and the online Meshy function (https://app.meshy.ai/). For Fantasia3D, only its appearance-modeling stage is used. We also show materialized renderings for public assets from Tripo3D (https://www.tripo3d.ai/app/ (accessed on 20 August 2026)) and TurboSquid (https://www.turbosquid.com/). These visual comparisons illustrate the different appearance behavior under relighting; Figure 13 provides additional views of assets processed by MaterialSeg3D.

Table 8. Quantitative evaluation on 20 Objaverse-1.0 assets. The first three rows use geometry/appearance reconstructed from one reference image and provide end-to-end context. The last two rows use the same predefined mesh and albedo UV; their difference isolates the effect of adding MaterialSeg3D-predicted PBR material. Metrics are averaged over the evaluated assets/views.

Method	Input Setting	CLIP Similarity↑		PSNR↑		SSIM↑
		Reference	Novel	Reference	Novel	Reference	Novel
Wonder3D [12]	Reconstructed	0.85	0.84	16.06	15.83	0.78	0.75
TripoSR [13]		0.93	0.90	16.93	16.14	0.79	0.76
OpenLRM [14]		0.92	0.87	16.30	15.37	0.77	0.76
Baseline	Predefined mesh + albedo	0.93	0.93	16.28	16.30	0.79	0.78
Baseline + Ours		0.98	0.97	20.72	18.39	0.85	0.84

The symbol “↑” indicates that higher values represent better performance. Bold text highlights the best performance among the compared methods.

further separates the two input settings quantitatively. We select 20 Objaverse-1.0 assets [60] with material information and render their reference and novel views as targets under a common fixed lighting setup. For each asset, three novel camera views are sampled at random while keeping the elevation within a valid viewing range. In the reconstructed input setting, Wonder3D, TripoSR, and OpenLRM reconstruct geometry and appearance from the single reference image; their outputs are resized to the scale of the target asset and aligned in pose before evaluation. These rows measure end-to-end reconstruction quality and should not be interpreted as isolating material prediction. In the predefined mesh + albedo setting, both “Baseline” and “Baseline + Ours” use the same Objaverse mesh and albedo UV. Baseline removes the original material information, whereas Baseline + Ours adds the PBR material maps predicted by MaterialSeg3D. This controlled pair isolates the contribution of the material assignment stage while keeping geometry and albedo fixed. CLIP similarity [30], PSNR, and SSIM are computed against the corresponding target renderings and averaged over the evaluated assets/views.

6.3. Weighted Voting and Region Unification

Appendix A and Figure A1 visualize how multi-view predictions are combined in UV space. A material region that is misclassified in one difficult view can still receive the correct label when the same surface is observed consistently from other views. The bottom examples also show the role of region unification: small local subclass shifts can remain after voting, and the post-processing step suppresses these isolated changes before the class-specific roughness and metallic values are assigned. The current paper uses the camera set, $\alpha =2$, and the $5\%$ region threshold as fixed pipeline settings. A systematic sensitivity study over these parameters would be useful for characterizing robustness, but is not required to apply the method and is left for future work, as discussed below.

6.4. Limitations and Open Issues

Several limitations remain in the current formulation. First, MaterialSeg3D predicts a discrete material class and then assigns a representative roughness/metallic pair to that class. This design is practical for semantic materialization, but it cannot represent continuous intra-class variation or high-frequency spatial detail. Two wooden regions, for example, may share the same semantic label while having different finish, wear, or local roughness. Future work could refine the class-level maps using albedo-conditioned spatial cues or replace the lookup stage with direct continuous PBR regression when suitable supervision is available.

Second, the quantitative experiments emphasize 2D material segmentation and rendered appearance rather than direct pixel-wise errors between predicted and ground-truth roughness/metalness maps. The Objaverse evaluation in Table 8 controls geometry and albedo in the Baseline/Baseline + Ours comparison and uses target materialized renderings, but authored material maps from heterogeneous public assets are not a calibrated continuous-material benchmark. A dedicated evaluation set with standardized ground-truth PBR maps would enable more direct measurements of roughness/metalness error, UV-space material accuracy, cross-view consistency, and final rendering quality. We consider this an important extension of the present evaluation rather than a claim already established by the current experiments.

Third, the multi-view camera configuration, the high-value-view weight $\alpha =2$, the $5\%$ region-unification threshold, class-balanced sampling, and SAM initialization are fixed implementation choices. The MIO-versus-MIO++ comparison in Table 7 evaluates the effect of the enlarged/finer dataset, but the present study does not perform a full factorial ablation over all pipeline hyperparameters. Such a study, including the number and elevation of views and alternative vote weights, would provide a more complete sensitivity analysis.

Fourth, the plane-simplification stage is introduced as auxiliary preprocessing for visibly irregular AI-generated meshes and is currently supported mainly by qualitative examples. A dedicated geometry study could quantify normal consistency, surface smoothness, geometric deviation, detail preservation, and rendering changes before and after simplification, and could separately ablate normal estimation, K-means clustering, RANSAC filtering, and mesh simplification. These measurements require a geometry-focused benchmark and are outside the material prediction evaluation used here.

Finally, a reverse-rendering ambiguity remains for AI-generated RGB textures. As shown in Figure 14, illumination can be baked into the generated appearance, making it difficult to determine whether a color variation belongs to the intrinsic albedo or to lighting. Estimating a clean albedo together with material properties under unknown illumination remains an important direction for future work.

7. Conclusions

In this paper, we formulate surface material assignment for 3D assets as material prediction in 2D views followed by fusion in UV space. We expand the original MIO dataset to MIO++, containing 115,542 images, 12 object themes, and 30 fine-grained material categories with independent class-level roughness and metallic assignments. Based on these 2D priors, MaterialSeg3D projects multi-view predictions to UV space, combines them through weighted voting and region unification, and converts the resulting labels to PBR material maps. The expanded MIO++ training data improve material segmentation on the backward-compatible MIO evaluation, while controlled and qualitative 3D experiments show that the assigned material maps can improve relightable appearance when geometry and albedo are available. We also extend the workflow to AI-generated meshes with an auxiliary plane-oriented preprocessing stage. The current class-level PBR mapping and fixed fusion settings favor a practical, lightweight workflow; continuous spatial material prediction, systematic sensitivity analysis, and direct evaluation against standardized PBR ground truth remain useful directions for future work.