SD-GS: Gradient-Semantic Analysis Based on Multi-State Scene 3D Gaussian Splatting

Abstract

By analyzing the semantic information of Direct Current (DC, the zeroth-order spherical harmonic coefficient) gradients during 3D Gaussian Splatting (3DGS) optimization, this paper achieves unsupervised state classification in scenes with discrete appearance states under the proposed State-Discovery Gaussian Splatting (SD-GS) framework via SVD dimensionality reduction and K-means clustering. To improve the stability of the clustering results, an appearance-difference-weighted refinement mechanism is further proposed to confirm high-confidence labels. To address the difficulty of distinguishing similar states when the number of states exceeds two, a sequential peeling strategy is proposed that decomposes a multi-class partition into several two-class separations. On four real-world scene datasets, SD-GS achieves 100% classification accuracy with reconstruction quality of 31.98–38.83 dB PSNR. Ablation studies validate the effectiveness of the gradient direction mode and the SVD dimensionality reduction strategy.

1. Introduction

Explicit scene representation based on 3DGS [1] has been widely adopted in computational imaging and complex scene reconstruction [2,3,4,5,6] thanks to its efficient optimization and real-time rendering. During 3DGS training, the parameters of the Gaussians are continuously optimized by gradient descent to fit the input images. The gradient is usually treated merely as an intermediate quantity for parameter updates, yet as an iterative quantity it in fact encodes structural and appearance information of the scene.

More importantly, such gradient information already carries scene-state-related cues at early training stages. Existing studies show that gradient features in the 3DGS optimization process can be used for geometric detail identification, Gaussian-level semantic segmentation, and density control [7,8,9], indicating that the gradient is not a mere numerical byproduct but a usable signal carrying scene structure and semantics.

Among the appearance parameters of 3DGS, the DC coefficient determines the base color of a Gaussian and directly corresponds to the scene’s appearance. Therefore, for research along the color-related direction, the DC gradient is a more relevant object of study than gradients associated with geometry or opacity.

A number of works have addressed reconstruction under appearance variation by encoding per-image appearance features to reconstruct scenes under different illumination or weather conditions (e.g., WildGaussians; see Section 2 for details). Such methods mainly target scenes with continuously varying appearance, mostly rely on additionally learned appearance embeddings, and do not exploit the gradient semantics of the reconstruction process itself. Compared with continuously varying appearance, the gradient semantics in discrete-state scenes are more pronounced. Such discrete-state scenes are widespread and of practical value, e.g., red/green/yellow switching of traffic lights and different displays of variable message signs in intelligent transportation, repeated captures of equipment under different display states in digital twins, and indicator on/off states and panel states under different operating modes in industrial equipment monitoring.

Building on this, we propose the State-Discovery Gaussian Splatting (SD-GS) framework for unsupervised state classification and 3D reconstruction of discrete appearance-state scenes. Given the number of discrete states K, SD-GS automatically identifies the discrete state of each image using DC gradient semantics at early stages of 3DGS training, without any prior annotation or an additional classification network.

The main contributions of this paper are as follows:

• We propose the SD-GS framework that achieves unsupervised image state classification during 3DGS training. By analyzing the gradient direction differences of DC coefficients induced by different state images on the Gaussian primitives, combined with SVD dimensionality reduction and clustering with voting, the framework automatically completes the discrete state partition of images at early training stages, without any image-level annotation or an additional classification network.
• We propose an appearance-difference-weighted refinement mechanism. Using the rendering differences after the divergence of state appearance branches, our framework constructs pixel-level weights that focus the comparison on the local regions where state changes occur, verifying and refining the clustering results to improve the stability and consistency of classification.
• We propose a sequential peeling strategy for multi-state ($K>2$) scenes. To address the problem that a single principal component cannot simultaneously distinguish multiple similar states when the number of states exceeds two, we adopt class-by-class peeling: the most separable state is first isolated and fixed, and the discriminative regions are then relocated for the remaining states before separating them individually, effectively alleviating the entanglement of similar states in multi-state scenes.
• We validate the effectiveness of the proposed method on multiple real-world discrete-state scenes. On four datasets (with both $K=2$ and $K=3$ settings), the method obtains stable state partition results and achieves a reconstruction quality of 31.98–38.83 dB PSNR.

2. Related Work

Novel view synthesis and 3D scene reconstruction are among the core problems in computer vision and computer graphics. Neural Radiance Fields (NeRFs) [10], proposed by Mildenhall et al., implicitly encode the radiance field of a 3D scene with a multilayer perceptron and achieve photorealistic novel view synthesis through differentiable volume rendering, driving rapid progress in this field. However, NeRF’s dense per-ray sampling incurs substantial training and rendering costs. To improve multi-scale rendering quality, Mip-NeRF [11] introduces a multi-scale scene representation that effectively suppresses aliasing across resolutions; to overcome the efficiency bottleneck, Instant-NGP [12] replaces conventional positional encoding with a multiresolution hash encoding, reducing the training time from hours to the order of seconds.

Unlike these implicit representations, 3DGS [1], proposed by Kerbl et al., explicitly represents a scene with a set of anisotropic 3D Gaussians and achieves differentiable rendering through rasterization, attaining real-time frame rates while maintaining rendering quality, and has become an important paradigm for novel view synthesis. Around its rendering quality, structural compactness, and geometric expressiveness, subsequent works have improved it from several directions. Mip-Splatting [13] introduces a scale-adaptive filtering mechanism combining 3D smoothing and 2D Mip filtering to suppress aliasing across sampling rates; Scaffold-GS [14] replaces the unstructured Gaussian distribution with an anchor-guided structured representation, substantially improving compactness and rendering quality; SuGaR [15] introduces a surface regularization term to align Gaussians with the scene surface and efficiently extracts editable triangle meshes from the Gaussian representation via Poisson reconstruction. Building on this, and targeting the illumination changes, weather differences, and exposure shifts commonly present in real captures, WildGaussians [16] introduces per-Gaussian and per-image appearance embeddings combined with an appearance MLP to describe inter-image illumination and tone differences. Gaussian in the Wild [17] decomposes each Gaussian’s appearance into intrinsic and dynamic components, separately modeling stable scene attributes and image-varying environmental factors. Wild-GS [18] further explicitly decomposes appearance into material, global illumination, and camera-response-related factors for finer appearance modeling. RobustSplat [19] suppresses the interference of transient objects and abrupt illumination changes through delayed densification and multi-scale transient mask bootstrapping.

These methods model appearance variation as per-image embeddings in a continuous latent space, suitable for scenes with gradually varying appearance. However, when scene appearance switches among a finite number of discrete states (e.g., indoor lighting on/off, or alternating sunny/rainy weather), continuous appearance representations cannot explicitly produce a discrete state partition, nor do they exploit the state-discriminative information embedded in the optimization process itself.

Recently, several works have begun to explore the rich information carried in 3DGS optimization gradients. Gradient-Direction-Aware Gaussian Splatting [7], proposed by Zhou et al., quantifies the consistency of sub-gradient directions through a Gradient Coherence Ratio to identify geometrically detailed regions and introduces a nonlinear dynamic weighting mechanism to guide adaptive densification, effectively alleviating over-reconstruction and over-densification. Gradient-Driven 3D Segmentation [8], proposed by Joseph et al., uses masked gradients as voting signals and achieves accurate Gaussian-level semantic segmentation through cross-view gradient accumulation, enabling the transfer of 2D annotations to 3D space. Opacity-Gradient-Driven Density Control [9], proposed by Elrawy and Mohammed, uses the opacity gradient as a lightweight proxy for rendering error to replace the conventional position-gradient heuristic for triggering densification and, combined with a conservative multi-stage pruning strategy, achieves a compact and efficient Gaussian representation in few-shot settings.

These works collectively show that gradients in the 3DGS optimization process are not merely numerical byproducts but usable signals carrying scene geometry and semantic discriminative information. However, existing gradient-semantic studies mainly serve geometry-level tasks (e.g., detail identification, semantic segmentation, and density control) and have not explored the intrinsic connection between gradient semantics and image appearance states, nor have they applied it to the unsupervised discovery and partitioning of discrete appearance states. In summary, existing appearance-aware methods rely on continuous per-image appearance embeddings and struggle to characterize discrete states, whereas existing gradient-semantic studies focus on the geometric level and do not address appearance states. Neither exploits the gradient semantics in the optimization process for unsupervised classification of discrete appearance states. This paper addresses this gap by leveraging the state semantics carried by DC gradients in early training to achieve unsupervised classification and multi-state reconstruction in discrete-state scenes.

3. Method

This section presents a framework for unsupervised image state classification and multi-appearance separation based on DC gradient direction differences in the 3DGS optimization process. By analyzing the DC gradient direction differences produced by different state images on the same Gaussian in discrete-state scenes, the framework achieves automatic separated optimization of K appearance branches at the early stage of 3D reconstruction. The following subsections introduce scene representation and parameterization (Section 3.1), DC gradient-semantic analysis (Section 3.2), and unsupervised state classification based on gradient direction differences (Section 3.3).

3.1. Unified Geometry and Decoupled Appearance Representation

For a set of N 3D Gaussians in the scene, the full parameterization of the i-th Gaussian is defined as

$${\Theta }_i=\{{\Theta }_i^{\mathrm{geo}},{\Theta }_i^{\mathrm{app},0},{\Theta }_i^{\mathrm{app},1},\dots ,{\Theta }_i^{\mathrm{app},K-1}\}$$

(1)

where the shared geometric parameters ${\Theta }_i^{\mathrm{geo}}$ include the Gaussian center ${\mathit{\mu }}_i\in {\mathbb{R}}^3$, scale ${\mathbf{s}}_i\in {\mathbb{R}}_{+}^3$, rotation quaternion ${\mathit{q}}_i\in \mathbb{H}$ with $\parallel {\mathit{q}}_i\parallel =1$, and opacity ${\alpha }_i\in [0,1]$. The covariance matrix is constructed as follows:

$${\Sigma }_i={\mathit{R}}_i{\mathit{S}}_i{\mathit{S}}_i^T{\mathit{R}}_i^T$$

(2)

where ${\mathit{R}}_i=\mathrm{quat}2\mathrm{rot}\left({\mathit{q}}_i\right)$ and ${\mathit{S}}_i=\mathrm{diag}\left({\mathit{s}}_i\right)$.

The independent appearance parameters ${\Theta }_i^{\mathrm{app},k}$ correspond to the spherical harmonic (SH) coefficients under the k-th state:

$${\Theta }_i^{\mathrm{app},k}={\mathbf{f}}_i^{\left(k\right)}={\left\{{\mathbf{c}}_{i,l}^{\left(k\right)}\right\}}_{l=0}^L$$

(3)

where ${\mathbf{c}}_{i,l}^{\left(k\right)}\in {\mathbb{R}}^3$ denotes the l-th order SH coefficient. The zeroth-order coefficient ${\mathbf{c}}_{i,0}^{\left(k\right)}$ represents the view-independent DC color component, while higher-order terms encode view-dependent variations. Since the DC component dominates the energy in the SH expansion, this work focuses on DC coefficient gradients for state analysis.

Geometric parameters are jointly optimized by all images to maintain multi-view consistency, while appearance parameters are maintained independently per state to avoid color interference across states.

3.2. DC Gradient-Semantic Analysis

In discrete-state scenes, the initial point cloud is obtained via joint Structure-from-Motion (SfM) reconstruction from mixed images, yielding initial colors ${\mathbf{c}}_i^{\left(0\right)}$ that reflect the average appearance across multi-state observations. At early training stages, Gaussian colors have not yet converged, and the DC gradients induced by different state images on the same Gaussian approximately point toward their respective target colors:

$${\mathbf{g}}_i^{\left(k\right)}\propto {\mathbf{c}}_i^{\left(k\right)}-{\mathbf{c}}_i^{\left(0\right)}$$

(4)

For Gaussians with significant appearance differences across states, the gradient directions exhibit clear separation determined by the physical appearance variation of the scene, and this separation is present from the early training stages. Define the cross-state color difference as follows:

$${\delta }_i=\underset{0\le j<k\le K-1}{\max }∥{\mathbf{c}}_i^{\left(k\right)}-{\mathbf{c}}_i^{\left(j\right)}∥$$

(5)

When ${\delta }_i$ is large, the angular separation between gradient directions of different states increases, while images from the same state produce similar gradient directions. Let ${\widehat{\mathbf{g}}}_i^{\left(j\right)}$ denote the normalized gradient direction, and define the directional variance:

$${\sigma }_i^{\mathrm{angle}}=1-∥{\displaystyle \frac{1}{M}}{\displaystyle \sum _{j=1}^M}{\widehat{\mathbf{g}}}_i^{\left(j\right)}∥\in [0,1]$$

(6)

A larger value indicates stronger state discriminability at the given Gaussian. Additionally, gradient magnitude also reflects appearance variation. By the triangle inequality, when ${\delta }_i>0$, there exists at least one state k such that $\parallel {\mathbf{g}}_i^{\left(k\right)}\parallel \ge {\delta }_i/2$. The magnitude variance

$$v_i^{\mathrm{value}}={\mathrm{Var}}_j\left(∥{\mathbf{g}}_i^{\left(j\right)}∥\right)$$

(7)

provides an alternative measure of state difference.

3.3. Unsupervised State Classification with Weighted Refinement

The analysis in Section 3.2 shows that Gaussians in regions with significant appearance differences exhibit separable gradient structures across different state images. Based on this property, this section employs gradient features for unsupervised image state classification.

The overall pipeline is illustrated in Figure 1, comprising four stages: gradient accumulation, clustering with voting, weighted refinement, and full training.

Figure 1. Overall pipeline of SD-GS.

First, the top N Gaussians ranked by ${\sigma }_i^{\mathrm{angle}}$ (or $v_i^{\mathrm{value}}$) are selected as the discriminative set $\mathcal{D}$. For each image n, its normalized gradient directions on $\mathcal{D}$ are concatenated into a feature vector ${\mathbf{f}}_n\in {\mathbb{R}}^{3N}$, followed by zero-mean normalization.

SVD decomposition is applied:

$$\mathbf{F}=\mathbf{U}\Sigma {\mathbf{V}}^T,p_n={\mathbf{U}}_{n,1}$$

(8)

The first principal component $p_n$ captures state-discriminative information. The singular value ratio ${\sigma }_1/{\sigma }_2$ reflects the degree of state separability. K-means clustering is applied on the one-dimensional projections $\left\{p_n\right\}$ to obtain initial state labels.

Multiple rounds of clustering results are accumulated through a voting mechanism. When image n receives more than $V_{\min }$ votes, its state label is confirmed, forming a preliminary confident subset. The remaining samples are left unlabeled for now.

Based on this preliminary confident subset, K sets of appearance parameters are trained independently. After sufficient iterations, distinct rendering differences emerge between appearance branches. Since appearance differences between states are typically concentrated in local regions, a uniform full-image loss comparison would dilute the discriminative signal. Therefore, the rendering differences between appearance branches are used to localize appearance-varying regions and weight the loss comparison:

$$\mathbf{W}=\underset{0\le j<k\le K-1}{\max }|{\mathbf{I}}^{\left(j\right)}-{\mathbf{I}}^{\left(k\right)}|$$

(9)

The weighted reconstruction error is computed under this weight, selecting the state with minimum error:

$$k_n^{*}=\arg \underset{k}{\min }{\displaystyle \frac{{\sum }_p{\mathbf{W}}_p·|{\mathbf{I}}_p^{\mathrm{gt}}-{\mathbf{I}}_p^{\left(k\right)}|}{{\sum }_p{\mathbf{W}}_p}}$$

(10)

$\mathbf{W}$ takes large values in appearance-varying regions and approaches zero in shared regions, focusing the comparison on state-relevant local areas. This weighted refinement is used to verify the preliminary confident subset obtained by voting. For each sample in this subset, the weighted reconstruction error under each state branch is computed; an image whose state with the minimum weighted error agrees with its clustering label is confirmed, while otherwise it is set as uncertain, thereby purifying the preliminary confident subset into a high-confidence subset. For all remaining uncertain images, each is rendered under all state branches, its PSNR against the ground-truth image is computed, and a voting mechanism assigns each image to the state with the highest PSNR, finally completing the state partition of the full dataset.

The above first-principal-component one-dimensional clustering effectively distinguishes two states when $K=2$. However, when $K>2$, the differences among multiple states are difficult to distinguish using a single principal component: the first principal component can only isolate a single high-confidence subset, while the discriminative information of the remaining subsets lies in lower-energy, noisier secondary principal components that cannot be separated simultaneously in the same principal-component space. Directly increasing the clustering dimensionality or the number of principal components does not solve this problem and instead becomes less stable due to the low signal-to-noise ratio of the secondary components.

To address this, we adopt a sequential peeling strategy that decomposes the K-state partition into several two-class separations. First, on all images, the most separable state is isolated using the first principal component, its high-confidence subset is confirmed through weighted refinement, and the remaining same-class images in the dataset are purified accordingly. Based on the local weighted-difference regions obtained in the first step, these regions are back-projected into 3D space to construct new discriminative points, on which gradients are re-accumulated and clustered for the remaining images to separate the next state, until all K states are classified. Compared with distinguishing all states at once in a single principal-component space, sequential peeling avoids the mutual entanglement of similar states in weak principal components, enabling the method to stably handle multi-state scenes with $K>2$.

4. Experiments

4.1. Experimental Setup

We evaluate the proposed method on four real-world scenes exhibiting discrete appearance state variations, whose basic information is summarized in Table 1. Ring Light and Duck are self-collected $K=2$ scenes, LOM-Bike is a $K=3$ scene from the public LOM dataset [20], and Keyboard is a self-collected $K=3$ scene.

Table 1. Dataset overview.

Scene	States K	Images	Description
Ring Light	2	677 (314/363)	Ring light on/off (local appearance variation)
Duck	2	667 (352/315)	Decorative light on/off (local appearance variation)
LOM-Bike	3	108 (36/36/36)	Three exposure conditions (global illumination variation)
Keyboard	3	393 (130/142/121)	Keyboard backlight three levels (local appearance variation)

Ring Light, Duck, and Keyboard are self-collected indoor scenes in which appearance varies only in local regions across states, while the background remains stable. Among them, Ring Light is a large-scale indoor scene, Duck is a small close-range object, and Keyboard corresponds to the three backlight levels of a keyboard ($K=3$), whose appearance variations are distributed across multiple local key regions and whose inter-state differences are relatively close, making it a more challenging multi-state scene. LOM-Bike is a public dataset captured under three exposure times, exhibiting overall global brightness variation.

All experiments were conducted on a workstation equipped with an NVIDIA RTX 4060 Ti GPU (16 GB). For all scenes, SfM was applied jointly, without state labels, to generate a unified camera pose set and a sparse initial point cloud.

Training used the Adam optimizer with exponential learning rate decay for 30,000 iterations. The SH degree was set to $L=3$. For state classification, gradient accumulation warm-up was set to 1500 iterations, clustering was performed every 100 iterations, the number of discriminative Gaussians was $N=1000$, and the voting threshold was $V_{\min }=4$. Appearance separation was activated at iteration 3000, and weighted refinement was performed 1000 iterations after activation.

Evaluation metrics included classification accuracy (computed under best-permutation matching) and reconstruction quality metrics: Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS).

4.2. Unsupervised State Classification Results

As shown in Table 2, for the $K=2$ scenes, Ring Light exhibits global illumination variation with gradient direction differences distributed over a larger area, yielding an initial accuracy of 100%, while Duck reaches an initial clustering accuracy of 99.6%. For the $K=3$ scenes, LOM-Bike, captured under three exposure conditions with pronounced state differences, reaches 100% in a single round of clustering, whereas Keyboard, whose three backlight levels are relatively close, suffers from mutual entanglement of similar states when distinguishing three states at once in a single principal-component space, giving an initial accuracy of only 70.3%, which improves markedly after sequential peeling separates the states one by one. The weighted refinement stage raises the accuracy of confirmed images to 100% across all scenes, and all scenes finally achieve fully correct classification.

Table 2. Classification accuracy at different stages.

Scene	Clustering	After Refinement	Final
Ring Light	100% (325/325)	100% (323/323)	100% (677/677)
Duck	99.6% (533/535)	100% (532/532)	100% (667/667)
LOM-Bike	100% (108/108)	100% (108/108)	100% (108/108)
Keyboard	70.3% (133/189)	100% (107/107)	100% (393/393)

Figure 2 shows the appearance difference heatmaps for all datasets, reflecting the spatial distribution of rendering differences between appearance branches. Keyboard and Duck show localized differences, Ring Light shows scene-wide differences, and LOM-Bike exhibits globally high response due to three-exposure variation. This heatmap corresponds to the spatial weight $\mathbf{W}$ used in weighted refinement in Section 3.3.

Figure 2. Appearance difference heatmaps: (a): Ring Light, (b) Duck, (c) LOM-Bike, (d) Keyboard. Brighter regions indicate larger appearance differences between states.

To further illustrate the advantage of the proposed method over image-level unsupervised methods, we add three comparison methods that group images solely based on the images themselves: the average image color, the color histogram, and image features extracted by the general-purpose vision model DINOv2 [21]. Each method performs clustering after extracting the corresponding features, and the classification accuracies are reported in Table 3.

Table 3. Classification accuracy of image-level unsupervised methods and SD-GS.

Scene	Average Color	Color Histogram	Vision Model	SD-GS (Final)
Ring Light	56.2%	71.3%	85.6%	100%
Duck	95.8%	100.0%	100.0%	100%
LOM-Bike	100.0%	100.0%	67.3%	100%
Keyboard	55.4%	51.2%	50.2%	100%

Image-level methods such as average color and color histogram achieve high discrimination only on Duck and LOM-Bike, whose overall appearance differs significantly, because state switching in these two scenes causes clear changes in the overall color or brightness of the image, so that image-level statistics suffice for separation. In contrast, Keyboard is the most challenging scene: the appearance differences among its three backlight levels occur only in a few local key regions, the global brightness of the scene remains essentially unchanged, and the brightness intensities of the varying regions across states are relatively close. This makes it difficult for any representation relying on whole-image statistics to capture the state differences, and all three image-level methods degrade to near-random performance on Keyboard (50.2–55.4%). SD-GS, by localizing the Gaussians where appearance changes occur through DC gradient semantics, still reliably distinguishes states even in such a complex scene, with almost no difference in global statistics. This indicates that the performance gain of the method stems from the ability of the proposed gradient-semantic mechanism to localize local state differences, rather than from the separability of the data itself.

4.3. Reconstruction Quality

The reconstruction quality of each scene after state classification is reported in Table 4.

Table 4. Reconstruction quality of each scene.

Scene	K	PSNR (dB) ↑	SSIM ↑	LPIPS ↓
Ring Light	2	31.98	0.959	0.078
Duck	2	37.62	0.976	0.067
LOM-Bike	3	38.83	0.948	0.101
Keyboard	3	33.48	0.934	0.098

Once state classification is correctly completed, all scenes achieve high reconstruction quality (PSNR above 31 dB and SSIM above 0.93). Among them, Duck and LOM-Bike attain the highest reconstruction quality (37.62 and 38.83 dB PSNR, respectively); Ring Light and Keyboard, whose appearance variations are distributed over more local regions, are relatively more difficult to reconstruct but still reach 31.98 and 33.48 dB, respectively. These results show that, once the states are correctly separated, each set of appearance parameters can produce high-quality independent reconstruction for its corresponding state.

To further illustrate the effect of state separation on reconstruction quality, we add two comparisons on the Keyboard scene: a standard single-branch 3DGS that does not distinguish states and trains on all mixed images jointly (lower bound), and an oracle setting that uses ground-truth state labels to separate the images of each state and trains an independent standard 3DGS for each (upper bound). The three are compared in Table 5.

Table 5. Reconstruction quality comparison on the Keyboard scene.

Method	Geometry	Labels	PSNR (dB)	SSIM	LPIPS
Standard 3DGS (mixed, lower bound)	Shared	None	31.12	0.951	0.090
SD-GS (ours, automatic)	Shared	None (automatic)	33.48	0.934	0.098
Oracle (GT labels, independent)	Independent	GT	37	0.961	0.062

The standard 3DGS that does not distinguish states trains on the images of multiple states jointly, forcing the shared appearance parameters to compromise across states and producing evident color mixing in the backlight region, as shown in Figure 3. Under the same shared geometry, SD-GS automatically separates the states and maintains independent appearance parameters for each, thereby avoiding such mixing and achieving reconstruction quality superior to the mixed-training lower bound. The oracle setting independently optimizes geometry and appearance for each state using ground-truth labels, without mutual constraints between states, representing the upper bound of reconstruction quality; its gap from SD-GS mainly stems from the fact that the multi-state data are captured separately for different states, whose camera trajectories cannot fully coincide, so that scene-boundary regions covered by only a single state lack multi-state information during training and appear blurred.

Figure 3. Color mixing of standard 3DGS mixed training on the Keyboard scene: (a) ground-truth image; (b) rendering result of mixed training.

Figure 4 shows the rendering results compared with the ground truth for all datasets. For the $K=2$ scenes (Ring Light, Duck), the ground-truth images and corresponding renderings under the two states are shown; for the $K=3$ scenes (LOM-Bike, Keyboard), the renderings of the three states under the same viewpoint are shown. The renderings remain highly consistent with the ground truth in color and structure, indicating that appearance separation optimization can produce high-quality independent renderings for each state once the classification is correct.

Figure 4. Multi-state rendering results: (a1,a2) Ring Light, (b1,b2) Duck, (c,c1–c3) LOM-Bike, (d1–d3) Keyboard. Numbers denote different states. GT denotes the input ground-truth image. Render denotes the corresponding rendered result.

4.4. Ablation Study

Our method uses gradient direction (angle) features combined with SVD dimensionality reduction before clustering. To validate these two design choices, we replace one of them at a time for comparison: replacing the angle feature with the value feature that captures only gradient magnitude, and removing SVD dimensionality reduction to perform K-means directly on the original $3N=3000$-dimensional features. The clustering-stage classification accuracies of these methods are reported in Table 6.

Table 6. Ablation on feature mode and dimensionality reduction strategy (clustering-stage classification accuracy).

Scene	Ours (Angle + SVD)	Value Feature	Direct High-Dim K-Means
Ring Light	100% (325/325)	84.1% (509/605)	66.8% (217/325)
Duck	99.6% (533/535)	95.9% (513/535)	75.3% (403/535)
LOM-Bike	100% (108/108)	87.0% (94/108)	60.2% (65/108)

4.4.1. Clustering Mode Comparison

The angle mode constructs features from gradient directions, while the value mode uses gradient magnitudes. As shown in Table 6, the two modes perform comparably on Ring Light and Duck, but on LOM-Bike the angle mode significantly outperforms the value mode. The state differences in LOM-Bike arise from exposure changes, producing clearly distinct gradient directions across states, whereas the value mode only captures magnitude changes and lacks discrimination in such scenes. This indicates that the value mode is more susceptible to noise during scene construction, and discriminative points with large magnitudes are difficult to localize precisely to the local varying regions, whereas the angle mode remains effective across all of the above scenes. Therefore, the gradient-direction-based angle mode exhibits more stable cross-scene applicability.

4.4.2. Dimensionality Reduction Strategy Comparison

The feature vector dimension is $3N=3000$. As shown in Table 6, SVD dimensionality reduction significantly outperforms direct high-dimensional K-means across all scenes, with an average improvement of approximately 25 percentage points; in particular, on the $K=3$ LOM-Bike scene, direct high-dimensional clustering reaches only 60.2%, indicating that the interference of high-dimensional noise on the discriminative signal becomes more severe as the number of states increases. In contrast, SVD effectively extracts the dominant state-difference direction, enabling stable clustering in the reduced space.

4.4.3. Effectiveness of Weighted Refinement

To validate the purifying effect of weighted refinement, we compare “weighted refinement” against “unweighted (verification by the uniform full-image PSNR)” on the $K=3$ Keyboard scene. The high-confidence subset confirmed by weighted refinement contains 107 images, all correctly classified (100%), whereas the unweighted setting confirms 121 images, of which only 75 are correct (62.0%). This indicates that, since the state differences are concentrated in local regions, a uniform full-image error is diluted by the large number of unchanged regions and leads to erroneous confirmations, while weighted refinement focuses on the local state-varying regions and screens out a high-confidence subset that provides reliable labels for appearance-branch training.

4.4.4. Back-Projected Discriminative Point Seeding

In the subsequent rounds of sequential peeling, the discriminative points are constructed by back-projecting the weighted appearance differences of the already-separated states into 3D space. To validate this design, we conducted a controlled experiment on the $K=3$ Keyboard scene, keeping all other settings identical and comparing only whether back-projected discriminative point seeding was used. All accuracies were computed on the 393 images under best-permutation matching. The results show that, with back-projected discriminative point seeding, the classification accuracy on Keyboard reaches 100% (393/393), whereas retaining the gradient-variance-based selection yields only 77.4% (304/393). Back-projection effectively reduces the noise introduced by the discriminative points and concentrates the new-round discriminative points on the regions that truly distinguish the remaining states, thereby markedly improving the separation of similar states.

4.5. Computational Cost

Since SD-GS maintains multiple sets of appearance parameters under a shared geometry and additionally performs gradient accumulation and clustering with voting in the early training stage, we measured its memory and time overhead relative to a standard single-branch 3DGS on the Keyboard ($K=3$) scene, as reported in Table 7.

Table 7. Computational cost comparison between SD-GS and standard single-branch 3DGS.

Method	Peak Memory (MiB)	Per-Step Time (ms)	Extra Classification Cost
Standard single-branch 3DGS	4905	24.9	—
SD-GS ($K=3$)	5341 (+8.9%)	24.9 (regular step)	∼30 s

In terms of memory, the multi-branch SD-GS consumes only about 8.9% more than the single-branch baseline, far below the branch-count ratio. This benefits from the shared-geometry design of our method: the geometric parameters (position, scale, rotation, and opacity) are shared across all states, and only the appearance parameters (DC and spherical harmonic coefficients) grow with the number of states, which account for a small fraction of the total parameters of a single Gaussian. Therefore, the storage cost of multi-state separation is small and does not grow linearly with the number of branches. In terms of time, the additional computation introduced by the classification mechanism is concentrated in the early clustering windows (about 250 rounds): a regular training step takes about 24.9 ms, whereas a classification step within a clustering window takes about 153.9 ms, and their cumulative difference amounts to about 30 s, a small fraction of the entire 30,000-step training process; gradient collection stops once completed in the early stage and does not affect the subsequent training speed.

5. Discussion

This section further analyzes the applicable conditions of SD-GS, the working mechanisms of its key designs, and its limitations, based on the experimental results.

Boundary of Multi-State Separability: The one-dimensional clustering of SD-GS based on the first principal component can stably distinguish two-state scenes. However, when the number of states exceeds two, the first principal component can typically isolate only one high-confidence state, while the discriminative information of the remaining states lies in lower-energy, noisier secondary principal components that cannot be separated simultaneously in the same principal-component space. Through sequential peeling, this work converts a single multi-class partition into several more tractable two-class separations, raising the accuracy of similar multi-state scenes such as Keyboard from about 70% with a single principal component to 100%. This analysis also delimits the applicable scope of the method: its effectiveness presupposes the existence of clear state differences in the scene that can be captured by gradient semantics.

Timing of the Discriminative Signal: The approximation relied upon by Equation (4) holds only at the early training stages: at this point, the DC colors of the Gaussians are still close to the SfM-initialized average color, the rendering residual is dominated by the difference between the current average color and the target state color, and the gradient direction therefore approximately points toward the target state color. As training proceeds and the state colors gradually converge, this directional signal weakens. Consequently, state clustering and voting must be performed within the window after the signal matures but before convergence, while the separated training of the appearance branches should be activated only after enough high-confidence labels have been accumulated. This timing dependency is a prerequisite for the reliability of the method, rather than a defect. As long as labels are confirmed within an appropriate window, the method works stably over a wide range of hyperparameters.

Role of Weighted Refinement: Appearance differences between states are often concentrated in local regions, and label verification based on a uniform full-image error would dilute the differences in the varying regions with the large number of unchanged regions. Weighted refinement constructs pixel-level weights from the rendering differences between state appearance branches, focusing the comparison on the state-varying regions and thereby providing a reliable basis for verification in scenes with local variations. The purpose of weighted refinement is to screen rather than to modify labels: it does not alter the clustering labels of images but instead confirms and locks the clearly reliable ones among them. Images that remain ambiguous under the weighted comparison are not confirmed and are instead left to the subsequent PSNR comparison across state branches for a more accurate decision. This both keeps the high-confidence labels used for training the appearance branches pure and avoids misjudging uncertain samples at this stage.

Relation to Shared Geometry: In our experiments, the reconstruction quality of the oracle setting is higher than that of SD-GS, and this gap stems from the data acquisition manner. The multi-state data in this work are captured separately for different states, and the camera trajectories of the separate captures cannot fully coincide, so scene-boundary regions covered by only a single state lack multi-state information during training and appear blurred; this does not affect the information in the key regions of the scene.

Limitations and Future Work: The method is applicable to scenes with a finite number of discrete appearance states and requires the number of states K to be specified in advance. For scenes with continuously varying appearance, the discrete modeling assumption no longer holds. Future work will further exploit the gradient semantics in the 3DGS optimization process and apply it to more generalized scenes, extending the applicability of gradient-semantic analysis to a broader range of reconstruction tasks.

6. Conclusions

This paper proposes the SD-GS framework, which leverages the state-discriminative information carried by DC gradient directions in the early stage of 3DGS training to achieve unsupervised state classification and multi-appearance reconstruction for scenes with discrete appearance states, without any image-level labels or an additional classification network. To address the entanglement of similar states in multi-state scenes, we further propose a sequential peeling strategy that decomposes a multi-class partition into several two-class separations. On four real-world scene datasets, SD-GS achieves 100% classification accuracy, with reconstruction quality of 31.98–38.83 dB PSNR.