Relaxing Accurate Initialization for Monocular Dynamic Scene Reconstruction with Gaussian Splatting

Abstract

Monocular dynamic scene reconstruction is a challenging task due to the inherent limitation of observing the scene from a single viewpoint at each timestamp, particularly in the presence of object motion and illumination changes. Recent methods combine Gaussian Splatting with deformation modeling to enable fast training and rendering; however, their performance in real-world scenarios strongly depends on accurate point cloud initialization. When such initialization is unavailable and random point clouds are used instead, reconstruction quality degrades significantly. To address this limitation, we propose an optimization strategy that relaxes the requirement for accurate initialization in Gaussian-Splatting-based monocular dynamic scene reconstruction. The scene is first reconstructed under a static assumption using all monocular frames, allowing stable convergence of background regions. Based on reconstruction errors, a subset of Gaussians is then activated as dynamic to model motion and deformation. In addition, an annealing jitter regularization term is introduced to improve robustness to camera pose inaccuracies commonly observed in real-world datasets. Extensive experiments on established benchmarks demonstrate that the proposed method enables stable training from randomly initialized point clouds and achieves reconstruction performance comparable to approaches relying on accurate point cloud initialization.

1. Introduction

Recently, 3D Gaussian Splatting (3DGS) [1] has emerged as a remarkably expressive scene representation, bringing great breakthroughs in novel view synthesis by enabling fast, high-fidelity model training and GPU/CUDA-friendly, real-time rendering. The position, covariance, opacity, and directional appearance (typically spherical harmonics) of Gaussians are optimized to capture the complex geometry and appearance of a scene. Due to its effectiveness, researchers explore extending 3DGS to other scene-related applications [2,3,4,5], including monocular dynamic scene reconstruction [6].

The datasets used in monocular dynamic scene reconstruction are usually categorized as two types: synthetic [7] and real-world [6,8]. As mentioned in 3DGS, the initialization plays an important role in real-world datasets, especially in scenes with complicated background and areas not well covered by training cameras [1]. We can infer that the accurate initialization (mostly referred to high-quality point clouds) becomes more vital in dynamic scenes, as the discontinuity is manifested both in time and space. High-quality point clouds in real-world datasets are commonly obtained by Structure-from-Motion (SfM) techniques [9]. The process for obtaining point clouds from the monocular dynamic real-world dataset involves annotating masks for the static part in each frame and inputting masked images into COLMAP. In this way, sparse point clouds representing the static part of scenes are generated. Some methods [10] generate dense point clouds by simply inputting all images into COLMAP and further implement image undistortion, depth maps calculation, and fusion, point cloud fusion, and surface reconstruction to improve the initialization quality. However, on the one hand, annotating mask is quite cumbersome, and there might be scenes where SfM techniques struggle to converge [11]. On the other hand, obtaining dense point clouds requires extra computational cost and makes the process more complex.

Recently, some methods focus on achieving monocular dynamic scene reconstruction without utilization of COLMAP. SplineGS [12] leverages masks from datasets to separate static and dynamic regions and proposes a motion-adaptive spline method to control the dynamic gaussians. GFlow [13] utilizes depth and optical flow to initialize the point cloud. However, additional information as assistance is necessary for these methods, which requires the consumption of extra computing resources or complex annotation efforts.

In this work, we propose a novel optimization strategy for monocular dynamic scene reconstruction with Gaussian Splatting, which relaxes the requirement of accurate initialization and supports random point cloud as initialization, as shown in Figure 1 and Figure 2. This allows us to bypass the reliance on precise initialization, which is often difficult to obtain, particularly in complex dynamic scenes. Inspired by the method of obtaining initialized sparse point cloud in monocular dynamic real-world datasets as mentioned above, we firstly take all the training views as input to reconstruct a 3D static scene starting from random initialization, without considering the dynamic transformation over time. This initial reconstruction serves as a coarse representation of the scene and results in a 3D scene where the static areas are reconstructed quite well, while the dynamic areas appear blurry due to the lack of consideration for temporal changes (i.e., inconsistent supervision would lead to results converging towards the average of all supervisions). In this situation, when evaluating each view, the discrepancy between dynamic areas and ground truth will be much greater than that of static areas. Therefore, in the next stage, the error between renderings and ground truth images for each Gaussian is calculated based on the former 3D reconstruction as performed in [14]. A portion of Gaussians with the highest error values are activated as dynamic, that is, their properties are conditioned on time. The static and dynamic areas in the scene are separated and optimized by different Gaussians to effectively capture the unique characteristics of each in this stage. Furthermore, since real-world datasets usually lack exact accuracy on camera poses, we utilize an annealing jitter regularization term to enhance the robustness of the model.

In summary, our main contributions are as follows:

• We introduce an innovative optimization approach that relaxes the requirement of initial point cloud for monocular dynamic scene reconstruction, providing the possibility of using Gaussian Splatting in situations where obtaining accurate point clouds is challenging.
• We propose an error-based method that separates the Gaussian representing static areas and dynamic ones, which helps with precise and diverse control over different Gaussians.
• Extensive experiments on different datasets are conducted to demonstrate the effectiveness of our method, showing that our method using randomly initialized point clouds achieves comparable or even better results compared to methods trained with accurate point clouds.

2. Related Work

2.1. Dynamic Scene Reconstruction

Reconstructing dynamic scenes and synthesizing novel views as well as interpolating time intervals are challenging tasks in computer vision and graphics. Early works [16,17,18] in this field focus on constructing dynamic primitives or interpolation.

With the advent of Neural Radiance Field (NeRF) [19], it has demonstrated remarkable success in capturing the complex geometry and appearance of static scenes from multiple viewpoints. The ability of NeRFs has inspired researchers to extend their application to dynamic scenarios. One type of NeRF-based approaches [20,21,22,23,24,25] leverages time t as an additional input. However, this strategy entangles position and time variations, which constrain their adaptability for downstream applications. Another type of methods [6,7,26,27,28] utilizes deformation, which focuses on modeling the motions in the scene by applying deformations to an initial static reconstruction. These methods are particularly powerful for handling dynamic scenes where objects move and deform over time, representing the dynamic components of the scene as a series of deformations applied to a base one.

Three-dimensional Gaussian Splatting (3DGS) quickly attracts a lot of attention by enabling real-time radiance field rendering. Follow-up works [10,15,29] turn to leverage 3D Gaussians for dynamic scene reconstruction. DeformableGS [15] employs an MLP to model the deformation for Gaussians, while [10] introduces a hexplane-based encoder to enhance the efficiency of deformation query. GauFRe [29] separates the static and dynamic components and optimizes Gaussians for each, respectively. However, their reconstruction for real-world scenes is greatly dependent on accurate point clouds generated by SfM methods. Some recent works [12,13] utilize additional information (such as dynamic object masks and optical flow) to handle situation without using SfM, but they require extra computing resources or complex annotation efforts. There might be situations where SfM struggles to generate point clouds. To relax the initialization condition and not rely on other conditions, we propose an optimization strategy to start the reconstruction from random initialized point clouds.

2.2. Initialization for 3DGS

The initialization plays an important role in 3DGS as mentioned in their experiment [1]. In 3D Gaussian Splatting (3DGS), the initialization phase lays the groundwork for how the Gaussians are positioned and distributed within the scene, which directly influences the accuracy. A good initialization can significantly speed up the convergence of the model and improve the quality of the final reconstruction. For synthetic datasets, the optimization starts from random initialization as the camera captures the scene uniformly, and the background is pure white. However, for real-world datasets, the background is complex, and there are areas that the camera has not captured well. Using random initialized point clouds leads to performance drops. Thus, 3DGS utilized SfM point cloud as a prior for real-world scene reconstruction.

Nevertheless, since real-world scenes are far more complex, SfM methods may not successfully capture the geometry of the scene and fail to generate point clouds. RAIN-GS [30] proposes a strategy to train with random initialized point clouds and achieves performance on-par with or even better than 3DGS trained with accurate SfM point clouds.

In this paper, we explore relaxing the initialization for dynamic scenes, especially for monocular dynamic scenes, a challenging task inherently involves sparse reconstruction from a single viewpoint.

3. Preliminary

3.1. Three-Dimensional Gaussian Splatting

Three-dimensional Gaussian Splatting [1] proposes to fit a 3D scene with a collection of anisotropic 3D Gaussians, where each Gaussian is defined by its mean (i.e., center position) $\mu$ and covariance $\Sigma$ and featured with its opacity $\alpha$ and a set of spherical harmonics (SH) coefficients. For rendering from a specific view, 3D Gaussians are projected to 2D via world-to-image transformation.

$${\Sigma }^{\prime }=JW\Sigma W^T{J}^T,$$

(1)

where ${\Sigma }^{\prime }$ is the 2D covariance in camera coordinates, J is the Jacobian of the affine approximation of the projective transformation, and W is the viewing transformation matrix. With the projected 2D Gaussians, the color is calculated via alpha blending as follows:

$$C=\sum _{i\in N}{c}_i{\alpha }_i^{\prime }\prod _{j=1}^{i-1}(1-{\alpha }_j^{\prime }).$$

(2)

Here, N is the number of Gaussians, $c_i$ denotes the view-independent color value of each 2D Gaussian calculated with SH coefficients, and ${\alpha }_i^{\prime }$ is determined by multiplying the opacity ${\alpha }_i$ of the i-th 3D Gaussian with the evaluation of the corresponding 2D Gaussian.

Given images captured from a scene, 3D Gaussians are optimized by minimizing the difference between renderings and ground truth images. During optimization, 3DGS utilizes an adaptive density control strategy to clone/split Gaussians in under-/over-reconstructed regions, which helps obtain representations with higher quality.

3.2. Deformation for Dynamics

Since there is only one frame available for each moment of a dynamic scene in monocular datasets, it is challenging to reconstruct the scene both spatially and temporally. Previous methods [6,8,10,15] seek to simplify the problem using deformation. Assuming the space at the starting time $t_0$ as canonical space, properties (e.g., position x, color c) representing the scene change along with time t:

$$x_t=T_x(x_{t_0},t),c_t=T_c(c_{t_0},t),$$

(3)

where $T_x$ and $T_c$ represent deformation in time for position and color, respectively. Please note that Equation (3) is just an example of how properties are deformed. The deformation could be applied to any properties, primarily depending on what the method is concerned about.

4. Method

There exist scenes that are challenging for SfM techniques to converge [11]. In such situation, accurate point clouds cannot be obtained for initialization, causing difficulty in reconstruction using Gaussian Splatting methods. As analyzed in 3DGS, real-world scenes with complex background and areas not well covered from training views are challenging to be reconstructed from randomly initialized point clouds. Thus, our method focuses on relaxing precise initialization on real-world datasets in monocular dynamic scene reconstruction with Gaussian Splatting.

Inspired by point clouds provided in real-world datasets [6,8], which are generated by running COLMAP using images with background masks, we first regard the scene as a static one and reconstruct it using each frame from the monocular video. This allows us to initially achieve a good reconstruction of the static areas, which are essentially the background, while inherently ignoring areas of the scene that change over time. These dynamic regions tend to exhibit poorer reconstruction results, typically appearing blurred. Subsequently, we evaluate the error of Gaussians in each view, and activate a portion of Gaussians with the highest error values as dynamic. The scene is now divided into static and dynamic parts, which are optimized separately. An additional regularization term—annealing jitter—is applied to reduce the deviation caused by inaccurate poses when test images are rendered. We will first discuss the scene modeling process in Section 4.1, and then introduce the regularization term in Section 4.2.

4.1. Dynamic Scene Modeling

4.1.1. Initial Scene Modeling as Static

Due to the fact that positions, shapes, and appearances of some objects in dynamic scenes change over time, there are differences between each frame in monocular dynamic videos. The differences between each frame in monocular dynamic videos can be subtle or significant, depending on the nature of the motion and the time interval between frames. However, the variations represent only a small portion of the scene, as most areas in real-world datasets are static, commonly referred to as the background. Given inconsistent frames, we regard the scene as static for initial reconstruction, which means we forcibly treat dynamic objects as static ones as well. Through this approach, consistent areas from different viewpoints (the background) can be well reconstructed, but it inevitably leads to artifacts in the inconsistent areas (i.e., dynamic areas), as the true motion are not yet accounted for.

Following RAIN-GS [30] where the constraint for accurate initialization of 3DGS is removed, we randomly generate a point cloud with sparse-large-variance Gaussians and use it as our scene initialization. After a few epochs of optimization, the static areas of the scene converge quickly to a stable solution, with the differences between renderings and the ground truth images being smaller compared to the dynamic ones.

4.1.2. Error-Based Dynamic Activation

Based on the scene modelled above, we can render an image where static areas are well reconstructed, while the originally dynamic areas are in low quality. Thus, we can distinguish between dynamic and static areas based on the varying degrees of difference between renderings and ground truth images. However, the error is commonly calculated in pixel level, e.g., mean average error (i.e., $L_1$ loss), and each pixel error entangles the contribution of multiple Gaussians. This entanglement makes it challenging to accurately attribute the error to specific Gaussian primitives within the scene, as the influence of each Gaussian is intertwined across the entire image. Since all the optimizations are carried out on Gaussians in Gaussian Splatting, we need to figure out the error for each Gaussian to distinguish static and dynamic areas in Gaussian level.

It is worth emphasizing that the terms dynamic and static in our formulation do not strictly correspond to semantically dynamic or static objects in the scene. Instead, they describe whether a region exhibits consistent appearance over time under monocular observations. The core objective of our method is therefore not to explicitly disentangle static objects from dynamic ones, but to first stabilize optimization by reconstructing the scene under a static assumption without accurate initialization, and subsequently apply dynamic correction to regions that consistently violate this assumption as indicated by elevated reconstruction errors.

To propagate per-pixel errors to per-Gaussian errors, we rethink the alpha-blending process in Gaussian Splatting. In Equation (2), the alpha-compositing coefficient could be represented as

$$w_i={\alpha }_i^{\prime }\prod _{j=1}^{i-1}(1-{\alpha }_j^{\prime }).$$

(4)

When distributing per-pixel errors to each Gaussian, the process is reversed and the distribution should be proportional to the contribution of each Gaussian to the pixel, i.e., $w_i$. Specifically, from a view $\pi$, the rendering error for pixel $\mathit{u}$ is ${\mathcal{E}}_{\pi }\left(\mathit{u}\right)$. The error for each Gaussian g under view $\pi$ could be calculated as followed:

$$E_g^{\pi }=\sum _{\mathit{u}\in \mathrm{Pix}}{\mathcal{E}}_{\pi }\left(\mathit{u}\right)w_g^{\pi }\left(\mathit{u}\right),$$

(5)

where the sum runs over the image pixels. For each Gaussian, we track the maximum value of the error $E_g^{\pi }$ across all training views, i.e.,

$$E_g=\underset{\pi \in \mathrm{View}}{max}{E}_g^{\pi }.$$

(6)

We now obtain per-Gaussian errors for all the Gaussians in the former reconstructed scene. Gaussians with high error values are inherently considered to be the dynamic parts of the scene. We sort the Gaussians based on their error values and activate the top scoring Gaussians as dynamic.

Notably, this error-based criterion also effectively handles objects undergoing fast and large-displacement motion across the scene. When such objects are forced to conform to the static assumption during the initial reconstruction, they tend to form elongated, low-opacity structures distributed along their motion trajectories. These structures consistently produce large reconstruction errors across multiple views, causing the associated Gaussians to be stably ranked in the high-error regime and selected for dynamic activation.

After the activation, the scene is represented by static Gaussians for background and dynamic Gaussians for motion, respectively. The properties of dynamic Gaussians are conditioned on time using deformation described in Section 3.2. By applying deformations that are time-dependent, we can track the movement of each Gaussian, reflecting the actual motion of the objects they represent.

4.2. Annealing Jitter Regularization

The most challenging problem for real-world datasets compared to synthetic datasets is their lack of accuracy in camera pose estimation, which can significantly impact the quality of scene reconstruction [6]. Imprecise camera poses can lead to overfitting of the training views. As a result, the reconstructed scene becomes tailored to the particularities of the training views. Previous methods like [15] only take the degradation of rendering quality at interpolated times into account, without considering the quality of interpolated views.

To improve our model’s robustness when rendered from an interpolated view and time, we propose to utilize a regularization term to reduce the effect of pose inaccuracies on the reconstruction quality:

$$\begin{array}{c}{\mathcal{X}}_t\left(i\right)=\mathbb{N}(0,1)·\beta ·\Delta t·(1-i/\tau ),\\ {\mathcal{X}}_p\left(i\right)=\mathbb{N}(0,1)·\beta ·\Delta p·(1-i/\tau ).\end{array}$$

(7)

Here, ${\mathcal{X}}_t\left(i\right)$ and ${\mathcal{X}}_p\left(i\right)$ represent the linearly decaying Gaussian noise at the i-th training iteration for time and camera pose, respectively, $\mathbb{N}(0,1)$ denotes the standard Gaussian distribution, $\beta$ is an empirically determined scaling factor with a value of 0.1, $\Delta t$ is mean time interval and $\Delta p$ is mean pose interval, and $\tau$ represents the threshold iteration for annealing smooth training (empirically set to 20k).

We represent how the regularization term is applied to our model using position x as an example:

$$x_t^{\prime }=T_x(x_{t_0},t+{\mathcal{X}}_t\left(i\right)).$$

(8)

The deviation of the original camera pose p as $p+{\mathcal{X}}_p\left(i\right)$ affects the rendering view with noise, simulating the inaccurate camera pose jitter and making the model more robust to inaccurate camera pose in training.

By incorporating this regularization term into our model, we can reduce the impact of pose inaccuracies on the reconstruction process. This leads to more reliable and accurate reconstructions, even when rendered from interpolated views and times. The result is a model that is not only robust to variations in the input data but also capable of generalizing well to new conditions, making it well suited for real-world dataset applications.

5. Results

5.1. Implementation Details

We incorporate our optimization strategy into two existing Gaussian-Splatting-based methods which use different deformation networks for dynamic scene reconstruction, i.e., Deformable-GS [15] utilizing MLP and 4DGS [10] utilizing hex planes. We replace their initialization with random point clouds, firstly consider the scene as static and then separate the scene into static and dynamic. Following RAIN-GS [30], we set our initial number of random Gaussians to $N=10$. After 5k epochs of training, we implement error-based dynamic activation and set Gaussians with top 15% error values as dynamic. These dynamic Gaussians are jointly trained with corresponding deformation network. The loss function and its hyper-parameters are consistent with the original version and we leave the adaptive density control unchanged. All the experiments are performed on a single NVIDIA RTX 3090 (24G) GPU. Our code will be released.

Datasets

We focus on real-world scenes. We utilize scenes provided by HyperNeRF [6] and NeRF-DS [8]. The division on training and testing sets and the image resolution align perfectly with the original paper.

5.2. Quantitative Results

We conduct quantitative comparisons with NeRF-based methods [6,8,21], which do not need any particular point cloud initialization, and Deformable-GS and 4DGStrained with SfM-initialized point cloud and with random-initialized point cloud using our optimization strategy.

We report PSNR, LPIPS, and SSIM as evaluation metrics between the reconstruction renderings and ground truth images as previous methods. These metrics provide a comprehensive assessment of the reconstruction quality by evaluating different aspects of the rendered images compared to the actual scene captures.

Table 1. Quantitative comparisons on NeRF-DS dataset. NeRF-based methods like TiNeuVox [21], HyperNeRF [6], and NeRF-DS [8] are trained without points initialization. SfM ¹ refers to sparse SfM points generated by COLMAP and SfM ² refers to dense ones.

Method	TiNeuVox	HyperNeRF	NeRF-DS	DeformableGS		DeformableGS + Our Strategy	4DGS			4DGS + Our Strategy
Init. Points	-	-	-	SfM 1	Random	Random	SfM 2	SfM 1	Random	Random
PSNR	21.61	23.45	23.60	24.11	23.7513	23.89	23.47	21.01	20.10	22.6695
LPIPS	0.2766	0.1990	0.1816	0.1769	0.1859	0.1901	0.1651	0.2887	0.3417	0.2259
SSIM	0.8234	0.8488	0.8494	0.8524	0.8461	0.8491	0.8288	0.7243	0.6683	0.8203

and

Table 2. Quantitative comparisons on HyperNeRF vrig dataset. SfM ¹ refers to sparse COLMA SfM pointsP and SfM ² refers to dense ones. Our strategy shows on-par or even better performance compared to methods trained with SfM points.

		3D Printer			Broom
Method	Init. Points	PSNR	LPIPS	SSIM	PSNR	LPIPS	SSIM
DeformableGS	SfM 1	20.78	0.2846	0.6579	20.03	0.7006	0.2679
	Random	20.36	0.3871	0.6165	19.56	0.8325	0.2351
DeformableGS + Our Strategy	Random	20.43	0.3137	0.6341	20.29	0.4074	0.3742
4DGS	SfM 2	21.94	0.3227	0.7084	22.24	0.5451	0.3858
	SfM 1	21.75	0.3144	0.6792	21.13	0.5802	0.3393
	Random	20.70	0.3915	0.6745	20.46	0.5821	0.3169
4DGS + Our Strategy	Random	21.63	0.3203	0.7036	21.46	0.5693	0.3483
DeformableGS	SfM 1	23.16	0.2286	0.6273	25.96	0.1603	0.8502
	Random	23.13	0.2322	0.6279	21.10	0.2502	0.7462
DeformableGS + Our Strategy	Random	23.26	0.3063	0.6533	25.04	0.1751	0.8136
4DGS	SfM 2	28.83	0.2771	0.8158	28.69	0.1842	0.8666
	SfM 1	27.79	0.2566	0.7855	22.72	0.2805	0.7439
	Random	27.11	0.3164	0.7152	24.52	0.2617	0.7721
4DGS + Our Strategy	Random	27.50	0.2729	0.7732	26.54	0.2369	0.8620

show that both Deformable-GS and 4DGS are highly dependent on the accuracy of initial point clouds. Their performances drop greatly when trained with random point cloud. In contrast, our strategy trained from random point cloud shows comparable or even better results compared to their methods trained with accurate SfM point clouds.

5.3. Qualitative Results

We show our results in Figure 3 and Figure 4 on NeRF-DS dataset implemented on DeformableGS and 4DGS, respectively. By comparing the last two columns, we demonstrate the effectiveness of our strategy, which reconstructs complete structure in the scene and removes high-frequency floaters or artifacts. In Figure 5, we show different timestamps and views of the scene peel-banana in HyperNeRF dataset carried out with DeformableGS. Thanks to our operation of training static and dynamic Gaussians separately, different Gaussians can be optimized for their respective roles within the scene. This allows us to refine the static Gaussians to accurately represent the unchanging parts of the environment, while the dynamic Gaussians are free to adapt and evolve in response to the motion and changes occurring over time. Moreover, by training static and dynamic Gaussians separately, we can more effectively manage computational resources during the training process. Static Gaussians, once converged, require less frequent updates, allowing us to allocate more computational power to the dynamic Gaussians that need more iterations to accurately model the evolving scene. A supplementary video demonstrating the qualitative results of our method is provided in the Supplementary Materials.

5.4. Ablation Study

In

Table 3. Ablation study on the ratio of activation, the epochs to start up activation, and the impact of the annealing jitter regularization term in our strategy. ✓ indicates that the annealing jitter regularization term is enabled, whereas ✗ indicates that the regularization term is not applied.

Activation Ratio	Activation Epoch	Reg.	PSNR	LPIPS	SSIM
0.1	5000	✗	22.64	0.2212	0.8284
0.2	5000	✗	22.82	0.2285	0.8241
0.25	5000	✗	22.41	0.2351	0.8192
0.15	3000	✗	21.34	0.2563	0.7891
0.15	4000	✗	23.16	0.2170	0.8185
0.15	6000	✗	19.91	0.2922	0.7666
0.15	7000	✗	22.18	0.2291	0.8204
0.15	5000	✗	23.20	0.2175	0.8329
0.15	5000	✓	23.89	0.1901	0.8491

, we validate the effectiveness of each setting in our strategy trained in the NeRF-DS dataset [8] with randomly initialized point clouds. Our ablation study systematically evaluates the contribution of different components of our approach, providing insights into how each aspect influences the final reconstruction quality. We compare the ratio of Gaussians for dynamic activation in the first part and explore the most suitable proportion of dynamic Gaussians that are activated and optimized separately. We also compare the epochs for initial scene reconstruction. Too few epochs might lead to an underfit and result in the inaccuracy of subsequent error-based dynamic activation. Conversely, too many epochs can lead to overfitting, reducing the efficiency for convergence. The annealing jitter regularization term is applied to reduce the impact of camera pose inaccuracies and increase robustness for real-world scene reconstruction. We present the quantitative results of reconstruction with and without the regularization in the last two rows. All three ablations show that using our settings performs better and using all three settings achieves the best.

We note that the current choice of a fixed activation ratio is primarily motivated by simplicity, stability, and reproducibility. Nevertheless, the error-based formulation of our method naturally lends itself to more adaptive strategies. For example, the activation threshold could be determined based on the statistical properties of the error distribution, such as bimodal separation of per-Gaussian errors, adaptive percentile selection, or even learnable gating mechanisms. Exploring such adaptive or data-driven activation criteria is a promising direction for future work and may further improve robustness across scenes with varying levels of dynamic complexity.

6. Discussion

Our work was motivated by the hypothesis that the strong dependence of Gaussian-Splatting-based monocular dynamic scene reconstruction on accurate point cloud initialization is not intrinsic, but instead arises from jointly optimizing static geometry and dynamic deformation from the beginning of training. The experimental results support this hypothesis, showing that by first reconstructing the scene under a static assumption and subsequently activating dynamic Gaussians based on reconstruction errors, comparable performance can be achieved even when starting from randomly initialized point clouds. In contrast to prior methods such as DeformableGS and 4DGS, which rely heavily on SfM-derived geometric priors, the proposed strategy alleviates initialization sensitivity without introducing additional supervision or external motion cues.

From the perspective of previous studies on dynamic scene reconstruction, the effectiveness of the proposed approach can be attributed to the predominance of static content in real-world monocular datasets. Similar to observations in NeRF-based dynamic reconstruction, enforcing a static model in the early optimization stage enables stable convergence of background geometry, while dynamic regions consistently exhibit higher reconstruction errors. By exploiting these residuals, the error-based dynamic activation mechanism provides an implicit yet representation-aligned way to disentangle static and dynamic components at the Gaussian level. The ablation results further indicate that both the timing and proportion of activated Gaussians are important for balancing reconstruction accuracy and optimization stability.

Another important finding is that the proposed annealing jitter regularization improves robustness to camera pose inaccuracies, which are common in real-world monocular capture. While prior work primarily emphasizes temporal deformation modeling, these results suggest that controlled stochastic perturbations during training can effectively reduce overfitting to noisy poses and enhance generalization to interpolated views. This observation is consistent with broader trends in representation learning and highlights pose robustness as a complementary factor to geometric initialization in dynamic scene reconstruction.

In a broader context, the proposed strategy lowers the practical barrier to deploying Gaussian-Splatting-based methods by reducing reliance on SfM pipelines and dense point cloud preprocessing. This has implications for applications such as robotics, augmented reality, and free-viewpoint video, where monocular input is often the most accessible data modality. Nevertheless, the method assumes that dynamic regions produce consistently higher reconstruction errors under a static model, which may not hold for scenes with subtle motion. Future work may explore adaptive or learnable activation criteria, joint optimization of camera poses and scene representation, and tighter integration with Gaussian densification strategies to further improve robustness and scalability.

7. Conclusions

In this work, we present a novel optimization strategy for monocular dynamic scene reconstruction with Gaussian Splatting, which starts from randomly initialized point clouds. We initiate the reconstruction process by treating the scene as static across monocular video frames and using sparse-large-variance Gaussians to model it. We then assess the error of Gaussians and activate the most erroneous Gaussians as dynamic and use them to simulate the motions and deformations of the scene. An additional regularization term is introduced to mitigate the impact of camera pose inaccuracies. Extensive experimental results demonstrate the effectiveness of our method on real-world scenes and shows the strength of our approach against state-of-the-art in terms of using random point clouds as initialization.