A Relative Orbital Motion-Guided Framework for Generating Multimodal Visual Data of Spacecraft

Highlights

What are the main findings?

• A novel, physically grounded framework is proposed to generate large-scale synthetic visual datasets for non-cooperative spacecraft. It incorporates relative orbital motion simulation and end-to-end imaging degradation to produce realistic multimodal data (RGB, mask, depth, normal) with precise 6-DoF pose annotations for four representative spacecraft types.
• The generated dataset, comprising 8000 high-fidelity samples, effectively bridges the domain gap between synthetic and real on-orbit imagery. It specifically addresses the critical data scarcity issue in spacecraft visual perception, which arises from limited and inaccessible real-world data.

What are the implications of the main findings?

• This work provides a crucial, open-access data foundation and simulation tool for the development, benchmarking, and validation of data-driven algorithms (e.g., for pose estimation, segmentation, tracking) in on-orbit servicing and space debris removal, potentially accelerating related research and technology maturation.
• The proposed integrated pipeline establishes a new paradigm for generating physically accurate and task-specific synthetic data in remote sensing by integrating motion physics, multimodal ground truth, and sensor degradation. This paradigm can be adapted to other space and Earth observation applications facing similar data scarcity challenges.

Abstract

The advancement of on-orbit servicing and space debris removal missions has established high-precision visual perception for non-cooperative spacecraft as a key research focus. However, the availability of high-quality, diverse spacecraft image datasets is severely limited due to extreme on-orbit imaging conditions, data confidentiality, and morphological diversity of targets, significantly constraining the advancement of data-driven algorithms in this domain. To address this challenge, we propose a relative orbital motion-guided framework for generating multimodal visual data of spacecraft. The proposed method integrates an orbital dynamics model into the synthetic data generation pipeline to simulate typical relative motion patterns between the camera and the target in a realistic orbital environment, thereby generating image sequences characterized by continuous spatiotemporal evolution. Targeting four representative spacecraft—Tiangong, Spacedragon, ICESat, and Cassini—this work simultaneously generates a dataset comprising 8000 samples, each containing four strictly aligned modalities: RGB images, instance segmentation masks, depth maps, and surface normal maps, along with precise 6-degree-of-freedom (6-DoF) pose ground truth. Furthermore, an end-to-end physical image degradation model is developed to accurately simulate the complete imaging chain—from optical diffraction and aberrations to sensor sampling and noise—thereby effectively narrowing the domain gap between synthetic and real data. By addressing three key aspects—physical motion modeling, synchronous multimodal ground truth, and imaging degradation simulation—this work provides a crucial data foundation for training, testing, and validating data-driven on-orbit perception algorithms.

1. Introduction

With the increasing prevalence of space missions such as on-orbit servicing and space debris removal [1,2,3], 3D reconstruction [4,5,6,7] and robust pose estimation [8,9,10] for non-cooperative spacecraft have become critically important. Accurately determining a target spacecraft’s six-degree-of-freedom (6-DoF) pose is a fundamental prerequisite for autonomous rendezvous and docking, on-orbit maintenance, and active debris removal, directly impacting mission safety and success. However, acquiring visual data of real spacecraft faces multiple challenges: first, extreme on-orbit imaging conditions characterized by high-contrast illumination, rapid relative motion, and complex background interference; second, the typically classified and non-public nature of space mission data; finally, the morphological diversity and structural complexity of non-cooperative targets, often without prior model support. These factors collectively result in a severe scarcity of publicly available, high-quality, and diverse spacecraft image datasets [11], which significantly constrains the development and application of data-driven visual perception algorithms in this domain.

To address these challenges, this paper proposes a relative orbital motion-guided framework for generating multimodal visual data of spacecraft. Our method integrates orbital dynamics models [12,13,14] into the image generation stage to simulate the relative motion between the camera and the target spacecraft in a realistic orbital environment, producing image sequences that exhibit continuous temporal evolution across multiple relative distances and modalities. Focusing on four representative spacecraft targets—Tiangong, Spacedragon, ICESat, and Cassini—our selection is designed to maximize morphological diversity, thereby establishing a benchmark dataset capable of effectively evaluating the generalization ability of perception algorithms: Tiangong represents large, symmetric modular space stations, characterized by structural complexity and severe self-occlusion; Spacedragon represents symmetric, compact crew capsules, which pose challenges due to limited surface texture and the inherent ambiguity in pose estimation caused by symmetry; ICESat represents science satellites equipped with large, flat solar panels, challenging due to their non-centroidal structure and extensive low-texture regions; Cassini represents deep-space probes with large parabolic antennas and deployable booms, whose extreme geometric complexity tests an algorithm’s robustness to irregular structures and protruding components.

Our framework generates four types of strictly synchronized multimodal data for each target, with 500 samples per modality for each. The result dataset comprises 8000 images in total (4 targets × 4 modalities × 500 samples), with all modalities precisely aligned spatially and temporally. By grounding the generation process in physically consistent relative motion patterns, our method ensures that the synthetic data reflects various real mission scenarios, ranging from approach tracking to docking, while simultaneously providing rich multimodal ground truth to support algorithms for pose estimation, instance segmentation, and 3D reconstruction.

The main contributions of this work are as follows:

(1)

Relative motion modeling with orbital constraints: Orbital dynamics equations are integrated into the data generation pipeline to construct physically plausible relative motion trajectories, ensuring the kinematic realism of simulated scenarios.

(2)

Multi-source spacecraft ground truth data generation: RGB images, instance segmentation masks, depth maps, and surface normal maps are rendered synchronously, accompanied by precise 6-DoF pose ground truth, establishing a benchmark for related algorithm research.

(3)

Physics-driven image degradation model: An end-to-end imaging chain model incorporating optical degradation and sensor noise is developed, effectively narrowing the domain gap between synthetic and real satellite imagery, thereby enhancing the generalization capability of models trained on synthetic data for real-world tasks.

2. Related Works

For space manipulation missions, academia and industry have released various spacecraft image datasets. A classic benchmark for spacecraft pose estimation is the Spacecraft Pose Estimation Dataset (SPEED) [15], which was provided for a satellite pose estimation challenge jointly organized by the European Space Agency and Stanford University. This dataset aims to support space object pose estimation tasks and comprises 15,300 images of the Tango satellite. Of these, 15,000 images are synthetic, generated using the OpenGL-based optical camera simulation software, while the remaining 300 are real images captured by the Rendezvous and Optical Navigation (TRON) facility at the Space Rendezvous Laboratory (SLAB). The SPEED provides images with a resolution of 1920 × 1200, annotated with quaternions and 3D depth information. In 2021, SPEED+ [16] was introduced as the next-generation dataset to address the cross-domain gap in spacecraft pose estimation. It contains 59,960 synthetic images generated by simulation software, split into training and validation sets in an 8:2 ratio. Additionally, it provides test set images captured by the TRON facility under different illumination sources: 6740 images under lampbox illumination and 2791 images under solar lamp illumination. Park et al. introduced the SHIRT dataset [17], which extends the SPEED+ dataset by providing continuous image sequences of a target mock satellite in simulated rendezvous scenarios. The dataset encompasses both synthetic images generated with OpenGL and Hardware-in-the-Loop (HIL) images captured from the TRON facility’s lightbox, corresponding to the same pose labels for two representative rendezvous orbital element (ROE) sets: ROE1 and ROE2. Subsequently, the authors proposed the SP3ER dataset [18], which contains over 60 distinct types of space objects. For each object, the dataset provides 1000 RGB images along with their corresponding binary masks.

Huang et al. constructed a synthetic Wide Depth Illumination-Composite Dataset (WDICD) [19] for the spacecraft segmentation task. They proposed a pixel-level automatic semantic annotation method and, based on this, developed a structure-aware semantic segmentation framework for spacecraft components. Sam et al. introduced the SwiM spacecraft segmentation dataset [20], which comprises two versions: a baseline version (28,917 images) and an enhanced version (63,917 images), covering a variety of spacecraft configurations ranging from classic satellites to deep-space probes. The dataset’s unique value is reflected in three aspects: it pioneered the simulation of optoelectronic distortion and noise characteristics in space imaging; it achieved comprehensive coverage of illumination conditions, spacecraft poses, and environmental backgrounds by blending real and synthetic data; and all annotations are provided in a unified YOLO-format polygon contour representation, ensuring consistency in algorithm evaluation. Cao et al. generated the Spacecraft-DS dataset [21] through hardware-in-the-loop acquisition for key component detection and segmentation. This dataset encompasses two types of spacecraft, nine key components, three illumination types, and two motion states: approach and hover.

Proença et al. generated the URSO dataset [22] comprising images of the Soyuz and Dragon spacecraft using the Unreal Engine 4 simulator, with an image resolution of 1080 × 960. However, both the SPEED and URSO datasets primarily focus on pose estimation and do not provide any segmentation annotations. To address this, Duang et al. proposed a novel space imagery dataset [23] containing 3117 annotated space-based images of satellites and space stations at a resolution of 1280 × 720. The dataset covers 10,350 components from 3667 spacecraft, with the size of spacecraft in the images varying from as small as 100 pixels to nearly occupying the entire frame. To standardize benchmarking for segmentation methods, the dataset is partitioned into 2516 training images and 600 test images, and is intended for tasks such as spacecraft object detection, segmentation, and component recognition. Hematulin et al. created a synthetic dataset for spacecraft pose estimation [24] using the Unreal Engine 5 simulator. The work provides a detailed algorithm for pose sampling and image generation. The dataset includes images captured under harsh lighting conditions and against high-resolution backgrounds, featuring models of the Dragon, Soyuz, Tianzhou, and the Chang’E-6 ascent vehicle, totaling 40,000 high-resolution images. Each image is annotated with a pose vector representing the spacecraft’s relative position and orientation with respect to the camera.

Hu et al. introduced the SwissCube dataset [25], generated using physics-based rendering. This dataset accounts for the 3D model of the satellite, as well as the influences of star background, the Sun, and the Earth. SwissCube is placed in its actual orbit at approximately 700 km above the Earth’s surface, and an “observer” is positioned in a slightly higher orbit to render image sequences with varying relative velocities, distances, and viewing angles. The experiments generated 500 scenes, with each scene containing a sequence of 100 frames, resulting in a total of 50 k images with a resolution of 1024 × 1024, split into training and test sets in a 4:1 ratio. Bechini et al. proposed a tool for generating spaceborne image datasets [26], capable of producing noise-free synthetic images using ray tracing. Gallet et al. constructed and made publicly available a large-scale dataset, named the RAPTOR dataset [27], comprising 120,000 images. Each image not only provides appearance information but is also accompanied by rich ground truth annotations, including masks, distance maps, celestial positions, and precise camera parameters. The data generation process ensures fine statistical balancing to facilitate the training and evaluation of pose estimation solutions. The method employs a physics-based camera model and meticulously simulates space-specific lighting and radiation characteristics—such as solar flux and spectrum, Earth albedo, and a vacuum environment without atmospheric effects—as well as texture scattering, to produce highly realistic images.

During the 2021 IEEE International Conference on Image Processing, the Interdisciplinary Center for Security, Reliability, and Trust organized the first Spacecraft Recognition competition leveraging space environment knowledge, and provided the SPARK dataset [28] for tasks such as space object classification and detection. The SPARK dataset utilizes realistic 3D models of 10 distinct satellites and spacecraft (along with 5 debris models) to simulate images of space objects. The dataset enhances the fidelity of the simulated data by compositing the rendered targets onto high-resolution Earth background images. Yang et al. [29] constructed the NASA 3D dataset, which includes point clouds, watertight meshes, occupancy fields, and corresponding points, utilizing 3D models available on the official NASA website.

Existing public datasets have made considerable progress in advancing tasks such as object detection [30,31,32], pose estimation [33,34], segmentation [35,36] and relative navigation [37,38]. Existing datasets offer valuable training resources, yet their generation frameworks are limited by two main shortcomings in theory and practice. First, in terms of motion modeling, current methods primarily depend on static scenes or overly simplified motion trajectories that lack physical realism. They do not incorporate orbital dynamics models, and therefore cannot reproduce the kinematically and dynamically consistent relative motion observed in actual on-orbit missions. Second, regarding data completeness, available resources are mostly restricted to RGB images. The absence of synchronized, pixel-level ground-truth data for geometry (depth, normals) and segmentation hinders the development and evaluation of data-driven algorithms in tasks such as 3D reconstruction, accurate pose estimation, and model-based tracking—tasks that rely on rich multimodal annotations.

3. Method

This work establishes high-fidelity 3D models of the Tiangong, Spacedragon, ICESat, and Cassini spacecraft [39] based on publicly available 3D model resources. A novel method for generating multimodal spacecraft visual data guided by orbital relative motion is proposed. This method departs from traditional random pose sampling by incorporating an orbital dynamics model, systematically defining four typical relative motion scenarios: elliptical co-orbit, spiral, teardrop, and drift. This approach generates continuous relative motion trajectories that are physically plausible. To achieve high-fidelity imaging simulation, a parameterized camera model is further developed, and stringent physical constraints for space-based imaging are analyzed, including the geometric and optical visibility of the target within the camera’s field of view. Building on this, an automated rendering pipeline is constructed within the Blender platform using its Cycles physically based rendering engine. By configuring physically based shaders and multi-source lighting models, the pipeline synchronously generates per-frame RGB images, instance segmentation masks, depth maps, surface normal maps, and 6-degree-of-freedom (6-DoF) pose ground truth data. The resulting spacecraft image dataset covers a wide range of distances, viewing angles, lighting conditions, and motion patterns, providing a robust data foundation for algorithm training and evaluation in spacecraft visual tasks such as pose estimation and 3D reconstruction.

3.1. Relative Orbital Motion Pattern Modeling

This section establishes a mathematical model for the relative orbital motion pattern between the target spacecraft and the observation spacecraft, which serves as the core foundation for subsequent analysis of observation constraints and optimization of imaging strategies. The applicable constraints of the C-W linearized equations used in this paper to simulate spacecraft relative motion are as follows: the reference orbit must be near-circular to avoid nonlinear interference from highly elliptical orbits; the relative distance between the two spacecraft must be much smaller than the reference orbit radius to ensure the relative motion is in the linear range; only two-body Keplerian dynamics is considered, ignoring various orbital perturbation effects, making it suitable for short-duration and close-range on-orbit relative motion simulation. To lay a solid basis for the subsequent modeling work, we first describe the observation scenario and define the reference coordinate system to accurately characterize the orbital motion states of the two types of spacecraft. Then, three key parameters are defined to describe the characteristics of the trajectory. Finally, a dimensionless parameter is defined as the criterion for classifying typical relative motions, and a schematic diagram of the relative motion trajectory is presented.

3.1.1. Scene Description

To characterize the orbital motion states of the target spacecraft and the observation spacecraft, the Earth-centered inertial (ECI) frame is employed. Its origin is located at the center of the Earth, as shown in the accompanying Figure 1. The $X$-axis points towards the vernal equinox, aligning with the Earth’s equatorial and ecliptic planes. The $Z$-axis points towards the North Pole. The $Y$-axis is perpendicular to the $X$-axis, following the right-hand rule.

The relative motion is described in the Local Vertical, Local Horizontal (LVLH) frame attached to the target spacecraft. The origin of this coordinate system is located at the center of mass of the observation spacecraft. The $x$-axis is aligned with the geocentric radius vector $r_1$, pointing from the Earth’s center to the spacecraft. The $y$-axis lies within the orbital plane, perpendicular to the $x$-axis, and points in the direction of motion. The $z$-axis is aligned with the spacecraft’s angular momentum vector. The average angular velocity of the observation spacecraft is $n=\sqrt{\mu /a^3}$, where $\mu =3.986\times {10}^{14} {\mathrm{m}}^3{/\mathrm{s}}^2$ is Earth’s gravitational parameter, and $a$ denotes the orbital semi-major axis.

Furthermore, since the relative distance between the target and observer spacecraft is significantly smaller than the target’s geocentric distance, the relative kinematic equation for the observer spacecraft, namely the Clohessy–Wiltshire (CW) equations [40], can be derived.

$$\left\{\begin{array}{l}\ddot{x}-2n\dot{y}-3n^{2}x=0\\ \ddot{y}+2n\dot{x}=0\\ \ddot{z}+n^{2}z=0\end{array}\right.$$

(1)

Here, $(x,y,z)$ and $(\dot{x},\dot{y},\dot{z})$ represent the position and velocity coordinates, respectively, of the observer spacecraft in the LVLH frame. The solution to Equation (1) can be obtained by solving the ordinary differential equations, resulting in the following expression:

$$\left\{\begin{array}{l}x(t)={\displaystyle \frac{{\dot{x}}_0}{n}}\sin nt-\left(2{\displaystyle \frac{{\dot{y}}_0}{n}}+3x_0\right)\cos nt+2\left(2x_0+{\displaystyle \frac{{\dot{y}}_0}{n}}\right)\sin nt\\ y(t)=2\left({\displaystyle \frac{{\dot{y}}_0}{n}}+3x_0\right)\sin nt+2{\displaystyle \frac{{\dot{x}}_0}{n}}\cos nt-3\left({\dot{y}}_0+2n{x}_0\right)t+\left(y_0-{\displaystyle \frac{2}{n}}{\dot{x}}_0\right)\\ z(t)={\displaystyle \frac{{\dot{z}}_0}{n}}\sin nt+z_0\cos nt\end{array}\right.$$

(2)

where $(x_0,y_0,z_0,{\dot{x}}_0,{\dot{y}}_0,{\dot{z}}_0)$ denotes the relative position and velocity at a given time $t_0$, and is the state transition matrix. For ease of analysis, the initial time is set to 0.

3.1.2. Feature Parameterization Formulation

The orbital component (x–y-plane) of the relative motion between the spacecraft and the out-of-plane motion ($z$-axis) are separated. The in-plane relative motion (ROMP) of the spacecraft is characterized by four parameters: $x_0,y_0,{\dot{x}}_0,{\dot{y}}_0$. To simplify the analysis, these four parameters are processed as follows: According to Equation (2), there exists a time $t_p$ such that $\dot{x}(t_p)=0$, at which point $x(t_p)$ reaches its maximum value. The state at this moment is defined as the characteristic point [41]: $\dot{x}(t_p)=0,x_p=x_{\max }$.

Here, $x_{\max }$ represents the maximum value of the radial $x$-axis motion. Thus, the solution to the C-W equations can be simplified as:

$$\left\{\begin{array}{l}x(t)=x_c(t)+x_l(t)\\ y(t)=y_c(t)+y_l(t)\\ z(t)={\displaystyle \frac{{\dot{z}}_p}{n}}\cdot \sin n(t)+z_p\cdot \cos n(t)\end{array}\right.$$

(3)

Taking the state of the characteristic point $P$ as new parameters, the solution to the C-W equations can be decomposed into the superposition of an elliptical periodic motion and a linear drift motion.

$$\left\{\begin{array}{l}{x}_c(t)=-{\displaystyle \frac{\alpha }{2}}\cdot \cos nt\hfill \\ y_c(t)=\alpha \cdot \sin nt\hfill \\ z_c(t)={\displaystyle \frac{{\dot{z}}_p}{n}}\cdot \sin nt+z_p\cdot \cos nt\hfill \end{array}\right.$$

(4)

$$\left\{\begin{array}{l}{x}_l(t)=-{\displaystyle \frac{2\beta }{3n}}\hfill \\ y_l(t)=\beta (t)+\gamma \hfill \\ z_l(t)=0\hfill \end{array}\right.$$

(5)

Three key parameters are defined: $\alpha =2\left(2{\displaystyle \frac{y_p}{n}}+3x_p\right)$, $\beta =-3\left({\dot{y}}_p+2n{x}_p\right)$, and $\gamma =y_p$. The parameters $\alpha$, $\beta$, and $\gamma$ are uniformly sampled within the physical ranges permitted by their definition. Here, $\alpha$ represents the elliptical size, determining the amplitude of the periodic motion; $\beta$ is the drift rate, determining whether the trajectory is closed and its direction of motion. When $\beta =0$, the trajectory is a closed ellipse; when $\beta \ne 0$, the trajectory is various types of open curves; $\gamma$ is the initial trajectory deviation.

3.1.3. Typical Relative Motion Classification

Based on the formula for $\beta$, a dimensionless parameter $\sigma \triangleq -{\displaystyle \frac{{\dot{y}}_p}{n\cdot x_p}}$ can be defined [41], which essentially represents the ratio of the along-track velocity to the radial position at the characteristic point. Different values of $\sigma$ correspond to four typical scenarios, namely elliptical trajectory, spiral trajectory, drip-drop trajectory, and drift trajectory, covering various categories of relative trajectories. The classification criteria and corresponding dynamic characteristics are summarized in

Table 1. Classification of relative orbital motion pattern based on normalized drift parameter ($\sigma$).

Trajectory Classification	$\mathit{\sigma }$ Range	Dynamic Characteristics
Elliptical trajectory	$\sigma =2$	$\beta$ = 0, the trajectory forms a closed ellipse, and the observer spacecraft moves periodically around the target spacecraft.
Spiral trajectory	$\begin{array}{l}1.8877<\sigma <2.2039, \\ \sigma \ne 2\end{array}$	Drift and periodic motion combine into a smooth, non-intersecting spiral, drifting along-track away from or toward the target.
Drip-drop trajectory	$1.75<\sigma <1.8877$ $\mathrm{or} \sigma >2.2039$	The trajectory shows periodic loops in the drift direction, resembling a water droplet, with two subcategories based on $y$-axis crossing.
Drift Trajectory	$\sigma \le 1.75$	With small periodic oscillation, the trajectory is nearly linear or gently curved, moving unidirectionally away from the target along-track, representing the simplest relative motion.

.

Based on the aforementioned classification, four types of relative motion can be constructed. The three-dimensional and two-dimensional schematic diagrams of their trajectories are shown in Figure 2 below.

For the elliptical trajectory, it forms a closed ellipse, where the observation spacecraft moves periodically around the target spacecraft, presenting a standard closed elliptical shape within the orbital plane. The spiral trajectory is a superposition of drift motion and periodic motion, forming a smooth non-intersecting spiral that drifts along-track either away from or toward the target. The drip-drop trajectory forms a waterdrop-like shape in the drift direction, combining periodicity with slight drift, thus exhibiting a unique quasi-periodic motion morphology. The drift trajectory is an approximately linear or gently curved curve with minimal oscillation amplitude, sweeping unidirectionally past the target along the track direction, representing the simplest form of relative motion.

3.2. Imaging Geometry Principles

This section introduces the camera imaging model, which provides the geometric foundation for pose estimation and visibility analysis. First, the camera imaging model is elaborated. The model describes the projective mapping from 3D world points to 2D image coordinates. Second, two common attitude representations used in spacecraft vision are then presented: Euler angles and quaternions. Finally, target visibility conditions are examined from two perspectives. Geometric visibility determines whether the target is within the camera’s line of sight and not occluded. Optical visibility depends on whether the target surface is sufficiently illuminated to be observed. Both are governed by the relative positions of the target, camera, and sun.

3.2.1. Camera Imaging Model

The camera imaging model is the foundation for target position and pose estimation. The imaging process of a target is illustrated in Figure 3. The imaging process can be summarized as the projection of a three-dimensional 3D point in space onto a two-dimensional 2D point on the image plane. Before describing the camera imaging model, it is necessary to define four coordinate systems: the world coordinate system $O_W-X_W{Y}_W{Z}_W$, the camera coordinate system $O_C-X_C{Y}_C{Z}_C$, the image coordinate system on the physical imaging plane $O_R-xy$, and the pixel coordinate system $O_I-uv$. In the attitude estimation problem, the rotation matrix $R_c$ is the quantity to be determined.

Typically, the origin $O_C$ of the camera coordinate system $O_C-X_C{Y}_C{Z}_C$ is the camera’s optical center. The $X_C$-axis points to the right, the $Y_C$-axis points downward, and the $Z_C$-axis points along the camera’s optical axis. The origin $O_R$ of the image coordinate system $O_R-xy$ is defined as the intersection of the optical axis with the image plane, typically the center of the image plane. The $x$-axis is aligned with the image plane pointing to the right, and the $y$-axis points downward. The origin $O_I$ of the pixel coordinate system $O_I-uv$ is defined as the top-left corner of the image. The $u$-axis points to the right along the image and is parallel to the $x$-axis, while the $v$-axis points downward and is parallel to the $y$-axis.

Consider a spatial point $\mathrm{P}$. Its coordinates in the camera coordinate system are ${\mathrm{P}}_C{(x_C,y_C,z_C)}^T$. According to the perspective projection principle, the corresponding image point $p$ on the image plane can be expressed as:

$$x={\displaystyle \frac{f\cdot x_C}{z_C}}, y={\displaystyle \frac{f\cdot y_C}{z_C}}$$

(6)

where $f$ is the focal length, defined as the distance from the optical center to the physical image plane. Coordinates in the image plane are expressed in physical dimensions (e.g., meters).

Since an image is composed of discrete pixel elements, the projection on the physical image plane must undergo sampling and quantization, i.e., a transformation from the image coordinate system to the pixel coordinate system. The coordinates of the image point $p$ in the pixel coordinate system are:

$$\left\{\begin{array}{c}u=f_x\cdot x+u_0\\ v=f_y\cdot y+v_0\end{array}\right.$$

(7)

Here, $f_x=1/d_x$ and $f_y=1/d_y$, where $d_x$ and $d_y$ are the physical dimensions of a pixel along the $x$ and $y$ axes of the image plane, $f_x$ and $f_y$ are the number of pixels per unit physical length, with units of pixel/m, and $(u_0,v_0)$ are the pixel coordinates of the principal point. Equation (7) can be written in matrix form using homogeneous coordinates as:

$$\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$$

(8)

Therefore, the transformation from the camera coordinate system to the pixel coordinate system can be expressed as:

$$z_C\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}{x}_C\\ y_C\\ z_C\end{array}\right]$$

(9)

The middle matrix in Equation (9) is defined as the camera’s intrinsic matrix, denoted as $K$. The intrinsic matrix is typically provided by the camera manufacturer but can also be obtained through calibration.

In the projection of a spatial point onto an image, the point’s coordinates in the camera coordinate system are used. However, as the camera moves, the coordinates of point $\mathrm{P}$ in the camera coordinate system change. Thus, the coordinates of point $\mathrm{P}$ in the camera coordinate system are obtained by transforming its coordinates from the world coordinate system $O_W-X_W{Y}_W{Z}_W$ based on the current camera pose $T$. Camera pose describes the transformation between the camera and world coordinate systems and is represented by a rotation matrix $R$ and a translation vector $t$. In summary, the transformation of a 3D spatial point $\mathrm{P}$ from the world coordinate system to the camera coordinate system can be expressed in homogeneous coordinates as:

$$\left[\begin{array}{c}{x}_C\\ y_C\\ z_C\\ 1\end{array}\right]=T\left[\begin{array}{c}{x}_W\\ y_W\\ z_W\\ 1\end{array}\right]=\left[\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{x}_W\\ y_W\\ z_W\\ 1\end{array}\right]$$

(10)

$T$ is also called the camera’s extrinsic parameter matrix. Then, the process of projecting a 3D spatial point from the world coordinate system to the pixel coordinate system can be expressed as:

$$z_C\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{f}_x& 0& u_0\\ 0& f_y& v_0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{cc}R& t\\ 0^T& 1\end{array}\right]\left[\begin{array}{c}{x}_W\\ y_W\\ z_W\\ 1\end{array}\right]=KT\left[\begin{array}{c}{x}_W\\ y_W\\ z_W\\ 1\end{array}\right]=KT{P}_W$$

(11)

The transformation between the camera coordinate system and the world coordinate system defines the relative pose and position of the target (in the world coordinate system) with respect to the camera, represented by the rotation matrix $R$ and the translation vector $t$.

It should be noted that the aforementioned camera imaging model represents an ideal configuration. In practice, however, the manufacturing and assembly processes of camera lenses—along with the handling of light transmission by optical elements and the integration of camera components—cannot achieve perfect precision as assumed in the ideal case. Furthermore, influences from the space environment and the operational lifespan of the camera can introduce deviations from the ideal optical state, leading to lens distortion in the imaging system. Lens distortion causes image points to deviate from their ideal positions, resulting in geometric deformation of the captured imagery. Under such conditions, the ideal imaging model described earlier can no longer accurately characterize the projective geometry. Therefore, distortion correction is required.

Distortion correction can generally be implemented in two ways: (1) applying a global distortion correction to the entire image first, and then conducting subsequent research based on the corrected image; (2) correcting only the specific image points of interest according to the distortion model. In the field of computer vision, approach (1) is more commonly adopted. Once the images have been corrected for distortion, the pinhole camera model can be applied to establish the projection relationship from 3D points in space to the 2D pixel coordinate system. Since this study focuses on the investigation of vision-based estimation algorithms, all images discussed in the subsequent sections are assumed to have undergone distortion correction.

3.2.2. Attitude Representation

In spacecraft visual perception, the accurate representation of attitude serves as a critical bridge connecting geometric observation with dynamic control. Attitude describes the rotational state of a rigid body (e.g., the spacecraft body) in space, with common representations including Euler angles, quaternions, rotation matrices, etc. [42]. This section systematically introduces the two most commonly used methods—Euler angles and quaternions.

(1)

Euler Angles

In the field of aerospace, Euler angles obtained by rotating the target’s body frame (world coordinate system) in a $\mathrm{zyx}$ sequence are typically used to represent the spacecraft’s attitude change, usually the attitude of the target’s body frame relative to the orbital coordinate system. The corresponding Euler angles are yaw $\psi$, roll $\phi$, and pitch $\theta$, respectively. When a spatial target is rotated in the $\mathrm{zyx}$ sequence, the corresponding rotation matrix $R$ is:

$$\begin{array}{l}R=\left[\begin{array}{ccc}\cos \theta & 0& -\sin \theta \\ 0& 1& 0\\ \sin \theta & 0& \cos \theta \end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& \cos \phi & \sin \phi \\ 0& -\sin \phi & \cos \phi \end{array}\right]\left[\begin{array}{ccc}\cos \psi & \sin \psi & 0\\ -\sin \psi & \cos \psi & 0\\ 0& 0& 1\end{array}\right]\\ \hspace{0.33em}\hspace{0.33em}=\left[\begin{array}{ccc}\cos \theta \cos \psi -\sin \phi \sin \theta \sin \psi & \cos \theta \sin \psi +\cos \psi \sin \phi \sin \theta & -\cos \phi \sin \theta \\ -\cos \phi \sin \psi & \cos \phi \cos \psi & \sin \phi \\ \cos \psi \sin \theta +\cos \theta \sin \phi \sin \psi & \sin \theta \sin \psi -\cos \theta \cos \psi \sin \phi & \cos \phi \cos \theta \end{array}\right]\end{array}$$

(12)

(2)

Quaternions

An attitude quaternion is defined as $q=q_0+q_1\mathrm{i}+q_2\mathrm{j}+q_3\mathrm{k}$, where ${q_0}^2+{q_1}^2+{q_2}^2+{q_3}^2=1$. Given a rotation axis vector $n$ and a rotation angle $\theta$ about that axis, the corresponding quaternion is:

$$q=\left[\cos \left({\displaystyle \frac{\theta }{2}}\right),\hspace{0.33em}\hspace{0.33em}\sin \left({\displaystyle \frac{\theta }{2}}\right)\cdot n_x,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\sin \left({\displaystyle \frac{\theta }{2}}\right)\cdot n_y,\hspace{0.33em}\hspace{0.33em}\hspace{0.33em}\sin \left({\displaystyle \frac{\theta }{2}}\right)\cdot n_z\right]$$

(13)

Therefore, the rotation matrix corresponding to the quaternion can be written as:

$$R=\left[\begin{array}{ccc}1-2q_2^2-2q_3^2& 2(q_1{q}_2+q_3{q}_0)& 2(q_1{q}_3-q_2{q}_0)\\ 2(q_1{q}_2-q_3{q}_0)& 1-2q_1^2-2q_3^2& 2(q_2{q}_3+q_1{q}_0)\\ 2(q_1{q}_3+q_2{q}_0)& 2(q_2{q}_3-q_1{q}_0)& 1-2q_1^2-2q_2^2\end{array}\right]$$

(14)

Euler angles are suitable for static coordinate transformations, while quaternions are better suited for dynamic attitude control. Currently, mainstream spacecraft visual datasets predominantly use quaternions as the standard representation for attitude annotation, as they more effectively capture the continuous nature of attitude changes.

3.3. Visibility Analysis

The prerequisite for characterizing a space object based on space-based visible light imagery is the acquisition of valid space-based images. Considering the unique characteristics of the space environment and the orbital motion patterns of space objects, it is essential to determine the target’s visibility before a space-based imaging device can capture an image. This means verifying whether, at the moment of imaging, the lighting conditions and the geometric positions of the space-based imager and the target satisfy the imaging constraints. Therefore, this section will briefly discuss the visibility conditions for space-based observation of objects from two aspects: geometric visibility and optical visibility.

3.3.1. Geometric Visibility

Geometric visibility refers to the condition that the space-based observation platform and the space target are mutually visible to each other. A schematic diagram of the observation geometry between the platform and the targets is shown in Figure 4.

As can be clearly seen from the figure, Target A is visible, Target B is in the critical region, and Target C is not visible. Let $r_C$ be the geocentric radius vector of the observation platform (camera), $r_T$ be the geocentric radius vector of the observation target, $r_{TC}$ be the distance vector from the observation target to the camera, $\theta$ be the angle between the line from the camera to the Earth’s center and the line from the camera to the target spacecraft (ranging from 0° to 180°), and $R_E$ be the Earth’s radius. The condition for geometric visibility between the observation platform and the target is:

$$\left\{\begin{array}{l}\cos \theta ={\displaystyle \frac{r_C\cdot r_{TC}}{\left|r_C\right|\cdot \left|r_{TC}\right|}}\hfill \\ \cos \theta <{\displaystyle \frac{\sqrt{{r_C}^2-{R_E}^2}}{\left|r_C\right|}}\hfill \end{array}\right.$$

(15)

Therefore,

$$r_C\cdot r_{TC}<\sqrt{{r_C}^2-{R_E}^2}\cdot \left|r_{TC}\right|$$

(16)

3.3.2. Optical Visibility

Optical visibility can be concisely summarized as the geometric visibility condition among the Sun, the observation platform, and the observation target. Optical visibility must satisfy two conditions: (1) during observation, the Sun does not directly shine into the platform’s lens; (2) the space target is not within the Earth’s shadow (umbra/penumbra) region.

Regarding condition (1), if sunlight directly enters the lens, the field of view will be flooded with a high-brightness background, making imaging of the target impossible and potentially damaging the camera. Figure 5 below shows the schematic diagram of the conditions for avoiding direct sunlight.

Regarding condition (2), whether the platform carries a CCD or CMOS camera, the imaging principle relies on receiving sunlight reflected from the target. The photosensitive elements convert this light signal into an electrical signal to form an image. If the target is within Earth’s shadow, it cannot reflect sunlight, and thus the platform cannot image it. Therefore, during target observation and imaging, it is essential to ensure the target is outside Earth’s shadow region. Figure 6 below shows the schematic diagram of the Earth shadow conditions that need to be satisfied.

Let the distance vector from the observation platform to the Sun be $r_{CS}$, the distance vector from the camera to the observation target be $r_{CT}$, the sum of the Sun’s apparent radius and the light scattering angle be $\alpha$, and the angle between $r_{CS}$ and $r_{CT}$ be $\beta$. Then the condition to avoid direct solar illumination can be expressed as:

$$0<\alpha <\beta <\pi$$

(17)

The above formula can be written as:

$$\cos \alpha >{\displaystyle \frac{r_{CS}\cdot r_{TC}}{\left|r_{CS}\right|\cdot \left|r_{TC}\right|}}$$

(18)

Meanwhile, optical visibility also requires that the space target is outside the Earth’s shadow region. The Earth’s shadow condition is shown in the figure. In the figure, the position indicated by the blue arrow is the critical position of the Earth’s shadow region. Let the central angle between the target’s geocentric vector $r_T$ and the Sun’s geocentric vector $r_S$ be $\lambda$. Then, if the target is to be outside the Earth’s shadow, the following condition needs to be met:

$$\cos \lambda >{\displaystyle \frac{-\sqrt{{r_T}^2-{R_E}^2}}{\left|r_T\right|}}$$

(19)

The above formula is equivalent to:

$$r_T\cdot r_S>-\sqrt{{r_T}^2-{R_E}^2}\cdot \left|r_S\right|$$

(20)

The aforementioned visibility analysis establishes theoretical support and a basis for evaluating dataset quality. In the future, it can serve as a key module for automatic filtering and grading of multimodal images, enabling efficient filtering of low-quality, low-information samples, thereby enhancing the overall signal-to-noise ratio of the training data and improving model convergence stability.

3.4. Multimodal Ground Truth Rendering and Imaging Simulation

This section systematically presents the multimodal ground truth rendering and space imaging simulation framework constructed in this paper. It elaborates on the simulation environment configuration, multimodal image generation pipeline, and space image degradation modeling method, which can provide sufficient, high-quality, and well-annotated datasets for the training, testing, and performance evaluation of subsequent pose estimation algorithms. The output of the orbital dynamics model serves as the precise input for the rendering pipeline. Specifically, the output of the dynamics model at time t is a 6-DoF relative pose $\left[R(t),T(t)\right]$, which is directly utilized as the extrinsic parameters for the virtual camera in the 3D rendering engine.

3.4.1. Simulation Environment and Configuration

In the synthetic data generation experiments conducted for this study, the hardware and software environment was configured as follows: The experimental platform was equipped with an Intel Core i7-12700H processor and an NVIDIA GeForce RTX 3060 Laptop GPU, running a 64-bit Windows operating system. All 3D scene construction, material configuration, camera parameter setup, and image rendering were performed within Blender 4.0 software, utilizing the Cycles rendering engine. The pipeline outputs four data types: RGB images, depth maps, and mask maps. The ground truth annotations for 6D pose were generated using the script compiler built into the Blender software.

3.4.2. Multimodal Image Rendering Pipeline

The flowchart of the designed simulation experiment is shown in Figure 7, which is divided into four core stages. First, in the orbital parameters and relative motion setup stage, the 3D trajectory (represented by in the LVLH frame) of the observer spacecraft relative to the target spacecraft is defined, along with the attitude angles of the target spacecraft relative to the camera. This establishes the relative motion constraints for constructing subsequent 3D scenes. Given the initial state, the relative orbital dynamics model can generate the 6-DOF relative pose $P(t)=\left[R(t),T(t)\right]$ of the observer spacecraft with respect to the target spacecraft at any time $t$, where $R(t)\in SO(3)$ is the rotation matrix and $T(t)\in R^3$ is the translation vector. This pose is directly utilized as the extrinsic matrix for the virtual camera in the subsequent rendering pipeline.

Second, in the high-fidelity 3D scene and material configuration stage, high-fidelity 3D models containing textures and materials are imported. The space background and lighting environment are configured to enhance the physical realism of the simulation scene. The third stage is camera parameters and rendering setup. Here, the camera’s intrinsic parameters, including focal length and sensor size, are configured. The Cycles rendering engine is selected to output images with a resolution of 1280 × 960, containing four modalities: RGB images, normal maps, depth maps, and mask maps. This ensures the rendering process conforms to the imaging characteristics of a real camera. Finally, in the Image and Annotation Generation stage, time-series image sequences are output, and 6D pose ground truth annotation files are synchronously generated. This provides a high-quality, labeled dataset for subsequent 3D reconstruction and pose estimation algorithms. The entire pipeline implements an end-to-end generation process from orbital dynamics parameters to synthetic imagery and ground truth annotations.

3.4.3. Image Physical Degradation Simulation

The objective of image degradation research for space target images is to construct an end-to-end imaging degradation pipeline that simulates various degradation effects introduced by the optical system and sensor characteristics of a real spacecraft camera during imaging. By authentically reproducing the actual imaging distortion, blurring, noise interference and color distortion of space targets in complex on-orbit observation scenarios, this research generates high-fidelity simulated space target image data that are consistent with real detection conditions, so as to provide reliable and realistic training and test samples for subsequent space target detection, recognition, feature extraction and image restoration algorithms. This aims to generate training and testing data that more closely approximate real observational scenarios. The degradation pipeline primarily encompasses two major categories of degradation factors: The first is optical system degradation modeling. This mainly includes effects such as the optical diffraction limit, aberrations, defocus, and motion blur. These degradations can be described by corresponding Point Spread Functions (PSFs) and simulated by convolving the ideal image. The second is sensor sampling and readout degradation modeling. This part simulates the degradation introduced during the sensor’s conversion of a continuous optical signal into a discrete digital signal. Most spacecraft cameras employ a single-chip sensor with a Bayer color filter array to acquire color images. Therefore, the high-resolution RGB image obtained from rendering is first downsampled into a single-channel RAW image according to the Bayer pattern. It is then reconstructed into an RGB image through a demosaicing algorithm (such as bilinear interpolation or adaptive homochromatic interpolation), a process that introduces color artifacts and high-frequency detail loss. Additionally, the sensor introduces various types of noise during signal conversion, such as shot noise and read noise, which can be modeled as the sum of Poisson noise and additive Gaussian noise.

$$I_{\mathrm{noisy}}=Possion(I_{\mathrm{clean}})+Gaussian(I_{\mathrm{clean}})+(I_{\mathrm{clean}}\times \mathrm{PSF})$$

(21)

where $I_{\mathrm{noisy}}$ is the degraded image, and $I_{\mathrm{clean}}$ is the original clear image.

4. Results

4.1. Synthetic Image Results

In a simulation scenario, it is essential to construct a complete imaging system model that incorporates the environment, the target, and camera motion. The International System of Units (SI) is uniformly adopted, with the “meter” serving as the scale baseline, ensuring consistency between the simulated environment and the real physical world. A directional light source simulating solar illumination is introduced into the scene, allowing for flexible configuration of its intensity and incident angle to replicate various lighting conditions in space.

We systematically analyzed the pose parameters, distance distribution, and lighting conditions of the dataset, with the results summarized in

Table 2. Dataset scenario parameter analysis.

Scene	Position Range (m)	Average Angular Variation (°)	Total Illumination Energy (W/m2)	Ambient Light Intensity (W/m2)
Tiangong	X ∈ [−19.80, 19.80] Y ∈ [−25.38, 25.38] Z ∈ [4.26, 4.26]	0.70	264.06	1.40
Spacedragon	X ∈ [−13.91, 13.15] Y ∈ [−13.08, 12.67] Z ∈ [7.39, 7.39]	0.73	40.08	1.00
ICESat	X ∈ [−41.26, 41.51] Y ∈ [−39.08, 20.47] Z ∈ [−19.05, 40.19]	1.52	100.54	0.80
Cassini	X ∈ [−30.06, 29.98] Y ∈ [−38.97, −24.30] Z ∈ [13.28, 13.29]	0.20	357.29	1.00

. The dataset covers a variety of typical scenarios, in which: the position range represents the spatial distribution of the camera relative to the target object in the Cartesian coordinate system; the average angular variation reflects the average magnitude of pose changes between consecutive frames (the frame sequence is uniformly sampled in time); the total illumination energy characterizes the total radiant flux density of all light sources in the scene; and the ambient light intensity describes the strength of global ambient light, simulating indirect lighting effects. Table 2 presents the key parameter statistics for four space target scenarios (Tiangong, Spacedragon, ICESat, and Cassini). In terms of camera poses, the positional variation ranges along the X and Y axes generally span from ±15 m to ±40 m, while the Z-axis remains relatively stable. The average angular variation ranges from 0.20° to 1.52°, indicating notable differences in trajectory smoothness. Regarding illumination conditions, the total illumination energy covers a wide spectrum from 40.08 W/m² to 357.29 W/m², whereas the ambient light intensity is maintained at a relatively low level between 0.80 W/m² and 1.40 W/m², simulating the weak contribution of indirect light in real scenarios. In summary, the dataset exhibits sufficient representational capacity in terms of camera pose variation, spatial coverage, and lighting diversity. It comprehensively covers typical ranges of motion and illumination variation, thereby ensuring the diversity and representativeness of the data. This makes it suitable for providing a multi-dimensional, highly robust test benchmark for vision-based tasks such as visual localization and 3D reconstruction.

This study has constructed a large-scale, high-fidelity multimodal synthetic dataset to support vision-based tasks in complex space scenarios. For each target object, a systematic data acquisition pipeline was employed to generate synchronized multimodal imagery under diverse viewing geometries, illumination conditions, and scene backgrounds. Specifically, four complementary image modalities were rendered in parallel for every scenario, including RGB, depth, semantic segmentation masks, and surface normal maps, with 500 samples generated per modality to ensure statistical sufficiency. The detailed data composition and scale are summarized in

Table 3. Statistics of the synthetic multimodal dataset.

Data Modality	Description	Samples per Object	Total Samples	Primary Use
RGB Image	24-bit true color image	500	2000	Network input
Segmentation Mask	Binary pixel-level annotation	500	2000	Segmentation tasks
Depth Map	Z-depth in camera space	500	2000	3D reconstruction; depth estimation
Normal Map	Surface normal vectors	500	2000	Normal estimation; lighting analysis

. Importantly, all modal images are strictly aligned both temporally and spatially, thereby forming four sets of perfectly matched multimodal data streams that enable pixel-wise correspondence across modalities. This alignment facilitates joint learning and cross-modal validation in downstream tasks such as 3D reconstruction, pose estimation, and scene understanding.

Figure 8 illustrates the synthetic rendering results of four typical space targets: Tiangong, Spacedragon, ICESat, and Cassini. For each target, four types of complementary data are provided: high-resolution RGB images, corresponding binary segmentation masks, grayscale depth maps, and surface normal maps. Among these, the RGB image faithfully reproduces the visual appearance of the target under space-based lighting and background conditions, providing intuitive photometric information for visual perception tasks. The segmentation mask clearly isolates the target spacecraft from the space background, serving as ground truth annotations for semantic segmentation and target detection algorithms. The depth map encodes the relative distance from the camera to each point on the target surface, offering critical geometric cues for 3D reconstruction and pose estimation. The surface normal map records the orientation of each point on the target surface, providing normal vector information for surface feature analysis and illumination simulation. Collectively, these multimodal data form a comprehensive dataset to support a variety of space target perception and analysis tasks.

During the image degradation modeling stage, a Point Spread Function (PSF) is employed to simulate motion blur resulting from the relative motion between the camera and the target, with the motion blur length set to 15 pixels and the motion direction set to 30 degrees. Poisson noise and Gaussian noise are subsequently added to the blurred image. Poisson noise models the statistical fluctuations of photons arriving at the sensor, with its intensity determined by the image brightness. Concurrently, Gaussian noise is introduced to simulate electronic sensor noise, with a mean of 0 and a variance of 0.01. The results of the image degradation are illustrated in Figure 9.

4.2. Ablation Study

To quantitatively assess the contribution of each degradation module within the proposed image degradation model and systematically analyze its impact on image quality, this study designs a progressive ablation experiment. Using the original high-definition synthetic images as the reference benchmark, three core degradation modules—motion blur, Poisson noise, and additive Gaussian noise—are sequentially introduced to construct four progressively degraded experimental groups: D1 includes motion blur only; D2 combines motion blur and Poisson noise; D3 combines motion blur and Gaussian noise; and D4 represents the complete degradation model, incorporating all three modules—motion blur, Poisson noise, and Gaussian noise—thereby forming a comprehensive end-to-end physical imaging pipeline.

For each experimental group, three objective image quality assessment metrics were computed between its output images and the original reference images: the Peak Signal-to-Noise Ratio (PSNR), the Structural Similarity Index (SSIM), and the Learned Perceptual Image Patch Similarity (LPIPS). PSNR reflects pixel-level error, SSIM gauges the preservation of structural information, while LPIPS assesses perceptual similarity from a deep feature space. Together, this triad of metrics provides a comprehensive evaluation of the degradation in image fidelity, structural integrity, and visual realism across different stages of the applied degradations. For each object, evaluate the performance metrics of different combinations of the four degradation modules, and then take the average. Ablation study results are shown in

Table 4. Ablation experimental results of degradation models (↑ means better performance for higher values, ↓ means better performance for lower values).

Experiment Groups	PSNR (dB) ↑	SSIM ↑	LPIPS ↓
D1	29.60 ± 2.59	0.90 ± 0.06	0.13 ± 0.05
D2	25.24 ± 2.60	0.58 ± 0.19	0.54 ± 0.26
D3	21.85 ± 0.60	0.11 ± 0.05	0.89 ± 0.08
D4	20.13 ± 1.23	0.06 ± 0.02	0.95 ± 0.05

.

The experimental results are presented in Table 3. As the degradation modules are progressively added (from D1 to D4), the PSNR and SSIM metrics exhibit a monotonic decrease, while the LPIPS metric shows a monotonic increase, which systematically validates the physical soundness of the proposed degradation model. Detailed analysis indicates that the introduction of Poisson noise (D1 → D2) leads to the most drastic changes in SSIM and LPIPS, demonstrating its strongest disruptive effect on structural information and perceptual quality. In contrast, the introduction of Gaussian noise (D2 → D3) causes the most significant attenuation in PSNR, highlighting its predominant impact on pixel-level fidelity. The complete model (D4) exhibits the most severe degradation across all metrics, confirming the necessity of joint modeling of multiple physical effects for realistic image degradation simulation. In summary, the experimental results quantitatively reveal the contribution of each degradation module and provide a reliable degradation benchmark for training and testing algorithms based on synthetic data.

5. Discussion

In this paper, a multimodal synthetic dataset for spacecraft visual perception is constructed, and an end-to-end physical degradation model is designed. The core contribution lies in filling the gap of existing test benchmarks for spacecraft visual perception in terms of multimodality and multi-scenario coverage. The dataset designs a variety of relative motion orbits and generates four-modal image data, which not only provides a multi-dimensional test benchmark for vision-based space missions but also solves the problems of difficult acquisition and high cost of real on-orbit data, offering reliable support for subsequent visual perception algorithms. Meanwhile, through systematic ablation experiments, we quantitatively analyze the mechanism of each degradation module, clarify its contribution weight to the degradation effect, and verify the rationality and effectiveness of the proposed physical degradation model, ensuring that it can truly reflect the degradation characteristics of spacecraft on-orbit visual imaging.

Combined with the research status in the field of spacecraft visual perception, the innovations of this work are highly consistent with industrial demands; however, there are still limitations, which also serve as the key breakthrough directions for follow-up research, specifically reflected in three aspects. First, although the dataset covers four typical on-orbit service scenarios, the relative motion modes are established under ideal orbital conditions without fully considering motion deviations caused by complex orbital perturbations such as Earth’s oblateness and atmospheric drag. This restricts the applicability of synthetic data in non-ideal orbital scenarios and leads to inconsistencies with the real relative motion characteristics of on-orbit spacecraft. Second, the illumination simulation is insufficient, as it fails to accurately model complex on-orbit illumination variations and only considers basic static illumination scenarios, making it impossible to verify the robustness of visual perception algorithms against illumination changes. Finally, the validation is only conducted based on full digital simulation without semi-physical simulation or hardware-in-the-loop testing. As a result, the practicality of synthetic data and the algorithm performance in real imaging chains have not been fully verified, making it difficult to ensure the smooth migration of algorithms from simulation to engineering applications, which is also a critical bottleneck for its engineering implementation.

Future work will focus on improving the physical fidelity and validation completeness of the simulation framework. Three main directions are proposed.

First, high-fidelity orbital dynamics will be integrated. This includes modeling key perturbations such as Earth’s oblateness (J₂), atmospheric drag, and third-body gravitational effects. These additions will allow a more accurate simulation of six-degree-of-freedom relative motion in complex space environments. The framework will be extended to cover challenging scenarios like highly elliptical orbits and long-duration missions.

Second, a dynamic, high-fidelity illumination model will be developed. This model will account for time-varying solar irradiation, Earth albedo, planetary illumination, and deep-space thermal radiation. It will realistically simulate the complex lighting conditions encountered in orbit, including scenes with high dynamic range, strong shadows, and extreme contrast. This will provide a more reliable test environment for evaluating the robustness of visual algorithms under real illumination variations.

Third, the validation pipeline will be extended to include hardware-in-the-loop (HIL) testing. Building on the current fully digital simulation data, real optical hardware will be integrated with synthetic scenes. This step will further verify the utility of the synthetic data and assess the performance of perception algorithms within a near-realistic imaging chain. The goal is to establish a closed-loop “digital–HIL” validation framework.

Through these extensions, a higher-fidelity and more complete simulation and verification platform will be established. This platform will support the development and testing of algorithms for vision-based navigation and on-orbit servicing in a wider range of demanding space scenarios.

6. Conclusions

This paper addresses the challenge of insufficient high-quality real-world data in the field of spacecraft visual perception. We propose a multimodal spacecraft data generation framework based on orbital relative motion constraints. This method integrates the laws of orbital relative motion into the synthetic data generation process. It simulates four typical spacecraft relative motion patterns: elliptical trajectory, spiral trajectory, drip-drop trajectory, and drift trajectory. It generates kinematically plausible image sequences. The constructed dataset contains 8000 images. Each image is accompanied by rich annotations. These annotations include RGB images, masks, depth maps, surface normals, and 6-DoF poses. The dataset covers various position ranges, angle changes, and light intensities. It provides dense supervision information for tasks such as pose estimation and 3D reconstruction. Meanwhile, we design an end-to-end physical degradation model. This model simulates real imaging artifacts from optical systems to sensors. It effectively narrows the gap between simulation and reality. In addition, this paper summarizes the shortcomings and limitations of the research. It also looks forward to future research directions. In summary, this study systematically improves the quality of spacecraft synthetic data from three key aspects: motion physical authenticity, annotation completeness, and visual realism. It lays a solid data foundation for the training, testing, and validation of spacecraft perception algorithms.