Interactive Maintenance of Space Station Devices Using Scene Semantic Segmentation

Abstract

A novel interactive maintenance method for space station in-orbit devices using scene semantic segmentation technology is proposed. First, a wearable and handheld system is designed to capture images from the astronaut in the space station’s front view scene and play these images on a handheld terminal in real-time. Second, the proposed system quantitatively evaluates the environmental lighting condition in the scene by calculating image quality evaluation parameters. If the lighting condition is not proper, a prompt message will be given to the astronaut to remind him or her to adjust the environment illumination. Third, our system adopts an improved DeepLabV3+ network for semantic segmentation of these astronauts’ forward view scene images. Regarding the improved network, the original backbone network is replaced with a lightweight convolutional neural network, i.e., the MobileNetV2, with a smaller model scale and computational complexity. The convolutional block attention module (CBAM) is introduced to improve the network’s feature perception ability. The atrous spatial pyramid pooling (ASPP) module is also considered to enable an accurate calculation of encoding multi-scale information. Extensive simulation experiment results indicate that the accuracy, precision, and average intersection over the union of the proposed algorithm can be better than 95.0%, 96.0%, and 89.0%, respectively. And the ground application experiments have also shown that our proposed technique can effectively shorten the working time of the system user.

1. Introduction

With the rapid development and deployment of China space station technologies, the daily workloads of astronauts in space stations in orbit [1] are increasing day by day. Like the International Space Station (ISS), on the one hand, astronauts need to accomplish the normal maintenance tasks inside the space station, such as cabin cleaning tasks, living material handling tasks, or safety inspection tasks [2], etc. On the other hand, the space station has also deployed a large number of scientific experimental cabinets [3] for conducting various kinds of scientific research in materials science, biological science, physical science, etc.; therefore, maintaining and operating those experimental cabinets will also occupy a lot of time and energy in astronauts’ daily work (Figure 1). Some practical applications have shown that engaging in monotonous and boring work for a long time can increase the mental fatigue of astronauts in space stations. Clearly, small operational errors and decision mistakes in orbit may lead to some significant flight safety accidents [4]. Therefore, to reduce the daily workload of astronauts and improve the reliability of in-orbit tasks, it is necessary to develop a device using artificial intelligence technology [5] to assist them in carrying out various scientific research and daily maintenance tasks [6].

During the operations of the ISS, artificial intelligence technologies were applied to assist astronauts in space stations in their daily work and life schedules from a long time ago [7]. Since 2019, space collaborative robots have been used to assist astronauts in their daily operations on the ISS. A small cubic flying robot called “Astrobee” [8] is designed to assist astronauts in completing daily management tasks, participating in various scientific experiments, and conducting safety monitoring of astronauts’ daily behaviors. To further improve the human–machine-friendly performance of collaborative robots, much research work [9,10] has been carried out around the design of interactive operation processes, scene understanding, and human–computer interface display methods. For example, in [11], scholars developed a kind of intelligent Polipo pressure sensing system for assessing injury inside the space suit. The sensitivity model was used for data analyses of sensing data. In [12], a wide-ranging stable motion control method for robot astronauts in space stations using human dynamics was reported. A viscoelastic dynamic humanoid model of parking under a microgravity environment was established. In [13], the virtual reality technique was also considered for improving the psychological and psychosocial health state of astronauts. The exergaming was proposed for application during the long-duration flight mission.

In the past, a large number of studies have been conducted regarding semantic segmentation [14] to alleviate the workloads of people. Semantic segmentation aims to accurately classify each pixel of an image into its corresponding category, thereby achieving a refined understanding of image content. The core feature of semantic segmentation is pixel-level classification. Each pixel is assigned to a predefined semantic category, but different instances of the same category are not distinguished. It is widely applied in autonomous driving [15] or medical image processing [16]. Its common models include the U-Net [17], DeepLab network [18], SegNet [19], etc. The encoder and decoder of U-Net are closely connected through skip connections, allowing the decoder to fully utilize the feature information of different levels of the encoder when restoring image resolution. The DeepLab series models introduce dilated convolution [20], which expands the receptive field of the convolution kernel without increasing computational complexity. It can better capture contextual information from images and has advantages in segmenting large-scale objects and complex scenes [21]. SegNet adopts an encoder–decoder structure, characterized in that the decoder part uses pooling indexes corresponding to the encoder for upsampling, reducing the number of model parameters and improving the running efficiency of the model. Clearly, although the above networks have shown good computational performance in their specific applications, it is still worth researching how to develop a real-time and high-accuracy scene understanding method for space station applications [22].

Regarding the semantic segmentation technology, from the perspective of network training, it can be classified into fully supervised, semi-supervised, and unsupervised networks [23]. The fully supervised network needs to label all training data before it can be trained. The semi-supervised network only needs a small amount of complete labeled data, supplemented by a large number of unlabeled data for network training. And the unsupervised network aims to achieve accurate image segmentation without relying on annotation data or only a small amount of annotation data. For example, the semi-supervised network includes the generative adversarial network (GAN)-based method [24], the multi-network architecture-based method [25], the multi-stage architecture-based method [26], and the single-stage end-to-end architecture-based method [27], etc., while the unsupervised networks include the adaptive adversarial learning-based method [28], the adaptive image segmentation transfer-based method [29], and the adaptive self-training-based method [30], etc. In comparison, the fully supervised network can accurately learn the mapping relationship between input and output data, so its calculation accuracy is highest. In addition, the design of a semi-supervised or unsupervised network is more difficult than that of a fully supervised network. In this article, because the working scene in the space station is all known and the ambient light is also controllable, as a result, the fully supervised network is selected for semantic segmentation calculation.

The semantic segmentation technology can also be classified into the traditional or the real-time computational network according to its processing speed. In general, the real-time semantic segmentation network refers to a model with a computational frame rate of 30 frames per second on a specific device. The traditional non-real-time semantic segmentation methods [31] rely on fine imaging details and rich spatial context information, which have the disadvantages of a large amount of calculation and a complex network structure. Differently, through the reasonable improvement of network structure, the real-time semantic segmentation network can retain more abundant spatial information with less computational cost and capture more effective multi-scale context so as to achieve the balance between the calculation speed and accuracy of the model. According to the network structure, the real-time semantic segmentation network can be roughly classified into the single-branch network [32], the double-branch network [33], the multi-branch network [34], the U-shaped network [35], and the neural structure search network [36], etc. The main disadvantage of the real-time semantic segmentation method is its limited fine segmentation ability. Considering that the application requirements of device maintenance guidance in this paper have higher requirements for calculation effect but little demand on the real-time calculation, a non-real-time network design method is adopted to reduce the difficulty of network design.

For the fully supervised non-real-time semantic segmentation networks, the common methods include the encoder–decoder model-based method, the dilated convolution model-based method, and the multi-task learning model-based method, etc. The encoder–decoder model-based method converts original data into low-dimensional representations, captures their key features, and then converts them back to the original data space. The common networks have RefineNet [37], U-Net, SegNet, etc. The dilated convolution model expands the receptive field of the convolution kernel by introducing dilated convolution; its typical model is the DeepLab series networks. The multi-task learning model combines the network framework with complementary tasks, and its network structure can make the parameters shared in the training process and reduce the maintenance cost of different tasks. Its common methods include the multi-scale convolutional neural network (MSCNN) [38], the prediction-and-distillation network (PAD-Net) [39], etc. In comparison, the DeepLab network has a fine calculation effect and fast processing speed, which is more suitable for the application of this article. The DeepLab series network is derived from VGG16 [40]. DeepLabV1 uses the full convolution network to replace the last full connection layer, removes the last two pooling layers, and expands the receptive field using dilated convolution. DeepLabV2 uses ResNet-101 [41] as the basic network and puts forward the pyramid pool structure. DeepLabV3 further improves the atrous spatial pyramid pooling (ASPP) structure to solve the problem of low computational efficiency of dilated convolution. DeepLabV3+ upgrades ResNet-101 to Xception, which further reduces the number of network parameters.

In this paper, an interactive maintenance technique of the space station complex devices using the scene semantic segmentation method is proposed. First, a glasses-style wearable device and a piece of handheld panel equipment are both designed and used to assist astronauts in space stations to collect and play the scene imaging information. In a microgravity environment, the handheld panel can be secured to an astronaut by a restraint strap. Second, the image quality evaluation parameters [42] are used for evaluating environmental lighting. Since the working environment inside of a space station cabin is entirely an artificial system, it will be very easy to achieve environmental lighting control by astronauts. Third, a semantic segmentation network, i.e., an improved DeepLabV3+ network [43], is proposed for scene understanding application. MobileNetV2 [44] is employed as the backbone network, a convolutional block attention module (CBAM) [45] is considered to participate in the feature computations, and a kind of ASPP module [46] is also used. The main contributions of this paper include (1) a new interactive task assistance system proposed for astronauts working in orbit during space station missions, which effectively reduces the complexity of astronauts’ work and improves mission reliability [47]. (2) An effective computational model for scene perception and understanding has been proposed in this paper, which has the advantages of lightweight deployment and high computational accuracy.

In the following sections, the key techniques of the proposed system and method will be presented in Section 2, some evaluation experiments will be illustrated in Section 3, further data analyses and discussions will be given in Section 4, and a conclusion will be made in Section 5.

2. Proposed Methods

2.1. Computational Flow Chart Overview

Figure 2 presents the application illustration and computational flowchart of the proposed scene perception system. After the astronaut in the space station wears the glasses-based environmental perception system and enters the working area of the space station, taking the handheld display system, first, our proposed system evaluates the ambient lighting intensity in front of astronauts. If the ambient lighting intensity is too low (for example, the main control computer has reduced the ambient lighting output in the working space in order to save energy consumption in the space station), our system prompts the astronaut to adjust the cabin lighting source to meet the normal working lighting brightness. Second, our system collects the forward view scene images of the astronaut in the space station and uses our improved DeepLabV3+ network to perform semantic segmentation on the complex cabinet components in the scene. Third, after completing the segmentation calculation, the astronaut can select the interested components in the handheld display system according to the repair outline of the complex scientific experimental cabinet based on the segmentation results above. Finally, according to the astronaut’s selections, our system can create the 3D models, audio and video introductions, and textual descriptions of the selected component in the software interface, thereby assisting the astronaut in understanding its maintenance principles.

2.2. Environment Lighting Perception

A robust environment lighting perception method is used in this paper [48]. First, the original input image is transformed from the RGB color space into the Lab color space [49]. Second, the mean subtracted and contrast normalized (MSCN) coefficient [50] is computed. The MSCN coefficient is extracted by subtracting the mean from the image and normalizing the contrast to extract statistical features of the image, which can be used to predict the distortion type and perceived quality of the image. Third, the information entropy indices are calculated. The image information entropy is calculated from the image histogram and can reflect the average information content of an image. The larger the entropy value, the more chaotic the image. And the gray-level histogram statistics can reflect the pixel distribution or brightness changes in an image. Therefore, this paper uses the information entropy to measure the overall brightness of an image, which has the advantages of fast calculation speed and high processing accuracy. Finally, the support vector machine (SVM) [51] is considered for environment lighting degree perception. Equations (1)–(4) show the computational method of information entropy, and a three-input and one-output SVM is used for lighting state evaluation. A binary classification problem is considered, dividing the ambient lighting into normal (ordinary and strong lighting conditions) and abnormal (dim lighting conditions).

Table 1. Definition of imaging luminance degrees.

	Imaging Luminance Condition
	Dim	Ordinary	Strong
Ambient lighting condition	~<50.0 lx	~>50.0 lx and ~<160.0 lx	~≥160.0 lx
Information entropy	4.5 > HL > 2.5, 5.2 > Ha > 5.0, and 2.5 > Hb > 2.1	5.0 > HL > 4.0, 5.0 > Ha > 4.5, and 2.5 > Hb > 2.1	19.0 > HL > 11.5, 5.4 > Ha > 5.2, and 3.3 > Hb > 2.8

presents the definition methods of imaging luminance degrees and their roughly corresponding information entropy distributions in this paper. Figure 3 shows the data samples of complex cabinet components captured from different ambient lighting conditions. From Table 1, the brightness division can be continuously optimized and adjusted with the accumulation of data, so the estimation of lighting brightness using SVM will be more meaningful currently.

$${I^{\prime }}_X\left(i,j\right)={\displaystyle \frac{I_X\left(i,j\right)-{\mu }_X\left(i,j\right)}{{\sigma }_X\left(i,j\right)+C}},$$

(1)

$${\mu }_X\left(i,j\right)={\displaystyle \sum _{k=-K}^K{\displaystyle \sum _{l=-L}^L{w}_{k,l}{I}_X\left(i+k,j+l\right)}},$$

(2)

$${\sigma }_X\left(i,j\right)=\sqrt{{\displaystyle \sum _{k=-K}^K{\displaystyle \sum _{l=-L}^L{w}_{k,l}{\left[I_X⌈i+k,j+l⌉-{\mu }_X\left(i,j\right)\right]}^2}}},$$

(3)

$$H_X=-{\displaystyle \sum _{m=0}^{255}{P}_X\left(m\right){\log }_2{P}_X\left(m\right)},$$

(4)

where I_X(i, j) is the image intensity of the X component of Lab color space in point (i, j), i = 1, 2, …, M, j = 1, 2, …, N; M and N are the height and weight of the input image; I′_X(i, j) is the MSCN estimation of I_X(i, j); μ_X(i, j) is the mean value of I_X(i, j); σ_X(i, j) is the standard deviation value of I_X(i, j); C is a constant, C = 1.0 in this paper; X ∈ {L, a, b}; w_k,l is a Gaussian filter; k and l are the sizes of the discrete Gaussian filter, k = −K, …, K, l = −L, …, L, K = 2 and L = 2 in this paper; H_X is the information entropy of the X component; and P_X(m) is the histogram value of I′_X(i, j) in gray intensity m.

2.3. Improved DeepLabV3+

The overall architecture of classic DeepLabV3+ is shown in Figure 4. The core idea of DeepLabV3+ is the introduction of a new encoder–decoder architecture. The main body of its encoder is a deep convolution neural network (DCNN) [52] with atrous convolutions, which can be dependent on the commonly used classification networks. The atrous convolution adds holes to the standard convolution kernel to increase the receptive field. By setting different dilation rates, different-sized receptive fields can be obtained, and the multi-scale information can be estimated. For example, a normal convolution is a dilated convolution with an expansion rate of 1; differently, a 3 × 3 convolution kernel with an expansion rate of 2 has the same receptive field as a 5 × 5 convolution kernel and only needs 9 parameters. At the end of the encoder module, DeepLabV3+ introduces the ASPP module with dilated convolutions to improve its segmentation capabilities for multi-scale objects. ASPP performs 1 × 1 convolution on the input feature maps, 3 × 3 convolution with dilation rates of 6, 12, and 18, and the global average pooling operations. After fusing the feature maps and performing 1 × 1 convolution, the number of channels is compressed to 256. Finally, the ASPP is able to extract and distinguish feature information of targets at different scales, achieving good segmentation of multi-scale targets. The goal of this module is to capture contextual information at different scales in order to better understand the semantic information of whole images. The decoder module in DeepLabV3+ is used to perform necessary downsampling operations, further fusing low-level features with high-level features. During the feature map restoration process, low-level features are fused to restore the boundary information of the target part. Linear interpolation is also considered for feature map restoration, ultimately improving the accuracy of network segmentation.

In this paper, a series of improvement studies are conducted on the DeepLabV3+ network to meet the application requirements of the scene understanding mission in the space station. Figure 4 illustrates the structure of our improved DeepLabV3+. The original backbone network is replaced with a lightweight MobileNetV2 with a smaller model scale and lower computational complexity. The CBAM is introduced to improve the network’s feature perception ability. And the ASPP module is also considered to enable an accurate calculation of encoding multi-scale information.

• Improvement of backbone network

The classic DeepLabV3+ uses the Xception [53] as the backbone network. The architecture of the Xception model is relatively complex and has strong expression ability, making it suitable for complex tasks and large-scale datasets. However, due to its relatively large model size, it requires more computing resources and storage space, and its training time is also long, making it less suitable for portable devices with limited resources. In this paper, our improved DeepLabV3+ network (Figure 5) replaces the classic backbone network with a lightweight convolutional neural network, i.e., MobileNetV2. Compared with the Xception network, MobileNetV2 has a smaller model size and lower computational complexity, making it suitable for mobile devices and embedded systems. Due to its lightweight design, MobileNetV2 also has a fast speed in the reasoning phase. It replaces ordinary convolution with deep convolution and introduces two hyperparameters, namely width factor and resolution factor, to flexibly control the size of the network model. With the optimization of network structure, the network accuracy exceeds that of most neural networks with fewer parameters and computational complexity. MobileNetv2 has two important modules, i.e., the linear bottleneck relationships between network layers and the residual connections between bottleneck blocks. The design of bottleneck blocks can effectively encode the feature information of the model at the input and output ends, and the inner layer is used to encapsulate the model’s transformation from lower-level information to higher-level abstract representations.

• Combination of CBAM

CBAM (Figure 6) is one of the widely used attention mechanisms in deep learning networks, aiming to improve the performance of the network model by focusing on important parts of the image. CBAM processes both channel and spatial dimension information, focusing on which channels are important through the channel attention module and then focusing on where the informative part is through the spatial attention module. This dual attention mechanism enables it to comprehensively capture key information in features. CBAM mainly has the following characteristics. First, it has a dual attention mechanism. It can simultaneously model the dependency relationship between channels and the importance of spatial positions, enhancing the model’s ability to capture the key features. Second, it utilizes a lightweight design method. It only requires a small number of additional parameters to significantly improve performances. Third, it has the advantage of convenient plug-and-play. It can be flexibly embedded into any convolutional blocks of deep learning networks and is suitable for tasks such as classification, detection, segmentation, etc.

The channel attention mechanism obtains two different feature descriptions, i.e., an average and a maximum, for each channel of the input feature map through global average pooling and global maximum pooling. These descriptions are processed by a multi-layer perceptron with the shared weights, and the output feature maps are then element-wise summed. The Sigmoid function is applied to generate a channel attention map M_c with a shape consistent with the number of input channels. Its computational formula is shown in Equation (5). The spatial attention uses channel attention to generate two 2D feature maps through average pooling and max pooling in the channel direction. These feature maps are stacked in the channel dimension, then passed through a 7 × 7 convolutional layer, and then a spatial attention map M_s can be generated using the Sigmoid function. Its computational formula is expressed in Equation (6). The output feature map F′ (Equation (7)) can be obtained by the element-wise multiplication, combining the input feature map F with the channel attention map M_c and spatial attention map M_s. Finally, through the structure above, the CBAM can effectively enhance the network’s response to important detailed features in images, improving the model’s response performance in various visual tasks, especially in image recognition and semantic segmentation of complex application scenes.

$$M_c=\sigma \left\{MLP\left[AvgPool\left(F\right)\right]+MLP\left[MaxPool\left(F\right)\right]\right\},$$

(5)

$$M_s=\sigma \left\{Con{v}_{7\times 7}\left[Concat\left(AvgPool\left(F\right),MaxPool\left(F\right)\right)\right]\right\},$$

(6)

$$F^{\prime }=\left(F\otimes M_c\right)\otimes M_s,$$

(7)

where MLP(·) is the multi-layer perceptron; AvgPool(·) and MaxPool(·) are the average pooling and maximum pooling processing; F is the first feature map; σ(·) is the Sigmoid activation function; Conv_7×7(·) means the 7 × 7 convolution operation; Concat(·) is the feature fusion operation; and ⊗ is the Kronecker product.

• Replacement of ASPP by DenseASPP

After extracting features in the backbone feature extraction network, a preliminary effective feature layer that has undergone four downsampling is obtained, and then multi-scale feature capture is performed on this effective feature layer. After multi-scale capture is completed, multiple feature layers are fused and then rolled up by 1 × 1 to obtain the feature output map. The traditional ASPP utilizes parallel AtrousConvolution to extract features using AtrousConvolution at different rates, which can capture different information at multiple scales through convolution of different receptive fields. The traditional ASPP format is shown in the figure. However, as the expansion rate increases (especially when it exceeds 24), the effective weights of dilated convolutions also decrease, and the feature extraction ability also decreases. Therefore, this paper adopts the DenseASPP [54] module (Figure 7) instead of the traditional ASPP module, which can better capture semantic information in images. DenseASPP adopts a dense connection and cascaded dilated convolutional layer design. The output of each dilated convolutional layer (Equation (8)) is not only passed to the next layer but also connected to all subsequent unvisited layers, forming a dense feature transfer path. The dilation rate of each layer in DenseASPP increases layer by layer, and the final output is a feature map generated by multi-scale dilated convolutions with multiple dilation rates.

$$y_i=H_{k,d_l}\left(\left\{y_{l-1},y_{l-2},\dots ,y_0\right\}\right)$$

(8)

where d_l represents the expansion rate of the lth layer of the network; H_k,dl(·) is the dilated convolution operation; and [y_l−1, y_l−2, …, y₀] is the feature map formed by connecting the outputs of all previous layers. DenseASPP not only accumulates multi-scale features through dense connections but also determines the receptive field through multiple continuous dilated convolutional layers, avoiding the problem of performance degradation caused by a single high expansion rate.

3. Experiment Results and Evaluations

A series of simulation tests are carried out on our PC (Intel Xeon Gold 6133 CPU (Intel Corporation, Santa Clara, CA, USA) 2.50 GHz, 40 GB RAM, and Tesla V100-SXM2-32 GB) by Python 3.8.20 to test the correctness of the proposed algorithm. In this paper, we use three comparative experiments to evaluate our proposed system and method, including the environmental lighting evaluation experiment, the improved DeepLabV3+ network comparison experiment, and the system application evaluation experiment. Due to the space station application limitations, all the evaluations are conducted using the ground imitation experimental methods currently.

3.1. Experiment Data and Organizations

Because we cannot directly obtain the massive imaging observation data of the actual experimental devices in the space station currently, in order to verify the correctness of the proposed system and method, in this paper we design a 3D model of a kind of complex device and use the 3D printing technology to produce its physical object to imitate the appearance of corresponding actual experimental devices in the space station. A Groudchat wearable camera whose image resolution is 3840 × 2160 is used to collect experimental data. The samples of the relevant image dataset are shown in Figure 8. As can be seen from Figure 8, the components in Figure 8 include multiple types of jars or boxes, round buttons, and smaller-sized pipelines and valves, which are similar in appearance to the experimental devices or daily environmental control and life support equipment (e.g., the urine recycler) in the space station. In order to increase data diversity, we collect and store these images of this device from different photographic angles (angle deviations from the center front view direction of this device are about −60°~60°) and distances (distance deviations from this device are about 20.0 to 50.0 cm). Meanwhile, considering that environmental light control is often required within the space station to save electricity, we also collect data under different lighting conditions to support testing the brightness evaluation indicators proposed in our method. Currently, we have accumulated more than 1500 images in our dataset. Among them, 750 images are collected under ordinary lighting conditions, while the remaining 750 images are captured in environments with strong lighting (Figure 8). The LabelMe software [55] (windows version) is used to label the original data. Regarding the image annotation task, currently, we classify the scene into three kinds of devices, including five tanks, three valves, and eight pipelines. In this paper, the occlusion processing is also considered. For the partially occluded part, the method of bypassing the occlusion object to mark the rear target is used. For the truncated part, the method of double-line segmentation is employed. In addition, we also consider the color specification. The color is strictly consistent with the pre-designed specified color, and LabelMe automatically assigns the color according to the category order.

3.2. Evaluation Experiments of Environment Lighting

This paper uses image quality evaluation parameters to evaluate the lighting brightness of the imaging environment. On one hand, we calculate the information entropy in the LAB color space to quantitatively describe illumination imaging features. Compared to the RGB color space, it is well known that the LAB color space is more in line with the human eye’s perception of color, making it suitable for objective calculation of environmental brightness. On the other hand, we also use the SVM for binary classification of imaging environment brightness. The SVM classifier has the characteristics of fast calculation speed, good calculation effect, and less training data requirement, making it particularly suitable for lightweight implementation of various computing applications.

Table 2. Classification accuracy comparison of environment lighting conditions using different SVM kernel functions.

Kernel Function	Polynomial Function	Linear Function	Radial Basis Function
Classification accuracy	0.75	0.875	0.8929

compares the classification accuracy of SVM using different kernel functions [56]. The training data amount is 600, and the testing data amount is 56. As shown in Table 2, using the radial basis function as the kernel function of SVM can achieve the best classification calculation accuracy in this application. Therefore, this kernel function will also be used for our subsequent experiments.

Table 3. Subjective and objective imaging brightness evaluation results of Figure 8 and Figure 9.

Image Name	HL	Ha	Hb	SVM Output *	Subjective Imaging Luminance Evaluation Result
Figure 8a-1	4.32	4.85	2.29	P	Normal
Figure 8a-2	4.24	4.52	2.12	P	Normal
Figure 8a-3	4.15	4.61	2.16	P	Normal
Figure 8a-4	4.36	4.88	2.29	P	Normal
Figure 8a-5	4.30	4.85	2.29	P	Normal
Figure 8a-6	4.40	4.58	2.17	P	Normal
Figure 8a-7	4.09	4.63	2.16	P	Normal
Figure 8a-8	4.01	4.92	2.47	P	Normal
Figure 8b-1	17.19	5.32	3.27	P	Normal
Figure 8b-2	18.04	5.32	3.28	P	Normal
Figure 8b-3	18.53	5.34	3.28	P	Normal
Figure 8b-4	14.41	5.28	2.88	P	Normal
Figure 8b-5	16.20	5.29	2.99	P	Normal
Figure 8b-6	13.13	5.32	2.82	P	Normal
Figure 8b-7	13.35	5.33	2.85	P	Normal
Figure 8b-8	11.75	5.26	2.83	P	Normal
Figure 9-1	3.25	5.08	2.43	N	Abnormal
Figure 9-2	3.83	5.05	2.40	N	Abnormal
Figure 9-3	3.26	5.08	2.45	N	Abnormal
Figure 9-4	3.64	5.06	2.30	N	Abnormal
Figure 9-5	3.69	5.07	2.20	N	Abnormal
Figure 9-6	3.82	5.07	2.14	N	Abnormal
Figure 9-7	2.87	5.02	2.12	N	Abnormal
Figure 9-8	3.87	5.14	2.48	N	Abnormal

* In this table, the symbol ‘P’ means the output of SVM is positive, i.e., the imaging quality is normal; the symbol ‘N’ indicates the output of SVM is negative, i.e., the imaging quality is abnormal.

shows the information entropy values corresponding to all the data in Figure 8 and Figure 9. From Figure 8 and Figure 9 and Table 3, it can be seen that the imaging illumination intensity calculation method in this paper can correctly evaluate the illumination environment and meet our system application requirements. Finally, according to the statistical calculations, the average calculation speed of the above method is only about 0.2 s. The above fast processing speed also ensures that our method can be applied to practical space station scene understanding.

3.3. Evaluation Experiments of Improved DeepLabV3+

An ablation experiment is performed in this paper to verify the effectiveness of the proposed improved DeepLabV3+ network.

Table 4. The ablation experiment results of the proposed DeepLabV3+ network.

Experiment ID	Network Module			Accuracy/%	Precision/%	MIoU/%
	CFF	CBAM	DenseASPP
1	×	×	×	97.69	96.57	91.4
2	√	×	×	97.68	96.64	91.34
3	×	√	×	97.76	96.74	91.62
4	×	×	√	97.71	96.7	91.45
5	√	√	×	97.71	96.78	91.48
6	√	×	√	97.72	96.7	91.51
7	√	√	√	97.73	96.68	91.53
8 (Ours)	×	√	√	97.76	96.7	91.66

presents the corresponding results. Regarding this experiment, the training set has 1170 data, and the test set has 130 data. Three evaluation metrics are used for segmentation performance evaluation, i.e., accuracy, precision, and MIoU. The accuracy is defined as the ratio of correctly classified pixels to the total number of pixels. The higher the accuracy, the better the segmentation performance of the model. The precision refers to the proportion of correctly predicted (true label positive) samples among all samples predicted as positive. In semantic segmentation, precision reflects the model’s ability to recognize positive samples. The MIoU is an indicator that measures the degree of overlap between predicted and actual regions. The higher the MIoU, the better the segmentation effect. In Table 4, all models use MobileNetV2 as the backbone network, and besides the CBAM and DenseASPP modules, a cascaded feature fusion (CFF) module [57] is also considered for evaluation. The CFF module is a deep learning architecture designed for multi-scale feature optimization. Its core idea is to improve semantic segmentation performance in complex scenes through a progressive feature integration mechanism. Because CFF often appears in the design of deep learning network structures, this paper also adds comparisons to our ablation experiment. From Table 4, although our method cannot perform the best in all the evaluation parameters; however, our model with CBAM and DenseASPP modules can achieve the relative best results.

To further verify the effectiveness of the proposed method in replacing the backbone network, attention mechanism, and application of the ASPP module, eight comparison experiments are also designed. In this experiment, the training set has 1100 data, and the test set has 120 data. In experiment 1, we use the classic DeeplabV3+ algorithm for computation. In experiment 2, we use the classic DeeplabV3+ and replace its backbone network with MobileNetV2 while keeping other parts unchanged. In experiment 3, we use the classic DeeplabV3+ and replace the backbone network with MobileNetV3 while keeping other parts unchanged. In experiment 4, we use the backbone network with MobileNetV2 and introduce the attention mechanism CBAM. In experiment 5, we use the backbone network with MobileNetV2 and introduce the attention mechanism of the squeeze-and-excitation (SE) module [58]. In experiment 6, we use the backbone network with MobileNetV2 and consider the mechanism of the context anchor attention (CAA) module [59]. In experiment 7, we use the backbone network with MobileNetV2 and employ both CBAM and the CFF module. In experiment 8, we use the backbone network with MobileNetV2 and utilize CBAM and DenseASPP modules. The evaluation metrics, including accuracy, precision, and MioU, are used again for processing effect evaluation. The corresponding results are illustrated in

Table 5. Semantic segmentation comparison experiments using different computational modules.

Experiment Type	Accuracy/%	Precision/%	MIoU/%	Time Consumption
Experiment 1	Overfitting	Overfitting	Overfitting	1.63 s
Experiment 2	94.96	97.19	89.45	1.52 s
Experiment 3	95.02	97.11	89.19	1.65 s
Experiment 4	95.19	97.23	89.52	1.60 s
Experiment 5	94.88	97.19	89.39	1.58 s
Experiment 6	94.94	97.21	89.47	1.64 s
Experiment 7	95.13	97.21	89.51	1.68 s
Experiment 8 (Ours)	95.05	97.24	89.59	1.74 s

. From Table 5, it can be seen that our proposed model with MobileNetV2, CBAM, and DenseASPP modules can achieve the relative best processing effect.

A series of comparison experiments among the common semantic segmentation networks are also performed. In this experiment, we compare the computational performance of the SegFormer network [60], UNet, SegNet, pyramid scene parsing network (PSPNet) [61], classic DeepLabV3+, and our improved DeepLabV3+ network. In this experiment, the training set has 1168 data, and the test set has 131 data. SegFormer is a semantic segmentation model based on Transformer [62], which adopts a hierarchical Transformer encoder structure and a lightweight multilayer perceptron decoder. The encoder part fuses features of different scales through overlapping patch merging operations, while the decoder part uses lightweight multilayer perceptrons to gradually upsample the feature map to the original resolution. This design not only improves the network efficiency but also reduces the computational load. The core design concept of UNet combines the encoder and decoder structures and skip connection mechanism, which is particularly suitable for scenarios with small sample data and high-precision segmentation requirements. Its main features include the small sample adaptability, fine segmentation ability, and lightweight network design. The core design of SegNet improves segmentation accuracy and reduces computational overhead through the pooling index upsampling mechanism. Its main features include efficient parameter design, pooling index upsampling, and lightweight real-time computation performance. The PSPNet is also a deep convolutional neural network semantic segmentation model. The core idea of this model is to use pyramid pooling modules to capture contextual information at different scales to improve the accuracy of image semantic segmentation. Its characteristic is that it can effectively utilize contextual information while integrating multi-scale information, etc. The corresponding experimental results are presented in

Table 6. Comparison experiment results of different semantic segmentation networks.

Experiment Type	Accuracy%	Precision%	MIoU%	Time Consumption
SegFormer	93.95	80.61	73.14	1.76 s
UNet	95.97	93.04	83.94	1.84 s
SegNet	94.65	93.67	87.43	2.46 s
PSPNet	93.17	78.36	70.83	1.67 s
Classic DeepLabV3+	97.03	96.35	88.73	1.63 s
Improved DeepLabV3+ (Ours)	97.27	96.57	89.86	1.74 s

and Figure 10. As can be seen from these results, the method proposed in this paper can perform the best on all indicators, which demonstrates the effectiveness of our method to some extent. The average processing speeds of all the above algorithms are basically less than 1.0 s, which can meet the requirement of human–computer interactive operation regarding the corresponding in-orbit maintenance guiding applications. When astronauts in space stations apply this technology in practice, they need to carefully observe the handheld panel for a very long time, so the impact of a few seconds of processing time on the mission is not significant.

3.4. Evaluation Experiments of System Application

This paper performs an application comparison experiment of our designed system and method in the ground environment. First, an equipment inspection and disassembly operation without using our system and method is performed. The system users directly employ the traditional handheld panel to learn the device’s working principles and try to dismantle and assemble it. The handheld panel is equipped with a complete device introduction video and text. Then the average operation time, the average longest operation time, and the average shortest operation time can be recorded. Second, our designed method is used to re-examine and disassemble the same equipment above, and the average operation time, the average longest operation time, and the average shortest operation time are calculated again. The corresponding results are recorded in

Table 7. Comparison experiment results using different equipment maintenance guidance methods.

Experiment Type	Average Operation Time/s	Longest Operation Time/s	Shortest Operation Time/s
Experiment without the application of our proposed system and method	~140.0	~164.0	~123.0
Experiment with the application of our proposed system and method	~25.0	~32.0	~21.0

. As shown in Table 7, the average operation time without using our method is more than 140 s longer than that using our intelligent interactive method. And the average maximum and average minimum operation times are also significantly longer than the method proposed in this paper. In addition, we also can observe that when our intelligent method is not used, the astronaut will occasionally make some mistakes during that mission, which will delay the task execution time and reduce task reliability. Figure 11 presents the application process example of our proposed system and method. Users can select objects of interest based on the semantic segmentation results of the scene returned by the handheld panel. The system comprehensively uses 3D model explosion figures, audio and video files, text, and other methods to provide maintenance prompts to users.

4. Discussion

The importance of space station technology is self-evident as the forefront base for human exploration of space. However, space station in-orbit devices face many serious challenges. The enclosed and narrow space characteristics of the space station make its internal air circulation dependent on the ventilation system. Meanwhile, astronauts in space stations breathing, sweating, and even the water vapor generated by equipment operation will make the devices in the space station prone to surface contamination and mold problems, seriously threatening the performance and service life of the space station and thus affecting the safe and stable operation of space missions as well as the health and work efficiency of astronauts in space stations. To ensure the long-term reliable operation of the space station, astronauts working on the station need to regularly maintain the equipment. Due to the heavy workload of astronauts, wearable glasses and handheld panels can be designed based on semantic segmentation methods to identify the devices within the astronaut’s line of sight, allowing astronauts to directly interact with the device models and view their working principles. The performance parameters of our system can include the scene understanding semantic segmentation accuracy, the system response speed, the system mean time between failure (MTBF), etc. Clearly, the application of this kind of interactive artificial intelligence technology can significantly improve the safety of space flight missions, reduce the workload of astronauts, and avoid many potential risks.

Numerous experiments have shown that ambient light can affect the computational performance of the semantic segmentation network proposed in this paper. Therefore, in this paper, we further evaluate and analyze the computational performance of our improved DeepLabV3+ on images with different ambient light ratios. The experiments use images collected under different proportions of strong and ordinary lighting sources for network training and then randomly extract data from the image dataset (including images under various lighting conditions) for testing the training network. Currently, a total of 5 experiments are tested, with the number of strong lighting environment images and ordinary lighting environment images being 600 and 0 (experiment 1), 400 and 200 (experiment 2), 300 and 300 (experiment 3), 200 and 400 (experiment 4), and 0 and 600 (experiment 5), respectively. The number of test datasets is 60, including 30 images from strong lighting environments and 30 images from ordinary lighting conditions. The relevant network calculation results are shown in

Table 8. The semantic segmentation results using different ratios of environmental light intensity data.

Experiment Type	Accuracy/%	Precision/%	MIoU/%
Experiment 1	97.87	96.72	91.71
Experiment 2	98.04	96.76	91.56
Experiment 3	97.86	96.36	91.33
Experiment 4	97.92	96.71	91.65
Experiment 5	98.07	97.05	92.11

and Figure 12. From the results in Table 8, it can be seen that using data under ordinary lighting conditions for network training can achieve the best computational performance. Strong (i.e., overly bright) lighting environments may not achieve the best segmentation results; the possible reason for this result is that the device surfaces segmented in this paper are mostly metallic, which are prone to producing bright areas or reflectors and therefore affect the subsequent calculations. Clearly, the final lighting output control also needs to consider the subjective visual perception of astronauts in space stations and cannot be determined solely based on the imaging semantic segmentation results.

Generally speaking, the device maintenance missions of space stations in orbit have gone through three stages of development. In the first stage, in order to assist astronauts in space stations in completing various tasks in space stations, the paper operation manual is universally used to provide guidance to astronauts on relevant work details. Clearly, using paper-based operation manuals will lead to an increasing number of accumulated operation manuals and data generated during the long-term implementation of space station missions. In addition, paper manuals are not conducive to the storage and electronic processing of information, nor are they convenient for the efficient analysis of complex information. During the second stage, the use of a fixed electronic manual to assist astronauts in orbit operations emerged. Conventional electronic manuals only provide some principal system introductions, typical schematic diagrams, photo playback, and other functions, and cannot perform online interactive analysis of device status in actual scenarios, so their application effectiveness is also relatively limited. In the third stage, interactive scene understanding techniques using imaging analysis have emerged [63]. Astronauts in space stations can perform online interactive operations anytime and anywhere based on the information they are actually capturing, which significantly enhances their work interest and reduces workload [64]. As a result, the application of these techniques will significantly simplify astronauts’ in-orbit operations and improve mission reliability.

The system and method proposed in this paper have at least the following advantages. First, the network structure of our improved DeepLabV3+ method is lightweight, computationally efficient, and fast to implement, which can meet the real-time application of scene understanding in space stations. The actual experimental results show that the processing speed of our method is better than 1.0 s per frame. Second, the designed method has high reliability and can be used independently, making it suitable for practical applications in space stations. Based on the imaging lighting range inside the space station, we conduct a series of semantic segmentation calculations under different lighting environments, and the experimental results all show the effectiveness of the proposed method. Third, the designed method has strong scalability and can easily be combined with other deep learning methods for computation or replaced with other networks with more powerful computing capabilities. Of course, our methods also have certain shortcomings. For example, our method currently does not further process the segmentation of the same object under occlusion conditions. The semantic segmentation of small-sized targets remains a challenge in this field. In the future, we will further optimize the design methods of semantic segmentation networks, combining pre-training, transfer learning [65] and reinforcement learning techniques [66] to improve network computing performance. And the proposed system and method are expected to be applied in the maintenance of the life support system of the space station.

5. Conclusions

A design scheme for an interactive application system and its related algorithms is proposed for the development of space station scene understanding technology. First, a wearable glasses-based imaging acquisition device of the astronaut in the space station front view scene is designed, and the image processing and result display are achieved through a handheld panel. Second, an information entropy image quality assessment parameter is used to estimate the brightness of the imaging environment. When the brightness of the imaging environment is too low, the system prompts the astronaut to increase the ambient lighting output. Third, an improved design method for the DeepLabV3+ network is proposed, which introduces MobileNetV2, the attention mechanism, the DenseASPP module, etc., achieving a lightweight and efficient scene semantic segmentation calculation. Fourth, an interactive astronaut mission assistance display system is also designed, which comprehensively uses audio, video, text, and other methods to guide and assist astronauts in maintaining their in-orbit equipment cabinets. A large number of ground experiments have shown the correctness and effectiveness of the proposed method. In the future, the designed system will be applied in real in-orbit missions, further improving the safety and reliability of space flight missions.