A Novel Shipyard Production State Monitoring Method Based on Satellite Remote Sensing Images

Abstract

Monitoring the shipyard production state is of great significance to shipbuilding industry development and coastal resource utilization. In this article, it is the first time that satellite remote sensing (RS) data is utilized to monitor the shipyard production state dynamically and efficiently, which can make up for the traditional production state data collection mode. According to the imaging characteristics of optical remote sensing images in shipyards with a different production state, the characteristics are analyzed to establish reliable production state evidence. Firstly, in order to obtain the characteristics of the production state of optical remote sensing data, the high-level semantic information in the shipyard is extracted by transfer learning convolutional neural networks (CNNs). Secondly, in the evidence fusion, for the conflict evidence from the core sites of the shipyard, an improved DS evidence fusion method is proposed, which constructs the correlation metric to measure the degree of conflict in evidence and designs the similarity metric to measure the credibility of evidence. Thirdly, the weight of all the evidence is calculated according to the similarity metric to correct the conflict evidence. The introduction of the iterative idea is motivated by the fact that the fusion result aligns more closely with the desired result, the iterative idea is introduced to correct the fusion result. This method can effectively solve the conflict of evidence and effectively improve the monitoring accuracy of the shipyard production state. In the experiments, the Yangtze River Delta and the Bohai Rim are selected to verify that the proposed method can accurately recognize the shipyard production state, which reveals the potential of satellite RS images in shipyard production state monitoring, and also provides a new research thought perspective for other industrial production state monitoring.

1. Introduction

The shipbuilding industry plays a crucial role in national defense security, transportation, and marine development [1]. Effective monitoring of the shipyard production state is essential for the timely understanding of profitability, thereby facilitating the healthy upgrading and transformation of the shipbuilding industry structure. Additionally, it enables the orderly reorganization of uncompetitive shipyards during prolonged periods of losses to prevent wastage of coastal resources. Satellite remote sensing (RS) data offer an efficient means to accurately monitor the state of shipyard production from both spatial and time series perspectives, thus compensating for the limitations associated with traditional methods used for collecting order data.

With the development of sensor technology, the RS data have found extensive applications in disaster monitoring [2,3,4,5,6,7], precision agriculture [8,9], environmental risk assessment [10,11], and oil spill detection [12]. Furthermore, leveraging the repeated earth observations facilitated by satellites, multi-temporal RS data have been utilized to enhance land cover classification accuracy [13,14,15,16,17,18,19,20,21], object recognition [22,23,24], as well as disaster monitoring and assessment [25,26,27]. In recent years, high-resolution RS (HRS) data together with thermal infrared and night light data have been effectively employed for global-scale to local-region monitoring of human economic activities [28,29,30,31,32,33,34]. However, there is a dearth of research articles specifically addressing the shipyard production state.

The fundamental purpose of utilizing satellite remote sensing data for monitoring the production status of shipyards lies in determining whether the shipyard is engaged in production activities at the time of imaging, based on its distinctive characteristics. In simpler scenarios, discerning the state of a scene typically involves integrating relatively straightforward middle- and low-level semantics with machine learning techniques [35,36,37]. In more complex scenes, two processing approaches can be employed. The first approach entails extracting research objects from intricate backgrounds through image processing or object recognition to eliminate their influence before conducting state recognition on these research objects [38]. The second approach utilizes deep learning methods to extract and classify middle- and high-level semantic features of the scene [7,39,40]. In recent years, the continuous improvement in optical RS image quality has led to a more pronounced distinction in texture features of ground objects depicted in these images. This advancement enables the provision of richer semantic information and enhances interpretability. For extensive areas such as shipyards, employing optical RS images is suitable for identifying their production state.

The shipyard scene is intricate, comprising multiple core work sites. By leveraging high-level semantic information and low/middle-level features extracted from RS imagery, it becomes possible to establish evidence of the shipyard production state. Convolutional neural networks (CNNs), such as AlexNet [41,42,43], VGG [44,45], ResNet [46,47,48,49,50,51], and Inception [20,52,53], among others, can effectively derive high-level semantic information [39,54,55,56] for extracting optical production state features from optical HRS data.

Given the lengthy shipbuilding procedure, this study proposes the utilization of multi-temporal RS data to effectively monitor the production state of shipyards. This approach aims to enhance the quantity of observation data and mitigate errors that may arise from relying solely on single-phase data.

In the shipyard scene, the dock, berth, material stacking area and assembly area are closely related to the production state of the shipyard. They are the core categories in the monitoring of the production state of the shipyard. Therefore, the production state of the shipyard can be inferred by monitoring the dock, berth, assembly area and material stacking area in the satellite RS image. There may be differences in the production state of shipyards pointed to by different core sites, which is related to the manufacturing cycle of different shipyards and different ship types, the preparation stage of shipyards, and the preparation work in the outfitting stage of docks. Henceforth, a fusion method is imperative for resolving conflicts arising from disparate interpretations of a shipyard’s production state based on various core sites.

In current research, there is a predominant focus on pixel-level and feature-level fusions, with less attention being given to the decision level. However, it should be noted that the decision-level fusion algorithm exhibits higher accuracy compared to the standard multi-source data fusion algorithm and has the potential to overcome limitations associated with multi-source data fusion technology [57,58,59]. For instance, in other fusion levels, the loss of one source of data can result in system failure; however, this is not the case for decision-level fusion systems. Moreover, this approach outperforms both aforementioned methods in terms of real-time performance [57]. Voting [60], Bayes inference [61], evidence theory [62], fuzzy integrals, which include ICA and the support vector machine (SVM) [63], as well as various other specific methods [64,65,66], are some important examples of algorithms related to this level [57]. The DS evidence theory [67,68], with its advantages over other fusion methods [69] in capturing the uncertainty of evidence and without using prior conditional probability densities, has been widely used in multi-source and multi-temporal data [70,71,72,73,74]. However, traditional DS evidence fusion fails to address highly conflicting evidence arising from the ambiguity of different core sites within shipyards and the uncertainty inherent in different shipbuilding stages during production state monitoring.

In order to achieve precise monitoring outcomes, a shipyard production state monitoring approach based on multi-temporal RS images is proposed. The primary contributions encompass the following: (1) The utilization of a transfer learning network is proposed to extract high-level semantic information from optical RS images in shipyards located in coastal areas or on river coasts and map it to the evidence value of their production state. (2) To address the limitation of single-phase observation data in fully reflecting the production state across different shipbuilding stages, a multi-phase data-driven approach is suggested for monitoring the production state of shipyards. (3) Aiming at the problem of evidence conflict in different core sites, the correlation metric and similarity metric are constructed, and the monitoring framework of the shipyard production state based on CNN and improved DS evidence theory is proposed. The problem of evidence conflict between different core sites in the shipyard production state monitoring is solved, and the accuracy of shipyard production state monitoring is improved.

2. Data

2.1. Experimental Area and Data

The Yangtze River Delta (26.6°N–35.3°N, 116.7°E–122.8°E) and the Bohai Rim (34.4°N–43.4°N, 113.5°E–125.8°E) regions of China, where the shipyards are densely located, are selected for this study.

Considering the shipbuilding period and the capacity to acquire RS data, quarter monitoring and semi-annual monitoring are determined. Based on information from the China Shipbuilding Industry Yearbook, the International Ship Network and RS images, the true shipyard production states are collected (see Figure 1 and Figure 2) as a reference. The production states of shipyards can be divided into two categories: normal and abnormal. The normal production state means the shipyard runs well with clear signs of production in the RS images. An abnormal production state means that the shipyard is in a state of shutdown, without clear production signs in the RS images.

ZY-3, GF-1 and Google Earth data are used in experiments, with parameters shown in

Table 1. Selected satellite RS data parameters.

Satellite	Sensors	Level	Revisit Cycle/Day	Spatial Resolution/m
ZY-3	Optical	True Color Image Products	5	2.1
GF-1	Optical	Level-1A	4	2
Google Earth	Optical	True Color Image Products	/	2

.

Due to weather factors in the coastal area and along the river area, effective optical HRS imagery can be acquired quarterly. The satellite RS data used in the experimental area are shown in

Table 2. Selected satellite data used in the experimental area ^#.

Yangtze River Delta	Time/Month	Jan	Feb	Mar	Apr	May	Jun
	Time/Quarter	I			II
	Optical (ZY-3)	⚪			⚪
Bohai Rim	Time/Month	Jan	Feb	Mar	Apr	May	Jun
	Time/Quarter	I			II
	Optical (GF-1)	⚪			⚪

^# ⚪ indicates data are available.

.

2.2. Characteristics of Shipyards in Different Production States

In Figure 3, the shipyard scene includes the docks, berths, assembly sites and material storage sites which are defined as core sites for production state monitoring. The docks/berths are defined as shipbuilding core sites (SCSs) and the material storage areas/assembly areas are defined as material utilization core sites (MUCSs).

In the normal production state shipyard scene, there are generally some hulls, whole ships or multiple ships under construction in the shipbuilding site, and there are usually block materials in the material utilization core site, which are arranged more regularly. In the shipyard scene with an abnormal production state, there are no hulls in the shipbuilding site, and there is no material stacking in the material utilization core site, which is similar to bare land. The optical images of core sites in different production states are shown in

Table 3. Images of core sites in different production states.

Core Site	The Normal Production State	The Abnormal Production State
	ZY-3 Optical Image	ZY-3 Optical Image
SCS (dock/berth)
MUCS (material storage area/assembly area)

.

2.3. Optical Sample Datasets

The quantity and quality of the sample datasets are important for the training and detection accuracy of CNNs. Training samples of SCSs and MUCS in different production states are made, respectively, by ZY-3 and Google Earth data from 2018 to 2020 (see

Table 4. Example of training sample datasets.

Core Sites	Production State	Amount	Sample Example
SCS	Normal	1548
	Abnormal	1548
MUCS	Normal	444
	Abnormal	444

). The number of SCS training samples is 3096, and the ratio of normal and abnormal samples in the production state is 1:1. The number of MUCS training samples is 888, and the normal-to-abnormal sample ratio in the production state is 1:1.

3. Methods

This paper proposes to apply multi-temporal RS data to shipyard production state monitoring. Firstly, high-level semantic information is extracted by the fine-tuned convolutional neural network to obtain the optical production state evidence. Secondly, a correlation metric and similarity metric are used to solve the highly conflicting evidence, calculate the weight of all evidence, correct the conflicting evidence, fuse all the evidence and iteratively correct the fusion results. Finally, the improved DS evidence fusion method is applied to obtain the production state of the shipyard. The flow chart is as follows (see Figure 4).

3.1. Neural Network

There are various types of docks and berths available, such as dry docks, water injection docks, and floating docks. In terms of berths, there are inclined berths and horizontal berths. However, it should be noted that the number of nationwide samples for interpretation in China is limited.

For the complex shipyard optical remote sensing scene composed of multiple land types, Inception v3 [75] and ResNet101 [76] are selected for training based on the characteristics of CNNs and the number of training samples.

The pre-trained Inception v3 and ResNet101 by the ImageNet datasets [77] are fine-tuned using core sites samples with different production states, respectively. The development of a custom architecture was not undertaken during the pre-training phase. The training samples undergo data augmentation, including random rotation, flipping, clipping, scaling, etc., in order to augment the sample size and enhance sample diversity. Manual parameter adjustments are made based on the loss curves of both the verification set and training set, while hyperparameters are determined through testing. In the training, a stochastic gradient descent method and a cross-entropy loss function are used. The learning rate drops to 0.1 times per 20 epochs. The additional training parameters are listed in

Table 5. Training parameters.

Sample Datasets Ratio (Training: Validation)	Training Optimizer	Momentum Factor	Batch Size	Epoch	Initial Learning Rate
8:2	SGDM	0.9	64	100	0.001

.

3.2. Optical Evidence

The fine-tuned CNNs are applied to compute the production state probabilities of the core sites in a shipyard separately. As the shipbuilding period is lengthy, some core sites may not be in a use stage at the satellite sensor imaging moment. Hence, the maximum of the normal production probabilities in core sites of each category is adopted as evidence, as in Formulas (1) and (2).

$${\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}}=\mathrm{m}\mathrm{a}\mathrm{x}({\mathrm{p}}_1,{\mathrm{p}}_2,{\mathrm{p}}_3,\dots {,\mathrm{p}}_{\mathrm{r}})$$

(1)

$${\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}}=\mathrm{m}\mathrm{a}\mathrm{x}({\mathrm{q}}_1,{\mathrm{q}}_2,{\mathrm{q}}_3,\dots {,\mathrm{q}}_{\mathrm{l}})$$

(2)

The $\mathrm{t}\mathrm{o}$ is the imaging time of optical data. In Formula (1), ${\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}}$ is the normal production probability of the SCS in optical data. ${\mathrm{p}}_{\mathrm{i}}(1\le \mathrm{i}\le \mathrm{r})$ is normal production probability of the ith SCS in optical data and $\mathrm{r}$ is the amount of SCSs in the shipyard. In Formula (2), ${\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}}$ is the normal production probability of MUCSs in the optical data. ${\mathrm{q}}_{\mathrm{j}}\left(1\le \mathrm{j}\le \mathrm{l}\right)$ is the normal production probability of the jth MUCSs in the optical data and $\mathrm{l}$ is the amount of MUCSs in the shipyard.

The optical SCS evidence ${\mathrm{P}}_{\mathrm{d}}$ consists of the single-phase observation ${\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}}$ as Formula (3). The optical MUCS evidence ${\mathrm{P}}_{\mathrm{m}}$ consists of the single-phase observation ${\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}}$ as Formula (4) and $\mathrm{h}$ is the amount of optical RS data.

$${\mathrm{P}}_{\mathrm{d}}=[{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{I}},{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{I}\mathrm{I}},\dots ,{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}},\dots ,{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{h}}],\mathrm{t}\mathrm{o}=\mathrm{I},\mathrm{I}\mathrm{I},\dots ,\mathrm{h}$$

(3)

$${\mathrm{P}}_{\mathrm{m}}=[{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{I}},{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{I}\mathrm{I}},{\dots ,{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}},\dots ,\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{h}}],\mathrm{t}\mathrm{o}=\mathrm{I},\mathrm{I}\mathrm{I},\dots ,\mathrm{h}$$

(4)

3.3. Decision-Level Fusion

3.3.1. DS Evidence Fusion Basic Theory

The DS evidence theory defines the recognition frame Θ, in which propositions are exhaustive and mutually exclusive. The $\mathrm{m}\left(\mathrm{A}\right)$ is the basic probability assignment of proposition A, indicating the degree of confidence in proposition A that is satisfied with $\mathrm{m}\left(\mathrm{\varnothing }\right)=0 \mathrm{a}\mathrm{n}\mathrm{d} \sum _{\mathrm{A}\subseteq \mathsf{\Theta }}\mathrm{m}\left(\mathrm{A}\right)=1$. For ∀$\mathrm{A}$ ⊆ Θ, the information fusion rule for evidence ${\mathrm{m}}_1,{\mathrm{m}}_2,\cdots ,{\mathrm{m}}_{\mathrm{n}}$ is:

$$\left({\mathrm{m}}_1\oplus \cdots \oplus {\mathrm{m}}_{\mathrm{n}}\right)\left(\mathrm{A}\right)=\frac{1}{1-\mathrm{K}}·\sum _{{\mathrm{A}}_1\cap {\mathrm{A}}_2\cap \cdots {\mathrm{A}}_{\mathrm{n}}=\mathrm{A}}{\mathrm{m}}_1\left({\mathrm{A}}_1\right)·{\mathrm{m}}_2\left({\mathrm{A}}_2\right)\cdots {\mathrm{m}}_{\mathrm{n}}\left({\mathrm{A}}_{\mathrm{n}}\right)$$

(5)

Among them,

$$\begin{gathered} \mathrm{K}=\sum _{{\mathrm{A}}_1\cap {\mathrm{A}}_2\cap \cdots {\mathrm{A}}_{\mathrm{n}}=\mathsf{\Phi }}{\mathrm{m}}_1\left({\mathrm{A}}_1\right)·{\mathrm{m}}_2\left({\mathrm{A}}_2\right)\cdots {\mathrm{m}}_{\mathrm{n}}\left({\mathrm{A}}_{\mathrm{n}}\right)= \\ 1-\sum _{{\mathrm{A}}_1\cap {\mathrm{A}}_2\cap \cdots {\mathrm{A}}_{\mathrm{n}}\ne \mathsf{\Phi }}{\mathrm{m}}_1\left({\mathrm{A}}_1\right)·{\mathrm{m}}_2\left({\mathrm{A}}_2\right)\cdots {\mathrm{m}}_{\mathrm{n}}\left({\mathrm{A}}_{\mathrm{n}}\right) \end{gathered}$$

(6)

3.3.2. Evidence Analysis

The proposed evidence is derived by two core sites from multi-temporal RS image data, so there may be conflicts between the evidence.

(1)

Due to the different core sites used at different stages of the shipbuilding, there may be conflicting evidence from the two core sites at the imaging moment.

(2)

Conflict evidence may also be caused by changes in the shipbuilding stage when the sensor continuously observes the same core site.

The DS evidence fusion theory fails when there is highly conflicting evidence. To address these issues, this paper defines two metrics to measure the degree of conflict and credibility of evidence and constructs an improved DS evidence theory calculation system.

The correlation metric is defined to measure the consistency between evidence as Formula (7). When the correlation metric $\mathrm{R}<0$, the evidence points to inconsistency with a high conflict. The similarity metric ${\mathrm{S}}_{\mathrm{j}}$ is defined in Formula (8). The more similarity there is in the evidence, the more credible it is.

$${\mathrm{R}}_{\mathrm{d}\mathrm{m}}=\left({\mathrm{M}}_{\mathrm{d}}-0.5\right)\left({\mathrm{M}}_{\mathrm{m}}-0.5\right)$$

(7)

$${\mathrm{S}}_{\mathrm{j}}=1-\sqrt{\frac{1}{\mathrm{n}-1}\sum _{\mathrm{i}=1}^{\mathrm{n}}{\left|{\mathrm{m}}_{\mathrm{i}}\left(\mathrm{A}\right)-{\mathrm{M}}_{\mathrm{j}}\right|}^2}$$

(8)

$${\mathrm{M}}_{\mathrm{j}}=\mathrm{A}\mathrm{v}\mathrm{g}\left({\mathrm{P}}_{\mathrm{j}}\right),\mathrm{j}\in [\mathrm{d},\mathrm{m}]$$

(9)

The symbols $\mathrm{d}$, $\mathrm{m}$, denote the SCS and MUCS data. ${\mathrm{P}}_{\mathrm{d}}$ is the evidence from the SCS. ${\mathrm{P}}_{\mathrm{m}}$ is the MUCS evidence. In Formula (7), ${\mathrm{R}}_{\mathrm{d}\mathrm{m}}$ is the correlation between ${\mathrm{P}}_{\mathrm{d}}$ and ${\mathrm{P}}_{\mathrm{m}}$. ${\mathrm{M}}_{\mathrm{d}}$, ${\mathrm{M}}_{\mathrm{m}}$ is the average of ${\mathrm{P}}_{\mathrm{d}}$, ${\mathrm{P}}_{\mathrm{m}}$, respectively, which is defined in Formula (9). In Formula (8), ${\mathrm{m}}_{\mathrm{i}}\left(\mathrm{A}\right)$ is the evidence value and $\mathrm{n}$ is the amount of evidence.

3.3.3. Evidence Fusion

The recognition framework of improved DS evidence fusion for the shipyard production state is $\mathsf{\Theta }=\left\{\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l},\mathrm{A}\mathrm{b}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\right\}$. We assume that proposition $\mathrm{A}$: {shipyard production state is normal}, $\mathrm{B}$: {shipyard production state is abnormal}. The evidence combining SCSs with MUCSs is shown below.

$${\mathrm{m}}_0\left(\mathrm{A}\right)=\left[{\mathrm{P}}_{\mathrm{d}},{\mathrm{P}}_{\mathrm{m}}\right]$$

(10)

According to the evidence analysis in Section 3.3.2, the following improved DS evidence theory calculation framework is proposed (see Figure 5).

(1)

Calculating the correlation ${\mathrm{R}}_{\mathrm{d}\mathrm{m}}$ of evidence between two core sites, the SCS and the MUCS. If the ${\mathrm{R}}_{\mathrm{d}\mathrm{m}}<0$, the SCS and MUCS indicate evidence conflict. The similarity metric is calculated and the weight of all evidence is calculated by the similarity metric (in formula (11)).

(2)

In Formula (12) and (13), the conflict evidence is corrected by weight. The greater the degree of similarity of evidence is, the stronger is the credibility and the greater the weight. The modified evidence is substituted into Formula (5) for evidence fusion, and the first evidence fusion result $E_0$ is obtained. The definition of m₀ is shown in Formula (13).

(3)

The greater the similarity of evidence is, the stronger the credibility is, the greater the weight is, and the more biased the evidence fusion result is for the evidence with greater weight. Therefore, the correlation metric between the corrected evidence with a larger weight and the fusion result is calculated. If $R_{m_0{E}_0}<0$, it indicates that there is still a conflict. Otherwise, the conflict problem has been solved and the fusion result is the output.

(4)

Considering that the fusion result is closer to the expected output, if $R_{m_0{E}_0}<0$ the weight of evidence is re-determined according to the fusion result; that is, the iterative idea is introduced to correct the fusion result. The ith (i $\ge$ 1) iteration process is to calculate the correlation metric and the similarity metric of the modified evidence and the i-1th evidence fusion result $E_{i-1}$. if $R_{m_0{E}_{i-1}}<0$, the new evidence weight is calculated by the similarity metric, and the conflict evidence is corrected according to the weight to obtain the ith fusion result $E_i$. The convergence condition of the iteration is to end the iteration and output when $R_{m_0{E}_i}>0$; otherwise, we continue the iteration until the condition is satisfied.

$${\mathsf{\omega }}_{\mathrm{j}}=\frac{{\mathrm{S}}_{\mathrm{j}}}{{\mathrm{S}}_{\mathrm{d}}+{\mathrm{S}}_{\mathrm{m}}},\mathrm{j}\in [\mathrm{d},\mathrm{m}]$$

(11)

$$\begin{gathered} {{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}}}^{\prime }={\mathsf{\omega }}_{\mathrm{d}}{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}}\cdots {{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}}}^{\prime }={\mathsf{\omega }}_{\mathrm{m}}{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}} \\ {\mathrm{P}}_{\mathrm{d}}^{\prime }=[{{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{I}}}^{\prime },{{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{I}\mathrm{I}}}^{\prime },\dots ,{{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}}}^{\prime },\dots ,{{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{h}}}^{\prime }],\mathrm{t}\mathrm{o}=\mathrm{I},\mathrm{I}\mathrm{I},\dots ,\mathrm{h} \\ {\mathrm{P}}_{\mathrm{m}}^{\prime }=[{{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{I}}}^{\prime },{{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{I}\mathrm{I}}}^{\prime },\dots ,{{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}}}^{\prime },\dots ,{{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{h}}}^{\prime }],\mathrm{t}\mathrm{o}=\mathrm{I},\mathrm{I}\mathrm{I},\dots ,\mathrm{h} \end{gathered}$$

(12)

$$\begin{gathered} {\mathrm{m}}_1=\sum _{\mathrm{t}\mathrm{o}=\mathrm{I}}^{\mathrm{n}}{\mathsf{\omega }}_{\mathrm{d}}{\mathrm{P}}_{\mathrm{d}\mathrm{o}}^{\mathrm{t}\mathrm{o}} \\ {\mathrm{m}}_2=\sum _{\mathrm{t}\mathrm{o}=\mathrm{I}}^{\mathrm{n}}{\mathsf{\omega }}_{\mathrm{m}}{\mathrm{P}}_{\mathrm{m}\mathrm{o}}^{\mathrm{t}\mathrm{o}} \end{gathered}$$

(13)

$${\mathrm{m}}_0=\left\{\begin{array}{c}{\mathrm{m}}_1 {\mathsf{\omega }}_{\mathrm{d}}>{\mathsf{\omega }}_{\mathrm{m}} \\ {\mathrm{m}}_2 {\mathsf{\omega }}_{\mathrm{m}}>{\mathsf{\omega }}_{\mathrm{d}}\end{array}\right.$$

(14)

After the above treatment, the probability of the shipyard production state is calculated.

$$\mathrm{E}\left(\mathrm{A}\right)=\left({\mathrm{m}}_1\oplus {\mathrm{m}}_2\oplus \cdots \oplus {\mathrm{m}}_{\mathrm{n}}\right)=\frac{{\mathrm{m}}_1{\mathrm{m}}_2\cdots {\mathrm{m}}_{\mathrm{n}}}{{\mathrm{m}}_1{\mathrm{m}}_2\cdots {\mathrm{m}}_{\mathrm{n}}+\left[{1-\mathrm{m}}_1\right]\left[{1-\mathrm{m}}_2\right]\cdots \left[{1-\mathrm{m}}_{\mathrm{n}}\right]}$$

(15)

$$\mathrm{E}\left(\mathrm{B}\right)=1-\mathrm{E}\left(\mathrm{A}\right)$$

(16)

$\mathrm{E}\left(\mathrm{A}\right)$ is the probability that the shipyard is in a normal production state.

The probability of a normal production state takes the value interval of [0, 1]; the closer it is to 1, the more likely it is that the shipyard is in a normal production state. The shipyard production state determination is defined and recognized in Formula (16).

$$\mathrm{Production} \mathrm{state} \mathrm{of} \mathrm{the} \mathrm{shipyard}=\left\{\begin{array}{c}\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}, \mathrm{E}\left(\mathrm{A}\right)\ge 0.5\\ \mathrm{A}\mathrm{b}\mathrm{n}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}, \mathrm{E}\left(\mathrm{A}\right)<0.5\end{array}\right.$$

(17)

3.4. Accuracy Evaluation

In order to validate the proposed method, the accuracy, precision, false alarm (FA), recall, missing alarm (MA) and F1-score are assessed, as follows:

$$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=(\mathrm{T}\mathrm{A}+\mathrm{T}\mathrm{N})/(\mathrm{T}\mathrm{A}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{A}+\mathrm{F}\mathrm{N})$$

(18)

$$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\mathrm{T}\mathrm{A}/(\mathrm{T}\mathrm{A}+\mathrm{F}\mathrm{N})$$

(19)

$$\mathrm{F}\mathrm{A}=\mathrm{F}\mathrm{N}/(\mathrm{T}\mathrm{A}+\mathrm{F}\mathrm{N})$$

(20)

$$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\mathrm{T}\mathrm{A}/(\mathrm{T}\mathrm{A}+\mathrm{F}\mathrm{A})$$

(21)

$$\mathrm{M}\mathrm{A}=\mathrm{F}\mathrm{A}/(\mathrm{T}\mathrm{A}+\mathrm{F}\mathrm{A})$$

(22)

$$\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times \mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}/(\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l})$$

(23)

where TA is the number of abnormal shipyards with correctly recognized and TN is the number of normal shipyards with correctly recognized. FA is the number of abnormal shipyards with production states falsely recognized as normal, and FN is the number of normal shipyards with production states falsely recognized as abnormal.

4. Results

4.1. Presentation of Results

Taking the Inception v3 network as an example, the state evidence of optical data is calculated, and the improved DS evidence fusion method is used to obtain the shipyard production state. The monitoring results are accurate as shown in

Table 6. Positive test results display.

Shipyard	Images	${\mathbf{P}}_{\mathbf{d}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}}$	${\mathbf{P}}_{\mathbf{d}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}\mathbf{I}}$	${\mathbf{P}}_{\mathbf{m}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}}$	${\mathbf{P}}_{\mathbf{m}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}\mathbf{I}}$	True State
i		0.99	0.99	0.99	0.99	Normal
ii		0.01	0.02	0.01	0.01	Abnormal
iii		0.14	0.02	0.90	0.92	Normal
iv		0.99	0.99	0.01	0.06	Normal
v		0.75	0.04	0.99	0.93	Normal
vi		0.04	0.16	0.82	0.85	Normal

.

The proposed method can effectively monitor the shipyard production state. The shipyard production state evidence, ${\mathrm{P}}_{\mathrm{d}\mathrm{o}}$, ${\mathrm{P}}_{\mathrm{m}\mathrm{o}}$, ${\mathrm{P}}_{\mathrm{d}\mathrm{s}}$, ${\mathrm{P}}_{\mathrm{m}\mathrm{s}}$, is generally consistent with the acquired RS images, as showing the normal shipyard (i) and the abnormal shipyard (ii).

The improved DS evidence fusion can effectively avoid the uncertainty associatedwith conflicting evidence. In shipyards (iii) and (iv), evidence is in conflict from the two core sites with the correlation ${\mathrm{R}}_{\mathrm{d}\mathrm{m}}$ < 0. The similarity metric is calculated, the conflict evidence is corrected, and the fusion result is corrected by iteration, so that the shipyard can be identified as the normal production state.

Taking the (iii) and the (vi) shipyards in Table 6 as an example, we compare the traditional DS evidence fusion, the voting fusion, and the Yager [78] fusion results with the improved DS evidence fusion method. The comparison results are shown in the following

Table 7. Comparison of fusion results.

Shipyard	${\mathbf{P}}_{\mathbf{d}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}}$	${\mathbf{P}}_{\mathbf{d}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}\mathbf{I}}$	${\mathbf{P}}_{\mathbf{m}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}}$	${\mathbf{P}}_{\mathbf{m}\mathbf{o}}^{\mathbf{t}\mathbf{o}=\mathbf{I}\mathbf{I}}$	$\mathbf{Voting}$	Yager	DS Evidence Fusion	Improved DS Evidence Fusion	True State
iii	0.14	0.02	0.90	0.92	--	0.13	0.47	0.58	Normal
vi	0.04	0.16	0.82	0.85	--	0.14	0.36	0.88	Normal

.

The improved DS evidence fusion method effectively addresses the issue of in different core sites, which remains unresolved by traditional DS evidence fusion, thereby enabling the accurate identification of a shipyard production state. As demonstrated by the experimental data Shipyard (iii) and Shipyard (vi) in Table 7, the traditional DS fusion obtains results of 0.47 and 0.36, respectively, both below 0.5, leading to erroneous classification of an abnormal production state for the shipyard. The voting method fails in this case. The fusion results obtained by the Yager fusion method are 0.13 and 0.14, respectively, both below 0.5, which also leads to the classification error of the abnormal production state of the shipyard. In contrast, the improved DS evidence fusion method involves calculating the source credibility to determine weights (as per Formula (8)), revising original evidence values (as per Formulas (11)–(13) following new evidence. Consequently, the improved fusion results are obtained as 0.58 and 0.88, respectively, accurately identifying the normal production state of the shipyard.

The monitoring results of the shipyard’s production state show that the proposed evidence is consistent with the actual situation and can objectively reflect the production state of the core sites in the multi-temporal data. The improved DS evidence fusion method can better deal with the problem of evidence conflict from the core sites.

4.2. Evaluation Results

The method proposed in this paper can effectively realize the semi-annual monitoring of a shipyard production state, and the evaluation results are shown in

Table 8. Evaluation of overall results of production state monitoring.

Area	CNN	Period	Accuracy	Precision	FA	Recall	MA	F1-Score
Yangtze River Delta	Inception v3	The 1st half-year	99.11%	100.00%	0.00%	94.12%	5.88%	96.97%
	ResNet101	The 1st half-year	100.00%	100.00%	0.00%	100.00%	0.00%	100.00%
Bohai Rim	Inception v3	The 1st half-year	97.67%	87.50%	12.50%	100.00%	0.00%	93.33%
	ResNet101	The 1st half-year	95.35%	85.71%	14.29%	85.71%	14.29%	85.71%

.

Through the proposed framework of CNNs and improved DS evidence fusion, the production state of shipyards can be well monitored. The Inception v3 model combined with the improved DS evidence fusion method performs well in the Bohai Rim research area. However, the result is slightly worse than the result from the Yangtze River Delta study area. The reason is that GF-1 optical RS data are used in the Bohai Rim rather than the ZY-3 data and Google Earth data involved in the training, and there are differences in resolution and imaging quality, resulting in decreased accuracy.

In addition, using this framework, the traditional DS evidence fusion method, voting method and Yager fusion method are compared with the method proposed in this paper. The evaluation results are shown in

Table 9. Fusion results of other methods in the study area.

Area	Fusion Method	CNN	Periods	Accuracy	Precision	FA	Recall	MA	F1-Score
Yangtze River Delta	Traditional DS evidence fusion	Inception v3	The 1st half-year	98.21%	89.47%	10.53%	100.00%	0.00%	94.44%
		ResNet101	The 1st half year	97.32%	85.00%	15.00%	100.00%	0.00%	91.89%
	Voting	Inception v3	The 1st half-year	90.18%	60.71%	39.29%	100.00%	0.00%	75.56%
		ResNet101	The 1st half-year	91.94%	65.38%	34.62%	100.00%	0.00%	75.56%
	Yager	Inception v3	The 1st half-year	89.29%	58.62%	41.38%	100.00%	0.00%	73.91%
		ResNet101	The 1st half-year	88.39%	56.67%	43.33%	100.00%	0.00%	72.34%
Bohai Rim	Traditional DS evidence fusion	Inception v3	The 1st half-year	97.67%	87.50%	12.50%	100.00%	0.00%	93.33%
		ResNet101	The 1st half-year	97.67%	87.50%	12.50%	100.00%	0.00%	93.33%
	Voting	Inception v3	The 1st half-year	88.37%	58.33%	41.67%	100.00%	0.00%	73.68%
		ResNet101	The 1st half-year	90.70%	63.64%	36.36%	100.00%	0.00%	77.78%
	Yager	Inception v3	The 1st half-year	86.05%	53.85%	46.15%	100.00%	0.00%	70.00%
		ResNet101	The 1st half-year	86.05%	53.85%	46.15%	100.00%	0.00%	70.00%

.

Compared with the traditional DS evidence fusion, the improved DS evidence fusion improves the precision and F1-score while maintaining high recall. In the Yangtze River Delta study area, for the selected two CNNs, the accuracy of the proposed method is improved by 0.9% and 2.68%, respectively, compared with the traditional DS evidence fusion method.

For the voting method and the Yager fusion method, although the high recall is also maintained, the overall accuracy, precision and F1-score are significantly reduced.

4.3. Discussion

The proposed method is limited in recognizing a small/micro shipyard production state. The partial failure results of the combination of Inception v3 and the improved DS evidence fusion method are shown in

Table 10. False test results display.

Shipyard	Images	True State
vii		Abnormal
viii		Abnormal
ix		Abnormal

.

The error reasons of the production state monitoring can be divided into two categories.

The first reason is the impact of small and micro shipyard features. Shipyard (vii) is located on mudflats with core sites that are similar in texture structures under different production states, which causes the CNNs to miss detection. Shipyard (viii) is in abnormal production state. However, the gantry cranes and their shadows around the two core sites disturb optical evidence extraction, which leads to missed detection of the shipyards in an abnormal state.

The second reason is the insufficient frequency of optical data acquisition and the problem of image quality. The production state monitoring of the experimental shipyard is semi-annual monitoring. The optical data acquisition frequency used in the monitoring is insufficient, and in shipyard (ix), the optical image is affected by the cloud layer, which ultimately leads to shipyard (ix) being mistakenly identified.

The analysis of the monitoring results shows that the method in this paper has high accuracy in the monitoring of the shipyard production state, but there are still some limitations, which need to be further improved in the future. The details are as follows:

(1)

The method in this paper needs to identify the shipyard and extract the core sites before monitoring, and the workload is large. There are few studies on the extraction of special scenes in shipyards and the automatic extraction of internal core sites. Subsequently, the comprehensive identification of the shipyard scene and state attributes can be carried out.

(2)

The method in this paper has a weak detection ability for the quarterly production state of small/micro shipyards. For the reason of misdetection, it can be improved from two directions. Firstly, we can increase the training samples of core sites of small/micro enterprises to improve the reliability of optical evidence; Secondly, more time-phase optical images can be used to monitor the production state of shipyards in the future to further improve the monitoring accuracy.

5. Conclusions

Under complex and volatile international circumstances, there is significant fluctuation in the demand for new shipbuilding. The utilization of multi-source satellite RS images enables the dynamic monitoring of the shipyard production state from both spatial and temporal perspectives, thereby enhancing the monitoring efficiency and providing comprehensive and timely insights into changes in the shipyard production status. It is of immense significance for industrial development, social stability, resource utilization, and ecological environment restoration.

This paper analyzes the characteristics of shipyards on HRS images and proposes a shipyard production state monitoring framework based on CNNs and improved DS evidence theory.

The innovations of this paper are as follows: (1) It proposes the use of multi-source and multi-temporal satellite RS images to monitor the production state of shipyards, which reflects the potential of HRS data in monitoring the production state of shipyards, and also provides a new research thought perspective for other industrial production state monitoring. (2) A solution strategy is presented that employs a constructed correlation metric and similarity metric to effectively enhance the accuracy of shipyard production state monitoring results through high-conflict evidence fusion.

The proposed monitoring framework for the shipyard production state is implemented in the Yangtze River Delta and Bohai Sea regions. The findings demonstrate that compared to the conventional DS evidence fusion method, the semi-annual monitoring of the shipyard production state can be conducted with greater accuracy using our approach. Furthermore, our proposed method exhibits excellent performance across various types of HRS images.

The article proposes a method that combines the advantages of HRS images and deep learning techniques, which can provide new ideas for the detection and identification of industrial sites and related research in remote sensing. This will broaden new perspectives and application directions for satellite remote sensing applications.