Application of Radar Image Fusion Method to Near-Field Sea Ice Warning for Autonomous Ships in the Polar Region

Abstract

Mastering the real-time dynamics of near-field sea ice is the primary condition to guaranteeing the navigation safety of autonomous ships in the polar region. In this study, a radar image fusion process combining marine radar and ice radar is proposed, which can effectively solve the problems of redundant information and spatial registration during image fusion. Then, using the fused radar images, this study proposes a set of near-field sea ice risk assessment and warning processes applicable to both low- and high-sea-ice-concentration situations. The sea ice risk indexes in these two situations are constructed by using four variables: sea ice area, sea ice grayscale, distance between sea ice and the own-ship, and relative bearing of sea ice and the own-ship. Finally, visualization processing is carried out according to the size of the risk index values of each piece of sea ice to achieve a better near-field sea ice risk assessment and warning effect. According to the example demonstration results, through the radar image fusion process and the set of near-field sea ice risk assessment and warning processes proposed in this study, the sea ice risk distribution in the near-field area of the ship can be well obtained, which provides effective support for the assisted decision-making of autonomous navigation in the polar region.

1. Introduction

In recent years, with the continuous warming of the global climate, much research data have shown that the extent of sea ice in the Arctic and Antarctica has decreased dramatically [1,2,3], and many studies have even predicted that the Arctic Ocean will be ice-free in summer within 15 years [4]. As the sea ice in the polar region gradually decreases, the number of voyages into the polar region also gradually increases. In the Arctic region, the Arctic shipping route has the economic benefit of greatly shortening the route distance between the three major economic circles of Northeast Asia, Europe, and North America. In the Antarctic region, the number of people engaged in scientific research and tourism activities has also increased year by year. Although global warming has increased the possibility of polar navigation, sea ice remains a key factor affecting navigation safety in the polar region. In order to ensure navigation safety, it is of great significance to conduct near-field sea ice perception and warning research, which is also one of the key technologies for the development of autonomous ships in the polar region.

To realize the autonomous and unmanned ship, the technology of perception, decision-making, and control must be developed, and the quality of perception information is the source and key to determine the effect of decision-making and control. The sensing equipment of autonomous ships can be divided into two types: shipboard sensors and satellite-based sensors [5]. Shipboard sensors include the very high frequency (VHF) radio-based automatic identification system (AIS), electronic chart display and information system (ECDIS) [6], marine radar, attitude indicator, and depth sounder. Satellite-based sensors include the global navigation satellite system (GNSS), satellite-based AIS [7], the meteorological and oceanographic (METOC) data receiver, synthetic aperture radar (SAR), and the long-range identification and tracking (LRIT) system. In general, different sensors have different detection objects, data types, and applicability, and not all of them are suitable for the ice detection requirements of polar autonomous ships.

This study summarizes and compares the related studies on sea ice risk warning, as shown in Table 1. According to the sensing range of sea ice by the sensing equipment, it can be divided into three categories: large-scale sea ice data, medium-scale sea ice data, and small-scale sea ice data. Among them, the large-scale sea ice data are acquired mainly through a satellite remote sensing method such as NOAA (the National Oceanic and Atmospheric Administration satellite) [8], RADARSAT (Canada’s satellite) [9], and MODIS (the moderate resolution imaging spectroradiometer on board NASA’s satellite) [10], which is not affected by polar night and weather, is widely used to retrieve the information of sea ice concentration [9,11,12,13,14,15] and thickness/grayscale [8,16,17,18,19], and monitors the spatial-temporal changes in sea ice conditions. However, its temporal resolution is usually one day, which is more suitable for grasping the large-scale sea ice range, but poor in grasping near-field and real-time sea ice conditions. The small-scale sea ice data are acquired mainly through the shipboard or airborne camera [20,21,22], which is the observation method closest to human eyes and can accurately obtain the close-range sea ice image in the near-field area of the ship. However, due to its short viewing distance and inability to obtain extensive and complete sea ice information, it is more suitable for verifying ice conditions. The medium-scale sea ice data are acquired mainly through the airborne electromagnetic induction device (i.e., a so-called EM-bird) [23], marine radar, ice radar, etc. Compared with the other two categories of data, it can better grasp the near-field sea ice conditions in real time.

In fact, radar is sensing equipment suitable for ice detection in the near-field waters [24]. It is clearly stated in the International Code for Ships Operating in Polar Waters (Polar Code) that ships sailing in polar waters should be actively equipped with radars with ice detection capabilities [25]. Compared with the research that only uses a single-sensor marine radar to obtain ice information [26], this study uses the fused image of the marine radar and the ice radar, which can provide more comprehensive sea ice information for the near-field sea ice perception.

In addition, what variables should be considered when conducting near-field sea ice risk warning studies? As can be seen from Table 1, although most studies have taken into account the impact of sea ice physical characteristics such as sea ice concentration and sea ice area on sea ice risk when conducting sea ice risk assessment [8,9,10,11,12,13,14,15,16,18,19,20,21,22,23,26,27], they have not considered the relationship between the own-ship and sea ice from the point of view of navigation safety and the perspective of navigators. As for the differences in different sea ice concentration situations, the sea ice risk assessment and warning methods should also be considered and dealt with separately.

Based on this, this study uses the fused radar image, takes into account the risk variables such as sea ice concentration, sea ice area, sea ice grayscale, and the newly added risk variables such as the distance between sea ice and the own-ship and the relative bearing of sea ice and the own-ship, and proposes a set of near-field sea ice risk assessment and warning processes that meet the needs of autonomous ships in the polar region and are suitable for two situations. Ships can choose to adopt nonpartition or partition sea ice risk assessment methods according to the actual sea ice concentration in the near-field waters to ensure effective warning of sea ice risks and guarantee navigation safety.

Table 1. Comparison of studies related to sea ice risk warning.

Related Studies	Sensing Range	Sensing Equipment	Multi-Source Data	SIC 1	SIA 2	SIG 3	Dist 4	Relative Bearing	Visual Warning
Haverkamp et al. (1995) [19]	Large	Satellite (SAR)	X	X	O	O	X	X	X
Spreen et al. (2008) [11]	Large	Satellite (AMSR-E sensor)	X	O	O	X	X	X	O
Gu et al. (2013) [8]	Large	NOAA satellite	X	X	O	O	X	X	O
Beitsch et al. (2014) [12]	Large	Satellite (AMSR-2 sensor)	X	O	X	X	X	X	O
Posey et al. (2015) [13]	Large	Satellite (AMSR-2 and SSMIS sensors)	O	O	X	X	X	X	O
Hebert et al. (2015) [14]	Large	Satellite (AMSR-2 sensor)	O	O	X	X	X	X	O
Wang et al. (2016) [9]	Large	Satellite (SAR- RADARSAT-2)	X	O	X	X	X	X	O
Zeng et al. (2016) [10]	Large	Terra satellite (MODIS sensors)	X	O	X	O	X	X	O
Chi et al. (2017) [15]	Large	Satellite (passive microwave sensors)	X	O	X	X	X	X	O
Yan et al. (2019) [18]	Large	COMS satellite (GOCI)	X	X	O	O	X	X	O
Dumitru et al. (2019) [27]	Large	Sentinel-1 satellite (SAR)	X	X	O	O	X	X	O
Xiao et al. (2021) [16]	Large	CryoSat-2 satellite	X	X	X	O	X	X	O
Weissling et al. (2009) [20]	Small	Camera observation	X	X	O	O	X	X	O
Zhang and Skjetne. (2015) [21]	Small	Camera observation	X	X	O	O	X	X	O
Parmiggiani et al. (2019) [22]	Small	Camera observation	X	X	O	X	X	X	O
Renner et al. (2013) [23]	Medium	EM-bird sensor and camera observation	O	X	O	O	X	X	O
Hsieh et al. (2021) [26]	Medium	Marine radar	X	X	O	X	O	X	O
This study	Medium	Marine radar and ice radar	O	O	O	O	O	O	O

¹ Sea ice concentration; ² Sea ice area; ³ Sea ice grayscale or thickness; ⁴ Distance between sea ice and the own-ship. Note: O—considered, X—not considered.

Table 1. Comparison of studies related to sea ice risk warning.

Related Studies	Sensing Range	Sensing Equipment	Multi-Source Data	SIC 1	SIA 2	SIG 3	Dist 4	Relative Bearing	Visual Warning
Haverkamp et al. (1995) [19]	Large	Satellite (SAR)	X	X	O	O	X	X	X
Spreen et al. (2008) [11]	Large	Satellite (AMSR-E sensor)	X	O	O	X	X	X	O
Gu et al. (2013) [8]	Large	NOAA satellite	X	X	O	O	X	X	O
Beitsch et al. (2014) [12]	Large	Satellite (AMSR-2 sensor)	X	O	X	X	X	X	O
Posey et al. (2015) [13]	Large	Satellite (AMSR-2 and SSMIS sensors)	O	O	X	X	X	X	O
Hebert et al. (2015) [14]	Large	Satellite (AMSR-2 sensor)	O	O	X	X	X	X	O
Wang et al. (2016) [9]	Large	Satellite (SAR- RADARSAT-2)	X	O	X	X	X	X	O
Zeng et al. (2016) [10]	Large	Terra satellite (MODIS sensors)	X	O	X	O	X	X	O
Chi et al. (2017) [15]	Large	Satellite (passive microwave sensors)	X	O	X	X	X	X	O
Yan et al. (2019) [18]	Large	COMS satellite (GOCI)	X	X	O	O	X	X	O
Dumitru et al. (2019) [27]	Large	Sentinel-1 satellite (SAR)	X	X	O	O	X	X	O
Xiao et al. (2021) [16]	Large	CryoSat-2 satellite	X	X	X	O	X	X	O
Weissling et al. (2009) [20]	Small	Camera observation	X	X	O	O	X	X	O
Zhang and Skjetne. (2015) [21]	Small	Camera observation	X	X	O	O	X	X	O
Parmiggiani et al. (2019) [22]	Small	Camera observation	X	X	O	X	X	X	O
Renner et al. (2013) [23]	Medium	EM-bird sensor and camera observation	O	X	O	O	X	X	O
Hsieh et al. (2021) [26]	Medium	Marine radar	X	X	O	X	O	X	O
This study	Medium	Marine radar and ice radar	O	O	O	O	O	O	O

¹ Sea ice concentration; ² Sea ice area; ³ Sea ice grayscale or thickness; ⁴ Distance between sea ice and the own-ship. Note: O—considered, X—not considered.

2. Radar Image Fusion Process

Radar is an effective equipment for autonomous ships in the polar region to detect the near-field sea ice targets. Among them, marine radar is a necessary navigation equipment for ships. Its detection range is farther, but the detection effect of sea ice is poor, and only part of the edge of sea ice can be delineated. Ice radar can reflect the overall ice texture better, but the detection range is closer and the noise interference is greater. Compared with a single sensor, the fusion of image information of these two radars will help to perceive the near-field sea ice scene more comprehensively. However, even if two radar images at the same time point are selected for radar image fusion, the problems of redundant information and spatial registration must be solved. The problem of redundant information is how to eliminate the interference of various redundant information and correctly extract the target echo information on the radar echo display area. The redundant information includes: (1) various menus, options, words, and numbers on the radar image; (2) the marking lines and frames on the radar echo display area; (3) the echo at the center of the radar distance scale ring may be the hull of the own-ship rather than sea ice. The spatial registration problems include: (1) as the radar display orientation may be head-up or north-up, the heading angle of the two radar images may be inconsistent; (2) as the selected range or interval of the radar distance scale ring may be different, the pixel distance corresponding to the same 1 nautical mile may be inconsistent; (3) as the radar display mode may be non-off-center or off-center, the pixel position of the center of the radar distance scale ring may be inconsistent.

In order to solve the above problems, this study proposed a radar image fusion process for near-field sea ice perception of polar autonomous ships. The process includes five steps, including radar image capture, basic information acquisition, redundant information elimination, spatial registration correction, and radar image fusion, as shown in Figure 1. These steps are explained as follows.

Figure 1. Schematic diagram of radar image fusion process.

2.1. Radar Image Capture and Basic Information Acquisition

The images of marine radar and ice radar at the same time point (note that there may be a time difference between the two radars or two sensors) were captured, respectively, as shown in Figure 2a and Figure 3a; both radar images were saved in three-dimensional matrix form. Through image measurement software, the basic information such as image size, the range of region of interest (ROI), display orientation, range and interval of the radar distance scale ring, the position of the own-ship, and the short-term target position were obtained.

Figure 2. The marine radar image: (a) the original radar image; (b) the obtained basic information.

Figure 3. The ice radar image: (a) the original radar image; (b) the obtained basic information.

The basic information obtained from the marine radar image is shown in Figure 2b: the image size is 1366 × 728 pixels (px); the ROI is a rectangular region composed of four coordinate points (310, 48), (310, 675), (1054, 48), and (1054, 675); the radar display orientation is head-up and the heading angle is 223.5°; the radar display mode is off-center (that is, the center of the ring is located in the center of the screen); the range of the radar distance scale ring is set to 6 nautical miles (nm) and the ring diameter with a radius of 2 nm corresponds to 105 px; the position of the own-ship is at the center of the distance scale ring, located at (690, 555); the short-term target position (i.e., the Goal in Figure 2a) is the intersection of the planning path (i.e., the red dashed line in Figure 2a) and the bearing scale, located at (605, 57). The basic information obtained from the ice radar image is shown in Figure 3b: the image size is 1366 × 728 px; the ROI is a circular area with (430, 350) as the center and 300 px as the radius; the radar display orientation is head-up and the heading angle is 223.5°; the radar display mode is non-off-center (that is, the center of the ring is located below the center of the screen); the range of the radar distance scale ring is set to 6 nm and the ring diameter with a radius of 2 nm corresponds to 100 px; the position of the own-ship is at the center of the distance scale ring, located at (430, 350). The acquisition of the above-mentioned basic information will be helpful for the processing and fusion of the radar image.

2.2. Redundant Information Elimination

In the original images of marine radar and ice radar, in addition to the target echo information that is most concerned by sea ice perception, there are also various redundant information such as words, numbers, and marking lines, which should be filtered and eliminated.

2.2.1. ROI Image Acquisition

According to the basic information such as image size and the range of the ROI obtained from the two original radar images, a mask matrix is created for each of them to display the inside of the ROI in white (with the value set to 1) and the outside of the ROI in black (with the value set to 0), as shown in Figure 4a,b. Then, the image of the ROI can be obtained through Equation (1), as shown in Figure 4c,d.

$$dst\left(I\right)=src1\left(I\right)\wedge src2\left(I\right) if mask\left(I\right)\ne 0$$

(1)

where $dst$ represents the output image matrix;$I$ represents the multi-dimensional index of elements in the image matrix; $src1$ and $src2$ represent two image matrices with the same size; $mask$ represents the mask matrix.

Figure 4. The radar ROI image acquisition: (a) the marine radar mask image; (b) the obtained marine radar ROI image; (c) the marine radar mask image; (d) the obtained ice radar ROI image.

2.2.2. Marking Lines and Frames Elimination

In addition to radar echo, there may also be occlusion and the interference of various marking lines and frames in the obtained image of the ROI. For example, in the image of the ROI of the ice radar, there are red heading lines and yellow distance scale rings, as shown in Figure 5a. After creating the mask matrix to mark the areas to be eliminated, as shown in Figure 5b, the result of elimination can indeed improve the influence of the marking lines on radar echo, as shown in Figure 5c.

Figure 5. The radar marking lines and frames elimination: (a) before elimination; (b) the elimination area; (c) after elimination.

2.2.3. Own-Ship Echo Elimination

When a small and independent echo is displayed at the center of the radar distance scale ring, the echo may be the clutter generated by the steel hull of the own-ship rather than sea ice or other objects. The contour range information of all echoes in the image of the ROI are detected and compared with the position of the own-ship. After confirming the overlapping contour range information, the echo of the own-ship can be eliminated, as shown in Figure 6a,b.

Figure 6. The radar own-ship echo elimination: (a) before elimination; (b) after elimination.

2.2.4. Radar Echo Extraction

The marine radar echo is displayed in orange-yellow and the ice radar echo is displayed in grayish-white. In order to better extract the radar echoes with specific colors, the color space of image of the ROI is converted from RGB to HSV. Then, after setting the corresponding color threshold, the binary radar image can be obtained through Equation (2), as shown in Figure 7a,b.

$$dst\left(I\right)=lowerb\left(I\right)\le src\left(I\right)\le upperb\left(I\right)$$

(2)

where $dst$ represents the output image matrix;$I$ represents the multi-dimensional index of elements in the image matrix; $lowerb$ represents the lower boundary threshold;$src$ represents the input image matrix; $upperb$ represents the upper boundary threshold.

Figure 7. The radar echo extraction: (a) the obtained binary marine radar image; (b) the obtained binary ice radar image.

2.3. Spatial Registration Correction

In order to carry out spatial registration correction, this study adopts the method of fixing the marine radar image and adjusting the ice radar image to make a geometric transformation of the ice radar image.

2.3.1. Image Rotation

When the heading angles of the marine radar and ice radar images are inconsistent, after confirming the rotation angle, the image can be rotated through affine transformation of Equation (3) or coordinate transformation of Equations (4) and (5).

For example, if the heading angle of the marine radar image is 0° and the heading angle of the ice radar image is 60°, after rotating the ice radar image counterclockwise by 60°, the same heading angle as the marine radar image can be obtained.

$$\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\\ 1\end{array}\right]=A\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$

(3)

$$u^{\prime }=u\cos \theta -v\sin \theta$$

(4)

$$v^{\prime }=u\sin \theta +v\cos \theta$$

(5)

where $\left(u^{\prime },v^{\prime }\right)$ represent the pixel coordinates in the transformed image; $A$ represents the affine matrix; $\left(u,v\right)$ represent the pixel coordinates in the original image; $\theta$ represents the rotation angle.

2.3.2. Image Zooming

When the pixel distances corresponding to the same 1 nm of the marine radar and ice radar images are inconsistent, after confirming the zooming ratio, the image can be zoomed through affine transformation of Equation (6) or coordinate transformation of Equations (7) and (8). For example, if 2 nm in the distance scale ring of the marine radar corresponds to 105 px, and 2 nm in the distance scale ring of the ice radar corresponds to 100 px, after magnifying the ice radar image by 1.05 times, the same corresponding pixel distance per nm as the marine radar image can be obtained.

$$\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\\ 1\end{array}\right]=A\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}{c}_u& 0& 0\\ 0& c_v& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$

(6)

$$u^{\prime }=c_{u}u$$

(7)

$$v^{\prime }=c_{v}v$$

(8)

where $\left(u^{\prime },v^{\prime }\right)$ represent the pixel coordinates in the transformed image; $A$ represents the affine matrix; $\left(u,v\right)$ represent the pixel coordinates in the original image; $c_u$ and $c_v$represent the zooming ratio.

2.3.3. Image Translation

When the pixel positions of the own-ship of the marine radar and ice radar images are inconsistent, after confirming the vertical and horizontal movement values, the image can be translated through affine transformation of Equation (9) or coordinate transformation of Equations (10) and (11). For example, if the pixel position of the own-ship of the marine radar is located at (690, 555), the pixel position of the own-ship of the ice radar is located at (430, 350), and the corresponding pixel distance per nm of the two radar images is the same, after translating the ice radar image to the right by 260 px and downward by 205 px, the same pixel position of the own-ship as the marine radar image can be obtained.

$$\left[\begin{array}{c}{u}^{\prime }\\ v^{\prime }\\ 1\end{array}\right]=A\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}1& 0& t_u\\ 0& 1& t_v\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]$$

(9)

$$u^{\prime }=u+t_u$$

(10)

$$v^{\prime }=v+t_v$$

(11)

where $\left(u^{\prime },v^{\prime }\right)$ represent the pixel coordinates in the transformed image; $A$ represents the affine matrix; $\left(u,v\right)$ represent the pixel coordinates in the original image; $t_u$ represents the vertical movement value; $t_v$ represents the horizontal movement value.

2.4. Radar Image Fusion

After eliminating redundant information and correcting spatial registration, the image fusion can be carried out through Equation (12) after confirming the respective weights of marine radar and ice radar images. The orange-yellow marine radar echo is less sensitive to sea ice perception, but it helps to outline the edge contours of the sea ice, while the grayish-white ice radar echo has a higher sensitivity to sea ice perception. The whiter and brighter the echo color is, the thicker the sea ice thickness is, which helps to master the structure and texture of the sea ice layer. The complementary and combination of the two kinds of radar image information will contribute to the subsequent development of sea ice perception application function.

$$dst\left(I\right)=src1\left(I\right)\times alpha+src2\left(I\right)\times beta+gamma$$

(12)

where $dst$ represents the output image matrix; $I$ represents the multi-dimensional index of elements in image matrix; $src1$ and $src2$ represent two image matrices with the same size; $alpha$ represents the weight of image 1; $beta$ represents the weight of image 2; $gamma$ represents the add scalar value.

3. Near-Field Sea Ice Risk Assessment and Warning Process

As the ship is sailing, in addition to considering the physical characteristics of near-field sea ice (such as concentration, area, and grayscale), it is more important to consider the relationship between the own-ship and near-field sea ice (such as distance and relative bearing) from the perspective of the navigator. Therefore, this study proposes a set of near-field sea ice risk assessment and warning processes that meet the needs of autonomous ships in the polar region and are suitable for two situations, as shown in Figure 8. By calculating the sea ice concentration of the fused radar image, the near-field sea ice scene with a sea ice concentration less than or equal to 50% is defined as the low-sea-ice-concentration situation, and the near-field sea ice scene with a sea ice concentration greater than 50% is defined as the high-sea-ice-concentration situation. Then, by using the risk variables such as sea ice area, sea ice grayscale, distance between sea ice and the own-ship, and relative bearing of sea ice and the own-ship, the sea ice risk index with low-sea-ice-concentration and the sea ice risk index with high-sea-ice-concentration are constructed. Finally, visualization processing is carried out according to the size of the risk index values of each piece of sea ice to achieve a better near-field sea ice risk assessment and warning effect.

Figure 8. Schematic diagram of near-field sea ice risk assessment and warning process.

3.1. Near-Field Sea Ice Risk Variables Calculation

3.1.1. Sea Ice Concentration

When calculating the sea ice concentration, the near-field area of the own-ship is defined as a circular area with the position of the own-ship at the center and a radius of 200 px. Then, the sea ice concentration in the near-field area of the own-ship can be obtained through Equations (13) and (14).

$$count={\displaystyle \sum }_{I:src\left(I\right)\ne 0}1$$

(13)

$$SIC=\frac{count}{\pi r^2}\times 100\%$$

(14)

where $count$ represents the total number of sea ice pixels (i.e., the elements whose value is not 0 in the image matrix) in the near-field area of the own-ship; $I$ represents the multi-dimensional index of elements in the image matrix; $src$ represents the image matrix of the near-field area of the own-ship; $SIC$ represents the value of sea ice concentration in the near-field area of the own-ship.

3.1.2. Sea Ice Area

When calculating the sea ice area, the total number of pixels contained in each piece of sea ice is the area value of the piece of sea ice, which can be obtained through Equation (15). For example, as shown in Figure 9, the sea ice image matrix numbered 1 contains 9 px in total, so the area value of this piece of sea ice is 9.

$$A_i={\displaystyle \sum }_{I:src\left(I\right)=i}1$$

(15)

where $A_i$ represents the area value of the $i$-th piece of sea ice; $I$ represents the multi-dimensional index of elements in the image matrix; $src$ represents the sea ice image marker matrix.

Figure 9. Schematic diagram of sea ice marking in a low-sea-ice-concentration situation.

3.1.3. Sea Ice Grayscale

When calculating the sea ice grayscale, the average grayscale value of each piece of sea ice is the grayscale value of the piece of sea ice, which can be obtained through Equations (16) and (17). For example, as shown in Figure 9, the sum of the grayscale values of all pixels in the sea ice image matrix numbered 1 is 990, and the area value is 9, so the average grayscale value of this piece of sea ice is 110. Related studies have shown that there is an exponential relationship between sea ice albedo and sea ice thickness, and some scholars have proposed a retrieval model based on the exponential relationship between sea ice albedo and sea ice thickness [28,29]. Therefore, the higher the grayscale value of sea ice measured from the radar image, the thicker the sea ice may be.

$$G_i\_sum={\displaystyle \sum }_{I:src\left(I\right)=i}G\_blend\left(I\right)$$

(16)

$$G_i=\frac{G_i\_sum}{A_i}$$

(17)

where $G_i\_sum$ represents the sum of grayscale values of the $i$-th piece of sea ice; $I$ represents the multi-dimensional index of elements in the image matrix; $src$ represents the sea ice image marker matrix; $G\_blend$ represents the grayscale image fused according to a certain weight; $G_i$ represents the average grayscale value of the $i$-th piece of sea ice; $A_i$ represents the area value of the $i$-th piece of sea ice.

3.1.4. Distance between Sea Ice and the Own-Ship

When calculating the distance between sea ice and the own-ship, the distance refers to the Euclidean metric from the centroid of the sea ice to the position of the own-ship. Each piece of sea ice is regarded as an area with uniform mass distribution. The centroid of each piece of sea ice can be obtained through Equations (18) and (19), and then the distance between each piece of sea ice and the own-ship can be obtained through Equation (20). For example, as shown in Figure 9, the distance value between sea ice numbered 1 and the own-ship is 15 px.

$$\overline{u_i}=\frac{{{\displaystyle \iint }}^{ }{u}_{i}d\theta }{{{\displaystyle \iint }}^{ }1d\theta }$$

(18)

$$\overline{v_i}=\frac{{{\displaystyle \iint }}^{ }{v}_{i}d\theta }{{{\displaystyle \iint }}^{ }1d\theta }$$

(19)

$$D_i=\sqrt{{\left(u_s-\overline{u_i}\right)}^2+{\left(v_s-\overline{v_i}\right)}^2}$$

(20)

where $\left(\overline{u_i},\overline{v_i}\right)$ represents the centroid coordinates of the $i$-th piece of sea ice; $\left(u\prime ,v\prime \right)$ represents the pixel coordinates of the $i$-th piece of sea ice; $\theta \subset A_i$; $D_i$ represents the distance value between the $i$-th piece of sea ice and the own-ship; $\left(u_s,v_s\right)$ represents the position coordinates of the own-ship.

3.1.5. Relative Bearing of Sea Ice and the Own-Ship

When calculating the relative bearing of sea ice and the own-ship, the relative bearing refers to the included angle between the course line and the bearing line (i.e., the included angle between the line from the position of the own-ship to the short-term target position and the line from the position of the own-ship to the centroid of the sea ice, which is measured to the right or left based on the course line, and the range is 0°~180°), which can be obtained through Equations (21)–(24). For example, as shown in Figure 9, the relative bearing value of sea ice numbered 1 and the own-ship is 64.4°.

$$a_i=\sqrt{{\left(u_s-\overline{u_i}\right)}^2+{\left(v_s-\overline{v_i}\right)}^2}$$

(21)

$$b_i=\sqrt{{\left(u_g-\overline{u_i}\right)}^2+{\left(v_g-\overline{v_i}\right)}^2}$$

(22)

$$c_i=\sqrt{{\left(u_s-u_g\right)}^2+{\left(v_s-v_g\right)}^2}$$

(23)

$$Q_i={\cos }^{-1}\left(\frac{a_i{}^2+c_i{}^2-b_i{}^2}{2a_i{c}_i}\right)$$

(24)

where $a_i$ represents the distance between the centroid of the $i$-th piece of sea ice and the position of the own-ship; $b_i$ represents the distance between the short-term target position and the centroid of the $i$-th piece of sea ice; $c_i$ represents the distance between the position of the own-ship and the short-term target position; $Q_i$ represents the relative bearing value of the $i$-th piece of sea ice and the own-ship; $\left(u_s,v_s\right)$ represents the position coordinates of the own-ship; $\left(\overline{u_i},\overline{v_i}\right)$ represents the centroid coordinates of the $i$-th piece of sea ice; $\left(u_g,v_g\right)$ represents the coordinates of the short-term target position.

3.2. Nonpartition Sea Ice Risk Assessment Method for Low-Sea-Ice-Concentration Situation

In the low-sea-ice-concentration situation, as shown in the bottom left corner of Figure 8, this study uses the fused binary radar image to mark the regional scope of each piece of sea ice, and calculates its area value, grayscale value, distance value from the own-ship, and the relative bearing value to the own-ship throughs Equations (15)–(24), to construct the near-field sea ice risk index with low-sea-ice-concentration.

The marking of the regional scope of each piece of sea ice is shown in Figure 9. The connected regions of each piece of sea ice are marked with positive integers starting from “1” in sequence, and the other background regions are marked with “0”. The regions marked with the same positive integer belong to the same sea ice region. Then, considering variables such as the area value, grayscale value, distance value from the own-ship, and the relative bearing value to the own-ship of each piece of sea ice, this study constructs the near-field sea ice risk index with low-sea-ice-concentration, as shown in Equation (25). Accordingly, the sea ice risk assessment and warning can be carried out based on the obtained risk index values of each piece of sea ice. The assessment method is to treat each piece of sea ice itself as a whole, without any further partition treatment.

$$LCR{I}_i=\frac{\left[\frac{A_i}{A_{max}}+\frac{G_i}{G_{max}}+\frac{\left(D_{max}-D_i\right)}{D_{max}}+\frac{\left(Q_{max}-Q_i\right)}{Q_{max}}\right]}{4}\times 100\%$$

(25)

where $A_i$ represents the area value of the $i$-th piece of sea ice; $A_{max}$ represents the maximum area value in all sea ice regions; $G_i$ represents the average grayscale value of the $i$-th piece of sea ice; $G_{max}$ represents the maximum average grayscale value in all sea ice regions; $D_i$ represents the distance value between the $i$-th piece of sea ice and the own-ship; $D_{max}$ represents the maximum distance value from the own-ship in all sea ice regions; $Q_i$ represents the relative bearing value of the $i$-th piece of sea ice and the own-ship; $Q_{max}$ represents the maximum relative bearing value to the own-ship in all sea ice regions; $LCR{I}_i$ represents the risk index value of the $i$-th piece of sea ice.

3.3. Partition Sea Ice Risk Assessment Method for High-Sea-Ice-Concentration Situation

In the high-sea-ice-concentration situation, as shown in the bottom right corner of Figure 8, if the risk index is still constructed for the whole sea ice, it is easy to cause the near-field area of the own-ship to be full of high-risk sea ices, and the effect of sea ice risk assessment and warning is poor. Thus, this study uses the fused grayscale radar image to partition all sea ice. The regions with a sea ice grayscale greater than 200 are defined as high grayscale regions, and the other regions with a sea ice grayscale less than or equal to 200 are defined as non-high grayscale regions. Then, for the high grayscale regions in each piece of sea ice, their area value, distance value from the own-ship, and the relative bearing value to the own-ship are calculated through Equations (15) and (18)–(24), to construct the near-field sea ice risk index with high-sea-ice-concentration.

The partition of each piece of sea ice is shown in Figure 10a. For example, there is a large intact sea ice in the front left of the own-ship. According to the size of the grayscale values, it can be partitioned into the high grayscale regions and the non-high grayscale regions. For the high grayscale regions, as shown in Figure 10b, considering variables such as the area value, distance value from the own-ship, and the relative bearing value to the own-ship, this study constructs the near-field sea ice risk index with high-sea-ice-concentration, as shown in Equation (26); although the other non-high grayscale regions do not involve the construction of the near-field sea ice risk index, they can still be partitioned into four sub-regions (e.g., (1,50], (50,100], (100,150], (150,200]) according to the size of the grayscale values, which is helpful for the visualization of subsequent risk warning, as shown in Figure 10c. Accordingly, the sea ice risk assessment and warning can be carried out based on the obtained risk index values of the high grayscale regions in each piece of sea ice. The assessment method is to partition each piece of sea ice to find out the priority regions that have the greatest impact on navigation safety in the whole sea ice.

$$HCR{I}_j=\frac{\left[\frac{A_j}{3A_{\sigma }}+\frac{\left(D_{max}-D_j\right)}{D_{max}}+\frac{\left(90-Q_j\right)}{90}\right]}{3}\times 100\%$$

(26)

where $A_j$ represents the area value of the $j$-th high grayscale region; $A_{\sigma }$ represents the standard deviation of the area values of all high grayscale regions; $D_j$ represents the distance value between the $j$-th high grayscale region and the own-ship; $D_{max}$ represents the maximum distance value from the own-ship in all high grayscale regions; $Q_j$ represents the relative bearing value of the $j$-th high grayscale region and the own-ship; $HCR{I}_j$ represents the risk index value of the $j$-th high grayscale region.

Figure 10. Schematic diagram of sea ice partition in a high-sea-ice-concentration situation: (a) grayscale image; (b) high grayscale region; (c) non-high grayscale regions.

4. Example Demonstration for Near-Field Sea Ice Perception and Warning

This study collected the replay videos of marine radar and ice radar of the polar research icebreaker Xuelong 2 from Shenzhen to Zhongshan Station for China’s 36th Antarctic expedition in 2019. The radar images used in subsequent examples of low-sea-ice-concentration and high-sea-ice-concentration were all captured from these radar replay videos of this voyage.

4.1. Example 1

Example 1 used image software to capture the marine radar and ice radar images at 20:39:26 on 18 November 2019, as shown in Figure 11a,b. At this time, Xuelong 2 approached 63° south latitude, and the sea ice distribution was relatively scattered and sparse. Through the radar image fusion process in Section 2, after redundant information elimination, spatial registration correction, and image fusion, the fused radar image obtained is shown in Figure 11c.

Figure 11. The radar image of example 1: (a) the original marine radar image; (b) the original ice radar image; (c) the fused radar image.

In the fused radar image of example 1, through Equations (13) and (14), it can be calculated that the total number of sea ice pixels in the near-field area with the position of the own-ship at the center and a radius of 200 px is 7883, and the obtained sea ice concentration is 6.2% (less than 50%), which belongs to the low-sea-ice-concentration situation. Therefore, by adopting the nonpartition sea ice risk assessment method, the regional scope of each piece of sea ice in the fused binary radar image is marked, and a total of 824 pieces of sea ice can be obtained. The area value, grayscale value, distance value from the own-ship, and the relative bearing value to the own-ship of each piece of sea ice can be calculated through Equations (15)–(24) in Section 3.1. Then, the risk index value of each piece of sea ice can be obtained through Equation (25) in Section 3.2.

By drawing scatter plots, the correlation between the obtained sea ice risk index and each variable is analyzed separately, as shown in Figure 12a,d; the obtained sea ice risk index value ranges from 16.6% to 76.3%. It can be seen from Figure 12a that the minimum value of sea ice area is 1 and the maximum is 253. The larger the sea ice area, the higher the risk, and the two are positively correlated. It can be seen from Figure 12b that the minimum value of sea ice grayscale is 0 and the maximum is 178.3. The greater the sea ice grayscale, the higher the risk, and the two are positively correlated. It can be seen from Figure 12c that the minimum distance value between sea ice and the own-ship is 11.7 and the maximum is 540. The closer the sea ice, the higher the risk, and the two are negatively correlated. It can be seen from Figure 12d that the minimum relative bearing value of sea ice and the own-ship is 0.2°, and the maximum is 179.8°. The smaller the relative bearing (that is, the closer the position is to the front of the own-ship), the higher the risk, and the two are negatively correlated. Based on the above analysis results, the near-field sea ice risk index with low-sea-ice-concentration proposed in this study can indeed reasonably reflect the influence degree of each piece of sea ice on the navigation safety of the own-ship.

Figure 12. The scatter plot of sea ice risk index vs. variables: (a) the risk index vs. sea ice area; (b) the risk index vs. sea ice grayscale; (c) the risk index vs. sea ice distance; (d) the risk index vs. sea ice relative bearing.

According to the size of the obtained risk index values, the regions of each piece of sea ice can be marked with different colors for sea ice risk assessment and visualization warning, as shown in Figure 13. When the sea ice risk index value is greater than 66% ($HCR{I}_j$ > 66%), the sea ice region is marked in red and belongs to high-risk sea ice that the own-ship should focus on. When the sea ice risk index value is less than or equal to 66% and greater than 33% (33% < $HCR{I}_j$ ≤ 66%), the sea ice region is marked in yellow and belongs to medium-risk sea ice. When the sea ice risk index value is less than or equal to 33% ($HCR{I}_j$ ≤ 33%), the sea ice region is marked in green and belongs to low-risk sea ice. In addition, the $A_{max}$ can be marked next to the sea ice with the largest area and the $G_{max}$ can be marked next to the sea ice with the greatest grayscale to more intuitively display the sea ice information that threatens the navigation safety of the own-ship.

Figure 13. Schematic diagram of sea ice risk assessment and visualization warning of example 1.

4.2. Example 2

Example 2 used image software to capture the marine radar and ice radar images at 13:45:53 on 19 November 2019, as shown in Figure 14a,b. At this time, Xuelong 2 approached 66° south latitude, and the sea ice distribution was relatively concentrated and significant. Through the radar image fusion process in Section 2, after redundant information elimination, spatial registration correction, and image fusion, the fused radar image obtained is shown in Figure 14c.

Figure 14. The radar image of example 2: (a) the original marine radar image; (b) the original ice radar image; (c) the fused radar image.

In the fused radar image of example 2, through Equations (13) and (14), it can be calculated that the total number of sea ice pixels in the near-field area with the position of the own-ship at the center and a radius of 200 px is 96,489, and the obtained sea ice concentration is 76.7% (greater than 50%), which belongs to the high-sea-ice-concentration situation. Therefore, by adopting the partition sea ice risk assessment method, all sea ice regions in the fused grayscale radar image are partitioned into the high grayscale regions with a sea ice grayscale greater than 200 and the non-high grayscale regions with a sea ice grayscale less than or equal to 200. For all high grayscale regions, the area value, distance value from the own-ship, and the relative bearing value to the own-ship can be calculated through Equations (15) and (18)–(24) in Section 3.1. Then, the risk index values of the high grayscale regions in each piece of sea ice can be obtained through Equation (26) in Section 3.2.

According to the size of the obtained risk index values of the high grayscale regions, the same criteria as in Section 4.1 (the risk index value is greater than 66%, less than or equal to 66% and greater than 33%, or less than or equal to 33%) can also be used to mark each high grayscale region in red, yellow, and green, respectively, for sea ice risk assessment and visualization warning, as shown in Figure 15a. Other non-high grayscale regions can also be partitioned into four sub-regions (e.g., (1,50], (50,100], (100,150], (150,200]) according to the size of the grayscale values, and then the high grayscale regions (also regard it as a sub-region) are added to draw the contour map of these five sub-regions and to mark them with gradient colors from blue to white for sea ice risk assessment and visualization warning, as shown in Figure 15b.

Figure 15. Schematic diagram of sea ice risk assessment and visualization warning of example 2: (a) the high grayscale region risk distribution; (b) the overall grayscale distribution.

Figure 15b shows the overall grayscale distribution of sea ice in the near-field of the own-ship. According to the relevant study results that there is an exponential relationship between sea ice albedo and sea ice thickness [28,29], the higher the grayscale value, the thicker the sea ice may be. Therefore, priority attention should be given to regions marked in white (i.e., the high grayscale regions with a sea ice grayscale greater than 200). Figure 15a shows the risk distribution of these high grayscale regions. These regions are partitioned into three levels according to the size of the risk index values, among which the region marked in red is the region that poses the greatest threat to the navigation safety of the own-ship and should be given more priority attention. It can be seen from the results of visualization warning that in the high-sea-ice-concentration situation, the sea ice situation in the near-field area of the own-ship can be fully mastered from the macro- and micro-perspective through the overall grayscale distribution map of sea ice and the risk distribution map of high grayscale regions.

In this example, if the nonpartition sea ice risk assessment method is still adopted, according to the size of the obtained risk index values, the same criteria as in Section 4.1 can also be used to mark the regions of each piece of sea ice in red, yellow, and green, respectively, for sea ice risk assessment and visualization warning, as shown in Figure 16. However, it can be seen from the results of visualization warning that in the near-field area of the own-ship, the own-ship is surrounded by high-risk sea ice marked in red, and it is impossible to distinguish the priority regions that have the greatest impact on navigation safety and should therefore be given more priority attention. Thus, the nonpartition sea ice risk assessment method adopted in the low-sea-ice-concentration situation is indeed not suitable for the high-sea-ice-concentration situation.

Figure 16. Schematic diagram of sea ice risk assessment and visualization warning of example 2 using the nonpartition sea ice risk assessment method.

4.3. Results and Discussion

From the above two examples, the radar image fusion process proposed in this study is demonstrated. In example 1, in addition to the occlusion and interference of the marking line and marking frame, the echo noise of the own-ship is particularly significant in the ice radar image. After eliminating redundant information, these problems have been improved. In example 2, the heading angle, the corresponding pixel distance per nautical mile, and the position of the own-ship are inconsistent between the marine radar and the ice radar images. The marine radar display orientation is head-up, the heading angle is 0°, the distance scale ring with a radius of 2 nm corresponds to 53 px, and the position of the own-ship is located at (648, 548); the ice radar display orientation is north-up, the heading angle is 102.4°, the distance scale ring with a radius of 2 nm corresponds to 75 px, and the position of the own-ship is located at (430, 350). After the correction of image rotation, zooming, and translation, these spatial registration problems have been solved obviously. It can be seen from Figure 11c and Figure 14c that the radar image fusion effect in these two examples is good.

In addition, from the above two examples, it also demonstrates a set of near-field sea ice risk assessment and warning processes proposed in this study that meets the needs of autonomous ships in the polar region and is suitable for two situations. In the situation of low-sea-ice-concentration in example 1, because the sea ice distribution is relatively scattered and sparse, each piece of sea ice itself is regarded as a whole without any further partition treatment and the nonpartition sea ice risk assessment method proposed in this study is used to calculate the risk index. In the situation of high-sea-ice-concentration in example 2, because the sea ice distribution is relatively concentrated and significant, the partition sea ice risk assessment method proposed in this study is used to partition each piece of sea ice according to the size of the grayscale values, and then the risk index is calculated for the high grayscale regions (i.e., the priority regions that have the greatest impact on navigation safety). It can be seen from Figure 13 and Figure 15 that the risk assessment and warning effects of near-field sea ice are good in both the low-sea-ice-concentration situation in example 1 and the high-sea-ice-concentration situation in example 2.

In fact, the key to near-field sea ice risk assessment and warning is to identify the “relatively” dangerous sea ice regions around the ship. Unlike obstacles of rigid structures such as islands, reefs, or sunken ships, sea ice belongs to flexible structures that ships can contact or break; and the stronger the icebreaking ability of the ship, the thicker the sea ice that can be contacted. For example, the hull structural strength of the Xuelong 2 meets the PC3 level, and it can continuously break ice at a speed of 2~3 knots under sea conditions of 1.5 m of ice and 0.2 m of snow. Therefore, the ship can set the parameter values in the near-field sea ice risk index calculation formula proposed in this study according to its own conditions. With the support of two kinds of radar image information, the near-field sea ice risk index value obtained in this study is based on variable data such as sea ice concentration, sea ice area, sea ice grayscale, distance between sea ice and the own-ship, and relative bearing of sea ice and the own-ship. Unlike the partial and subjective judgment of the navigator, it can determine the sea ice risk situation in the near-field area of the ship more comprehensively and objectively, providing effective support for the assisted decision-making of autonomous navigation in the polar region.

5. Conclusions

For the navigation safety of autonomous ships in the polar region, this study chooses marine radar and ice radar as the near-field sea ice sensing equipment and proposes the radar image fusion process. The process can integrate the two kinds of radar image information well, solve the problems of redundant information and spatial registration during image fusion effectively, and perceive the sea ice information in the near-field area of the ship more comprehensively than a single sensor can.

Considering the relationship between the own-ship and near-field sea ice from the perspective of the navigator, this study proposes a set of near-field sea ice risk assessment and warning processes that meet the needs of autonomous ships in the polar region and are suitable for two situations. In the low-sea-ice-concentration situation, each piece of sea ice itself is regarded as a whole, and the nonpartition sea ice risk assessment method is adopted to calculate the risk index value of each piece of sea ice. In the high-sea-ice-concentration situation, the partition sea ice risk assessment method is used to find out the high grayscale regions that have the greatest impact on navigation safety in the whole sea ice, and calculate the risk index value of the high grayscale regions in each piece of sea ice.

According to the example demonstration results, through the integrated radar image fusion process and a set of near-field sea ice risk assessment and warning processes proposed in this study, the sea ice risk distribution in the near-field area of the ship can be well obtained in both low- and high-sea-ice-concentration situations. These processes and visualized warning maps can provide effective support for the assisted decision-making of autonomous navigation in the polar region.

On the basis of this study, follow-up research can carry out related work on how to maneuver ships to avoid sea ice at a lower risk. In fact, ship navigation is a dynamic process. The process proposed in this study can be used to master the sea ice risk distribution at each time point in the current navigation environment, and can also be used to evaluate the effect of avoiding sea ice.