Randomly Testing an Autonomous Collision Avoidance System with Real-World Ship Encounter Scenario from AIS Data

Abstract

Maritime Autonomous Surface Ship (MASS) is promoted as the future of intelligent shipping. While autonomy technologies offer a solution for MASS, they have also resulted in new challenges for performance validation. To address this, a scenario-based validation method to test the autonomous collision avoidance system is proposed in this paper, including mining ship encounter scenarios from massive historical AIS data and randomly generated virtual test scenarios according to the parameter probability distributions from the collected real scenarios, as well as the final experiments: a total of 2900 generated scenarios including single ship and multi-ship encounter situations are created and applied to conduct testing experiments on the further assessment of our collision avoidance algorithm. The results indicate that the proposed method has the ability to quickly create appropriate testing scenarios according to AIS records, which are helpful to catch potential defects in a collision avoidance algorithm of MASS and to further analyze its navigating features. As a result, the research forms a systematic set of validation procedures from data gathering to practical experiments conduction, incorporating both the real statistics and the random generation method.

1. Introduction

Interest in Maritime Autonomous Surface Ship (MASS) is growing rapidly as the high levels of safety and efficiency that can be reached by such ships are thought to have the potential to solve problems in the maritime industry, such as the prevention of marine accidents and the improvement of the environment [1]. The IMO’s Maritime Safety Committee (MSC) has classified the MASS into four categories [2], where the most advanced MASS is able to make decisions to navigate autonomously.

For such an autonomous ship, many manufacturers, experts, and scholars in the shipping field, as well as the global authoritative ship classification societies generally agree that MASS should at least be as safe as current conventional manned ships [3,4,5]. Therefore, it is very important to carry out tests to ensure the safety, security, and reliability of the MASS navigation. MASS is scheduled to be put into practical use as a coastal vessel in some areas overseas from as early as the first half of the 2020s [6]. The demand for verification and evaluation of MASS’s autonomous navigation system is increasingly urgent.

In terms of field tests for autonomous navigation systems, several industrial projects have been carried out such as the “NYK MASS Trials” project in Japan [7], the “Mayflower Ship” project in the UK [8] and the “Electric Autonomous Container Ship YARA Birkeland” project in Norway [6]. However, the costs and the risks of the full-scale testing of a real ship are usually high.

In the field of autonomous vehicles on land, scenario-based testing is one of the prevailing methods and is broadly supported by the automotive community [9,10]. This method proposes to create a scenario database from real-world driving data and use it in the safety assessment of autonomous driving systems. Achievement in the field of autonomous vehicle testing can be a great benefit to smart shipping. Therefore, how to set up a suitable testing scenario library is the key to the evaluation and verification of an autonomous navigation system in different stages.

Collision avoidance is a key operation of ship navigation. In most existing algorithms for autonomous collision avoidance, tests are usually conducted in the last section to validate the effectiveness of their methods. Besides the common three types of encounters (head-on, crossing, and overtaking), more complicated situations are also tested [11,12,13].

H. Imazu [14] put forward 42 encounter situations called the “Imazu problem” to specifically study how to carry out a validation. Zhou and Ahmed [15,16] made use of the first set of 22 encounter situations to test their collision avoidance algorithms. Johansen [17] proposed 15 collision avoidance scenarios for ships, according to the number of target ships, the wind, and the currents at sea. Chen [18] defined 15 different situations of two ships by considering the captain’s advice, the relative course, speed ratio, and true heading angle between the target ship and the own ship. Paul [19] created a test suite by the handcrafted COLREGs (International Regulations for Preventing Collisions at Sea) scenarios and scored the performances of each test experiment. The initial setting parameters of the scenario are important and fundamental in scenario designing because any change in a parameter will lead to a variation in the collision risk [20] and the encounter scenario will be probably different. The parameters in these above testing scenarios are all set artificially or expert based. The weakness is that the quantity is generally not more than one hundred predefined instances, which the collision avoidance algorithm can be particularly tuned to pass, but the algorithm may fail under broader or more unknown conditions, and the programmed machine may contain bugs in not only the stereotypes of encounter situations.

Random scenario testing is intuitive and can make up for the flaw in expertise-designed scenarios. One shortcoming is that the search space is infinitely large. It is impossible to test all the scenarios. To solve this problem, Liu and Henriksen [21,22] attempted to set parameter boundaries to filter out invalid scenarios. While it is difficult to define if a scenario is invalid, in fact, an algorithm might even fail to pass any kind of test scenario for various reasons. Woerner [23] randomly perturbed the geometry parameters of both the own ship and target ship to generate similar testing scenarios. Similarly, Minne [24] changed the geometry of target ships, while the own ship was fixed at zero coordinates; this method may limit the testing scenario library to a very little range, because all the initial positions of the target ships emerge around the predefined positions.

A number of researchers hold that testing scenarios should reflect the real statistical features of ship navigation in different water areas. An Automatic Identification System (AIS) provides rich information including ship type, position, course over ground (COG) speed over ground (SOG) [25], etc. AIS has been applied in a series of studies on data fusion, abnormal detection, and risk detection [26,27,28]. Inspired by this, a few studies are attempting to mine AIS data for MASS designing testing scenarios.

Pedersen [13] proposed that a promising method might be testing collision avoidance in scenarios that best represent the probable traffic encounter a vessel might meet. Historical cases using AIS data could be used as input to generate test scenarios. Vagale [29] introduced the AIS historical data and other real statistics to set up the multiplied maps for validating collision avoidance risks. While the method is prospective and relies on a vast information source, generating virtual scenarios will be conducive in case of a lack of data. Hwang [30] identified and classified various navigation encounter situations by applying a clustering model from AIS data in the Yellow Sea; the results are listed in percentage by their frequency and reflect the possibility of which type of encounter the ships are most likely to come across and how well they are likely to interact. Bakdi [31] has proposed a method to obtain test scenarios from AIS data and real-coast geographic information and other obstacles, considering both risks of collision and grounding, and thus has set up a more comprehensive and mature validation platform with better authenticity; in his most current study [32], COLREGs is also taken into consideration. Snijders [33] investigated the feasibility of using AIS data in the Strait of Dover for the identification and classification of real-life scenarios. The probability density functions describe a series of navigation parameters, including encounter time duration, target ship distance, relative speed, etc., in give-way scenarios. The scenarios from AIS data have the best advantage of fidelity, whilst the method also has some drawbacks.

The studies above have made good progress on validation by using AIS data, while these real records are also affected by various factors. The accessibility of AIS data around the world is not equal to all the peer researchers for various reasons: AIS will not always provide ships’ exact behaviors because not all the ships will turn on the AIS devices with correct information. The historical trajectories display the TSs’ behaviors interacting with the real world at that particular time instead of how they will act when meeting with the ship under the current test. In the real world, urgent situations constitute a limited part of all the records, while special types of encounters that lead to an accident need to be constructed to ensure our avoidance system will work when facing diverse situations. It is usually difficult to find a large number of these corner cases by experiences because it is difficult to define what kind of encounter is more stressing for a specific collision avoidance algorithm. For various programmed algorithms, they may become stuck or demonstrate irrational steering behaviors in any encounter, perhaps even not only in those complicated test instances.

While the discussion is not in any position to reject AIS, we proposed to incorporate the techniques of random maps because there are no concerns about the flaws from AIS data. Random maps are more flexible to adjustments in terms of ship density, obstacle contour, encounter types and other disturbances, which additionally enables sorting the validation processes into varying difficulty levels, which is helpful for researchers to quickly identify underlying defects in basic scenarios and make a comprehensive performance assessment in more stressed tests. The drawback of these randomly generated maps is that the search space is unlimited; they need some rules to accelerate the testing process and thus exploit its full advantages.

Therefore, the proposed method is a combination of the random generating and real-data-driven approach to gain larger coverage, while limiting the search space to a reasonable range and finally put the method into practice. The random process is required to make the coverage of testing scenarios larger than the reality, while the motion parameters of the target ships should be scattered in a limited range; the generated instances should not be either too close to nor far from the reality. Another merit is that the combined method has a better generalization ability to various validation requirements of different water areas, limited AIS data size, easy adjustment of the test cases and quick application in practical experiments.

Scenario generation is not the last step of validation. We have actually put these randomly generated and real-data-modified scenarios into practical testing experiments with results analysis. To our best knowledge, there has not been a systematic set of validation procedures from data gathering to practical experiments conduction that incorporates the real statistics and the random generation method.

2. Overview of the Framework

As shown in Figure 1, the remainder is organized as follows: Section 3 introduces the methods of scenario identification and extraction (step 1–step 3). In Section 4, case studies and numerical outputs are made from AIS data (step 4). In Section 5, probability distributions are applied to generate simulated testing scenarios (step 5). Experiments are conducted to test the proposed methodology in Section 6 (step 6), and the results of experiments can be used to optimize the methods or algorithms adopted in steps 1–5. The conclusions are discussed in Section 7.

3. Encounter Scenarios Identification and Extraction

3.1. Data Preprocessing and Trajectory Extraction

Firstly, we decoded the AIS messages to obtain the Maritime Mobile Service Identity (MMSI), latitude (Lat), longitude (Lon), SOG, COG, and other information of each ship. Secondly, the decoded AIS data were filtered and cleaned. Thirdly, each ship’s trajectory was organized and managed by MMSI. Finally, overlapping trajectories for all ship pairs were extracted by the same time frame.

3.2. Scenario Identifying

In our framework, the scenarios identified and extracted from AIS data can be further designed to test the autonomous collision avoidance decision-making scheme. The quantitative analysis of COLREGs, such as collision risk, the situation and the stage of the encounter, is important for finding valuable encounter scenarios.

3.2.1. Analysis of Ship Collision Risk

Many experts and scholars have carried out research on the quantification of collision risk. Some of them put forward the synthetic and objective indicators-based approach [34,35], where the relative distance (RD), the relative speed (RS), the Distance of Closest Point of Approaching (DCPA), and the Time to Closest Point of Approaching (TCPA), etc., are considered indicators to identify collision risk.

For the sake of simplicity, we accepted the DCPA and TCPA-based approach to determine collision risk in this paper. Considering the sailing practice, if the RD between the own ship (OS) and the target ship (TS) is above 6 nautical miles (nm), the collision risk is ignored [36]. Referring to the existing literature [33,37,38,39], the determination model of collision risk states (CRS) at time $t$ can be expressed as follows:

$$\begin{array}{c}\mathrm{CRS}\left(t\right)\hfill \\ =\{\begin{array}{c}1, \mathrm{if} (\mathrm{RD}\left(t\right)\le 6.0 \mathrm{nm}, \mathrm{DCPA}\left(t\right)<2.0 \mathrm{nm} \mathrm{and} 0 \min <\mathrm{TCPA}\left(t\right)<30\min )\hfill \\ 0, \mathrm{otherwise}\hfill \end{array}\hfill \end{array}$$

(1)

where $\mathrm{CRS}\left(t\right)=1 \mathrm{or} 0$, respectively, indicate that the collision risk in an encounter situation exists or does not exist.

3.2.2. Classification of Encounter Situations

The rules 13–15 in COLREGs Part B only qualitatively describe the three types of encounter situations in sight of each other, namely overtaking, head-on and crossing. The angles which are adopted to divide the different types of encounter situations are not fixed but are similar in the existing literature [36,37,38]. Therefore, we use the same angles as used in the study [38], as shown in Figure 2.

The motivation of this paper is to test the ship collision avoidance decision-making schemes with real world ship encounters from massive empirical AIS data. Encounter situations with collision risk may be more valuable for testing autonomous collision avoidance algorithms. Hence, the encounter situations (ES) at time $t$ can be expressed according to the relative bearing (RB) and CRS as follows:

$$\begin{array}{c}\mathrm{ES}\left(t\right)\hfill \\ =\{\begin{array}{c}0, \mathrm{if} \left(\mathrm{RB}\left(t\right)\in \left[0°, 5.7°\right]\bigsqcup \left[354.3°, 360°\right) \mathrm{and} \mathrm{CRS}\left(t\right)=1\right)\hfill \\ 1, \mathrm{if} \left(\mathrm{RB}\left(t\right)\in \left[112.5°, 247.5°\right] \mathrm{and} \mathrm{CRS}\left(t\right)=1\right)\hfill \\ 2, \mathrm{if} \left(\mathrm{RB}\left(t\right)\in \left(5.7°, 112.5°\right)\bigsqcup \left(247.5°, 354.3°\right) \mathrm{and} \mathrm{CRS}\left(t\right)=1\right)\hfill \end{array}\hfill \end{array}$$

(2)

where $\mathrm{ES}\left(t\right)=0, 1 \mathrm{or} 2$, respectively, indicate that $\mathrm{ES}\left(t\right)$ is the head-on situation, the overtaking situation or the crossing situation.

3.2.3. Quantification of Ship Encounter Stages

The entire process of the two vessels approach can be divided into four stages directed by COLREGs and good seamanship [40]. The ship collision avoidance decision-making schemes are different in different encounter stages. Hence, it is necessary to quantify the ship encounter stages in the scenario for testing the autonomous collision avoidance system. The four stages include no risk of collision, risk of collision, close-quarter situation and immediate danger. During the no risk of collision stage, the distance between two ships is very far (above 6 nm in practice sails) and the risk of collision stage is not formed. Therefore, the testing scenarios in this paper focus on the remaining three stages.

The distance between two ships at which the different stages begin to apply will vary considerably. The 6 nm, which is the range of Radar detection, is generally accepted for the beginning of the risk of collision stage [40,41]. That is to say, the two ships are in the no risk of collision stage when the distance between two ships is above 6 nm. The close-quarter situation can be considered to be between 1 nm and 3 nm [40,42]. In other words, 1 nm can be suggested to determine whether an immediate danger is developed. Therefore, 6 nm, 3 nm and 1 nm are taken as the division boundaries of the four encounter stages in this paper.

At time t, the ship encounter stages $\mathrm{SES}\left(t\right)$ between OS and TS are determined according to the relative distance (RD) and CRS as follows:

$$\mathrm{SES}\left(t\right)=\{ \begin{array}{c}0, \mathrm{if} (\mathrm{RD}\left(t\right)>6 \mathrm{nm}) \hfill \\ 1, \mathrm{if} (6 \mathrm{nm}\ge \mathrm{RD}\left(t\right)>3 \mathrm{nm} \mathrm{and} \mathrm{CRS}\left(t\right)=1)\hfill \\ 2, \mathrm{if} (3 \mathrm{nm}\ge \mathrm{RD}\left(t\right)>1 \mathrm{nm} \mathrm{and} \mathrm{CRS}\left(t\right)=1) \hfill \\ 3, \mathrm{if} \left(\mathrm{RD}\left(t\right)\le 1 \mathrm{nm} \mathrm{and} \mathrm{CRS}\left(t\right)=1\right) \hfill \end{array}$$

(3)

where $\mathrm{SES}\left(t\right)=0, 1, 2 \mathrm{or} 3$, respectively, indicate that $\mathrm{SES}\left(t\right)$ is the no risk of collision stage, the risk of collision stage, the close-quarter situation or the immediate danger stage.

Although we use discrete encounter stages here in this paper, it should be noted that the division of the encounter stages is subjective and depends on many other factors in practice.

3.2.4. Scenario Parameters

Table 1. Test scenario parameters.

Category	Name	Description	Unit
General	The duration time of the scenario	The period between the beginning and end of the scenario	s
	Encounter situations	Head-on, overtaking, crossing
	Encounter Stages	no risk of collision, risk of collision, close-quarters situation and immediate danger stage
Own ship	Ship type	Container ship, fishing ship, tanker, etc
	Length	Length of ship	m
	SOG	Speed over ground	kn
	COG	Course over ground
Target ship	Ship type	Container ship, fishing ship, tanker, etc.
	Length	Length of ship	m
	Relative distance	Distance relative to OS	nm
	Relative course	Course relative to OS
	Relative speed	Speed relative to OS	kn
	Relative bearing	Bearing relative to OS
	Safety passing distance	The minimum relative distance to OS during the scenario	nm
	DCPA	The Distance to Closest Point of Approaching	nm
	TCPA	The Time to Closest Point of Approaching	min

describes the different parameters for each navigation testing scenario. In order to reduce the impact of sensor error, we assume that the duration of the encounter situation is at least 120 s, so as to become a valid test scenario.

4. Case Study and Numerical Statistics from AIS Data

This section is mainly divided into two parts, of which the first part consists of the two encounter case studies on the output of extracted AIS data, then a series of parameter density distributions (including relative distance, relative speed, duration time, and passing distance) are shown.

4.1. Case Study from AIS Data

The Laotieshan water area selected as the experimental area is an important channel for ships to enter and leave the ports of Tianjin, Qinhuangdao, Yingkou and others in Bohai Bay of China. The AIS traffic flow based on an electronic chart system on 31 May 2019 is shown in Figure 3.

The cleaned-up research data collected from 1 May 2019 to 31 May 2019 in Laotieshan waters includes a total of 55,327,865 ship position reports from 9440 ships, which contain container ships, passenger ships, fishing ships, and other ships, shown as in

Table 2. Ship types included in the experimental data.

Ship Type	Number	Percentage
Cargo ship	2936	31.1%
Tanker	585	6.2%
Container ship	699	7.4%
Passenger ship	66	0.7%
Fishing ship	330	3.5%
Other types	4824	51.1%
total	9440	100%

.

Table 3. Number of testing scenarios.

Scenario Type	Number	Percentage
Overtaking	7449	13.1%
Head-on	135	0.2%
Crossing	49,368	86.7%
Total	56,952	100%

shows that a total of 56,952 valid scenarios were found.

To analyze the characteristics of the encounter situations, we selected the typical single-ship encounter situations and multi-ship encounter situations identified by the proposed method, as shown in flowing 3 figures.

Figure 4 presents a correctly classified situation, where OS (tanker, MMSI: “413474690”) and TS (container, MMSI: “413146000”) meet in a head-on encounter situation. As shown in Figure 4a, OS and TS are on the opposite course, with OS moving up and TS moving down. Figure 4d–f indicate that RD is 6.0 nm, DCPA is 0.1 nm and TCPA = 15 min at the start of this head-on situation (t = 0 s). Therefore, the OS and TS are in the risk of collision stage. TS engages in a course-change action at t = 60 s when the RD is about 5.5 nm, shown in Figure 4c,d. This maneuver increases the DCPA and decreases the TCPA shown as in Figure 4e,f. At last, OS and TS pass within 0.8 nm of each other when the situation is resolved. Figure 4g presents the scenario parameters aspects. The duration time of this head-on scenario is 950 s.

In this case, it may be argued that OS and TS should both engage in a maneuver. However, Figure 4b,c show that only TS takes the course change maneuvers (t = 60 s), while the OS still keeps the course and speed. This is likely due to the OS considering the TS as a give-way ship in a small-angle crossing situation.

Figure 5 presents a multi-ship encounter situation, where OS (container, MMSI: “412713000”) encounters TS1 (tanker, MMSI: “412377520”) in a head-on situation, OS (give-way ship) and TS2 (cargo vessel, MMSI: “413700110”), TS3 (tanker, MMSI: “413441230”), TS4 (tanker, MMSI: “413697340”), respectively, meet under crossing encounter rules and OS (stand-on ship) and TS5 (cargo vessel, MMSI: “412766340”, give-way ship) meet in a overtaking encounter situation. Figure 5d indicates that RD (t = 0 s) between OS and TS1, TS2, TS3, TS4 are, respectively, 3.5 nm, 4.4 nm, 1.9 nm and 2.2 nm. Therefore, the OS, TS3, and TS4 are in the close-quarter situation. As shown in Figure 5a, the OS does not substantially alter its course to starboard. On the contrary, the OS keeps her main course (t = 0~210 s) and passes on the starboard side of TS3 and TS4. The OS engages in a course-change action at t = 210s, shown in Figure 5c. This maneuver increases the DCPA of TS2, TS3 and TS4, and decreases the risk of collision, as shown in Figure 5f. Meanwhile, Figure 5d–f) indicate that the OS keeps clear of TS3 (t = 410 s), TS4 (t = 460 s) and TS1 (t = 660 s) within 0.7 nm of each other when TCPA is less than zero. Figure 5g presents the scenario parameters aspects. The duration time of this multi-ship encounter scenario is 690 s.

In this case, it may be argued that OS as a give-way ship is obliged to make a right turn maneuver to keep enough passing distance to TS2, TS3 and TS4 at the start of this multi-ship encounter situation. However, Figure 5b,c show that OS takes no action and keeps her main course and speed at t = 0~210 s. That is to say, the OS violates COLREGs and gives up the purpose of passing on the port side of TS3 and TS4. This maneuver obviously decreases the RD and TCPA and increases the risk of collision. The likely reason is that it may lead to a new urgent collision risk with TS2 if the OS alters its course to starboard. The OS and TS1-4 may communicate and coordinate with each other before the OS adopts this collision avoidance decision-making scheme.

From the above two cases, we can know that not all vessels in the real world strictly abide by COLREGs, often leading to dangerous situations. However, the behavior of rule-violating vessels is difficult to design and model manually. Hence, we hope to use the real statistical results from AIS data to realize the generalization of the test scenario.

4.2. Statistical Results from AIS Data

In this section, we show the statistical results of all detected single-ship encounter scenarios (n = 56,952). The Probability Density Functions (PDF) of parameters in different types of encounter scenarios (overtaking, head-on, crossing) are obtained. The Kernel Density Estimation (KDE) method is adopted to calculate the PDF in this paper. The statistical results of different types of scenarios in terms of the RD (relative distance), the RS (relative speed), the duration time, and the safety passing distance, are shown, respectively, in following 4 figures.

A similar method has been used in [43], where the RD and RS are calculated for DCPA and TCPA and distributions of ship maneuvers when facing near collision situations are plotted.

Figure 6 shows the distribution of RD at the start of different encounter scenarios. The head-on scenarios on average have a substantially higher distance (4.7 nm) than crossing scenarios (3.3 nm), which again, on average, have a significantly higher distance than overtaking scenarios (1.8 nm); even the median RD in head-on scenarios is 5.7 nm, which is close to the max RD (6 nm). The result indicates that ships with a high RS in head-on situations, so far as possible, usually keep enough distance to take early action considering the limit of ship handling and ship inertia.

Figure 7 indicates the distribution of RS at the start of different encounter scenarios. The head-on scenarios on average have a minor higher speed (21.4 kn) than crossing scenarios (19.3 kn), which again on average have a significantly higher speed than overtaking scenarios (3.9 kn). This is to say, ships approach much slower during overtaking scenarios. As Figure 7 shows, there are only minor differences in the median when comparing head-on situations to crossing situations. However, it is evident that there are significant differences in the median when comparing overtaking situations (3.2 kn) to head-on situations (21.2 kn) and crossing situations (21.3 kn).

Figure 8 presents the empirical distribution of duration time for different encounter scenarios. The average duration time is different in different encounter types. The crossing scenarios on average have a lower duration time (6.8 min) than head-on scenarios (12.6 min), which again, on average, have a lower duration time than overtaking scenarios (20.6 min); even the max duration time of overtaking scenarios reaches 115 min. In other words, the testing of one overtaking scenario will cost much more time.

Figure 9 shows the distribution of safety passing distance in different encounter scenarios. The average safety passing distance is also different in different encounter types. The crossing scenarios, on average, have a substantially higher distance (2.7 nm) than overtaking scenarios (1.2 nm), which again, on average, have a minor higher distance than head-on scenarios (1.0 nm); even the median distance in head-on scenarios is 0.6nm, which is very close to the min passing distance (0.1). In other words, the safety passing distance in most head-on scenarios is under 0.6 nm. The reason is perhaps that the OS and TS in head-on scenarios have negotiated in advance and understood their respective sailing intentions. Therefore, the two ships may accept a short passing distance to change their original routes as little as possible.

The probability distributions of these scenario parameters can be not only used to generate a huge number of test scenarios that conform to the characteristics of real marine traffic, but are also useful to analyze and evaluate the autonomous collision avoidance algorithm, such as using the safety passing distance to assess the safety of collision avoidance algorithms.

5. Scenarios Generation and Evaluation

The outputs of the sections above are the probability distributions for a series of scenario parameters, i.e., the duration the encounters will last, the initial RD between OS and TS when an encounter exists and the RS and the minimal passing distance between OS and TS. Thus, a scenario generation program was designed to create similar encounters complying with the probabilities. For instance, if in most encounters the TS has a relative speed of 15 kn, the scenario generation program should imitate the statistical pattern to create more scenarios where the relative speed between TS and OS is around 15 kn. The process of generating is actually the inverse of the AIS data mining procedures.

In Figure 6 and Figure 7, the RD and the RS distributions depict how frequently the TS will appear at a distance from the OS, and what its speed is likely to be. Inspired by this, we designed a scenario generation program that set the initial target ship at a specific location with a certain velocity to simulate such probabilities. The program works to virtually generate a target ship, during every loop, the target ship’s position and velocity are randomly set according to the possibility in Figure 6 and Figure 7, but within the limitations of the Equation (1).

Note that the passing distance and duration as shown in Figure 8 and Figure 9 are not used in this part to design a scenario; however, they will be applied to compare with the outcomes of tests in Section 6.

For a clear demonstration, we initially created 20 cases in each encounter category, as shown in Figure 10. The OS is set on (0,0). Note that the 20 scenarios are each single-target-ship encounters and will be tested, respectively, instead of multiple-target-ships encounters. In these figures, it can be seen that a number of target ship locations are near to each other. This is because the probability distribution value is high at these particular positions.

To test an autonomous collision avoidance algorithm completely, a large number of test cases should be conducted; therefore, a total of 900 scenarios are generated in head-on, crossing and overtaking encounters, 300 per encounter situation, as illustrated in Figure 11.

The distributions of RD and RS are recounted to validate the simulated scenarios, (in Figure 12 and Figure 13); it is obvious that the generated scenarios conform to the real AIS data. The testing experiments will be carried out and the results are analyzed in the next section. For length limitations, parts of the test cases are listed in

Table 4. The virtual scenarios generated for head-on encounter.

Head-On Virtual Scenarios
Case No.	Y (nm)	X (nm)	Heading (°)	Velocity (kn)	DCPA (nm)	TCPA (min)
1	−0.27	5.08	179.48	11.63	0.30	14.08
2	0.51	5.91	159.14	8.55	1.49	18.87
3	−0.14	5.96	181.76	7.88	0.22	19.99
4	−0.13	5.37	151.36	15.76	1.50	12.37
5	−0.21	5.92	208.96	10.60	1.73	17.04
…
296	−0.17	5.83	202.61	9.38	1.27	17.98
297	0.02	1.97	197.23	15.45	0.34	4.62
298	−0.39	5.46	1803.1	9.72	0.41	16.60
299	−0.41	5.47	207.66	12.64	1.85	14.08
300	−0.32	5.52	166.60	11.56	0.37	15.45

,

Table 5. The virtual scenarios generated for crossing encounter.

Crossing Virtual Scenarios
Case No.	Y (nm)	X (nm)	Heading (°)	Velocity (kn)	DCPA (nm)	TCPA (min)
1	2.44	2.71	79.97	17.23	0.67	10.07
2	3.59	0.30	105.58	19.21	0.62	11.15
3	1.57	0.86	139.05	25.99	1.52	2.89
4	2.28	1.68	117.77	22.00	1.72	6.95
5	3.02	3.39	47.10	9.98	1.42	14.13
…
296	5.65	−1.86	131.73	29.96	0.60	14.51
297	3.21	0.91	69.57	12.07	1.95	8.96
298	4.86	−1.89	121.77	25.44	1.11	13.95
299	1.78	3.64	32.75	14.83	0.46	10.14
300	2.44	1.92	78.24	15.61	0.13	9.22

and

Table 6. The virtual scenarios generated for overtaking encounter.

Overtaking Virtual Scenarios
Case No.	Y (nm)	X (nm)	Heading (°)	Velocity (kn)	DCPA (nm)	TCPA (min)
1	−0.29	−1.64	3.42	19.37	0.09	10.62
2	−1.03	−0.93	21.79	12.73	0.50	15.34
3	1.56	−1.11	0.52	12.94	1.61	21.37
4	−0.08	−2.41	13.50	13.64	1.63	23.47
5	0.79	−0.95	347.86	14.34	0.07	14.71
…
296	−1.21	−1.45	22.39	12.57	0.99	19.14
297	1.67	−0.91	0.95	13.12	1.73	15.19
298	1.23	−1.94	7.58	14.35	1.91	16.48
299	−0.04	−2.37	11.71	11.72	1.99	27.65
300	0.20	−1.61	8.93	11.89	1.31	22.63

.

6. Test Experiments Using the Generated Scenarios

In this section, the 900 virtual scenarios in Section 5 are, respectively, deployed to test the collision avoidance algorithm [15], similar to our previous study which aimed to solve how to autonomously avoid collisions for large intelligent ships in open waters; the algorithm passed the Imazu problem [14] in more than 20 scenarios with both single and multiple target ships. As shown in Figure 14, the previous research [15] relieves the turning delay caused by inertia on large ships and eliminates the parallel lock problem (as shown in Figure 14), which, if not solved, the own ship can not return back to its next waypoint. A similar phenomenon is also noted by Torben [43].

This experiment attempts to find potential defects in the algorithm and observe maneuver features in terms of duration and passing distance. In all the scenarios, the OS starts at the waypoint (0,0) with a speed of 10 knots, heading to 0° (north up). The task for the OS is to navigate to the next waypoint (0,12 nm) while avoiding any potential collision, as shown in the samples (Figure 15, where there are plotted 3 cases in heading on, crossing and overtaking encounters, each case is added by DCPA, TCPA and rudder angle graph).

The test results are listed in

Table 7. Test results in head-on scenarios.

Test Results (Head-On)
Case No.	Passing Distance (nm)	Total Duration (min)	Near Miss	Collision
1	0.86	14.02	0	0
2	1.49	18.88	0	0
3	1.07	20.45	0	0
4	1.50	12.38	0	0
5	1.73	17.07	0	0
…
296	1.27	18.00	0	0
297	0.81	4.35	0	0
298	1.02	16.68	0	0
299	1.85	14.08	0	0
300	0.98	17.18	0	0

,

Table 8. Test results in crossing scenarios.

Test Results (Crossing)
Case No.	Passing Distance (nm)	Total Duration (min)	Near Miss	Collision
1	0.88	9.35	0	0
2	0.76	7.68	0	0
3	1.52	2.90	0	0
4	1.72	6.97	0	0
5	1.41	14.15	0	0
…
296	0.87	10.68	0	0
297	1.95	8.97	0	0
298	1.11	13.97	0	0
299	0.88	10.02	0	0
300	0.84	7.97	0	0

and

Table 9. Test results in overtaking scenarios.

Test Results (Overtaking)
Case No.	Passing Distance (nm)	Total Duration (min)	Near Miss	Collision
1	0.63	7.68	0	0
2	0.42	17.02	0	0
3	1.61	21.32	0	0
4	1.62	23.48	0	0
5	0.77	19.75	0	0
…
296	0.99	19.35	0	0
297	1.73	15.17	0	0
298	1.91	16.48	0	0
299	1.99	27.73	0	0
300	1.31	22.65	0	0

which show the passing distance, duration time, action time and marks of near miss and collision. Near miss denotes that the OS has avoided the collision, but with a passing distance smaller than 0.3 nm and larger than 0.2 nm. If the passing distance is smaller than 0.1nm, the OS is considered to have collided with the TS.

Out of the total 900 scenarios, there are three cases where the autonomous ship fails to avoid the collision; they are case 74, case 79 and case 142, as listed in

Table 10. Initial settings of failed scenarios.

Failed Scenarios
Case No.	Y (nm)	X (nm)	Heading (°)	Velocity (kn)	Collision	Near Miss	Encounter
74	−4.51	−0.70	63.94	27.49	1	0	crossing
79	−2.16	1.65	111.74	25.48	1	0	crossing
142	−3.69	0.37	77.22	23.40	1	0	crossing
211	−3.65	−1.51	50.89	32.06	0	1	crossing
128	−0.13	−0.50	6.29	13.82	0	1	overtaking
253	0.01	−0.79	11.35	15.20	0	1	overtaking
270	−0.06	−0.55	358.82	14.62	0	1	overtaking

, all in crossing encounters. Our initial inference is that the TS in each scenario is at around the left beam, the OS has to turn starboard to abide by the COLREGs, but the TS has a higher speed, OS’s course change does not take effect. Additionally, there are another four near-miss cases that OS navigates too close to TS with less than 0.2 nm. This might be caused by the initial settings in which the TS’s relative distance and DCPA are too small for OS to act. According to Section 4.1, it can be seen that these near-miss cases exist in the real world, where the target ship may turn its heading abruptly because of its scheduled waypoints, thus a more dangerous situation might occur. Under such conditions, seafarers on human-operated ships will generally confirm each other’s sailing intentions by VHF and reciprocal actions will be taken to make sure ships are able to overcome such a situation. Further investigations are needed to enhance OS’s performance in these cases. Therefore, the proposed testing method reveals that there are still potential defects in the collision avoidance algorithm, although it has safely passed all the Imazu problems [14].

The passing distance and the duration distributions calculated from test results are illustrated in Figure 16 and Figure 17. The passing distances are all less than 2 nm, in contrast to Figure 8 and Figure 9 from real AIS, larger than 2 nm. The reason is that during AIS data gathering, the scenarios are considered as ended when the ships take actions to avoid the collision, the distance at the very time the ships taking action is marked as the passing distance, while in the experiment, the definition of the passing distance is the least distance between the two ships through the whole test experiment.

Most of the experimental cases concentrate on the passing distance around 0.8 nm, which is slightly larger than that of human-operated ships (around 0.6 nm). The duration distributions (Figure 17) are similar to those of real AIS statistics (Figure 8); most of the durations last from 15 to 20 min in head-on encounters, from 5 to 15 min in crossing encounters and 5 to 20 min in overtaking. From this perspective, the autonomous collision avoidance algorithm is, to some extent, similar to human-operated vessels.

Multiple ship encounters are a more challenging task for the automatic collision avoidance algorithm. This paper collected scenarios involving three and five target ships from the AIS data. Validation of total 2000 cases (1000 cases for 3 and 5 target ships, respectively) has been conducted. The statistics gathering, probability distribution plotting, virtual test generating, and experiment conduction are similar to those on single target ship encounters. Considering length limitation, we placed part of the test results from multiple target ships experiments in Figure 18 and Figure 19 (where five cases are plotted, respectively, for three and five target ships, with the DCPA, TCPA and rudder angle changing data).

Out of the 1000 cases from the three target ships encounter experiments, the collision avoidance algorithm passed 963 instances, of which 32 scenarios set initial target ships too close to the OS under test. This is caused by a similar reason to that discussed in the single-encounter experiments.

The remaining five cases are important, since they find potential defects in the algorithm, which has revised the parallel lock problem (as shown in Figure 14) and indeed works all well under most similar situations (in Figure 20). The own ship could make an opposite turn and wait for the appropriate opportunity to return to the waypoint by sailing from the stern of the target ship.

In Figure 21, the own ship does not collide with any ship, while it actually fails to pass the cases, because it is not able to turn back to the waypoint and face the parallel problem. This indicates that there still exists a bug in the algorithm of the previous study. Our initial guess is that there is a problem with the identification programming function, which requires further debugging. This situation, in total, happens only 5 times out of the 1000 cases but it is essential that our research is continued. If the test quantity is too small or if the target ship locations are too regular, we will not be able to notice these conditions.

In regard to the five target ships encounter experiments, the number of successful passing cases is 860 out of a total of 1000. In the failed 140 cases, 80 failures are caused by collision, in 30 cases, the own ship passed the target ships, while with a near miss. Our basic observation is that these 110 cases were actually caused by initial settings of target ship locations that were too close to the own ship under test that the collision avoidance algorithm could not avoid accidents, even when taking the full rudder. The remaining 30 cases are stuck in the parallel problem, similar to the three ship encounter experiments.

There are some cases where the collision avoidance algorithm under test actually does not take any action (for example, case 3 in Figure 18 and other cases by single target encounter in Figure 15). This indicates that there is no collision risk in these cases. This is because, during random generation, the scenario searching space is larger than the real data set collected from the AIS. The relative distance and relative velocities probability densities are plotted, respectively, while they are actually independent in the real world, thus there may exist some scenarios that do not contain a collision risk. From Figure 18 and Figure 19, it could be concluded that the collision avoidance algorithm under test could generally handle some multiple target ship encounter scenarios; the deficiency is that the rudder may shake back and forth at some specific time (for example, in case 5 of Figure 18). Our inference is that the decision process frequently changes at the edge values, such as the encounter section division edge to decide if the ship should turn port or starboard, the threshold to determine whether the OS should return back to its next waypoint or just keep its current course. The trajectories of these test cases do not obviously show the frequent rudder waving phenomenon because of the ship’s delay on rudder maneuvering, though we will continue our investigation on the boundary value problem to improve performances.

7. Conclusions and Discussion

Autonomous techniques are being rapidly developed on MASS and the validation of these techniques is demanded to guarantee safety. Conventionally, small amounts of test cases cannot cover all the situations, while completely random tests require a large search space, which is too time-consuming.

We proposed a method to test a collision avoidance algorithm with random scenarios generated from real world AIS data. There are three advantages to our validation method. First, this method will only search the candidate test case within the possibility distributions but not lose much testing coverage, thus dropping a large number of invalid scenarios which might be generated in an arbitrary method. Compared to setting scenarios manually, it is more compelling to set the parameters according to the real situations in marine traffic. Second, the framework as a whole from extracting information by AIS and drawing probability distributions to accordingly design test scenarios have a certain flexibility and can be generalized to other water areas covered by AIS across the world. The source of the ship autonomous navigation testing scenario dataset in this paper can include not only scenarios mined from AIS data, but also those designed based on expert experience, or from accident data. These scenarios can complement each other. Third, we have actually put these randomly generated and real-data-modified scenarios into practical testing experiments; merits and weaknesses from the test results are also briefly analyzed. Therefore, the constructed ship autonomous navigation testing scenario dataset in this method is practical for engineering applications. Test results will always be the best feedback for debugging during algorithm development and further scientific research, such as assessing and optimizing the functionality of developed autonomous collision avoidance algorithms from a statistical analysis view.

Still, there are many aspects open to be discussed, of which we have listed several points below. Collision risk is an important argument in the proposed scenarios generation method. Although the collision risk in equations (Equation (1)) based on the DCPA and TCPA methods is widely used, other types of collision risk formulas may affect the distributions and the scenario generation.

During drawing distribution graphs from AIS, the value range is divided into 8–10 smaller sections, the shape of the distribution line in Figure 6, Figure 7, Figure 8 and Figure 9 may change according to various section numbers. This will also affect how simulated scenarios are generated, because in each small section, the parameters of TS relative distance and relative speed are sampled with a uniform distribution.

All the target ships in the virtually generated cases keep their course and speed along their origin course. This is not the exact steering manner in the real world, where generally all the ships are responsible for avoiding urgent situations following COLREGs, as investigated in [32], where scenarios of multi-ship situations are extracted by the rules. Although a few ships may not take necessary actions or even, by mistake or on purpose, take adverse behaviors, a more efficient way in reality is to communicate by VHF. The proposed random generation method is not aimed at imitating the perfect reality, the reference from AIS data is a method helpful to limit the search space of the arbitrary sampling process. We need the random process to make the testing coverage larger for scenarios than the reality. While the motion parameters of the target ships should vibrate in a limited range according to the reality, the generated method should not be either too close to or far from the reality. Further studies are needed to balance such tradeoffs.

The manned ship may not abide completely to the COLREGs in testing scenarios as implied by the case study, where the OS and the TS identify the encounter differently in Figure 4. This can happen among autonomous ships, since there are various collision risk models and there is not a clear angle division between encounters (e.g., small angle crossing and head-on). This phenomenon is also indicated in some simulated experiments, where the OS has to turn to left rudder to avoid risk because the give-way TS does not take any action. However, all these specific “against COLREGs” situations mentioned above should be temporary and not lead to any collision accidents. On the other hand, negotiation is an efficient approach to cope with complicated circumstances in a few failed cases (as listed in Table 10). The speed of OS is too slow to avoid collision by taking any course changes; communication is a need for mitigating an urgent situation.

The elements of ship maneuverability, visibility, etc., are not considered in this paper. This will affect the evaluation of the timing and magnitude of the give-way ship maneuver when we deeply analyze the evacuation maneuver. In future research, we may extend the testing scenario parameters with ship maneuverability, ship size, visibility, and additional environmental elements from an electronic navigational chart. Additionally, we aim to extend this proposed framework with the use of AIS data from different waters and to set up a benchmark dataset for performance and risk evaluation of the MASS or autonomous navigation system.