Artificial Intelligence-Based Prediction Model for Maritime Vessel Type Identification

Abstract

This paper presents an artificial intelligence-based model for the classification of maritime vessel images obtained by cameras operating in the visible part of the electromagnetic spectrum. It incorporates both the deep learning techniques for initial image representation and traditional image processing and machine learning methods for subsequent image classification. The presented model is therefore a hybrid approach that uses the Inception v3 deep learning model for the purpose of image vectorization and a combination of SVM, kNN, logistic regression, Naïve Bayes, neural network, and decision tree algorithms for final image classification. The model is trained and tested on a custom dataset consisting of a total of 2915 images of maritime vessels. These images were split into three subsections: training (2444 images), validation (271 images), and testing (200 images). The images themselves encompassed 11 distinctive classes: cargo, container, cruise, fishing, military, passenger, pleasure, sailing, special, tanker, and non-class (objects that can be encountered at sea but do not represent maritime vessels). The presented model accurately classified 86.5% of the images used for training purposes and therefore demonstrated how a relatively straightforward model can still achieve high accuracy and potentially be useful in real-world operational environments aimed at sea surveillance and automatic situational awareness at sea.

1. Introduction

In the past few decades, we have witnessed an increased use of artificial intelligence (AI) in every area of life, ranging from smartphones that can provide access to an unimaginable amount of information in an instant to smart houses that you can converse with and that know when you run out of milk. One of the areas most influenced by these intelligent information systems has undoubtedly been the military domain. Long past are the days of using smoke and mirrors, such as dazzle camouflage [1,2], in order to confuse or disorient the opponent—today’s armies are equipped with advanced information systems and surveillance capable of easily seeing through such distortion techniques. NATO armies are gradually shifting towards more and more automation [3], and it is reasonable to expect that other military forces and alliances are doing the same. In the interest of national security and overall safety, it is therefore of paramount importance to keep up with these movements towards automation in order to maintain the status quo or gain a competitive advantage on the battlefield.

Artificial intelligence use in the military spans multiple areas, from dynamic decision support [4,5] to ethically questionable lethal autonomous weapons systems (LAWS) [6,7]. It can be applied to almost every area belonging to the military domain, but in this paper, we specifically focused on its application in the real-time surveillance of territorial waters via automatic classification of maritime vessels. We concentrated our research on one specific geographical area, namely the Croatian part of the Adriatic Sea, and built a custom image dataset to support the development of our classification model. The final model presented in this paper is therefore specifically designed for the surveillance of Mediterranean and Mediterranean-like waters. However, its adaptable architecture allows for straightforward upgrades and expansions to accommodate other geographical regions where the types of maritime vessels encountered may differ in both variety and frequency compared to those typically found in Mediterranean waters.

The proposed maritime vessel classification model is based on computer vision and data mining (DM) and combines advanced AI techniques such as deep learning for initial image representation with more traditional machine learning algorithms such as k-Nearest Neighbors (kNN) and linear regression. The model is implemented in the Orange Data Mining tool [8] that allows for rapid prototyping and design. This capability is particularly relevant in the context of information systems, which are of great importance for the safety of activities at sea [9]. They can be used in automatic collision avoidance [10], vessel tracking [11], search and rescue operations [12], detection of illegal activities [13], etc. While we based our classification model on computer vision techniques, it is important to note that not all models aimed at maritime surveillance do the same. Some of them employ different sets of detection and identification systems and can commonly include synthetic-aperture radars (SAR), automatic identification systems (AIS), and visual information obtained by human operators or surveillance cameras. Radars are used for the detection and monitoring of different objects, while AIS provides navigational and other vessel information [14]. Because the information obtained through radars and AIS alone is often not enough for accurate identification of a vessel, visual identification plays a key role in confirming the identity of the vessel [15,16]. The combination of radar, AIS, and visual data and their subsequent aggregation can be very useful for maritime security [17], and this is especially important in military applications [18,19]. Technological development has enabled the application of AI and data mining methods in the analysis of various types of data, including digital images, thus ensuring a more efficient and reliable operation in various domains [20].

Even though highly reliable results in maritime vessel classification can be obtained by combining different types of detection and identification systems, as discussed previously, our approach of using visual data alone has one tremendous advantage over this: cameras are relatively inexpensive. This makes them accessible to the vast majority of researchers and organizations looking to develop and test information systems for automatic monitoring of territorial waters or the high seas.

The research presented in this paper is concentrated on the detection of maritime vessels in one specific geographical area, that being the Croatian part of the Adriatic Sea. According to the Croatian Bureau of Statistics [21], Croatia’s mainland coast spans 1880 km, while the island coastline spans 4398 km, totaling 6278 km of coastline. Given the extensive length of the Croatian coastline, human surveillance alone is insufficient for effective monitoring, and automatic or semi-automatic video surveillance is recommended. Croatia is strategically significant as a NATO member state and one of the bases for NATO operations due to its numerous naval ports and significant naval infrastructure, and this emphasizes the need for effective surveillance of critical areas. Additionally, Croatia has many national parks and nature reserves situated on islands and along the coastline, such as Lastovo Archipelago Nature Park and Brijuni, Mljet, and Kornati National Parks. These protected areas are not only beautiful, but also a home to numerous protected and critically endangered marine species, including the Mediterranean monk seal (lat. Monachus monachus), noble pen shell or fan mussel (lat. Pinna nobilis), and loggerhead sea turtle (lat. Caretta caretta). The protection of these species, as well as the coastline and Croatian waters, is essential for ensuring safety and conserving the balance of marine ecosystems.

The rest of the paper is organized as follows: Section 2 provides an overview of related research and challenges that can often be encountered in automatic maritime surveillance and monitoring, especially in the military domain. Section 3 presents artificial intelligence background, including the principles of computer vision, traditional machine learning and deep learning, and data mining. Section 4 presents an artificial intelligence-based maritime vessel classification model. It elaborates on the process of image dataset construction and model design and evaluation. In Section 5 we present a conclusion and discuss future work.

2. Related Work

Vessel detection and identification are procedures by which the identity of the vessel is determined and confirmed based on specific characteristics and information [22]. This process is crucial for the regulation of maritime traffic, safety, and management of maritime resources. The number of new vessels and the density of maritime traffic are increasing every day, and therefore the process of detecting, classifying, and identifying given vessels is becoming more and more difficult and time-consuming.

The detection procedure is the first step in the identification of the vessel. Detection refers to the process of locating or identifying an object on or beneath the sea surface. Vessel detection plays a key role in maritime security, protection, and prevention of illegal activities such as smuggling, illegal fishing, or piracy. Vessel detection technologies also assist in search and rescue operations, providing critical information for rapid response in emergency situations. Detection can be performed in several ways with the help of different devices such as radar, sonar, or AIS or with the help of different visual cameras and satellite images. By fusing data obtained from different sensors, it is possible to unambiguously identify a specific vessel [23].

Automatic maritime vessel classification algorithms can broadly be divided into five main categories, depending on the type of data used during their training and testing: algorithms based on the analysis of AIS data, algorithms based on the analysis of SAR data, and algorithms based on the analysis of optical data. While the classification model presented in this paper clearly belongs to the third category, it remains essential to discuss, at least briefly, AIS- and SAR-based algorithms as well. The comparison between these three types of algorithms can highlight the advantages that can be achieved by utilizing optical data instead of or in addition to AIS and SAR, particularly regarding military surveillance and reconnaissance operations.

2.1. Classification Approaches Based on AIS Data

The International Maritime Organization (IMO) defines automatic identification systems (AISs) as transponders that “are designed to be capable of providing position, identification, and other information about the ship to other ships and to coastal authorities automatically” [24]. In 2004, the requirement for the following three classes of maritime vessels to be equipped with AIS became effective: (1) ships of at least 300 gross tonnage that are engaged in international voyages, (2) cargo ships of at least 500 gross tonnage that are not engaged in international voyages, and (3) all passenger ships regardless of their size [24].

Even though AIS can provide important information for vessel identification, McCauley et al. [25] emphasize two main problems associated with its usage: only a small number of vessels are actually required to be equipped with AIS, and vessels can manipulate AIS information and turn off transponders. Additionally, some vessels may be exempt from carrying AIS [25] or retain the option to turn it off in specific conditions (e.g., military vessels during combat or covert operations) [26].

In spite of these AIS shortcomings, there are methods that are used for the automatic classification of maritime vessels. For example, Huang et al. [27] use AIS and machine learning methods for the automatic classification of ships into bulk carriers, containers, general cargo, and vehicle carriers, Sheng et al. [28] use logistic regression and ship trajectories obtained from AIS to classify vessels into cargo ships and fishing boats, and Baeg and Hammond [29] use ship trajectories obtained from AIS and classify them via different machine learning algorithms into fishing, passenger, tanker, and cargo classes.

While we do acknowledge the potential benefits of AIS when used correctly, we also believe that it has several limitations, especially in the military surveillance and reconnaissance domains. While AIS is not the primary focus of this paper, those seeking more information on its use in maritime research can consult [30].

2.2. Classification Approaches Based on SAR Data

High-resolution synthetic-aperture radars (SARs) are imaging radars capable of high-resolution mapping of the Earth’s surface under certain conditions. They can be mounted on elevated places on land and on manned and unmanned aircraft and spacecraft [31]. They can operate in all-day and all-weather conditions [32] and are able to penetrate clouds, fog, and sometimes even leaves, snow, and sand [31]. These abilities make SARs highly appealing for maritime surveillance, as they can remain unaffected by certain conditions that could potentially impede optical sensors such as surveillance cameras.

There are many existing AI-based methods that employ SAR imaging in the automatic classification of maritime vessels. For example, Sannapu et al. [33] compared the performance of convolutional neural network (CNN), ResNet50, and VGG-16 deep learning architectures for the classification of maritime vessels from SAR images and obtained an accuracy of over 99% when using the CNN architecture, and Wang et al. [34] proposed a ship classification model based on the VGG-16 architecture that obtained an average classification accuracy of more than 95%.

While we do recognize the advantages of SAR-based models for maritime vessel classification, we also identify some serious drawbacks that are particularly evident in the military domain. These key challenges primarily include security concerns and the problem of real-time information gathering required for swift decision-making in rapidly evolving battlefield environments. The process of timely data acquisition from SAR-mounted satellites can be problematic as it depends on their orbits, and although utilizing SAR-mounted aircraft can help mitigate this issue, it can also introduce another one: these aircraft can represent potential targets for adversaries due to their immense strategic importance. Even if data from SAR-mounted satellites or aircraft are obtained in a timely manner, there still exists the problem of the interpretation of the acquired data that usually require experts in the field, therefore prolonging the decision-making process.

Since battlefields represent dynamic environments where even the slightest delay can have grave consequences, timely decision-making is crucial. There is a well-known saying amongst military personnel that states that speed kills [35], meaning that in order to have better chances of defeating the adversary, one must operate faster than the adversary, thereby placing the adversary in an unfavorable position. This principle, therefore, serves to highlight the importance of speed in military decision-making processes (MDMPs) and is further detailed in the well-known OODA (Observe, Orient, Decide, Act) loop initially proposed by the U.S. Air Force fighter pilot John R. Boyd [36,37,38].

2.3. Classification Approaches Based on Optical Data

Algorithms for maritime vessel classification from optical data usually rely on the analysis of data gathered from surveillance cameras or imaging systems mounted on satellites. These cameras can either have fixed (e.g., being mounted on land) or dynamic (e.g., being mounted on an unmanned aerial vehicle or UAV, ship, buoy, or satellite) positions and can operate in the visible or non-visible part of the electromagnetic (EM) spectrum.

Cameras also have a key role in visual data collection used to identify vessels. They enable detailed recording of the object and its surroundings, which is necessary for recognizing the vessel’s characteristics and monitoring its movements. Regardless of whether they are mounted on a ship, a coastal station, an unmanned system, or a satellite, cameras can provide a realistic picture of an object that is difficult to visually conceal or alter its shape [39]. Some types of cameras in use are RGB (Red, Green, Blue) or RGB-D (Depth), IR (Infrared), and HDR (High Dynamic Range). All mentioned types of cameras utilize specialized types of sensors and have specific applications. In the context of this paper, we limited our research to studies oriented on the analysis of visible parts of the EM spectrum and divided classification approaches to maritime vessel classification from optical data into three main categories: approaches based on traditional machine learning methods, approaches based on novel deep learning methods, and approaches that combine both.

It is important to note that due to the extensive number of existing AI-based approaches to maritime vessel classification, we have chosen to highlight only those that we deemed most relevant to our research. For readers seeking a comprehensive review of the latest advancements in maritime surveillance, we recommend referring to [40].

2.3.1. Approaches Based on Machine Learning

Approaches to maritime vessel classification that belong to this category of algorithms are based on traditional machine learning techniques such as support vector machine (SVM), k-Nearest Neighbors, or Naïve Bayes algorithms. These methods do not employ convolutional neural networks nor deep learning in general. Since the appearance of the deep learning paradigm in AI, these types of methods have decreased in frequency. Some of the algorithms that belong to this category include [41,42,43].

Mattyus [41] presented an approach for the automatic detection of maritime vessels from optical satellite images. Their method was based on pattern recognition and computer vision techniques and consisted of two parts: a binary classifier trained on images of maritime vessels and backgrounds and a cascade classifier based on Haar-like features and the Gentle AdaBoost algorithm.

Feineigle et al. [42] proposed an approach for ship recognition in harbor surveillance environments. Their method was based on a SIFT (Scale Invariant Feature Transform) [44] keypoint extractor and descriptor that they used to extract interest points from training images. These interest points were subsequently saved into a database, and when a novel image arrived, its SIFT keypoints were compared to the ones already in the database in order to perform the final ship recognition. The authors tested their method on images of tugboats and cargo carriers.

Du et al. [43] proposed a method for ship classification based on the extraction of ship shape eigenvalues from images, the construction of a library featuring the most separable types of those eigenvalues, and a ship target classification approach based on a kNN algorithm. They obtained the highest final classification rate of 96% and used the following ship classes in their research: bulk carrier, container vessel, tanker, passenger ship, and speedboat.

Approaches to maritime vessel classification based solely on traditional machine learning approaches are becoming increasingly rare, as the research in this field is shifting towards the development of deep learning-based classification methods.

2.3.2. Approaches Based on Deep Learning

Approaches to maritime vessel classification that belong to this category of algorithms are based on deep learning techniques alone and do not combine them with traditional machine learning methods such as SVM or kNN (or at least not to a great extent). Some of the algorithms that belong to this category include [45,46,47,48].

Leclerc et al. [45] used transfer learning and fine-tuned pre-trained CNNs based on Inception and ResNet deep learning architectures in order to construct a maritime image vessel classifier. They used the MARVEL image dataset [49] in their experiments that divided images into 26 super-classes, e.g., container ship, sailing vessel, patrol vessel, etc. Their final model obtained 78.73% classification accuracy.

Mishra et al. [46] used transfer learning and fine-tuned a pre-trained CNN based on the VGG-16 deep learning architecture. The final outcome of their proposed model was the classification of images of maritime vessels into one of four classes: aircraft carriers, crude oil tankers, cruise ships and liners, and destroyers. Their model obtained the average classification accuracy of 97.08%.

Leonidas and Jie [47] proposed a ship classification method based on an improved ResNet-152 architecture. Their proposed model achieved the best average accuracy of 95.8% on a public image dataset from Kaggle [50] consisting of 8932 images belonging to five classes: cargo, military, carrier, cruise, and tanker ships. The authors also tested their method on the MARVEL dataset and achieved an accuracy of 91.35%.

Zhenzhen et al. [48] proposed two CNN-based models for ship classification. They used the following ship classes in their research: aircraft carrier, warship, and a civil ship. If the ship is classified as a warship, it is further classified into one of the five types of warships: coastal combat ships, shipyard transport ships, amphibious assault ships, submarines, and destroyers. The first model that the authors presented was based on AlexNet, and the second one on GoogLeNet. The authors also presented two image datasets, one for coarse-grained and the other for fine-grained ship classification. Their AlexNet-based model achieved an accuracy of 94.7% on the testing part of the images from the first dataset and 91.0% on the testing part of the second dataset. Their GoogLeNet-based model achieved the accuracies of 98.0% and 95.0%, respectively.

2.3.3. Approaches That Combine Machine Learning and Deep Learning

Approaches to maritime vessel classification that belong to this category of algorithms are based on a combination of traditional machine learning methods and deep learning techniques. Some of the algorithms that belong to this category include [51,52,53].

Gallego et al. [51] presented a novel approach for the classification of ships and small unidentified floating objects. Their method was based on CNNs and used optical aerial imagery acquired in the visible part of the EM spectrum. They used the Xception CNN for feature extraction and the kNN algorithm for final classification. The final output of their method was information about the existence or non-existence of a ship in an image. The best F₁-measure they obtained for their approach was 99.05%.

Liu et al. [52] proposed an approach consisting of image processing-based detection of ship candidates and subsequent classification of ships into predefined categories via a CNN. Their CNN model was evaluated on 240 images from Google Earth, where it achieved 99% accuracy for ship detection and 92% accuracy for ship classification. Additionally, they evaluated their model on another dataset consisting of 1500 images across 10 categories of ships, although these specific categories were not disclosed. On this supplementary dataset, their method achieved classification accuracy of 95%. They compared the classification accuracy of SVM and a neural network on the same dataset, and the results they obtained were 87% accuracy for SVM and 81% accuracy for a neural network.

Polap and Wlodarczyk-Sielicka [53] presented an approach for ship classification using a bag-of-words (BoW) mechanism, SURF (Sped-Up Robust Features) [54] keypoint extractor, and a CNN. The best accuracy they achieved with their proposed method was 84.38% for a two-class classification problem (sailing ships and other classes) and 79.58% for a five-class classification problem (cargo, military, tanker, yacht, and motorboat).

2.4. Concluding Remarks

Image analysis used for vessel identification has a crucial role in the process of creating a Recognized Maritime Picture (RMP) [55]. The RMP presents the relevant maritime information based on the process that involves detection, classification, and assessment [56]. All of these processes are necessary elements in establishing situational awareness (SA) that provides a real-time view of activities in a specific area and forms a basis for the decision-making and response [57,58]. The establishment of SA in the information environment is a key requirement, and advanced solutions that enable it are important assets for organizations [59].

In light of the previous discussion, it can be concluded that the research in automatic maritime vessel classification has significantly shifted from traditional machine learning methods to methods that employ deep learning techniques. We expect that this trend will continue, as deep learning-based approaches can typically yield superior classification results.

Additionally, it is important to note that many of the existing methods for maritime vessel classification utilize different image datasets and classify different types of maritime vessels. Therefore, it does not appear that a universally accepted standard dataset for testing and comparison of various methods exists, which in turn makes the comparison of different classification approaches extremely challenging.

3. Artificial Intelligence Background

Automatic maritime vessel classification is a complex and interdisciplinary challenge that necessitates collaboration among experts from various fields. To address this, we aim to ensure that the content presented in this paper is accessible and understandable to a wide audience, including not only AI researchers but also military researchers who may not be as familiar with advanced AI terminology and techniques. It is therefore necessary to outline and discuss the various tools and techniques that we will employ in developing the proposed model for automatic maritime vessel classification.

In this section we will therefore explore the fundamental principles of computer vision and its diverse applications. We will also examine some of the traditional machine learning algorithms that we utilized in this paper, as well as the core concepts of deep learning. Finally, we will discuss the data mining process employed in the development of our maritime vessel classification model.

3.1. Computer Vision Principles

Computer vision is a branch of artificial intelligence that enables computers and systems to recognize information from digital images, videos, and other image-type input data and take actions or provide recommendations based on this information [60]. It can be stated that if artificial intelligence allows computers a certain type of thinking, computer vision allows them to see and interpret the content of an image. The way computer vision works is comparable to human vision, as artificial intelligence largely mimics biological systems [61,62].

Computer vision techniques use machines to perform these functions, from the initial phase of capturing the input and its transformation into a form suitable for processing up to the point where, based on what has been learned, conclusions can be drawn about the previously unknown [63]. A requirement placed before information systems is to perform these tasks on relatively large data collections in a relatively short time with a permissible or relatively small margin of error [64]. The use of computer vision techniques has applications in the manufacturing sector (e.g., defect detection), security (e.g., surveillance), healthcare (e.g., diagnostics), transportation (e.g., traffic management), and more [65]. Additionally, military applications of computer vision techniques are becoming more common [66,67], not just in maritime surveillance but in various other fields (e.g., in camouflage detection [68] and target tracking [69,70]).

The beginnings of digital image processing date back to the early twenties of the 20th century. Devices used to take photos are called cameras. A digital camera is a hardware device consisting of a high-resolution sensor with a fast signal processor for image processing. Recorded images are stored in memory, from where they can be sent to other devices through the communication interface and further processed. Images obtained by a camera sensitive to the visible part of the spectrum represent the illumination of objects on the scene. The image is basically a 2D function of light intensity f(x,y), where x and y denote spatial coordinates and the value of function f at any point (x,y), as shown in Figure 1.

It is useful to consider the function of the image with respect to illumination as a third axis. If we follow the rule that proportionally larger values are assigned to lighter areas, more components in such a drawing will correspond to lighter areas. A digital image is an image f(x,y) that is discretized by spatial coordinates and illumination. A digital image can be represented by a matrix whose columns and rows correspond to the columns and rows of the image. The value of an individual element of the matrix is determined by the level of illumination at that point of the image. The elements of such a digital field are called picture elements, or pixels. A 2D representation of a digital image I of dimensions MxN is displayed in a matrix form in Equation (1). In this equation, symbols a_i,j represent the illumination of a pixel located at position (i, j).

$$I=\left[\begin{array}{cc}\begin{array}{cc}{a}_{\mathrm{0,0}}& a_{\mathrm{0,1}}\\ a_{\mathrm{1,0}}& a_{\mathrm{1,1}}\end{array}& \begin{array}{cc}\dots & a_{0,M}\\ \dots & a_{\mathrm{1,2}}\end{array}\\ \begin{array}{cc}\dots & \dots \\ a_{N,0}& a_{N,1}\end{array}& \begin{array}{cc}\dots & \dots \\ \dots & a_{N,M}\end{array}\end{array}\right]$$

(1)

EM radiation itself does not contain any colors. Understanding the human perception of EM waves in the visible range includes the physiology of the eye, the conversion of the light spectrum into an RGB signal, and the human experience of that signal as hue, saturation, and brightness. RGB is simply a signal of the photosensor’s response to electromagnetic radiation, but the perception of colors is entirely a creation of the brain. The information allocated to specify a pixel’s color is a blend of these three primary colors. Figure 2 shows the decomposition of an image from the Split Port Ship Classification Dataset (SPSCD) [71] into red, green, and blue channels. The decomposition of the image into RGB channels was accomplished with Python and the OpenCV [72] image processing library.

The resolution of a digital image is measured using its pixels. An image of M pixels in height by N pixels in width has a resolution of MxN, where the first variable represents the number of rows and the second represents the number of columns. Higher resolutions mean that there are more pixels per inch (PPI), containing more information and resulting in a higher-quality image. The pixel depth determines the range of values it can store; it is based on the formula 2ⁿ, with n indicating depth. Color depth, on the other hand, indicates the number of bits used for each color component of a single pixel. Considering the mentioned parameters, the image file size is calculated and usually expressed in kilobytes (KB) or megabytes (MB). Image format refers to the structure and organization of data used to store a digital image; the most common types include JPEG, PNG, GIF, and others.

3.2. Machine Learning and Deep Learning Principles

The concept of machine learning refers to the development and implementation of computer programs capable of learning from data and making informed decisions. It is generally divided into five categories: supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning.

Supervised learning encompasses algorithms trained on labeled data. Labeled data refer to data that have been examined and labeled with appropriate classes, preferably by experts in the associated field. For example, medical doctors could look at a dataset containing magnetic resonance (MRI) images of a number of patients and label each image with a “healthy” or “non-healthy” diagnosis. A machine learning-based algorithm could then be constructed that could have the ability to differentiate, more or less successfully, between healthy and non-healthy individuals.

Unsupervised learning encompasses algorithms trained on unlabeled data. These kinds of algorithms are not provided with the names of classes that can be found in the dataset, so they operate by identifying patterns and similarities within the data itself. For example, if an unlabeled image dataset consisted of images of fishing boats and ocean liners, the algorithm would have to learn the difference between these two classes by itself, perhaps on the basis of different image features such as size, shape, and color. The algorithm would still lack the knowledge associated with the names of the classes but should still be able to, more or less successfully, accurately classify images containing fishing boats into class “A” and ocean liners into class “B”.

Semi-supervised learning refers to a combination of supervised and unsupervised learning and encompasses algorithms trained on partially labeled datasets. These kinds of datasets exist because manual data labeling is time-consuming and expensive, and it can result in errors or missing information due to human error.

Reinforcement learning incorporates “rewards” for the algorithm if it reaches a desired conclusion and “punishments” otherwise. This feedback mechanism is used to steer the algorithm towards the desired behavior.

Deep learning can be defined as an advanced form of machine learning that operates on a significantly larger scale. This approach typically involves the use of much larger datasets and much more powerful hardware infrastructure. With the rise of high-performance computing (HPC) and the increased availability of powerful computing resources available to researchers (e.g., advanced computing resources Supek and Vrančić in Croatia [73]), deep learning has gained widespread adoption across multiple fields.

3.3. Data Mining Process

Data mining is the process of discovering interesting and useful patterns and relationships in large amounts of data [74]. The field combines tools from statistics and artificial intelligence with database management to analyze large digital collections, known as datasets [75]. Essentially, data mining involves a step beyond just collecting and managing data, as it uses sophisticated algorithms to transform these large amounts of data into meaningful insights that can support decision-making. The data mining process is a very important part of model creation and follows clearly defined stages. For the purpose of this study, the Cross-Industry Standard Process for Data Mining (CRISP-DM) [76] was applied.

CRISP-DM is a widely accepted methodological framework for conducting data mining and data analysis projects [77]. This model provides a comprehensive process that helps users design, build, test, and execute data mining projects [78]. CRISP-DM consists of six consecutive phases covering the entire data mining life cycle [79]: business understanding, data understanding, data preparation, modeling, evaluation, and deployment.

In the field of data science, the CRISP-DM model has become the most widely used standard for executing projects [80,81]. This is not at all surprising considering its flexibility and reliability. It offers the benefits of agile principles and practices and can be followed easily even by those with less expertise in the field of data science.

4. Proposed Model for Maritime Vessel Classification

Maritime surveillance is a pivotal task in marine protection. It can aid in the identification of illegal fishing or shell poaching hotspots, in the detection of hostile vessels, in the monitoring of nature reserves situated on islands, and in the surveillance and protection of habitats that support protected species. This is especially important for countries with extensive coastlines, such as Croatia.

In the following sections we will detail the design and evaluation of our proposed model for maritime vessel classification, which is specifically tailored for Mediterranean and Mediterranean-like waters. Our primary focus is on the Croatian waters of the Adriatic Sea, and the image dataset that was constructed for model training and evaluation is specifically curated to reflect this unique maritime environment.

4.1. Dataset Preparation

Thorough preparation and an in-depth understanding of input data are an extremely important and time-consuming phase of model development and have an immense influence on the model’s final performance and overall results.

Despite the absence of a standardized dataset of maritime vessel images, several datasets do exist, with one of the largest being the MARVEL [49] dataset. The MARVEL dataset contains two million images categorized into 26 superclasses (e.g., container ship, sailing vessel, patrol vessel) and 109 vessel type classes. As stated by Teixeira et al. [40], the MARVEL dataset exhibits significant class imbalance, as some of the classes contain significantly fewer images than others. Teixeira et al. also state that this imbalance is a “natural reflection of a realistic environment, where there are many more ships of one type than of another” [40]. This principle has been integrated into the dataset that we constructed as well, where the number of images per category approximately corresponds to the actual occurrences of maritime vessels in their respective environments.

With the absence of a singular dataset that is generally considered standard for maritime vessel classification, we constructed our own and obtained maritime vessel images from various open sources available on the Internet, as well as our own independent photography. The images were classified into one of the 11 predefined categories: cargo, container, cruise, fishing, military, passenger, pleasure, sailing, special, tanker, and non-class (objects that can be encountered at sea but do not represent maritime vessels).

From the point of view of the relationship between the camera and the object being recorded, practically in all cases this is a conditionally static scene. This conditionality arises because the vessel is an object that moves relatively slowly, and the cameras used to record it are mounted in a fixed position (in the case of a coastal station) or on a platform (floating or flying). Furthermore, these elements are usually so distant from each other that the speed of movement of the objects in relation to the speed of the camera sensor is negligible, and therefore we consider the images to be static.

When it comes to the perspective or the shooting angle, it is the position of the camera in relation to the shooting object, more precisely the angle formed by the axis of the lens with the visible or imaginary horizontal of the shooting space. The input set of images, with regard to the relationship of the objects, contains images in which the camera is approximately at the height of the object being recorded (image from another vessel), slightly at the upper angle (image from a coastal station or flying platform), or, less often, at the zenith of the imaged object (the angle of 90°, which means that the overflight of the object has been carried out). All this contributes to the diversity of the input dataset, ensuring effective algorithm learning and more reliable model prediction.

It should be noted that the images that form the input to the model were created using different equipment in diverse meteorological and light conditions, which contributed to the representativeness of the collected dataset. Given that the implemented model represents a prototype of a future real information system, it is necessary for it to be able to process images of different formats and quality. This indicates that data can be obtained through our own sensors, received from external systems via the interface, or manually imported into the system. The model should be capable of processing all of these data sources effectively.

The organization of input data into distinct categories is a prerequisite for the model to be able to successfully learn how to classify types of maritime vessels based on their images. This is the responsibility of the analyst, who must determine the appropriate number and types of categories in alignment with business requirements while also ensuring that each category contains a sufficient number of instances. Classification of vessels into general types used in the model has been derived from AIS Ship Types [82] and NOAA Vessel Type and Group Codes [83] and adapted to meet the specific environmental requirements associated with the Adriatic Sea.

The presented division of vessel types is general and suitable for the needs of the proposed model, but it can be further refined according to the specific requirements of the future production system. For the purposes of building a predictive model, a total of 2715 images were collected, approximately half from open sources and half by independent photography. Distribution of maritime vessel images per class in this dataset is given in Figure 3 (generated with Python and Matplotlib [84]) and further detailed in

Table 1. Maritime vessel type classes and number of instances for each class in the constructed dataset.

Type	Description	No. of Images
Cargo	Vessel designed to transport cargo, goods, and materials in bulk.	245
Container	Vessel that carries all its load in truck-size intermodal containers.	276
Cruise	Vessels designed for providing passengers with accommodation and entertainment.	241
Fishing	Vessels used to locate, catch, and preserve fish and other aquatic animals in the sea.	142
Military	Naval vessel used to project power onto the sea or for naval warfare.	197
Passenger	Vessel specially designed for those transports of persons and their cargo.	429
Pleasure	Vessels designed for navigation and recreational purposes while providing comfort.	358
Sailing	Vessel provided with sufficient sail area for navigation used for cruising or racing.	151
Special	Self-propelled vessel serving diverse and often multiple functions (e.g., tugboat and dragger).	214
Tanker	Vessels designed to transport or store liquids or gases in bulk.	357
Non-Class	Non-vessels, floating or other objects.	105
		2715

. The ratio of collected images attempts to reflect the typical proportion of the number of vessels per type observed in the considered environment. This requirement was set during the model design phase.

The constructed dataset consists of 11 categories in total, 10 of which represent the corresponding type of maritime vessel. The last category is reserved for objects that can be encountered at sea but do not belong to maritime vessels. This category encompasses objects such as buoys, docks, submarines, aircraft, and drones, i.e., objects that can be captured by marine surveillance cameras. This category was incorporated into the model to ensure that it can handle a certain amount of noise that can be present in the input data. More specifically, the model should be able to distinguish maritime vessels from non-vessels and further classify vessels by type. Non-vessels, floating objects, and other artifacts should be assigned to a non-class category without further consideration.

The image dataset consisting of 2715 images was further divided into two subsets—one designated for the training of machine learning algorithms and the other one for testing. In the context of this paper, we will refer to this process as data sampling. Data sampling is basically a technique of dividing the input images into subsets, one for model training and the other for model evaluation. Using this approach, it is possible to make inferences about the data being analyzed and the selected learning algorithms. A fixed sample size option was chosen in which 90% of the input data were used for training (i.e., 2444 images) and the rest (i.e., 271 images) for testing machine learning algorithms.

In addition to the previously discussed 2715 maritime images in the input set, an additional set of 200 images is utilized to evaluate the model’s predictive performance during the final assessment stage. Distribution of maritime vessel images per class in this dataset is given in Figure 4 (generated with Python and Matplotlib).

4.2. Model Design

The proposed model for the classification of maritime vessels consists of three main phases:

• Image embedding and clustering. In this stage of model development, the input images are transformed into vectors via different deep learning-based image embedding methods (e.g., Inception v3, SqueezeNet, VGG-16, etc.). After the comparison of these methods, Inception v3 [85,86] was selected as optimal for the problem at hand. Following this comparison, hierarchical clustering is performed on the input images in order to evaluate the model’s ability to separate instances into categories based on a selected similarity measure and image embedding. After comparing different similarity measures (e.g., Euclidean, Manhattan, and Cosine distances), Euclidean distance was chosen as optimal.
• Comparison of different machine learning algorithms. In this stage of model development, different machine learning algorithms are trained, tested, and evaluated in order to determine the optimal one for maritime vessel classification. We focused on the evaluation of well-known, traditional machine learning algorithms such as logistic regression, a neural network, SVM, kNN, Naïve Bayes, and decision tree.
• Model evaluation. The optimal machine learning algorithm chosen in a previous stage of model development is evaluated on 200 previously unseen maritime images.

The implementation of the proposed model was realized using the Orange Data Mining tool, an open-source data visualization, machine learning, and data mining toolkit [8]. The presented model was created using the latest tool version, 3.38.1, on the x64-based system using an Intel(R) Core(TM) i5-3320M CPU and running MS Windows 10 Enterprise Build 19045.

Each phase of the proposed model’s development is elaborated in detail in the following subsections:

4.2.1. Image Embedding and Clustering

Image embedding is a technique used in many fields of data processing and artificial intelligence, including computer vision. It is used to convert high-dimensional data into lower-dimensional vectors. Vector representations of visual elements, namely images or video frames, are used in tasks such as object recognition, image search, data clustering, and image classification. Image embeddings are therefore numerical representations of digital images that condense the complexity of visual data, capturing their features and semantics, and encoding them into lower-dimensional vector representations that are basically image descriptors. Consequently, embedding is the process of deep learning. In the case of images, embeddings are calculated using computer vision models that are trained on very large datasets. The training of deep neural networks in this case encompasses automatic learning of image features through layers of convolution operations, activation functions, and fusion layers. This encodes the information that represents the content of the image in a condensed form, incomprehensible to humans in its raw form, but suitable for machine learning models to carry out further processing.

Several different image embedding methods (Inception v3, SqueezeNet, VGG-16, etc.) were evaluated, and Inception v3 was finally chosen based on its performance on the maritime vessel image dataset used in this paper. Inception v3 is Google’s 48-layer-deep convolutional neural network for image classification, originally published in [85] and pretrained on the ImageNet dataset [86]. By using Inception v3, each image in the dataset was represented by a vector of 2048 elements, i.e., features.

Next, hierarchical clustering is conducted in order to determine the effectiveness of image embeddings in distinguishing between 11 classes of maritime images. During this process, several different distance metrics that were discussed previously were compared, and Euclidean distance was chosen as optimal.

Euclidean distance D_E represents the shortest distance between two data points or vectors in a Cartesian coordinate system. It is the square root of the sum of the squares of the differences between the corresponding elements. Most machine learning algorithms use this distance metric to measure the similarity between observations [87]. The equation for the calculation of Euclidean distance D_E between points (x₁, y₁) and (x₂, y₂) in two-dimensional space is given in Equation (2).

$$D_E=\sqrt{{\left(x_1-y_1\right)}^2+{\left(x_2-y_2\right)}^2}$$

(2)

For a space with n dimensions, the Euclidean distance is calculated as shown in Equation (3), where n is the number of dimensions, and x_i and y_i are data points.

$$D_E=\sqrt{{\sum }_{i=1}^n{\left(x_i-y_i\right)}^2}$$

(3)

Drawing from the findings of the hierarchical clustering analysis, it can be seen that the images from the same class are usually grouped together, but that the algorithm also encounters some problems when distinguishing between similar images belonging to different classes. For example, some of the images representing yachts were classified as cruisers, but this is not surprising considering that there are similarities between these objects and that these similarities can be greatly emphasized depending on the scale of images.

This initial phase of model development concentrated on unsupervised learning, specifically hierarchical clustering. During this stage, raw input data were transformed into numerical vectors suitable for further processing, allowing us to evaluate the distinctiveness of these vectors across 11 different maritime classes. The final results obtained during this stage were promising, as they indicated that the numerical representations of images generated by the Inception v3 model are indeed valuable for further analysis, as they do exhibit significant differences across 11 classes of maritime images.

The final workflow of the first phase of model development is shown in Figure 5.

4.2.2. Comparison of Different Machine Learning Algorithms

In order to determine the machine learning algorithm that would be optimal for the automatic classification of image embeddings obtained in the previous stage of model development, six machine learning algorithms were evaluated. These algorithms included logistic regression, a neural network, SVM, kNN, Naïve Bayes, and a decision tree. The parameters used in the training process of these algorithms within the Orange Data Mining tool were determined during the development phase. This procedure entailed empirical testing and a comprehensive comparison of multiple hyperparameter values using the Orange Data Mining tool. The optimal hyperparameters were those that consistently yielded the best results. For each model, a range of key parameter values was tested (e.g., C and kernel type for SVM, the number of neighbors k for kNN, and the number of layers and activation functions for neural networks) using a validation dataset. These configurations were compared based on accuracy, precision, recall, and F₁-score metrics, and the final parameters were selected because they consistently achieved the best results across multiple experiment repetitions. Additionally, the Adam optimizer was chosen for training neural networks, as it has been proven to demonstrate high robustness and convergence speed in image classification tasks [88,89] (Dogo et al.; Kartowisastro and Latupapua, De et al.). The selected hyperparameters are detailed below:

• Logistic regression—L2 or ridge regularization was employed, and the strength of regularization was set at 1.
• Neural network—the number of hidden layers was set at 100, ReLU (Rectified Linear Unit) activation function and Adam optimizer were used, and the maximum number of iterations was set at 200.
• SVM—the cost C was set at 0, the kernel was set to RBF (Radial Basis Function), numerical tolerance was set at 0.0010, and the iteration limit was set at 100.
• kNN—number of neighbors was set at 5, Euclidean metric was used, and the weight of data points was uniform.
• Naïve Bayes—no parameters were set during training.
• Decision tree—the minimum number of instances in leaves of the tree was set at 2, subsets smaller than 5 were not split, binary trees were included, maximum tree depth was limited to 100, and the classification would stop when the majority threshold would reach 95%.

After the completion of training and evaluation of selected machine learning algorithms, their evaluation was performed. The focus was placed on the assessment of their suitability for the specific input dataset of maritime images, as well as their prediction accuracy. Five different performance measures were utilized for this purpose: area under the curve (AUC), classification accuracy (CA), F₁-score, recall, and precision. These measures are useful in the comparison of performances of different models, with higher scores usually indicating better performance.

For the task of maritime vessel classification, precision—defined as the proportion of true positives among instances classified as positive—was selected as the decisive criterion [90]. Precision is a key indicator in computer vision applications, especially when false positives can be costly, which is precisely the case for this model. The performance results obtained by using these measures are shown in

Table 2. Evaluation measures for selected machine learning algorithms.

Model	AUC	CA	F1-Score	Precision	Recall
SVM	0.991	0.900	0.902	0.914	0.900
Neural network	0.992	0.904	0.904	0.907	0.904
Logistic regression	0.992	0.882	0.881	0.885	0.882
kNN	0.968	0.856	0.858	0.861	0.856
Naïve Bayes	n/a	0.779	0.790	0.821	0.779
Decision tree	0.769	0.609	0.619	0.641	0.609

.

The results presented in Table 2 reflect the average performance across classes and demonstrate that the selected models achieve high classification accuracy. It should be noted that the SVM stands out with the highest precision. The effectiveness of the SVM model is further validated by its Receiver Operating Characteristic (ROC) value, utilized as an additional method for the evaluation and comparison of models’ performance [90]. Figure 6 shows the ROC curve for cargo, fishing, tanker, and sailing classes of maritime vessels. The area under a given ROC curve, or AUC, represents the probability that the prediction will be accurate, and it is often used as a measure of aggregated classification performance [91]. All tested models achieved excellent AUC scores, but since AUC is a relative measure that does not specify the exact number of errors the classifier commits and for which instances [92], it is advisable to combine it with other measures such as CA and precision. Considering these factors, the SVM algorithm was selected as optimal for the purpose of maritime image classification.

In order to further demonstrate the effectiveness of the SVM-based model for maritime image classification, the associated confusion matrix is presented in Figure 7. The confusion matrix is a common way to summarize the classifier results [93]. It shows the exact distribution of evaluated test data as the model based on the SVM algorithm assigns their belonging to a particular class [94]. This is therefore an objective measure of its ability to deal with unknown data. Numbers that are not on the diagonal indicate classification errors. The results clearly indicate that the model achieves high accuracy, and when it does make errors, it typically confuses similar types of maritime classes (e.g., it mistakes cargo as tanker or passenger vessels). This is not surprising given that the prediction is based on the recognition of distinctive features in images, and the objects in images can appear similar even if they belong to different classes.

In conclusion, several commonly used machine learning algorithms were evaluated during the model development phase. The trained models showed an overall high classification reliability, which indicated that they can be used for the maritime image classification task. Among the evaluated models, SVM exhibited the best performance and was therefore selected as the predictive algorithm that would be incorporated into the proposed model for maritime image classification.

Figure 8 illustrates the final workflow for the second phase of the model development process oriented towards maritime image classification.

4.2.3. Model Evaluation

With the proposed model for maritime image classification fully constructed, the final step that remains is to evaluate its performance on previously unseen data. These data encompass 200 images whose class distribution is shown in Figure 4.

The final workflow of the prediction framework is shown in Figure 9, and a confusion matrix detailing the average classification accuracy per class is shown in

Table 3. Final classification results achieved by the proposed model. The input set of images refers to the collection of 2715 maritime images, and the test set refers to the collection of 200 maritime images.

Type	Input Set		Test Set		Percentage [%]
	Train	Test	Successful	Unsuccessful
Cargo	226	19	10	5	66.67
Container	253	23	13	2	86.67
Cruise	216	25	16	1	94.11
Fishing	132	10	11	2	84.61
Military	175	22	13	2	86.67
Passenger	380	49	20	1	95.24
Pleasure	316	42	22	4	84.62
Sailing	137	14	17	0	100.00
Special	190	24	10	5	66.67
Tanker	321	36	25	2	92.59
Non-Class	98	7	16	3	84.21
Total	2444	271	173	27	86.50

.

From the results shown in Table 3, it can be seen that on the sample of 200 maritime images, the model accurately identified 173 instances, which corresponds to an overall accuracy of 86.50%, which is a high score. However, when looking at precision by individual image class, it can be observed that the success rate ranges from 66.67% (for the classes cargo and special) up to 100.00% (for the class sailing).

The relatively low prediction rate for cargo and special classes is not unexpected and can be attributed to several reasons. Different instances of vessels in these classes sometimes actually resemble the shape of vessels in other classes, which leads to an increase in classification errors. Specifically, the class special is characterized by significant variability in shape and other attributes, which consequently makes classification more difficult. Additionally, it is important to consider the specifics of the dataset on which the model was constructed and evaluated, but this is a standard property of all such predictive models.

Alternatively, the model achieved 100% accuracy in classifying maritime vessels belonging to the sailing class. This result can be attributed to the distinctive shape of these vessels, by which it is possible (on our dataset) to unambiguously determine their class.

Given that our research was conducted with a military context in mind, it is noteworthy that the model achieved a classification accuracy of 86.67% for images belonging to the military class. This achievement underscores the potential reliability of a similar model in real-world applications, in which the accurate detection of this type of vessel would be of utmost importance.

Finally, in

Table 4. Comparison of the proposed model with existing methods for maritime image classification.

Model	Maritime Classes	Average Accuracy [%]
[95]	barges, container ships, cargo ships, and tankers	94.63
[95]	barges, container ships, cargo ships, and tankers	86.87
[95]	merchant ships, sailing ships, medium passenger ships, medium “other” ships, tugboats, and small boats	85.07
[96]	container ships, speedboats, tanker ships, tugboats, cruise ships, and fishing boats	88.1
[53]	cargo, military, tanker, yacht, and motorboat	79.58
Proposed model	cargo, container ships, cruise, fishing, military, passenger, pleasure, sailing, special, tanker, and non-class	86.50

we compared our proposed model for maritime image classification to several existing models. It is important to note, however, that not all models focus on the identification of the same types of vessels, utilize the same image datasets for training and testing, employ the same artificial intelligence techniques, or use the same evaluation measures. Therefore, this comparison should be viewed as a preliminary assessment that is liable to change.

In conclusion, the proposed model for maritime image classification exhibited a high prediction accuracy, even for the non-class category, which was included in the model to introduce a controlled level of noise. Overall, the model demonstrated its ability to differentiate vessels from non-vessels and its effectiveness in determining the particular type to which the vessel belongs. Alternatively, the effectiveness in differentiating separate types of non-vessel objects was not the focus of this research and remains an area for future exploration.

5. Conclusions

The preceding analysis confirms the feasibility of independently developing and deploying models for maritime surveillance that are based on the analysis of images collected by optical sensors already in use in many seaports. The introduction of decision-making process automation based on artificial intelligence algorithms can ensure a high level of reliability and relieve operators, leaving them room to intervene only in doubtful situations. This proves that a similar model can be integrated as a crucial part of the information system used to maintain the maritime picture. The implementation of this model into real-world applications could automate the maritime surveillance process and thus contribute to maintaining situational awareness at sea.

The primary limitation of the presented model is the dataset on which it was constructed. This dataset was created by combining multiple sources that predominantly reflect conditions specific to the Croatian part of the Adriatic Sea. While the model is tailored for analyzing this region, any attempts to replicate similar systems in different areas would necessitate retraining on localized datasets.

To enhance the model’s performance in the future, we recommend the following strategies:

• Dataset augmentation. By increasing the number of images in the dataset, a more robust foundation for model training can be established.
• Refinement of maritime vessel categories. Enhancing the dataset by incorporating additional maritime vessel categories could ensure it is not restricted to one specific geographic area. This would make it more versatile and applicable to a wider number of applications.
• Evaluation of additional machine learning algorithms. Testing the performance of additional algorithms on the newly constructed dataset could provide valuable insights about the type of classification method most suited for the task at hand. The evaluation should prioritize not only classification accuracy but also the speed of processing, as real-time performance is essential for effective maritime vessel surveillance, especially in military domains.

In conclusion, we believe that the approach presented in this paper provides a robust new method for maritime vessel classification by emphasizing traditional machine learning algorithms over deep learning-based algorithms. Deep learning techniques usually require an extensive amount of training data and a substantial computational infrastructure, i.e., things that are arguably not readily available to the majority of users. However, many people do have access to home computers that can be used for training simple machine learning algorithms, and this accessibility makes our proposed approach more practical and widely applicable.