Cost-Sensitive Laplacian Logistic Regression for Ship Detention Prediction

Abstract

Port state control (PSC) is the last line of defense for substandard ships. During a PSC inspection, ship detention is the most severe result if the inspected ship is identified with critical deficiencies. Regarding the development of ship detention prediction models, this paper identifies two challenges: learning from imbalanced data and learning from unlabeled data. The first challenge, imbalanced data, arises from the fact that a minority of inspected ships were detained. The second challenge, unlabeled data, arises from the fact that in practice not all foreign visiting ships receive a formal PSC inspection, leading to a missing data problem. To address these two challenges, this paper adopts two machine learning paradigms: cost-sensitive learning and semi-supervised learning. Accordingly, we expand the traditional logistic regression (LR) model by introducing a cost parameter to consider the different misclassification costs of unbalanced classes and incorporating a graph regularization term to consider unlabeled data. Finally, we conduct extensive computational experiments to verify the superiority of the developed cost-sensitive semi-supervised learning framework in this paper. Computational results show that introducing a cost parameter into LR can improve the classification rate for substandard ships by almost 10%. In addition, the results show that considering unlabeled data in classification models can increase the classification rate for minority and majority classes by 1.33% and 5.93%, respectively.

1. Introduction

Maritime transportation is the backbone of international trade [1]. To guarantee maritime safety, as a complement to numerous international regulations and conventions, the port state control (PSC) is seen as the last line of defense against substandard ships [2]. During a PSC inspection, a ship condition found not to be compliant with the relevant regulations and conventions is recorded as a deficiency item. If critical deficiency items are found by port state control officers (PSCOs), the port state can detain the high-risk ship and require the ship to rectify these deficiency items before its departure. For ship operators, ship detention is the most severe outcome of a PSC inspection because it delays shipping schedules [3], and maritime transportation delays are costly [4]. For port states, identifying as many ships as possible that should be detained is their principal task because failing to identify these high-risk ships may lead to marine accidents with serious social and environmental damages. Hence, identifying ships with a high probability of being detained is of great practical significance. However, inaccuracies and inefficiencies are always present at several stages during a PSC inspection, from selecting a ship to making a final decision [3]. For example, the current ship selection scheme is not efficient in targeting high-risk ships, and to what extent the ships should be inspected and the decision regarding detention are highly dependent on the subjectivity of PSCOs. Therefore, designing an intelligent decision support system to help predict ship detention risk is an urgent need.

Nowadays, many studies use machine learning (ML) methods and PSC data to design inspection schemes for port authorities, examining how to implement PSC inspections more efficiently and discussing the effects of PSC inspections [5]. However, to the best of our knowledge, studies relevant to ship detention prediction are somehow limited. Different from most existing studies that focus on predicting the number of deficiencies of a ship, predicting ship detention is a more challenging task. As reported in the Annual Report on Port State Control in the Asia-Pacific Region 2021, out of 22,730 inspections which involve 14,951 individual ships operating in the region, 11,567 inspections found ships with deficiencies, while only 526 ships were eventually detained  [6]. From this report, we identify two challenges when developing classification models for ship detention prediction. First, in the region, the deficiency rate was 51%, while the detention rate was only 2.31%, showing that the ship detention rate is much more imbalanced than the ship deficiency rate. This result requires us to learn from imbalanced data, where the distribution of examples is skewed since representatives of undetained ships appear much more frequently [7]. In addition, less than 60% of all ships are inspected, so we must learn from unlabeled data; that is, not all visiting ships are labeled with inspection results. To address each of the two challenges, this paper studies two machine learning paradigms for ship detention prediction: cost-sensitive learning and semi-supervised learning.

Class imbalance and cost-sensitive learning are closely related to each other [8,9]. Cost-sensitive learning aims to make the optimal decision that minimizes the misclassification cost [10,11,12,13]. For our studied problem, misclassifying a ship that should be detained and misclassifying a ship that should not be detained have different influences. Therefore, the two kinds of misclassifications should be considered with different costs. Several studies have shown that cost-sensitive methods performed better than sampling methods in certain application domains [14,15,16,17]. Semi-supervised learning aims to improve learning performance by exploiting the unlabeled data in addition to the labeled data [18,19]. The lack of an assigned inspection class for each foreign visiting ship is the issue we are faced with when using observational data in PSC-related studies. This naturally occurs because ships that appear at high risk of detention have high possibilities of being inspected, while ships at lower risk may not. For example, the Paris Memorandum of Understanding (MoU) and Tokyo MoU adopt ship risk profile (SRP) to select ships that are more likely to have larger numbers of deficiencies and detained possibilities. Therefore, the PSC dataset used for prediction is a biased collection of all foreign visiting ships, which may ignore some important and valuable information underlying the unlabeled data.

The use of unlabeled data in cost-sensitive learning has attracted growing attention and many frameworks have been developed [20,21,22,23]. To the best of our knowledge, there has not been an attempt to apply both semi-supervised learning and cost-sensitive learning to help improve ship detention prediction. In our paper, we aim to address unequal misclassification costs and utilize unlabeled data simultaneously based on an extension of logistic regression (LR) by introducing a cost parameter and using data-dependent graph regularization [24]. Through computational experiments, the superiority of the proposed cost-sensitive semi-supervised learning framework in ship detention prediction is verified.

The remainder of this paper is organized as follows. Section 2 reviews related studies on ship detention prediction. Section 3 introduces the preliminaries of the ML framework to be extended, including the basic concepts of a learning problem and the Laplacian operator of a graph. Section 4 develops the cost-sensitive semi-supervised learning framework for ship detention prediction and introduces the performance metrics of prediction models. Section 5 conducts computational experiments to show the benefits of our developed cost-sensitive semi-supervised learning framework. Finally, Section 6 concludes this study and proposes future research directions.

2. Literature Review

Because current SRP schemes do not efficiently identify substandard ships, most PSC-related studies have focused on improving inspection efficiency by developing methods for identifying ships with more deficiencies or higher detention probabilities. Recall that the detention rate is much more imbalanced than the deficiency rate, and predicting the total number of deficiencies of a ship or predicting the number of deficiencies under each category is much easier than predicting the detention probability. The summary of the literature on ship risk prediction is shown in

Table 1. Literature on ship risk prediction.

Literature	Prediction Target	Prediction Method	Data Imbalance	Unlabeled Data
Xu et al. [25,26]	Ship deficiency number	SVM	– ${}^1$	No
Gao et al. [27]	Ship deficiency number	kNN and SVM	–	No
Wang et al. [28]	Ship deficiency number	TAN	–	No
Chung et al. [29] and Yan et al. [30]	The type and sequence of inspected deficiency codes	Apriori	–	No
Yan et al. [31]	Ship deficiency number	Random forest	–	No
Yan et al. [32]	Ship deficiency number	Xgboost	–	No
Cariou and Wolff [35]	Ship detention outcome	Quantile regression	No	No
Yang et al. [42]	Ship detention outcome	BN	No	No
Tsou [39]	Ship detention outcome	Association rule mining techniques	No	No
Wu et al. [34]	Ship detention outcome	SVM	No	No
Yan et al. [3]	Ship detention outcome	BRF	Yes	No
Our work	Ship detention outcome	Cost-sensitive Laplacian logistic regression	Yes	Yes

¹ –: Not applicable.

. We first briefly review studies focusing on predicting ship deficiency number. Early studies, such as Xu et al. [25] and Xu et al. [26], used a support vector machine (SVM) to predict the overall ship risk considering the ship deficiency number. Gao et al. [27] combined k-nearest neighbor (kNN) and SVM. In recent years, there are an increasing number of studies developing more advanced ML methods for ship deficiency number prediction. Wang et al. [28] developed a tree-augmented naive (TAN) Bayes classifier to predict the number of deficiencies of each foreign visiting ship. Chung et al. (2020) [29] and Yan et al. [30] adopted the Apriori algorithm to determine the type and sequence of deficiency codes that should be inspected. In recent years, ship deficiency prediction models have also been used to solve inspection assignment problems, as studied by Yan et al. [31] and Yan et al. [32]. For more studies related to ship deficiency number prediction, readers can refer to Yan et al. [33].

We now proceed to review related studies on ship detention prediction. According to Wu et al. [34], there is a stream of studies focusing on analyzing the relationship between the identified ship deficiencies and the detention outcome, such as Cariou and Wolff [35], Chen et al. [36], and Yan and Wang [37]. However, these studies did not consider the influence of ship parameters (e.g., ship age and ship type) on ship detention outcome. Ship parameters serve as critical factors in ship detention. Cariou et al. [38] pointed out that ship age is the main contributor to the ship detention. The same observation has also been shown by Tsou [39] and Şanlıer [40]. Hänninen and Kujala [41] identified that the ship type, the deficiency number, and the PSC inspection type are the most important factors attributed to the ship detention. Yang et al. [42] proposed a data-driven Bayesian network (BN) model based on TAN learning to predict the ship detention probability of bulk carriers under the Paris MoU. To determine the optimal inspection policy for a port, Yang et al. [43] incorporated the results of the ship detention prediction model into a game model that considered both port authorities and ship operators. Wu et al. [34] used an SVM model to predict ship detention, where they selected input features by using an analytic hierarchy process (AHP) and a grey relational analysis (GRA) to improve prediction accuracy. However, the ship detention prediction models seldom considered the imbalanced ship detention rate. To remedy this issue, Yan et al. [3] used a balanced random forest (BRF) model to predict ship detention to address the issues caused by the low-probability detention outcome, which is based on the sampling technique.

In summary, although some studies have detention prediction models, the two identified challenges mentioned before are seldom considered by them. First, they did not consider that the dataset for ship detention prediction is highly imbalanced, so we should attach different misclassification costs for minority and majority classes. Second, they did not consider that the used dataset is a biased collection of all foreign visiting ships, so we should further consider the value of unlabeled data. Therefore, to bridge these two research gaps, this paper develops the cost-sensitive semi-supervised learning framework for targeted and cost-effective ship detention prediction.

3. Preliminaries

This section introduces preliminaries that are related to this research. We first illustrate the importance of the regulation term in a learning problem in Section 3.1. We then introduce the smoothness of a decision function and the Laplacian operator of a graph in Section 3.2. The introduction generally follows the approaches from Vapnik [44], Shalev-Shwartz and Ben-David [45], and Melas-Kyriazi [46].

3.1. The Learning Problem and Regularization

Before proceeding to the introduction of a semi-supervised learning problem, this section first focuses on a supervised setting to lay some foundations of ML algorithms that would be extended in subsequent sections. Consider we have a dataset $S=\{({\mathit{x}}_1,y_1),$ …, $({\mathit{x}}_i,y_i),\dots ,({\mathit{x}}_n,y_n)\}$, where ${\mathit{x}}_i\in \mathcal{X}\subset {\mathbb{R}}^d$ is the feature vector and $y_i\in \mathcal{Y}\subset \mathbb{R}$ is the corresponding label. We assume that our data $({\mathit{x}}_i,y_i)$ are independently drawn and identically distributed from a probability space $\mathcal{X}\times \mathcal{Y}$ with measure $\rho$. Learning is to find a function f that generalizes from this finite dataset S to the space $\mathcal{X}\times \mathcal{Y}$. This task can be achieved by minimizing the expected loss $ϵ$, also called the risk:

$$ϵ(f,\mathit{x},y)=\mathbb{E}\left[L(y,f\left(\mathit{x}\right))\right]={\int }_{\mathcal{X}\times \mathcal{Y}}L(y,f\left(\mathit{x}\right))d\rho (\mathit{x},y),$$

(1)

where $L(y,f(\mathit{x}\left)\right)$ denotes the loss function that measures how well the function $f\left(\mathit{x}\right)$ predicts the outcome. If $f\left(\mathit{x}\right)=y$, $L(y,f(\mathit{x}\left)\right)=0$ and this prediction is perfect. Then, the objective of learning is to find a function by minimizing the risk:

$$f^{*}=\underset{f\in \mathcal{F}}{argmin}{\int }_{\mathcal{X}\times \mathcal{Y}}L(y,f\left(\mathit{x}\right))d\rho (\mathit{x},y),$$

(2)

where $\mathcal{F}$ denotes the class of all measurable functions. Because we only have finite data and computing (2) is impossible, we approximate it by using the empirical risk:

$$\widehat{ϵ}(f,\mathit{x},y)=\frac{1}{n}\sum _{i=1}^{n}L(y_i,f\left({\mathit{x}}_i\right))\approx {\int }_{\mathcal{X}\times \mathcal{Y}}L(y,f\left(\mathit{x}\right))d\rho (\mathit{x},y).$$

(3)

If we could minimize the empirical risk over all functional functions $f\in \mathcal{F}$, we would be able to approximate ${\int }_{\mathcal{X}\times \mathcal{Y}}L(y,f\left(\mathit{x}\right))d\rho (\mathit{x},y)$ and obtain a function $\widehat{f}={argmin}_{f\in \mathcal{F}}\widehat{ϵ}(f,\mathit{x},y)$ resembling the perfect function $f^{*}$. However, this is not possible because we are learning in an entirely unconstrained setting where the distributions of the dataset and the class of functions are not assumed in advance. To overcome this issue, we can take either of two ways accordingly to constrain the learning problem: (1) by assuming that the dataset follows a class of probability distributions $\theta$, namely letting $\rho \in \theta$, or (2) by assuming that learning functions follow a limited class of target functions $\mathcal{H}$, namely letting $f\in \mathcal{H}$, where $\mathcal{H}$ is a subset of $\mathcal{F}$. Because the probability distribution of $\rho$ is hardly known in practice, this paper adopts the second approach, as is common in ML theory.

When we optimize over a restricted class of functions $\mathcal{H}$, we would like $\mathcal{H}$ to be large so that we can learn complex functions. However, these complex functions may not work well with new data if they fit our training data nearly perfectly, causing a problem known as overfitting. Generally, when we have a lot of data, we hope to learn complex functions. Conversely, when we have little data, we prefer simpler functions. In order to achieve this aim, the regularization term is always introduced into the loss function, which takes the form of a penalty R and biases learning toward simpler and smoother functions. With regularization, the learning problem is transformed into:

$$\underset{f\in \mathcal{H}}{min}\widehat{ϵ}(f,\mathit{x},y)+\lambda R(f,\mathit{x},y),$$

(4)

where $\mathcal{H}$ is the predefined space, and $\lambda >0$ is a hyperparameter that balances the empirical risk term and the regularization term. Generally, the regularization term only depends on the function f and its parameters (e.g., L1/L2 norms). We call this regularization term a data-independent regularization term. In addition, in order to enforce that a prediction model should output similar prediction results for a pair of similar examples no matter whether they are labeled or unlabeled, we introduce a graph regularization term into the semi-supervised learning problem that this paper studies. Graph regularization is an example of data-dependent regularization because it is formulated by examining the geometric features of labeled and unlabeled data. We will introduce it in Section 3.2 and Section 4.3.

3.2. Smoothness and Laplacian

Our research problem focuses on the semi-supervised learning setting, where we wish to learn the probability distribution based on which both labeled and unlabeled examples are drawn. Assuming that the underlying distribution may have non-trivial geometric features, we hope to design algorithms that can exploit these geometric features to facilitate the learning outcome [24]. Because we have discrete and finite data, we assume that the probability distribution may have support on a graph, where we regard data examples as nodes. To examine the geometric features of a graph, we need to introduce the definition of the smoothness of a decision function on a graph and the Laplacian operator of a graph.

Consider $\mathcal{G}=(\mathcal{N},\mathcal{E})$ as a connected and undirected graph with edges $\mathcal{E}=\{1,\dots ,E\}$ and nodes $\mathcal{N}=\{1,\dots ,N\}$. We denote by $\mathit{W}=\left({\left(w_{ij}\right)}_{i,j=1}^N\right)$ the symmetric weighted adjacency matrix on graph $\mathcal{G}$, where $w_{ij}$ measures the similarity between nodes i and j [47], and $\mathit{D}$ = diag $(d_1,\dots ,d_N)$ the degree matrix with degrees $d_i={\sum }_{j=1}^N{w}_{ij},i=1,\dots ,N$ on the diagonal. A real-valued function on $\mathcal{G}$ is a map $f:\mathcal{N}\to \mathbb{R}$ defined on the nodes of the graph. Intuitively, a function on a weighted graph is smooth if its value on a node is similar to its value on each of the node’s neighbors; that is, this function does not make too many “jumps’’ [46]. Following this notion, we use

$$\sum _{i=1}^N\sum _{j=1}^N{w}_{ij}{\left(f_i-f_j\right)}^2$$

(5)

to measure the smoothness of a function on a weighted graph, which measures the weighted differences between the function value of a node with those of its neighbors. Because (5) is symmetric and quadratic, we can rewrite it as:

$$\begin{array}{cc}\hfill \sum _{i=1}^N\sum _{j=1}^N{w}_{ij}{\left(f_i-f_j\right)}^2& =\sum _{i=1}^N\sum _{j=1}^N\left(w_{ij}{f}_i^2-2f_i{f}_j{w}_{ij}+w_{ij}{f}_j^2\right)\hfill \\ \hfill & =\sum _{i=1}^N{d}_i{f}_i^2-2\sum _{i=1}^N\sum _{j=1}^N{f}_i{f}_j{w}_{ij}+\sum _{j=1}^N{d}_j{f}_j^2\hfill \\ \hfill & =2\sum _{i=1}^N{d}_i{f}_i^2-2\sum _{i=1}^N\sum _{j=1}^N{f}_i{f}_j{w}_{ij}\hfill \\ \hfill & =2{\mathit{f}}^{\top }\mathit{D}\mathit{f}-2{\mathit{f}}^{\top }\mathit{W}\mathit{f}\hfill \\ \hfill & =2{\mathit{f}}^{\top }(\mathit{D}-\mathit{W})\mathit{f}\hfill \\ \hfill & =2{\mathit{f}}^{\top }\mathit{L}\mathit{f},\hfill \end{array}$$

(6)

where $\mathit{f}=(f_1,f_2,\dots ,f_N)$, and $\mathit{L}=\mathit{D}-\mathit{W}$. We call $\mathit{L}$ the Laplacian of the graph $\mathcal{G}$ [24,48].

4. Models and Algorithms

Based on the preliminaries mentioned above, this section introduces the cost-sensitive semi-supervised learning framework for ship detention prediction considering missing labels and class imbalance, namely the cost-sensitive Laplacian logistic regression (CosLapLR) framework. We first introduce the basic logistic regression model for supervised classification in Section 4.1 and expand this model by incorporating a graph regularization term and cost-sensitive techniques to develop a cost-sensitive semi-supervised learning framework in Section 4.2. Finally, we introduce performance metrics of classification models in Section 4.3.

4.1. Predictive Model

Recall that in Section 3.1 we have a dataset $S=\{({\mathit{x}}_1,y_1),\dots ,({\mathit{x}}_i,y_i),\dots ,({\mathit{x}}_n,y_n)\}$. Here, in this study, we denote by ${\mathit{x}}_i\in \mathcal{X}\subset {\mathbb{R}}^d$ the feature vector of an inspected ship i and $y_i\in \mathcal{Y}=\{\pm 1\}$ the corresponding binary detention outcome of ship i. For every feature vector ${\mathit{x}}_i,i=1,\dots ,n$, the detention outcome is either $y_i=1$ or $y_i=-1$, where 1 corresponds to a detention result and $-1$ otherwise. We use logistic regression models to predict detention probabilities. The discriminative model for logistic regression is as follows:

$$\mathbb{P}(y_i=\pm 1|{\mathit{x}}_i,\mathit{\beta })=\frac{1}{1+\exp \left(-y_i{\mathit{\beta }}^{\top }{\mathit{x}}_i\right)},$$

(7)

where $\mathit{\beta }\in {\mathbb{R}}^d$ is the parameter vector that we should estimate by maximum likelihood estimation (MLE) by minimizing the conditional negative log-likelihood as follows:

$$-\log \mathbb{L}\left(\mathit{\beta }\right)=-\log \prod _{i=1}^n\mathbb{P}(y_i=\pm 1|{\mathit{x}}_i,\mathit{\beta })=\sum _{i=1}^n\log \left(1+\exp \left(-y_i{\mathit{\beta }}^{\top }{\mathit{x}}_i\right)\right).$$

(8)

By scaling (8) of factor $1/n$ and assuming that $f\left({\mathit{x}}_i\right)=\mathit{\beta }{\mathit{x}}_i$, we can obtain the empirical risk minimization problem with logistic loss, given as $L\left(y_i,f\left({\mathit{x}}_i\right)\right)=\log \left(1+\exp \left(-y_i\mathit{\beta }{\mathit{x}}_i\right)\right)$, as follows:

$$\underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{n}\sum _{i=1}^n\log \left(1+\exp \left(-y_i{\mathit{\beta }}^{\top }{\mathit{x}}_i\right)\right).$$

(9)

Finally, to avoid training a complex and overfitting decision function, we add a data-independent regularization term, ${\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta }$, into (9) as follows:

$$\underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{n}\sum _{i=1}^n\log \left(1+\exp \left(-y_i{\mathit{\beta }}^{\top }{\mathit{x}}_i\right)\right)+{\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta },$$

(10)

where ${\lambda }_{\mathcal{H}}>0$ is a hyperparameter balancing the empirical risk term and the regularization term.

4.2. Cost-Sensitive Semi-Supervised Learning Framework

To consider missing data for ships that did not receive PSC inspections, we add a graph regularization in (10) to impose an intrinsic constraint requiring that the decision function $f\left(\mathit{x}\right)$ should be smooth in predicting the detention outcomes for ships with similar features. In addition to our existing dataset ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$, we assume that there is another dataset ${\left\{{\mathit{x}}_j\right\}}_{j=n+1}^{n+u}$ capturing the feature vectors of u uninspected ships. Because we do not know the marginal distribution from which uninspected ships are drawn, the empirical estimates of the geometric features in uninspected ships are represented as a graph. In this graph, there is a total of $n+u$ nodes corresponding to all ships. These nodes represent the inspected (labeled) and uninspected (unlabeled) ships, and edge weights between nodes represent pairwise relationships between ships. If ship i is among the k-nearest neighbors of ship j, or vice versa, where k is a hyperparameter, these two ships are connected by a weighted edge, whose assigned weight is denoted by $w_{ij}$ (the approach to determining the value of $w_{ij}$ for two neighbors i and j will be introduced in Section 5); for two ships i and j that are not among the neighbors of each other, we set $w_{ij}=0$. Therefore, by measuring the similarity of all ships (including inspected and uninspected ships), we can obtain a symmetric weighted adjacency matrix $\mathit{W}=\left({\left(w_{ij}\right)}_{i,j=1}^{n+u}\right)$ and the Laplacian matrix $\mathit{L}$ of this graph, denoted as $\mathit{L}=\mathit{W}-\mathit{D}$, where $\mathit{D}=\mathrm{diag}\left({\sum }_{j=1}^{n+u}{w}_{1j},\dots ,{\sum }_{j=1}^{n+u}{w}_{(n+u)j}\right)$ represents the degree matrix. Then, adding the graph regularization term into (10), we can obtain

$$\underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{n}\sum _{i=1}^n\log \left(1+\exp \left(-y_i{\mathit{\beta }}^{\top }{\mathit{x}}_i\right)\right)+{\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta }+{\lambda }_{\mathcal{G}}{\mathit{\beta }}^{\top }{\mathit{X}}_{n+u}^{\top }\mathit{L}{\mathit{X}}_{n+u}\mathit{\beta },$$

(11)

where ${\mathit{X}}_{n+u}=({\mathit{x}}_1,\dots ,{\mathit{x}}_n,{\mathit{x}}_{n+1},\dots ,{\mathit{x}}_{n+u})$ is an $(n+u)\times d$-dimensional matrix, and ${\lambda }_{\mathcal{H}}$ and ${\lambda }_{\mathcal{G}}$ represent the parameters that control the $\mathcal{H}$ norm and the intrinsic norm, respectively. Regarding (11), the log term measures the prediction error, the $\mathcal{H}$ norm term requires that the decision function should not be complex, and the Laplacian term (intrinsic norm) requires that the decision function should output similar detention outcomes for two ships with high similarity regardless of their inspection status.

From above analysis, we have illustrated how to consider uninspected ships in our prediction model. We now proceed to introduce how to address the issue of class imbalance in the ship detention prediction problem. Class imbalance is caused by the fact that among inspected ships, only a small proportion of ships are detained due to their identified serious deficiencies. The original formulation of (11) does not consider the class imbalance because it attaches the same costs of misclassifying a ship that should be detained and misclassifying a ship that should not be detained. However, in reality, the above two cases should be considered differently, i.e., attached with distinguished misclassification costs, because they have different outcomes and impacts. Specifically, if a ship with serious deficiencies is not detained and is regarded to be seaworthy, this ship is more likely to cause serious accidents while sailing, leading to possible economical and environmental damages. Conversely, if a ship without serious deficiencies is detained, it will cause a shipping delay to this ship. Generally, from a socially beneficial perspective, the overall possible cost of the former case is much larger than that of the latter case. To consider two different cases in the loss function, we introduce two partial losses, denoted as $L_1\left(f\left(\mathit{x}\right)\right)=\log \left(1+\exp (-{\mathit{\beta }}^{\top }\mathit{x})\right)$ and $L_{-1}\left(f\left(\mathit{x}\right)\right)=\log \left(1+\exp \left({\mathit{\beta }}^{\top }\mathit{x}\right)\right)$, to construct a cost-sensitive classification loss given by $L_{\varphi }:\{-1,1\}\times \mathbb{R}\to [0,\infty ]$ with cost parameter $\varphi \in (0,1)$ as:

$$L_{\varphi }=\varphi {\mathit{1}}_{\{y=1\}}\log \left(1+\exp \left(-{\mathit{\beta }}^{\top }\mathit{x}\right)\right)+(1-\varphi ){\mathit{1}}_{\{y=-1\}}\log \left(1+\exp \left({\mathit{\beta }}^{\top }\mathit{x}\right)\right),$$

(12)

where ${\mathit{1}}_{\{·\}}$ is an indicator function that equals 1 if the condition in the brackets is true and 0 otherwise. Accordingly, the cost-sensitive formulation of optimization problem (11) is as follows:

$$\begin{array}{c}\hfill \underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{n}\sum _{i=1}^n\left[\varphi {\mathit{1}}_{\{y_i=1\}}\log \left(1+\exp \left(-{\mathit{\beta }}^{\top }{\mathit{x}}_i\right)\right)+(1-\varphi ){\mathit{1}}_{\{y_i=-1\}}\log \left(1+\exp \left({\mathit{\beta }}^{\top }{\mathit{x}}_i\right)\right)\right]\\ \hfill +{\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta }+{\lambda }_{\mathcal{G}}{\mathit{\beta }}^{\top }{\mathit{X}}_{n+u}^{\top }\mathit{L}{\mathit{X}}_{n+u}\mathit{\beta }.\end{array}$$

(13)

In order to linearize the indicator function, the equivalent optimization problem of (12) is finally as follows:

$$\begin{array}{c}\hfill \underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{2n}\left[\varphi {(\mathit{1}+\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left(-{\mathit{X}}_n\mathit{\beta }\right)\right)+(1-\varphi ){(\mathit{1}-\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left({\mathit{X}}_n\mathit{\beta }\right)\right)\right]\\ \hfill +{\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta }+{\lambda }_{\mathcal{G}}{\mathit{\beta }}^{\top }{\mathit{X}}_{n+u}^{\top }\mathit{L}{\mathit{X}}_{n+u}\mathit{\beta },\end{array}$$

(14)

where ${\mathit{X}}_n=({\mathit{x}}_1,\dots ,{\mathit{x}}_n)$ is an $n\times d$-dimensional matrix, $\mathit{1}$ is an $n\times 1$-dimensional vector with elements 1, and $\mathit{y}=(y_1,\dots ,y_n)$ is an $n\times 1$-dimensional vector with labels $y_i,i\in \{1,\dots ,n\}$. In summary, the developed CosLapLR framework is given in Algorithm 1.

Algorithm 1: Cost-sensitive Laplacian Logistic Regression (CosLapLR).

Require:n inspected ships ${\left\{({\mathit{x}}_i,y_i)\right\}}_{i=1}^n$, u uninspected ships ${\left\{{\mathit{x}}_j\right\}}_{j=n+1}^{n+u}$. Ensure: Estimated decision function $f:{\mathbb{R}}^d\to \mathbb{R}$. 1: Step 1: Choose the hyperparameter k and construct the symmetric weighted adjacency matrix $\mathit{W}$ with $n+u$ nodes and compute edge weights $w_{ij}$ by k-nearest neighbors. 2: Step 2: Compute the Laplacian matrix $\mathit{L}=\mathit{D}-\mathit{W}$, where $\mathit{D}$ = diag $({\sum }_{j=1}^{n+u}{w}_{1j},\dots ,{\sum }_{j=1}^{n+u}{w}_{(n+u)j})$. 3: Step 3: Choose the regularization parameters ${\lambda }_{\mathcal{H}}$, ${\lambda }_{\mathcal{G}}$, and the cost parameter $\varphi$. 4: Step 4: Solve (14) to obtain ${\mathit{\beta }}^{*}$. 5: Step 5: Output function $f^{*}\left(\mathit{x}\right)={\mathit{\beta }}^{*\top }\mathit{x}$.

4.3. Performance Metrics

Common evaluation metrics for binary classification problems are true negative (TN), false positive (FP), false negative (FN), and true positive (TP) [3]. For classifiers used for balanced datasets, model accuracy, denoted as $acc$, is the most commonly used metric to evaluate the model performance, which is defined as follows:

$$acc=\frac{TP+TN}{TP+FP+FN+TN}.$$

(15)

However, this metric is not suitable to evaluate the prediction performance of classifiers developed for imbalanced and cost-sensitive learning. Because our aim is to obtain a higher identification rate for substandard ships that should be detained without greatly compromising the classification of ships that are not substandard, we mainly use sensitivity and specificity to evaluate the classification performance of classifiers [49]. Sensitivity, or true positive rate ($\frac{TP}{FN+TP}$), denoted as $sen$, represents the accuracy on the positive class; specificity ($\frac{TN}{FP+TN}$), or true negative rate, denoted as $spe$, represents the accuracy on negative class. Furthermore, based on the cost-sensitive perspective, we design a cost-sensitive evaluation metric, called decision loss (DL) metric, to evaluate the overall misclassification costs of a prediction model, which is related to the cost parameter $\varphi$ and the values of false negative (FN) and false positive (FP), shown as follows:

$$DL=\varphi \times FN+(1-\varphi )\times FP.$$

(16)

5. Computational Experiments

This section conducts extensive computational experiments to verify the superiority of the proposed cost-sensitive semi-supervised learning framework. We first introduce the dataset we use and the parameter settings in Section 5.1. We then draw trade-off curves to determine Pareto-optimal prediction models in Section 5.2.1 and then determine cost-effective models based on the minimum value of DL in Section 5.2.2.

5.1. Data Description

The dataset we use in this study consists of two parts, namely the labeled one and the unlabeled one. The labeled one contains 3026 records of PSC initial inspections during January 2015 to December 2019 period at the Hong Kong Port and the corresponding ship-related factors. Note that among 3026 inspections, only 100 inspections were labeled with detention outcomes. The PSC inspection records are retrieved from the Asia Pacific Computerized Information System (APCIS) (http://apcis.tmou.org/public/, accessed on 15 October 2022) provided by Tokyo MoU, and the ship-related factors are obtained from the World Shipping Register (WSR) (http://world-ships.com/, accessed on 15 October 2022) database. The unlabeled one contains 2675 records of incoming foreign ships during January 2021 to December 2021 at the Hong Kong port (the corresponding database only provides records starting from 12 September 2020) and the corresponding historical inspection information and ship-related factors. The records of incoming foreign ships are downloaded from the website of the Hong Kong government (https://data.gov.hk/en-data/dataset/hk-md-mardep-vessel-arrivals-and-departures, accessed on 15 October 2022), the corresponding historical inspection information is retrieved from the APCIS database, and the ship-related factors are obtained from the WSR database as well.

We consider 14 input features that are strongly related to the ship condition in the literature [3,5,28,30,32], including ship age, gross tonnage, length, depth, beam, type, ship flag performance, ship recognized organization performance, and ship company performance in Tokyo MoU, last PSC inspection time in Tokyo MoU, the number of ship deficiencies in the last inspection in Tokyo MoU, the number of detentions in all historical PSC inspections, the number of flag changes, and whether a ship has a casualty in last 5 years [3,5,28,30,32]. We follow the data preprocessing procedures of Yan et al. [31]. All continuous attributes were normalized to a mean of zero and a standard deviation of one. We use the scipy.optimize package (https://docs.scipy.org/doc/scipy/reference/optimize.html, accessed on 15 October 2022) in Python as the optimization solver.

We define the weight matrix $\mathit{W}$ by k-nearest neighbors for CosLapLR models as follows [24,47]:

$$w_{ij}=\left\{\begin{array}{c}{e}^{-\gamma ∥{\mathit{x}}_i-{\mathit{x}}_j∥},\mathrm{if}{\mathit{x}}_i,{\mathit{x}}_j\mathrm{are}\mathrm{neighbors}\hfill \\ 0,\mathrm{otherwise}\hfill \end{array}\right..$$

(17)

We assume that in practice the cost parameter $\varphi$ is predefined by the stakeholders (i.e., port authorities) who generally weigh the misclassification of substandard ships much higher than low-risk ships. We will conduct experiments for varying choices of cost parameter $\varphi$ to show its effect in Section 5.2.1. Therefore, under each cost parameter $\varphi$, there are four other hyperparameters in total in CosLapLR models, namely, $\gamma$, k, ${\lambda }_{\mathcal{H}}$, and ${\lambda }_{\mathcal{G}}$. Before training ML models, we randomly divide the labeled dataset into a training set (70%, 2118 records) and a test set (30%, 908 records). We adopt 3-fold cross-validation (CV) in the model training process. Following the literature [49,50], the nearest neighbor parameter k is chosen from set {3,5,7}, the weight parameter $\gamma$ from set $\left\{2^i\right|-11,-9,-7,-5,-3,-1,1,3,5\}$, and the regularization parameters ${\lambda }_{\mathcal{H}}$ and ${\lambda }_{\mathcal{G}}$ from set $\left\{2^i\right|-11,-9,-7,-5,-3,-1,1,3,5\}$ and set $\left\{2^i\right|-17,-15,-13,-11,-9,-7,-5,-3\}$, respectively.

5.2. Results

5.2.1. Pareto Frontier Graphs of Binary Classification Models

Recall that the aim of this study is to obtain a higher identification rate for substandard ships that should be detained without greatly compromising the classification of ships that are not substandard. Therefore, we draw trade-off curves to determine Pareto-optimal models based on sensitivity and specificity. Under Pareto optimality, a model is considered dominated if there is another model that has a higher sensitivity or a higher specificity [49]. For CosLapLR models, we draw Pareto frontier graphs consisting of non-dominated models for varying choices of the cost parameter $\varphi$ based on the 3-fold CV performance. We conduct experiments for $\varphi \in \{0.5,0.7,0.9,0.95,0.97,0.99\}$, whose Pareto frontier graphs are shown in Figure 1, respectively. Figure 1 shows that increasing $\varphi$ can improve sensitivity, namely the classification performance of substandard ships, without significantly sacrificing specificity, namely the classification performance of ships that are not substandard.

Furthermore, we also draw Pareto-frontier graphs of the following learning methods: LR, cost-sensitive logistic regression (CosLR), and Laplacian logistic regression (LapLR), whose learning problems are shown as follows:

[LR]

$$\begin{array}{c}\hfill \underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{2n}\left[{(\mathit{1}+\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left(-{\mathit{X}}_n\mathit{\beta }\right)\right)+{(\mathit{1}-\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left({\mathit{X}}_n\mathit{\beta }\right)\right)\right]+{\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta },\end{array}$$

(18)

[CosLR]

$$\begin{array}{c}\hfill \underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{2n}\left[\varphi {(\mathit{1}+\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left(-{\mathit{X}}_n\mathit{\beta }\right)\right)+(1-\varphi ){(\mathit{1}-\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left({\mathit{X}}_n\mathit{\beta }\right)\right)\right]\\ \hfill +{\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta },\end{array}$$

(19)

[LapLR]

$$\begin{array}{c}\hfill \underset{\mathit{\beta }\in {\mathbb{R}}^d}{min}\frac{1}{2n}\left[{(\mathit{1}+\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left(-{\mathit{X}}_n\mathit{\beta }\right)\right)+{(\mathit{1}-\mathit{y})}^{\top }\log \left(\mathit{1}+\exp \left({\mathit{X}}_n\mathit{\beta }\right)\right)\right]+{\lambda }_{\mathcal{H}}{\mathit{\beta }}^{\top }\mathit{\beta }\\ \hfill +{\lambda }_{\mathcal{G}}{\mathit{\beta }}^{\top }{\mathit{X}}_{n+u}^{\top }\mathit{L}{\mathit{X}}_{n+u}\mathit{\beta }.\end{array}$$

(20)

For these learning models, corresponding hyperparameters, such as $\gamma$, k, ${\lambda }_{\mathcal{H}}$, ${\lambda }_{\mathcal{G}}$, and $\varphi$, follow the same search sets stated above.

After observing the Pareto frontier graph of LR models and CosLR models, as shown in Figure 2, we can find that introducing a cost parameter into LR can improve the value of sensitivity by almost 0.1, leading to a higher classification rate of substandard ships. Furthermore, after observing the Pareto frontier graph of LR models and LapLR models, as shown in Figure 3, we can find that considering unlabeled data into LR can improve the value of sensitivity without greatly compromising the value of specificity. Therefore, the significance of our proposed method is verified.

5.2.2. Prediction Performance of Binary Classification Models

From the above analysis, we have shown from Pareto frontier graphs that considering unlabeled data can expand the coverage of the Pareto frontier, and introducing the cost parameter can improve the classification ability for substandard ships without greatly compromising the classification ability for low-risk ships. We further pick up the models with the lowest DL value in each binary classification framework (i.e., LR, LapLR, CosLR, and CosLapLR) and list their hyperparameters and prediction performance metrics in

Table 2. Prediction performance of binary classification models.

$\mathit{\varphi }$	Model	k	$\mathit{\gamma }$	${\mathit{\lambda }}_{\mathit{h}}$	${\mathit{\lambda }}_{\mathit{g}}$	$\mathit{TN}$${}^1$	$\mathit{FP}$${}^1$	$\mathit{FN}$${}^1$	$\mathit{TP}$${}^1$	$\mathit{acc}$${}^1$	$\mathit{spe}$${}^1$	$\mathit{sen}$${}^1$	$\mathit{DL}$${}^1$
0.5	LR	–	–	2	–	555	325	9	19	0.632	0.631	0.679	167.00
	LapLR	3	32	$2^{-13}$	$2^{-7}$	591	289	6	22	0.675	0.672	0.786	147.50
0.7	CosLR	–	–	1/2	–	555	325	9	19	0.632	0.631	0.679	103.80
	CosLapLR	7	8	$2^{-7}$	$2^{-5}$	591	289	6	22	0.675	0.672	0.786	90.90
0.9	CosLR	–	–	$2^{-5}$	–	554	326	8	20	0.632	0.630	0.714	39.80
	CosLapLR	3	32	$2^{-7}$	$2^{-3}$	570	310	6	22	0.652	0.648	0.786	36.40
0.95	CosLR	–	–	$2^{-13}$	–	583	297	5	23	0.667	0.663	0.821	19.60
	CosLapLR	7	2	$2^{-13}$	$2^{-17}$	582	298	5	23	0.666	0.661	0.821	19.65
0.97	CosLR	–	–	$2^{-13}$	–	576	304	6	22	0.659	0.655	0.786	14.94
	CosLapLR	7	32	$2^{-13}$	$2^{-7}$	581	299	5	23	0.665	0.660	0.821	13.82
0.99	CosLR	–	–	$2^{-13}$	–	575	305	6	22	0.657	0.653	0.786	8.99
	CosLapLR	3	32	$2^{-13}$	$2^{-5}$	554	326	5	23	0.635	0.630	0.821	8.21

¹TN: true negative; FP: false positive; FN: false negative; TP: true positive; acc: accuracy; spe: specificity; sen: sensitivity; DL: decision loss.

.

As shown in Table 2, from a broad sense, the average increasing ratios of specificity and sensitivity of six groups of experiments are 1.33% ($\frac{(0.672-0.631)+(0.672-0.631)+(0.648-0.630)+(0.661-0.663)+(0.660-0.655)+(0.630-0.653)}{6}\times 100\%=1.33\%$) and 5.93% ($\frac{(0.786-0.679)+(0.786-0.679)+(0.786-0.714)+(0.821-0.821)+(0.821-0.786)+(0.821-0.786)}{6}\times$$100\%=5.93\%$), respectively. This result verifies that considering unlabeled data in classification models can increase the classification ability for both minority and majority classes. Consequently, the value of DL decreases. Furthermore, by comparing the values of sensitivity and specificity of LR and CosLR with the increase in cost parameter $\varphi$, we find that more substandard ships are identified even without compromising the classification ability for low-risk ships, verifying the general benefits of introducing a cost parameter when considering unbalanced data. However, comparing the prediction performance of CosLR and CosLapLR models under each cost parameter, we find that the classification ability for low-risk ships declines with the consideration of unlabeled data when the cost parameter $\varphi$ increases above around 0.9. This is because that the second term in Equation (14) used for measuring the classification loss for low-risk ships may be unimportant when the cost parameter increases, compromising the effectiveness of unlabeled data on the classification for low-risk ships and decreasing the value of specificity. Therefore, to maximize the effect of unlabeled data and to consider the imbalance of two classes, $\varphi =0.95$ is a good choice for the studied problem.

The above results have shown that introducing a cost parameter to consider the class imbalance can increase the classification ability for substandard ships and considering unlabeled data can also present more information to learning algorithms. These results first inspire port states to not only collect information for those inspected ships but store all visiting ships’ information in the database. This transformation would enable better information collection for all foreign visiting ships, so as to facilitate the use of unlabeled data and the development of more advanced semi-supervised learning frameworks. Second, regarding the cost parameter $\varphi$, it can be further determined by evaluating the adverse impacts of two kinds of misclassifications by consulting more experts. While determining $\varphi$, policymakers can consider not only the economic impacts but also the environmental and safety impacts.

6. Conclusions and Future Research Directions

This paper develops a cost-sensitive semi-supervised learning framework, namely the CosLapLR learning framework, to predict the ship detention outcome in PSC inspections. Different from previous studies, this paper innovatively solves two important challenges arising from the unlabeled and imbalanced data in the studied problem. In order to consider the unlabeled data, we introduce a graph regularization term in the traditional LR model, which requires that the decision function outputs similar results for two ships having similar features regardless of their inspection status. Furthermore, to address the different misclassification costs for substandard ships and low-risk ships, we introduce a cost parameter. Using collected PSC data, we conduct extensive experiments to verify the superiority of the developed CosLapLR framework. First, by comparing Pareto frontier graphs of LR models, CosLR models, and CosLapLR models, we find that introducing a cost parameter into LR can improve the classification rate of substandard ships, and considering unlabeled data can greatly improve the value of sensitivity without compromising the value of specificity. Second, by comparing the prediction performance of models with the lowest DL value in each binary classification framework (i.e., LR, LapLR, CosLR, and CosLapLR), we suggest that when the cost parameter equals 0.95, the effect of the unlabeled data is maximized and the class imbalanced is fully considered.

Our study is the first to incorporate unlabeled data into the prediction of ship detention outcomes using the semi-supervised learning paradigm. Nowadays, there are many state-of-the-art ML methods, which may show higher prediction abilities than LR. Regarding different ML methods, introducing a cost parameter and considering unlabeled data in the original learning algorithms may require different techniques. Future research may examine how to consider unbalanced data and unlabeled data simultaneously in other ML methods, such as random forest and artificial neural networks, and compare them with our proposed framework. In addition, we can try more variants of the proposed framework. For example, in this paper, we assume a linear relationship between the input features and the output target. However, they may have a high-dimensional relationship in practice. Therefore, we could consider this feature by transforming the original LR into kernelized LR. Finally, the prediction results can be integrated into mathematical programming models [51,52,53,54,55,56] to generate ship inspector routing and scheduling decisions.