All three optimizers address the same constrained optimization problem, but differ in their computational strategy and intended role within the framework.
Backtracking and Dynamic Programming are employed as exact solvers within the discretized speed space, providing globally optimal solutions and serving as reference methods. Backtracking offers a transparent tree-based benchmark, while Dynamic Programming enables scalable exact optimization on a structured time lattice.
Proximal Policy Optimization is introduced as a learning-based alternative, aimed at approximating optimal speed-selection policies through repeated interaction with the physics-based voyage simulator.
3.4.1. Backtracking (Depth-First Search with Pruning)
Backtracking is employed as an exact optimization method over the discrete speed space
defined in
Section 3.3.6. The algorithm performs a depth-first search on a segment-wise decision tree, in which each level corresponds to one route segment and each branch represents the selection of a candidate speed:
- (a)
Principle
Unlike heuristic routing methods, Backtracking explicitly explores all feasible speed combinations unless eliminated by admissible pruning rules. It therefore provides a globally optimal solution within the discretized speed space and serves as a transparent reference solver against which the Dynamic Programming and reinforcement learning approaches are benchmarked.
Decision-tree formulations of this type are classical in discretized maritime speed optimization, ship energy management, and voyage planning problems, particularly when the objective function exhibits additive structure and monotonic cost accumulation [
3,
25,
30].
- (b)
Decision Structure and State Representation
The decision tree has depth , corresponding to the number of route segments. At level , the partial solution is defined by the fixed speed sequence: ).
For any such partial profile, the unified physics-informed model introduced in
Section 3.2 and
Section 3.3 is applied to evaluate the corresponding partial route quantities:
Partial travel time: ;
Partial accumulated CO2 emissions: ;
Partial economic and regulatory cost:
From these quantities, a partial diagnostic CII value can be computed consistently with Equation (17).
The root node (level 0 corresponds to an empty profile with zero accumulated time, emissions, and cost. A leaf node (level
) represents a complete feasible speed profile:
for which
and
are evaluated according to
Section 3.3.6.
At each expansion step, a candidate speed
and the corresponding incremental physical and economic quantities are computed. The partial solution is retained only if all hard constraints defined in
Section 3.3.5 are satisfied.
Decision-tree formulations of this type are classical for discretized maritime speed optimization problems [
25,
30]. The corresponding structure is illustrated in
Figure 3.
- (c)
Constraint Enforcement and Admissible Pruning
All feasibility constraints introduced in
Section 3.3.5 are enforced explicitly during node expansion:
ETA feasibility;
Installed propulsion power limit;
CII/CO2 cap (when activated);
Metocean operability constraints.
In addition, Backtracking employs admissible pruning rules that preserve global optimality while significantly reducing the explored search space:
- (d)
ETA Lower-Bound Pruning
An optimistic lower bound on the remaining travel time
(Equation (27)) is computed. If
the branch is discarded.
- (e)
Emission/CII Pruning
When a regulatory cap is enforced, any partial solution for which the implied route-level CII cannot satisfy (Equation (30)) is terminated:
- (f)
Power and Operability Pruning
Candidate speeds violating the installed power limit (Equation (29)) or the metocean operability filters are discarded locally and not expanded.
- (g)
Cost-based Branch-and-Bound
Since all incremental cost components defined in Equations (19)–(24) are non-negative, any partial solution satisfying the following:
cannot lead to a better solution and is pruned.
All pruning rules are admissible and therefore do not exclude any globally optimal solution.
- (h)
Optimality and Computational Properties
Because all feasible combinations in
are explored unless eliminated by admissible bounds, Backtracking yields the global optimum of the discretized problem defined in
Section 3.3.6.
Without pruning, the computational complexity is
In practice, the combined effect of ETA, emission, power, metocean, and cost pruning reduces the explored tree by several orders of magnitude. For short- and medium-length routes, the method remains computationally efficient and provides a reliable exact benchmark for validation and sensitivity analysis [
11,
28].
- (i)
Role within the Proposed Framework
Within the proposed hybrid framework, Backtracking fulfils three complementary roles:
- (i)
Exact reference solver for the unified optimization problem;
- (ii)
Validation benchmark for Dynamic Programming discretization and PPO learning outcomes;
- (iii)
Transparent audit tool, providing full per-segment traceability of speed, power, fuel, emissions, and cost, consistent with ISO 19030 and ITTC recommended practices.
3.4.2. Dynamic Programming (DP) on the Time Axis for the Discretized Speed Profile
Dynamic Programming (DP) is employed as an exact optimization method formulated on a discretized cumulative-time axis. In contrast to Backtracking, which explores the combinatorial decision tree explicitly, DP propagates optimal partial solutions over a structured lattice of states indexed by segment and cumulative time.
Under the same speed discretization
(
Section 3.1) and a chosen time-grid resolution
, DP guarantees global optimality within the adopted discretization, while operating under the identical physics-informed mapping (
Section 3.2 and
Section 3.3) and the same hard/soft constraint logic (
Section 3.3.5).
DP formulations of this type are widely used for discrete-control maritime speed optimization and routing under time and emission constraints [
25,
28].
Figure 4 illustrates the lattice-based propagation of optimal partial solutions and the parent-pointer reconstruction mechanism.
- (a)
Principle
DP relies on the principle of optimality: an optimal speed profile on the full route is composed of optimal partial profiles on its prefixes.
For each segment index and each admissible cumulative-time bucket, DP computes the minimum accumulated cost consistent with the selected objective (Equations (25) and (26)), while enforcing all hard constraints (Equations (28)–(30)).
Soft constraints do not restrict feasibility; instead, they enter exclusively through incremental costs including ETA penalties (Equations (23) and (24)) and FuelEU penalties, Equation (22).
- (b)
State Space Definition
Let denote the adopted time discretization step.
The DP lattice is defined on states: where is the segment index and is the index of the cumulative-time bucket .
The DP value table stores the minimum accumulated cost required to reach the end of segment
at cumulative-time bucket
:
Initialization and for
When a hard CII/CO2 cap is activated, an auxiliary cumulative-emission register consistent with Equations (12)–(18), tracking cumulative CO2 emissions up to segment , may be maintained for feasibility screening, without altering the emissions model.
- (c)
Transition Mechanism
From any reachable state , DP evaluates all candidate speeds for segment .
The continuous-time update is as follows:
This value is projected onto the discretized time grid:
- (d)
A transition is accepted only if all hard constraints are satisfied.
Using the same lower-bound estimate
(Equation (27)):
This is equivalent in meaning to Equation (28), applied at the transition level.
(Equation (29)).
Transitions are retained only if the cumulative emissions remain compatible with Equation (30), evaluated consistently with Equation (12)–(18).
Metocean restrictions are enforced during preprocessing (
Section 3.3.5). DP does not propagate through excluded segments or infeasible speeds.
All soft constraints enter exclusively through and therefore influence the objective but not the feasibility.
- (e)
Bellman Recursion
For each feasible transition, DP applies the Bellman update:
This recursion guarantees optimal substructure and preserves global optimality under discretization.
- (f)
Terminal Selection and Reconstruction
At the final segment
, the algorithm inspects all admissible time buckets satisfying
and selects:
This defines the optimal terminal state
The optimal speed profile is reconstructed through backward traversal of parent pointers:
The propagation and reconstruction logic are shown in
Figure 4.
- (g)
Computational Properties
DP has pseudo-polynomial complexity driven by the number of segments N and the time-grid resolution . Compared to Backtracking’s exponential scaling in DP trades exact combinatorial enumeration for structured lattice propagation.
This makes DP significantly more scalable for long routes or finer speed discretizations while preserving global optimality within the adopted discretization [
25,
28].
- (h)
Role Within the Framework
Within the hybrid framework, DP serves three complementary purposes:
3.4.3. Proximal Policy Optimization (PPO) for Segment-Wise Speed Selection
Proximal Policy Optimization (PPO) is introduced as a reinforcement learning (RL) approach for learning a segment-wise speed-selection policy within the same physics-informed environment used by the exact solvers (Backtracking and Dynamic Programming). As summarized in
Table 1, PPO differs from the exact methods in that it approximates a decision rule rather than enumerating the entire discrete decision space.
Unlike heuristic RL formulations, PPO operates strictly on the unified physical, economic, and regulatory model defined in
Section 3.2 and
Section 3.3. No surrogate propulsion laws, simplified emission models, or alternative constraint structures are introduced. This guarantees direct comparability with Backtracking and Dynamic Programming.
Figure 5 schematically illustrates the closed-loop interaction between the PPO agent and the physics-informed route simulator, highlighting the sequential decision structure over route segments.
- (a)
Reinforcement Learning Environment and MDP Formulation
The fixed-route speed optimization problem is formulated as a finite-horizon Markov Decision Process (MDP) [
45], where each episode corresponds to the traversal of a route discretized into
quasi-stationary segments (
Section 3.1).
State Representation
At segment
i the observation vector is as follows:
These variables correspond exactly to the segment descriptors defined in
Section 3.1.
The simulator internally maintains cumulative quantities (elapsed time, fuel consumption, CO2 emissions, CII, and total cost) computed using Equations (1)–(26). These quantities are not redefined for PPO and ensure consistency with the deterministic solvers.
- (b)
Action Space and Transition Dynamics
The action corresponds to the through-water speed on segment
:
where
is the same discrete speed set used by Backtracking and Dynamic Programming (
Section 3.1).
After selecting
, the environment transitions to state
and all physical and economic quantities are computed using the unified mapping defined in
Section 3.2 and
Section 3.3:
Propulsion power (Equation (4));
Fuel consumption (Equations (7)–(11));
CO2 emissions (Equations (12)–(15));
Incremental cost (Equations (19)–(24)).
No alternative transition model is introduced for PPO.
- (c)
To ensure fairness in benchmarking, all hard constraints are enforced via action masking, using identical feasibility logic to that employed as pruning rules in Backtracking and Dynamic Programming:
ETA feasibility (Equation (28));
Installed propulsion power limit (Equation (29));
Optional CII/CO2 cap (Equation (30));
Metocean operability constraints.
Infeasible speeds are removed from the admissible action set before sampling.
Soft constraints (ETA penalties and FuelEU penalties) are incorporated exclusively through the incremental cost and therefore affect the reward but not the feasibility.
This unified constraint enforcement guarantees methodological consistency across all optimization methods.
- (d)
Reward Design and Objective Alignment
To align PPO with the deterministic objective functions (Equations (25) and (26)), the per-step reward is defined as follows:
where
is the incremental cost consistent with Equations (19)–(24), including any allocated route-level penalties, and
κ is a scaling constant (e.g.,
introduced for numerical stability.
Under feasibility-based action masking, the reward remains well shaped and does not require artificial penalty terms.
- (e)
PPO Objective and Learning Mechanism
The PPO learning rule follows the canonical actor–critic structure, in which the policy is updated using a clipped surrogate objective while a separate value function estimates the expected return.
The PPO algorithm follows the clipped surrogate objective proposed by Schulman et al. [
13]. The clipped objective is as follows:
with probability ratio:
The full actor–critic loss includes a value-function regression and regularization:
Advantages are estimated using Generalized Advantage Estimation (GAE) [
47]:
The discount factor (e.g., 0.99) and (e.g., 0.95), reduce variance while preserving stability.
The implementation follows the Stable-Baselines3 library [
54], ensuring transparent and reproducible training.
- (f)
Policy Approximation and Network Architecture
Both the policy
and value function
are implemented as multilayer perceptrons with normalized inputs and categorical output over
. Network architectures and hyperparameters are reported in
Supplementary Files—File S6, while implementation details are documented in
Supplementary Files—File S3.
- (g)
Training Setup and Reproducibility Protocol
The PPO agent was implemented using the Stable-Baselines3 framework [
54] and trained within the same deterministic, physics-informed simulation environment employed by the exact solvers. No surrogate models or simplified transition dynamics were introduced.
All training hyperparameters are explicitly reported in
Supplementary Files—File S6, including learning rate, discount factor (γ), GAE parameter (λ), clipping parameter (ε), rollout length (n_steps), batch size, number of optimization epochs, entropy coefficient, total training timesteps, and network configuration. The default Stable-Baselines3 MLP actor–critic architecture was used.
A fixed random seed was adopted to ensure reproducibility of both training and policy evaluation. No route-specific hyperparameter tuning was performed. All PPO training experiments were conducted using the same deterministic simulator configuration and fixed random seed, ensuring full reproducibility of the reported optimization results.
The environment dynamics are fully deterministic, and infeasible actions are removed through strict feasibility-based action masking. Under these conditions, the PPO training process exhibits stable convergence behavior for the reported configuration.
The performance indicators presented in
Section 6 correspond to the fixed training configuration defined in
Supplementary Files—File S6 (including the reported hyperparameters and fixed random seed) and are directly comparable with the deterministic solutions obtained via Backtracking and Dynamic Programming.
This setup establishes a transparent and reproducible benchmarking framework for evaluating the learning-based optimizer within the proposed hybrid methodology.
- (h)
Evaluation and Role within the Hybrid Framework
Backtracking and Dynamic Programming provide globally optimal solutions within the discretized decision space and serve as deterministic reference benchmarks.
PPO is evaluated under identical conditions:
Numerical results (
Section 6) show that PPO consistently produces solutions that are numerically close to the exact DP solutions in terms of fuel consumption and emissions, while often achieving shorter voyage durations through smoother segment-wise speed modulation. These results indicate empirical near-optimal performance within the discretized space, without claiming theoretical optimality.
Backtracking acts as a transparent exact benchmark;
Dynamic Programming provides scalable exact optimization;
PPO enables scalable policy inference after offline training, allowing rapid scenario analysis under varying economic and regulatory conditions.
As summarized in
Table 1, PPO differs from the exact solvers not in the underlying physical or economic model, but in the computational strategy, replacing exhaustive enumeration with learned policy approximation.