Lashing Force Prediction Model with Multimodal Deep Learning and AutoML for Stowage Planning Automation in Containerships

The calculation of lashing forces on containerships is one of the most important aspects in terms of cargo safety, as well as slot utilization, especially for large containerships such as more than 10,000 TEU (Twenty-foot Equivalent Unit). It is a challenge for stowage planners when large containerships are in the last port of region because mostly the ship is full and the stacks on deck are very high. However, the lashing force calculation is highly dependent on the Classification society (Class) where the ship is certified; its formula is not published and it is different per each Class (e.g., Lloyd, DNVGL, ABS, BV, and so on). Therefore, the lashing result calculation can only be verified by the Class certified by the Onboard Stability Program (OSP). To ensure that the lashing result is compiled in the stowage plan submitted, stowage planners in office must rely on the same copy of OSP. This study introduces the model to extract the features and to predict the lashing forces with machine learning without explicit calculation of lashing force. The multimodal deep learning with the ANN, CNN and RNN, and AutoML approach is proposed for the machine learning model. The trained model is able to predict the lashing force result and its result is close to the result from its Class.


Consideration of Stowage Planning
Stowage planning is a highly complex process with the goal to achieve cost efficiency and safety of crews and containership at the same time. This is done by ensuring that containers are loaded in the appropriate places on the containership, with consideration of the infrastructure limitation of all terminals in the round trip port rotation of the subject vessel, container composition to be loaded at each terminal, necessary segregations of the dangerous goods cargo, adherence to navigation visibility requirement, maximum number of cranes that can work concurrently, fulfilment of special stowage requirements from shippers, safety of containers and vessels, such as stability, strength, lashing, etc. In Figure 1, more considerations are categorized by the goal of stowage planning.

Literature Review
Ding [1] and Avriel [2,3] studied and developed heuristic algorithms for automated stowage planning in order to reduce a number of restows/shifts which are categorized as "Low Cost". Low [4] proposed and developed a system with consideration for crane intensity and a number of rehandles (restows/shifts), which are categorized as "Low Cost", and stability, categorized as "Safety". Ambrosino [5] studied the master bay plan problem (MBPP) with the Linear Programming model and also presented a heuristic approach to relax and solve the combinatorial optimization problem. This MBPP is related to "Space Utilization" of "High Revenue" and "Low Overstow" and "Low Restow/Shift" of "Low Cost". Korach [6] studied an efficient mathematical programming technique within a heuristic framework for the slot planning problem which is categorized as "High Revenue", especially for the "DC HC (normal or high cubic container) mix under deck" case. Rahsed [7] applied a rule-based greedy algorithm to solve the unnecessary Restow/Shift movement which is related to the "Low Restow/Shift" of "Low Cost" category. Shen [8] introduced the Deep Q-Learning Network (DQN) as a model to solve the stowage planning problem, and this study showed the possibility to apply Machine Learning, Deep Learning or Reinforcement Learning for stowage planning. The introduced features are mostly under "High Revenue" and "Low Cost". Rathje [9] introduced the new lashing rule of Germanischer Lloyd (GL) to offer containership operators more flexibility in on-deck container stowage without compromising safety. However, there is no study on Lashing Forces, taken into consideration stowage planning automation for the "Safety" category despite the importance of lashing forces on large containerships.

Lashing in Containership
As the containerships become larger, container stacks on deck become higher. Today, the largest containerships in the world can carry as many as 23,964 Twenty-foot Equivalent Units (TEUs) [10]. Back in the 1950s, the first generation of containerships had only two tiers on deck. Today, containerships are routinely carrying containers on deck up to eleven (11) tiers high. As such, lashing on containers becomes increasingly important. The lashing is the securing arrangements onboard to prevent containers from moving from their places or falling off into the sea when the vessel is in motion, especially during rough weather. Its effectiveness is measured by the magnitude of the various forces that act on

Literature Review
Ding [1] and Avriel [2,3] studied and developed heuristic algorithms for automated stowage planning in order to reduce a number of restows/shifts which are categorized as "Low Cost". Low [4] proposed and developed a system with consideration for crane intensity and a number of rehandles (restows/shifts), which are categorized as "Low Cost", and stability, categorized as "Safety". Ambrosino [5] studied the master bay plan problem (MBPP) with the Linear Programming model and also presented a heuristic approach to relax and solve the combinatorial optimization problem. This MBPP is related to "Space Utilization" of "High Revenue" and "Low Overstow" and "Low Restow/Shift" of "Low Cost". Korach [6] studied an efficient mathematical programming technique within a heuristic framework for the slot planning problem which is categorized as "High Revenue", especially for the "DC HC (normal or high cubic container) mix under deck" case. Rahsed [7] applied a rule-based greedy algorithm to solve the unnecessary Restow/Shift movement which is related to the "Low Restow/Shift" of "Low Cost" category. Shen [8] introduced the Deep Q-Learning Network (DQN) as a model to solve the stowage planning problem, and this study showed the possibility to apply Machine Learning, Deep Learning or Reinforcement Learning for stowage planning. The introduced features are mostly under "High Revenue" and "Low Cost". Rathje [9] introduced the new lashing rule of Germanischer Lloyd (GL) to offer containership operators more flexibility in on-deck container stowage without compromising safety. However, there is no study on Lashing Forces, taken into consideration stowage planning automation for the "Safety" category despite the importance of lashing forces on large containerships.

Lashing in Containership
As the containerships become larger, container stacks on deck become higher. Today, the largest containerships in the world can carry as many as 23,964 Twenty-foot Equivalent Units (TEUs) [10]. Back in the 1950s, the first generation of containerships had only two tiers on deck. Today, containerships are routinely carrying containers on deck up to eleven (11) tiers high. As such, lashing on containers becomes increasingly important. The lashing is the securing arrangements onboard to prevent containers from moving from their places or falling off into the sea when the vessel is in motion, especially during rough weather. Its effectiveness is measured by the magnitude of the various forces that act on containers, comparing against their limits and displayed in a percentage. Each Classification society has a slightly different method of measurement. In severe sea conditions, as well as in the case of improperly stowed containers and overweight containers, these forces may become excessive, causing, for example, failure of twist locks or collapse of lower-stacked containers. Consequently, whole container stacks may collapse and go overboard, which is not just an economic issue but also an issue for safe passageway, as these containers may be floating on the sea surface. Besides this, deck containers may be loaded with dangerous goods. Thus, containers going overboard also pose significant environmental implications [11]. According to the report of the World Shipping Council, the industry loses as many as 10,000 containers a year at sea [12].
The calculation of lashing forces is one of the important aspects in terms of cargo safety as well as slot utilization. It is solely dependent on the Classification society (Class), that the ship is certified under. Unlike other calculations such as Stability, Strength and DG (Dangerous Goods) check, the lashing calculation formula is not published and differs from Class to Class (e.g., Lloyd, DNVGL, ABS, BV, and so on). The stowage planner has to rely on the Class-certified Onboard Stability Program (OSP) to ensure his/her stowage plan is lashing compliant as described by the process in Figure 2. containers, comparing against their limits and displayed in a percentage. Each Classification society has a slightly different method of measurement.
In severe sea conditions, as well as in the case of improperly stowed containers and overweight containers, these forces may become excessive, causing, for example, failure of twist locks or collapse of lower-stacked containers. Consequently, whole container stacks may collapse and go overboard, which is not just an economic issue but also an issue for safe passageway, as these containers may be floating on the sea surface. Besides this, deck containers may be loaded with dangerous goods. Thus, containers going overboard also pose significant environmental implications [11]. According to the report of the World Shipping Council, the industry loses as many as 10,000 containers a year at sea [12].
The calculation of lashing forces is one of the important aspects in terms of cargo safety as well as slot utilization. It is solely dependent on the Classification society (Class), that the ship is certified under. Unlike other calculations such as Stability, Strength and DG (Dangerous Goods) check, the lashing calculation formula is not published and differs from Class to Class (e.g., Lloyd, DNVGL, ABS, BV, and so on). The stowage planner has to rely on the Class-certified Onboard Stability Program (OSP) to ensure his/her stowage plan is lashing compliant as described by the process in Figure 2.

Machine Learning in Lashing of Containership
As a trend of Machine Learning (ML), especially for Deep Learning nowadays, the idea is that ML can fulfil the needs of stowage planners to get the lashing force values. Instead of calculating the lashing forces by navel architecture engineering, this study proposes multimodal deep learning with ANN, CNN and RNN to train machines to predict the lashing forces.
As illustrated in Figure 3, the idea is that the stowage plan, consisting of stowage (e.g., container weight, height, slot position, etc.), condition (e.g., GM, Wind Speed, Roll Angle), and containership structure (e.g., bays, rows, tiers), is given to one of the appropriate ML models, trained per each Class (e.g., DNVGL, ABS, Lloyd, BV, etc.), and its ML Model predicts and returns Lashing Forces as a result during the stowage planning in the stowage planning tool. Without relying on OSP, stowage plans can be generated within the same system swiftly, making lashing forces compliant. This study proposes the Multimodal Deep Learning [13] model with AutoML [14][15][16] approach to predict Lashing Forces as a part of the process of stowage planning automation.

Machine Learning in Lashing of Containership
As a trend of Machine Learning (ML), especially for Deep Learning nowadays, the idea is that ML can fulfil the needs of stowage planners to get the lashing force values. Instead of calculating the lashing forces by navel architecture engineering, this study proposes multimodal deep learning with ANN, CNN and RNN to train machines to predict the lashing forces.
As illustrated in Figure 3, the idea is that the stowage plan, consisting of stowage (e.g., container weight, height, slot position, etc.), condition (e.g., GM, Wind Speed, Roll Angle), and containership structure (e.g., bays, rows, tiers), is given to one of the appropriate ML models, trained per each Class (e.g., DNVGL, ABS, Lloyd, BV, etc.), and its ML Model predicts and returns Lashing Forces as a result during the stowage planning in the stowage planning tool. Without relying on OSP, stowage plans can be generated within the same system swiftly, making lashing forces compliant. This study proposes the Multimodal Deep Learning [13] model with AutoML [14][15][16] approach to predict Lashing Forces as a part of the process of stowage planning automation.

Idea and Process
The process of stowage planning automation with lashing force prediction is depicted as follows.
1. As part of auto stowage planning process, the stowage planning tool slots containers on deck. 2. The conditions, as input parameters for lashing force prediction, are set from both the stability result (e.g., GM, Draft, Trim, etc.; subject to Class), calculated by the stowage planning tool and inputted values (e.g., Wind Speed, Roll Angle, etc.; subject to Class) by the stowage planner. 3. The stowage system requests the lashing force result for one of the embedded lashing force prediction models, trained per each Class. 4. The model returns the lashing force percentage for each lashing component. 5. If any of the returned lashing force values is greater than 100%, the stowage planner or stowage planning tool changes the containers with lighter ones and repeats from step no. 2.
A 10,000 TEU containership, belonging to one of biggest shipping lines, has been chosen and her real life, fully loaded stowage, especially On-Deck for the last port of region, has been selected as the input to train the above-mentioned model, as depicted in Figure 4. Almost all Rows in each On-Deck of Bay are fully loaded up to capacity. The lashing force percentage in this stowage is close to 100%. The Classification society is ABS and the lashing rule is the In-House Lashing Rule.

Idea and Process
The process of stowage planning automation with lashing force prediction is depicted as follows.

1.
As part of auto stowage planning process, the stowage planning tool slots containers on deck.

2.
The conditions, as input parameters for lashing force prediction, are set from both the stability result (e.g., GM, Draft, Trim, etc.; subject to Class), calculated by the stowage planning tool and inputted values (e.g., Wind Speed, Roll Angle, etc.; subject to Class) by the stowage planner. 3.
The stowage system requests the lashing force result for one of the embedded lashing force prediction models, trained per each Class. 4.
The model returns the lashing force percentage for each lashing component.

5.
If any of the returned lashing force values is greater than 100%, the stowage planner or stowage planning tool changes the containers with lighter ones and repeats from step no. 2.
A 10,000 TEU containership, belonging to one of biggest shipping lines, has been chosen and her real life, fully loaded stowage, especially On-Deck for the last port of region, has been selected as the input to train the above-mentioned model, as depicted in Figure 4. Almost all Rows in each On-Deck of Bay are fully loaded up to capacity. The lashing force percentage in this stowage is close to 100%. The Classification society is ABS and the lashing rule is the In-House Lashing Rule.

Containership Structure
As illustrated in Error! Reference source not found., the structure of containerships is well standardized because the container itself is standardized with several dimensional types (e.g., commonly 20 ft or 40 ft in length and normal or high cubic in height). Generally, one Hatch consists of two physical 20 ft bays (depicted Bay 25 and Bay 26) and one logical 40 ft bay (depicted Bay 26). This means that two 20 ft containers or one 40ft can be stacked in one slot. The bay consists of Rows and Tiers as in the table, and each square is called Slot. One Bay is divided into Under Deck and On Deck, and the lashing is needed On Deck only. There is a big number of Slot differences in the Bay between the small and large containerships. Typically, the size of the dimension needs to be fixed in order to train the machine, therefore, the maximum size of the dimension is defined by 26 Rows (horizontal) and 13 Tiers (vertical) which are able to accommodate the largest containership in the world. Since lashing forces are independent per each Hatch with the given ship level condition, such as GM, in this study, one dataset is defined by 1 Hatch (3 Bays) and On Deck. Each slot is presented by three-dimension array where; (1)

Containership Structure
As illustrated in Figure 5, the structure of containerships is well standardized because the container itself is standardized with several dimensional types (e.g., commonly 20 ft or 40 ft in length and normal or high cubic in height). Generally, one Hatch consists of two physical 20 ft bays (depicted Bay 25 and Bay 26) and one logical 40 ft bay (depicted Bay 26). This means that two 20 ft containers or one 40ft can be stacked in one slot. The bay consists of Rows and Tiers as in the table, and each square is called Slot. One Bay is divided into Under Deck and On Deck, and the lashing is needed On Deck only. There is a big number of Slot differences in the Bay between the small and large containerships. Typically, the size of the dimension needs to be fixed in order to train the machine, therefore, the maximum size of the dimension is defined by 26 Rows (horizontal) and 13 Tiers (vertical) which are able to accommodate the largest containership in the world. Since lashing forces are independent per each Hatch with the given ship level condition, such as GM, in this study, one dataset is defined by 1 Hatch (3 Bays) and On Deck. Each slot is presented by three-dimension array S brt where; (1)

Features
Lashing forces are calculated with container stacking profiles and containership structures. The following six features are proposed to represent the factors that influence lashing forces, illustrated in Error! Reference source not found..  In addition to the container stacking profiles on each Bay, there are vessel conditions that influence lashing forces. The following three conditions are extracted and modelled as auxiliary ANN for the multimodal modelling.

Features
Lashing forces are calculated with container stacking profiles and containership structures. The following six features are proposed to represent the factors that influence lashing forces, illustrated in Figure 6. In addition to the container stacking profiles on each Bay, there are vessel conditions that influence lashing forces. The following three conditions are extracted and modelled as auxiliary ANN for the multimodal modelling.
• GM (Metacentric Height)-this is the result condition when stability is calculated.

Lashing Force
As illustrated in Figure 7, there are 10 lashing force components per each Row and two of them, the Lashing Special Corner H Forces and Lashing Special Corner V Forces, are not applicable for this containership. These 10 values are answer labels for train and test data.

Lashing Force
As illustrated in Figure 7, there are 10 lashing force components per each Row and two of them, the Lashing Special Corner H Forces and Lashing Special Corner V Forces, are not applicable for this containership. These 10 values are answer labels for train and test data.

Dataset
Fully loaded stowage for thelast port of region is selected as the input to train as depicted in Figure 4. The lashing force percentage in this stowage is close to 100%, so this stowage is used as a baseline dataset. Since the prediction model is supervised learning, the label is needed for every dataset and the label comes from OSP. Therefore, over 100,000 training datasets are generated by interface between Stowage Planning Tool and OSP represented both in the following strategy and in Table 1 In this study, Multimodal Deep Learning is applied with Artificial Neural Network (ANN), Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN). It is common nowadays to adopt multimodal to predict results more accurately; for instance, Video-Audio input to recognize human emotion [17]. First of all, the vessel conditions are used as the auxiliary input of the ANN. General deep learning network, as ANN, is adopted for the stowage plan because each slot position itself can be considered as a meaningful feature. In addition, the Bay structure, presented in the Stowage Planning Tool-as illustrated in Figure 5-is already very similar as an image, i.e., 26 × 13 pixels with six features as channels; therefore, CNN is adopted as one of the inputs. Additionally, the lashing force is affected by the adjacent Rows because the outer row can protect the wind force to the inner row. This means that the sequence of the stacked container might impact the lashing force values for the next rows. This is the reason why the RNN model is used in this study. The features described in Section 2.2.2 are used for all ANN, CNN and RNN models, except for the auxiliary model.
As illustrated in Figure 8, the first auxiliary input is one dimension to accommodate the ship conditions, GM, Wind Speed and Roll Angle. The second ANN input is four dimensions to represent Bays, Tiers, Rows and Features. The third input is three dimensions to represent Tiers, Rows and Bays × Features as Channels of the CNN input. The last input is two dimensions to represent Tiers and Bays × Rows × Features as nodes of the RNN input.

Training
In total, 80% from all datasets are used for training. The specification of the training machine is: The configuration parameters for modeling and fitting are depicted in Table 2 and the training result of the best model, together with validation, is described in Table 3. The validation loss, normalized MSE, is 0.0023243355099111795, which gives very good results and the train and validation MSE and test MSE, scaled MSE, are 0.8656454525367441 and 0.8769041770067165. In consideration of the percentage values for the lashing force of components, less than 1 for the MSE indicates the variance is about 1%. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on these data at the end of each epoch The best hyperparameters to be found during the training in the AutoML approach are described in Table 4.

Training
In total, 80% from all datasets are used for training. The specification of the training machine is: The configuration parameters for modeling and fitting are depicted in Table 2 and the training result of the best model, together with validation, is described in Table 3. The validation loss, normalized MSE, is 0.0023243355099111795, which gives very good results and the train and validation MSE and test MSE, scaled MSE, are 0.8656454525367441 and 0.8769041770067165. In consideration of the percentage values for the lashing force of components, less than 1 for the MSE indicates the variance is about 1%. The model will set apart this fraction of the training data, will not train on it, and will evaluate the loss and any model metrics on these data at the end of each epoch The best hyperparameters to be found during the training in the AutoML approach are described in Table 4.
The training curve in Figure 9 depicts the learning curve of the training and validation in the best model. Within small steps, the loss became significantly reduced.

Testing
In total, 20% of all datasets are used for testing with the best model and the overall test result is illustrated in Figure 10. The test result of each lashing force component is illustrated in Figure 11. The X axis is the label lashing results from OSP and the Y axis is the predicted lashing force value from the best model.

Testing
In total, 20% of all datasets are used for testing with the best model and the overall test result is illustrated in Figure 10. The test result of each lashing force component is illustrated in Figure 11. The X axis is the label lashing results from OSP and the Y axis is the predicted lashing force value from the best model.

Testing
In total, 20% of all datasets are used for testing with the best model and the overall test result is illustrated in Figure 10. The test result of each lashing force component is illustrated in Figure 11. The X axis is the label lashing results from OSP and the Y axis is the predicted lashing force value from the best model.

Result Evaluation
After embedding the trained model into the Stowage Planning tool, the lashing forces result from OSP and the predicted result from the trained model has been compared with the new fully loaded stowage. Figure 12 describes the vessel top view of the entire Bay and the highest lashing force value of each Row is presented. If the lashing value is over 100%, then the red color is depicted; over 90%, the yellow color is presented; otherwise the green color is displayed. Overall, the color patterns of two results are pretty similar. The detail lashing force values are listed in Table 5

Result Evaluation
After embedding the trained model into the Stowage Planning tool, the lashing forces result from OSP and the predicted result from the trained model has been compared with the new fully loaded stowage. Figure 12 describes the vessel top view of the entire Bay and the highest lashing force value of each Row is presented. If the lashing value is over 100%, then the red color is depicted; over 90%, the yellow color is presented; otherwise the green color is displayed. Overall, the color patterns of two results are pretty similar. The detail lashing force values are listed in Table 5 and each value is different between OSP and the predicted result in the percentage scale. From the stowage planner point of view, normally an overall 5% variance is acceptable and manageable during the planning process. Therefore, the results that the trained model predicted are acceptable. Figure 12. Result comparison between OSP interface and model prediction in graphical top overview. The red color means that the lashing force value is over 100%, the yellow color is over 90%, otherwise the green color is depicted.  Figure 12. Result comparison between OSP interface and model prediction in graphical top overview. The red color means that the lashing force value is over 100%, the yellow color is over 90%, otherwise the green color is depicted.

Discussion
In this study, we consider the calculation of lashing forces on containerships to be one of the most important aspects in terms of cargo safety, as well as slot utilization, especially for large containerships. This study defines the idea and process for the lashing force prediction in stowage planning; extracts the features from stacked profiles in containership structures; prepares datasets to train, validate, and test the model; models Multimodal Deep Learning with ANN, CNN, and RNN; trains with the AutoML approach. This trained model predicts the lashing forces without an explicit calculation of the lashing force, and the result of it is acceptable and workable.
We think that the proposed approach is valuable in terms of stowage planning automation, and one of the future directions of this study should be to extend to other Classes (e.g., Lloyd, DNVGL, ABS, BV, and so on) by training with the different datasets of each Class.
Author Contributions: The idea for this paper was conceived by all three authors, C.L. and M.K.L. conducted the analysis and wrote the most of the text. C.L. and J.Y.S. developed, tested, and evaluated the model together, all authors commented, polished and agreed on the final manuscript. All authors have read and agreed to the published version of the manuscript.