Assessment Framework of Smart Shipyard Maturity Level via Data Envelopment Analysis

Abstract

The fourth industrial revolution (“Industry 4.0”) has caused an escalating need for smart technologies in manufacturing industries. Companies are examining various cutting-edge technologies to realize smart manufacturing and construct smart factories and are devoting efforts to improve their maturity level. However, productivity improvement is rarely achieved because of the large variety of new technologies and their wide range of applications; thus, elaborately setting improvement goals and plans are seldom accomplished. Fortunately, many researchers have presented guidelines for diagnosing the smartness maturity level and systematic directions to improve it, for the eventual improvement of productivity. However, most research has focused on mass production industries wherein the overall smartness maturity level is already high (e.g., high-level automation). These studies thus have limited applicability to the shipbuilding industry, which is basically a built-to-order industry. In this study, through a technical demand survey of the shipbuilding industry and an investigation of existing smart manufacturing and smart factories, the keywords of connectivity, automation, and intelligence were derived and based on these keywords, we developed a new diagnostic framework for smart shipyard maturity level assessment. The framework was applied to eight shipyards in South Korea to diagnose their smartness maturity level, and a data envelopment analysis (DEA) was performed to confirm the usefulness of the diagnosis results. By comparing the DEA models, the results with the smart level as an input represents the actual efficiency of shipyards better than the results of conventional models.

1. Introduction

Owing to changes in the global manufacturing environment, the shipbuilding industry currently faces challenges of survival and sustainability [1]. The global economic recession has induced shrinking international trade, while marine resource development projects are also decreasing or being canceled owing to declining oil prices, which are definitely unfavorable conditions for the shipbuilding and marine industry [2]. Analysts predict that it will be difficult to return to the early boom period of orders for ships and offshore plants, such as around 2010, which is referred to as the “super cycle” in the shipbuilding industry. Under these circumstances, shipyards worldwide are devoting various efforts to establish corporate strategies for the future. One such strategy is the active adoption of smart manufacturing technology, which numerous shipbuilding companies are considering [3,4].

Global manufacturing companies are expressing great interest in smart manufacturing and smart factories, with the goal of unmanned production through automation and connectivity among enterprise resources with the help of the IoT. The application of the smart factory concept is also accelerating with the adoption of Industry 4.0 in Germany. This concept is expanding with the combination of information and communications technology and automation solutions throughout the entire production process [5,6,7].

Although manufacturing companies seek to enhance the level of smart manufacturing through surveys on the latest technologies related to smart factories, adopting smart technologies often does not lead to practical effects (e.g., improved productivity). In fact, inelegantly applying smart technologies without sufficient accurate and practical analysis may adversely affect the existing production system. To solve this problem, a multidimensional and quantitative assessment of the smartness maturity level of manufacturing companies should be performed beforehand [8,9]. However, beyond simple products, it is necessary to consider both quantitative and qualitative criteria and interdependence in a complex manufacturing environment. Therefore, appropriate assessment of the manufacturing level of companies has long been an important issue. With rising interest in smart factories, researchers have proposed various definitions and criteria related to smart factories depending on the scale and type of the manufacturing company. Nevertheless, research on the assessment of smart factories is limited.

This growing emphasis on the importance of smart factories has led scholars to re-examine prior literature on the development of production systems and continuously publish new studies on the definition of a smart factory and the assessment of manufacturing systems. However, since most research has been focused on the general manufacturing industry, it has limited direct applicability to shipyards, which possess different industry characteristics. Basically, shipbuilding is a built-to-order industry. The products of the shipbuilding industry, i.e., ships—consists of several millions of similar but mostly unique intermediate products. At the early stages of production, the similarity among intermediate products is high enough to automate the processes, but the similarity plummets as the production progresses. Therefore, mostly, the automation level of shipyards are relatively low and disproportionally biased to earlier stages [10].

To evaluate smart shipyard and maturity level (SSML) assessment methods, the literature has been extensively surveyed. First, several studies examined smart factories in the manufacturing industry [6,7,11] Furthermore, studies on maturity level assessment of the manufacturing system can be largely divided into studies on evaluation of the manufacturing system in its current state and those on the evaluation of future manufacturing systems aimed at smart factories. In this regard, there have been several case studies on the evaluation of the manufacturing system in its current state [12,13,14,15,16]. In addition, since the advent of smart manufacturing owing to the fourth industrial revolution (4IR), maturity level assessment models for evaluating future manufacturing systems have been studied since 2013 [17,18,19,20]. Detailed descriptions of the literature on the definition of smart factories and on manufacturing system assessment are shown in Appendix A.

In South Korea, the Korea Production System (KPS) was developed, which is suitable for the South Korean manufacturing environment across representative manufacturing sectors such as automotive and consumer goods. [21]. Furthermore, following the development of KPS, to successfully promote the spread of smart factories, the Smart Manufacturing Innovation Planning Division of the Bureau of small and medium-sized enterprises (SMEs) developed a diagnostic tool that can represent plans for smart factory construction, with the goal of objectively diagnosing and assessing the smart maturity level of the manufacturing industry [22]. In this study, they derived the criteria and modules for smart factory assessment from the framework for the smart factory operation system (vision, goal, enterprise, factory, machine, and control). In addition, a questionnaire for the assessment items comprising each module was defined according to the maturity level definitions. Detailed configurations of KPS and smart factory assessment modules are shown in Appendix B.

The smart factory assessment module is a diagnostic tool suitable for the general machinery sector and is actively used by South Korean manufacturing companies [22]. Accordingly, this study also attempted to assess South Korea’s large and medium-sized shipbuilding companies using this smart factory assessment tool in the initial stage. However, the composition and contents of the assessment items defined in the existing model did not reflect the characteristics of the shipbuilding industry and differed from the shipbuilding production system, thus, degrading the reliability of the assessment results.

Concerning multivariate analysis, mathematical analysis techniques such as analytic hierarchy process (AHP), data envelopment analysis (DEA), and multivariate regression analysis are widely used for the assessment of multiple targets derived from various input variables. A meaningful productivity predictor was proposed based on a regression analysis of various indicators related to shipyard productivity [23]. The competitiveness of shipyards was analyzed through DEA, using capacity, technology level, and the industrial environment of various shipyards as input and productivity and building times as output [24]. Their studies extended beyond a qualitative analysis; through DEA, they expressed the relative levels of the most advanced shipyards and those lagging behind them, thereby presenting directions for improvement and quantitative improvement levels. A shipyard efficiency analysis using DEA was also introduced [25]. Turnover/cost values were derived by combining various shipyard production indicators to compare the actual competitiveness of shipyards [26]. Moreover, an assessment methodology for the shipyard block assembly process was developed using process mining and DEA, and they described a practical case applying the methodology to shipyards [27]. In addition, shipyard production plans were assessed using the AHP method considering sales, dock turnover, and quay load [28]. Most recently, a comparative analysis was performed for assessing the productivity of 21 shipyards in South Korea, China, and Japan using DEA and the metafrontier framework [29]. As such, researchers have performed a variety of studies on shipyard competitiveness and productivity, with DEA being applied the most.

In this study, we propose a SSML assessment framework that considers ship production characteristics and diverse production environments. To this end, we devised a modified assessment framework appropriate for the SSML assessment based on the categories of the smart factory assessment framework proposed by Lee et al. [22]. For this purpose, the SSML assessment framework was defined through expert surveys on South Korean shipbuilders of various sizes and technology levels and reflected in the composition of the detailed assessment items. Furthermore, we applied the developed SSML assessment framework to real shipyards and their subcontractors. To apply the proposed framework, this study adopted DEA techniques [24,30] to analyze productivity in the shipyard industry. Using DEA, we built a model that set the maturity level of the shipyard and its subcontractors as input, and sales and order quantity as output. The model confirmed that the smartness maturity level can serve as useful information for assessing the capabilities of companies and deriving quantitative improvement indicators.

The remainder of this study is structured as follows. In Section 2, we present how to develop the SSML and, in Section 3, we provide the results and analysis of the assessment from the interview. In Section 4, we introduce DEA and describe the method of quantitative analysis using DEA. In Section 5, we present the results and discuss the practical contributions of this study. In Section 6, we draw conclusions and suggest future research.

2. Development of Smart Shipyard Maturity Levels

While these previous maturity models present meaningful diagnostic criteria, they either lack specificity or are difficult to apply to diagnosing the SSML. As incorrect assessment criteria lead to incorrect assessment results, smartness maturity levels that reflect the characteristics of shipbuilding production systems must be defined.

With few studies on the assessment of shipyards, this study is the first step for developing a smart maturity level assessment system for shipyards. Technical demand surveys were conducted with large and medium-sized shipyards in South Korea, module manufacturing subcontractors, and related research institutes. The surveys were conducted by e-mail on May 2018, with the shipbuilding industry partners in South Korea including large shipyards, subcontractors, and research institutes. Through these online and offline demand surveys, we identified approximately 450 technology demand surveys from smart shipyards, which were categorized into five areas, as shown in Figure 1. Next, an industry-academia-research expert group was formed to devise a technical roadmap for realizing smart shipyards, see Figure 2, based on the technical demand survey results. This roadmap comprised a bottom-up process for selecting four major fields from the technology demands of the 450 candidates, and a top-down process for classifying the detailed tasks through the expert group. The technical roadmap was largely classified into infrastructure technology, including IoT, big data, process automation for unmanned production and logistics activities, intelligent technology for an array of production management tasks, and production design automation technology.

Figure 1. Technology demand survey for the smart shipyard realization. (a) Online survey; (b) Offline survey.

Figure 2. Technical roadmap for smart shipyard realization.

Next, to prepare the standard for the SSML from the technical roadmap, we used the concept revealed from the existing research focusing on the automation concept It is difficult to define the levels for complex production systems such as shipyards because existing level definitions for smart factories are defined on a single-level scale. The concept of automation was examined from a human-centered perspective rather than a physical device [31]. Accordingly, we defined the types of automation as follows:

• Control automation assists humans in the guiding process of executing the task and the machine movement through dangerous tasks. Control automation plays the role of an observer of the whole subsystem.
• Management automation allows humans to exercise demands oriented to technological actions and activities, and also a strategic point of the automation process.
• Information automation, a type of system that is changing very rapidly, provides the system with information about the progress and the execution of certain tasks.

We used these three types of automation, i.e., physical, intellectual, and information transaction, as keywords for smart shipyards (see Table 1). First, physical automation corresponds to control automation and automation of human physical labor. Next, intellectual automation refers to automation of human knowledge labor and corresponds to the management automation. Finally, connectivity refers to automation of information transfer, and corresponds to information automation [31]. Additionally, the assessment criteria for each maturity factor and level, together with the specific integrated form of smart shipyard concerning each level, are shown in Appendix C.

Table 1. Definition of smart shipyard maturity level.

Level	Connectivity	Physical Automation	Intellectual Automation
Level 5	Hyper connection	Unmanned	Hyper intelligence
Level 4	IoT connection	Fully automatic	Artificial intelligence
Level 3	Wireless connection	Partially automatic	Information and knowledge
Level 2	Wire connection	Mechanized	Data
Level 1	Face-to-face connection	Manual	Experience

On the basis of this definition of SSML and on the smart factory assessment framework [22], an assessment framework for SSML was developed. In this process, the number of inquiry items in the existing model increased from 46 (from [22]) to 61 (assessment framework for SSML) to reflect the characteristics of the shipbuilding industry. In particular, assessment items for the logistics operation and information system modules were added and modified (Table 2). Furthermore, Figure 3 shows the structure of the criteria, modules, and assessment items of the proposed assessment framework. This diagnosis framework was divided into the following four criteria groups: leadership and strategy, process, system and automation, and performance. In addition, each criteria group consisted of submodules; in particular, the process criteria group included product development, production planning, process management, quality management, facility management, and logistics operation modules in consideration of the shipbuilding process. Each module included assessment items, and based on those, the smart shipyard maturity level was diagnosed.

Table 2. Example of changed assessment items (partially selected).

Smart Factory Assessment			Smart Shipyard Assessment
Module 3 (Production planning): Inquiry 4	Short-term production planning	→	Module 3 (Production planning): Inquiry 4	Mid-term planning (quarter/month) and short-term planning
	Level of collaboration between sales and production and operational procedures levels Rolling planning system process operation status (weekly/daily) Whether to conduct what-if simulation to validate the production planning			Level of collaboration with upper planning departments and business and operational procedures levels Rolling planning system process operation status (weekly/daily) Whether to operate an evaluation and optimization system (simulation, operations research, etc.) to establish production planning Information sharing level of production planning information (manual, intranet, web, etc.)
Module 7 (Logistics): Inquiry 4	Warehousing	→	Module 7 (Logistics): Inquiry 4	Stockyards
	Whether to use a rack system to increase storage efficiency Rack address system Analysis of warehouse operation productivity indicators such as location management, warehouse space utilization, and inventory level Storage productivity level such as rack, transport tool, picking tool, and display			Whether to use various types of stock area to increase storage efficiency Lot numbering system for stockyards Analysis of storage-related efficiency indicators such as area management, area utilization, and level of materials and blocks Area efficiency level
	N/A		Module 8 (Info. System): Inquiry 8	Advanced planning and scheduling system
				Level definition using aggregate production scheduling The composition of a comprehensive system for production planning and service levels concerning information systems

Figure 3. Diagnosis framework for the assessment of smart shipyard maturity level.

3. Assessment of Smart Shipyard Maturity Level

3.1. Methodology for Assessment of SSML

We conducted an assessment of shipyards to verify the developed assessment framework. Regarding the assessment procedure, first, the assessment material was delivered to 15 companies, including large (L1~L5), mid-sized shipyards (M1~M3), as well as subcontractors such as hull block assembly (B1 and B2), outfitting material manufacturers (O1 and O2), and steel fabrication companies (P1 and P2), and the person-in-charge of the respective module at each company completed the assessment material through a self-assessment. As shown in Figure 3, since there were 10 evaluation modules, 10 employees of each company participated in the assessment. In order to eliminate personal bias, two or three consulting experts who had over 10 years of experience in shipbuilding production area visited each company afterwards and corrected those self-assessment scores through in-depth interviews with the person-in-charge of the respective module and site inspections. Then, reports were written on the interview, site inspection results, and corrected diagnoses and delivered to each company, after which feedback was received. The assessment results were converted into a score of 5 points and Table 3 shows the results of SSML of each assessment module.

Table 3. Results of the SSML of each assessment module.

	Shipyard									Subcontractor
	Large Shipyard					Mid-Sized Shipyard				Block		Outfitting		Processing
	L1	L2	L3	L4	L5	M1	M2	M3	M4	B1	B2	O1	O2	P1	P2
1. Leadership strategy	3.8	4.1	3.0	3.4	3.3	3.0	2.9	2.8	2.5	1.8	1.5	3.3	1.5	3.5	3.8
2. Product development	3.6	3.8	3.4	3.4	3.8	2.0	3.1	2.6	2.5	2.4	1.3	2.6	1.3	3.0	2.8
3. Production planning	2.9	3.4	3.0	3.3	3.0	2.3	3.1	2.9	2.4	2.0	2.3	2.4	2.1	3.0	2.0
4. Process management	2.6	3.4	3.0	3.0	2.7	2.3	2.8	2.9	2.6	2.2	2.4	2.2	2.3	3.2	2.8
5. Quality management	2.7	2.7	3.0	2.7	2.7	2.1	2.8	2.8	2.0	2.4	2.3	2.6	2.2	2.3	2.0
6. Facility management	2.5	3.5	2.9	2.8	2.5	2.2	2.3	2.8	1.9	2.4	2.1	1.7	2.1	2.5	2.2
7. Logistics operation	2.9	3.1	2.6	2.5	3.1	2.1	2.6	2.5	2.0	2.4	2.1	1.9	1.8	3.0	2.4
8. Information system	3.4	3.8	3.6	3.2	3.2	2.1	3.3	3.0	2.5	2.0	1.9	1.4	1.8	3.0	1.6
9. Production automation	2.5	2.6	3.1	2.2	2.4	1.3	1.7	2.6	1.2	1.2	1.4	1.5	1.2	1.2	2.4
10. Performance	3.3	3.7	3.4	3.4	3.4	2.7	2.9	3.0	2.6	3.2	3.0	2.3	2.6	3.0	2.0
Smartness level	3.0	3.4	3.1	3.0	3.0	2.2	2.7	2.8	2.2	2.2	2.0	2.2	1.9	2.8	2.4

3.2. SSML Assessment Results and Discussion

In contrast to general business consulting, the assessment results were expressed as scores from a sophisticated assessment that reflects the widely known scale and qualitative level of companies. Furthermore, they are meaningful as much as this is the first assessment of detailed production factors in the field of shipbuilding production.

Figure 4 shows the average score for each module of the companies investigated. Production automation, the most noteworthy module, showed the lowest assessment at 1.9 (approximately 40 points based on 100 points) owing to the high dependence on workforce in the shipbuilding industry. Most shipyard and subcontractor processes are manual processes. All construction can be performed manually, excluding processing and some assembly processes in large and mid-sized shipyards. Considering the weight of the number of hours by process, large shipyards also have very low automation rates, at approximately 30% for processing and 10% for assembly.

Figure 4. Average maturity level of each assessment module.

Next, the levels of quality management, facility management, and logistics operations were diagnosed as low. Concerning quality management, although domestic shipyards are somewhat competitive internationally [32], this module was assessed to be low because the assessment focused on the computerization aspect of data collection, processing, and sharing, rather than the level of quality itself. However, the numerous complaints in the on-site interviews from quality management employees about the inadequate data management systems suggests that, in addition to the quality management capabilities of domestic shipyards, the informatization systems that support quality management must be improved. The facility management assessment signifies inadequate management concerning smart maturity of large transport equipment (gantry cranes, transporters, etc.) and production equipment (cutting, welding, painting equipment, etc.). Advanced technologies such as connectivity and predictive maintenance, which are pursued in smart production, were not adopted, and most maintenance procedures consisted of responding to problems after they occurred. Hence, there is a need for advanced facility management by applying predictive maintenance and IoT connectivity technologies, which automatically collect equipment information in real time. As one of the main targets of shipyard management, logistics was recognized as a component of production that enables the smooth flow of production activities by connecting processes, rather than as a simple transport activity. Nevertheless, shipyards remain in the process of adopting logistics technology for tracking work in process (steel plate, hull block, outfitting module, etc.) items; therefore, from the perspective of smart production that pursues connectivity and automation in transport, the low score for this module is reasonable. Aside from these modules, the scores for product planning, production planning, process management, and the information system (approximately 2.7–2.8) were higher than the overall average smart maturity level.

Management’s proactiveness concerning smart technology, in terms of leadership and strategy and performance measurement, was scored relatively high at 2.9 and 3.0, respectively. However, as these two modules are considerably closer to qualitative assessment than other modules that can be quantitatively diagnosed (systems, facilities, informatization, etc.), they can be regarded as assessment items not supported by concrete evidence. Therefore, since the weights among the modules must be considered, future studies must apply the AHP technique to derive more reliable assessment results.

Next, we conducted a clustering analysis of each company based on the maturity level scores. Excluding leadership and strategy and performance measurement, which directly impact the production level, the average scores of the “process” and “system and automation” groups based on the criteria were set as the x- and y-axes, respectively, and the position of each company is shown in Figure 5. In Figure 5, the size of each circle indicates a company’s relative sales in 2019. As the low tide of the international shipbuilding market continued in 2019, the relative sales of each company indicated in the graph are not proportional to their production capacity. Therefore, although M2/M3 and M1/M4 are classified as mid-sized shipyards, the difference in actual production capacity is very large. Nevertheless, the scale of sales shown in Figure 5 are similar owing to the influence of market conditions; therefore, they should be considered only for the purpose of classifying company type. This is shown to classify large and mid-sized enterprises/SMEs.

Figure 5. Bubble chart for the comparison of smart shipyard maturity level concerning process and system and automation.

The large L1–L5 enterprises formed a relatively higher cluster, while the level of M2 and M3 companies among the mid-sized enterprises was similar to that of large shipyards. Although M1 and M4 are also classified as mid-sized shipyards along with M2 and M3 because the sizes of M2 and M3 are close to large, the difference in size is also reflected in the production level. The outfitting manufacturers (O1, O2) and block manufacturers (B1, B2), as well as mid-sized shipyards M1 and M4, formed a cluster separate from the large enterprise cluster. P1 and P2 located between these two clusters reflect the characteristics of companies that form curved hull plates, which showed rather exceptional results. These results suggest that, as basic research on the automation of curved hull plate forming has recently matured to some degree, curved hull plate forming companies have also attempted to replace operator-dependent tasks with automated machines, thus, resulting in a movement toward automation reflected in the smart maturity level.

By analyzing the company type and SSML score as above, we could quantitatively analyze the smart maturity level of large- and mid-sized companies in the South Korean shipbuilding industry. We confirmed that the developed assessment framework for the SSML can reasonably quantify the types and levels of shipbuilding companies.

4. Data Envelopment Analysis

4.1. DEA Method

The proposed DEA method measures efficiency based on linear programming in the decision-making process [33]. This method is mainly applied when it is difficult to identify and compare direct relationships among multiple inputs and outputs and can be used to develop a more suitable decision-making model than AHP in environments with insufficient information. AHP and DEA methods have been applied to manufacturing supply chain companies and their performances have been compared to verify the practical effectiveness of each technique [34]. According to the findings, the AHP technique can provide detailed and gradual decision-making results through pairwise comparison in cases with large amounts of information, whereas the DEA method can provide effective decision-making strategies for situations with insufficient decision-making factors in a new environment. Accordingly, this study focused on a situation with a lack of specific grounds and information for smart shipyard technology and directions and noted very fast and uncertain technological change in the shipyard industry, unlike existing manufacturing technologies. Therefore, rather than AHP, which is time-consuming in the development process, DEA was applied using the assessment results of the SSML as input, with the goal of verifying whether it could be useful for calculating the efficiency of shipyards and deriving quantitative improvement indicators.

This method calculates the relative efficiency of assessment targets, where multiple inputs and outputs are considered. It derives the most efficient decision-making units (DMUs) from all DMUs to be assessed; consequently, the relative efficiency of each DMU is calculated using linear programming. DEA models are largely categorized into models that assume constant returns to scale (CRS) and those that assume variable returns to scale (VRS). The CRS model assumes that the relationship between input and output is the same at a constant rate, regardless of scale, and was used when the DEA methodology was first proposed. The CRS model is also referred to as the Charnes–Cooper–Rhodes (CCR) model, after the first proposed model. The VRS model [30] relaxes the assumption of CRS in the CCR model and is also referred to as the Banker–Charnes–Cooper (BCC) model.

DEA models can also be categorized into input-oriented and output-oriented models according to their orientation to input or output. The input-oriented model seeks to minimize input with a fixed output, whereas the output-oriented model seeks to maximize output with a fixed input.

DEA models are based on linear programming and can be explained by the following equations: First, assume that n DMUs exist. If $DM{U}_k$ (k = 1, …, n) means that m inputs $x_{ij}$ (i = 1, …, m; j = 1, …, n) are input to output s outputs $y_{rj}$ (r = 1, …, s; j = 1, …, n), then, the efficiency of the $k$th observation $DM{U}_k$ is obtained through the linear programming solution of Equations (1) and (2) assuming an output-oriented CCR model. In the equation, $\theta$ is the efficiency value, and $s^{-}$ and $s^{+}$ are slack variables for the input and output, respectively. If the value of ${\theta }^{*}$ obtained as the solution to this equation is 1 and both slack variables are 0, then the DMU is an efficient (efficiency 100%) DMU. Equations (1) and (2) are as follows:

$$max\left\{{\theta }_k+ϵ\left({{\displaystyle \sum }}_{i=1}^m{s}_i^{-}+{{\displaystyle \sum }}_{r=1}^s{s}_r^{+}\right)\right\}$$

(1)

$$s.t.\left\{\begin{array}{c}{{\displaystyle \sum }}_{j=1}^n{x}_{ij}{\lambda }_j+s_i^{-}=x_{ik},i=1,\dots ,m\\ {{\displaystyle \sum }}_{j=1}^n{{\displaystyle \sum }}_{j=1}^n{y}_{rj}{\lambda }_j-s_r^{+}={\theta }_k{x}_{ik},r=1,\dots ,s\\ {\lambda }_j,s_i^{-},s_r^{+}\ge 0,\forall i,j,r\end{array}\right\}$$

(2)

As the BCC model assumes VRS, a constraint is added such that the sum of $\lambda$ is 1. Accordingly, the output-oriented BCC model can be expressed as in Equations (3) and (4) as follows:

$$max\left\{{\theta }_k+ϵ\left({{\displaystyle \sum }}_{i=1}^m{s}_i^{-}+{{\displaystyle \sum }}_{r=1}^s{s}_r^{-}\right)\right\}$$

(3)

$$s.t.\left\{\begin{array}{c}{{\displaystyle \sum }}_{j=1}^n{x}_{ij}{\lambda }_j+s_i^{-}=x_{ik},i=1,\dots ,m\\ {{\displaystyle \sum }}_{j=1}^n{{\displaystyle \sum }}_{j=1}^n{y}_{rj}{\lambda }_j-s_r^{+}={\theta }_k{x}_{ik},r=1,\dots ,s\\ {{\displaystyle \sum }}_{j=1}^n{\lambda }_j=1\\ {\lambda }_j,s_i^{-},s_r^{+}\ge 0,\forall i,j,r\end{array}\right\}$$

(4)

If the efficiency value of the CCR model is ${\theta }_{CCR}^{*}$ and that of the BCC model is ${\theta }_{BCC}^{*}$, then, the constant relationship ${\theta }_{CCR}^{*}$≤${\theta }_{BCC}^{*}$ is established, and the difference in efficiency between the two models originates from whether the scale is optimal. This difference is the scale efficiency (SE), and the following relationship is established in Equation (5):

$$SE={\theta }_{CCR}^{*}÷{\theta }_{BCC}^{*}$$

(5)

If SE is 1, then, it is in a CRS state and there is no inefficiency owing to scale; if SE is less than 1, then, it is in an increasing or decreasing returns to scale state, signifying that there is inefficiency owing to scale. The efficiency of the CCR model (${\theta }_{CCR}^{*}$) is referred to as the technical efficiency, while the efficiency of the BCC model (${\theta }_{BCC}^{*}$) is referred to as pure technical efficiency to emphasize that inefficiency owing to scale is excluded.

4.2. Quantitative Analysis Using DEA

Next, using the DEA method, we conducted a quantitative analysis, compared the companies, and presented target levels of quantitative improvement for relatively low-level companies. DEA is an analysis method that measures the relative efficiency of companies with multiple input and output factors. It measures the relative efficiency between the same groups and provides information on benchmarking targets to improve efficiency for those that appear inefficient. The DEA technique is useful for the following problems.

• It is useful when there are many inputs and outputs, but it is difficult to integrate them into the single index in an appropriate way.
• DEA provides a basis for benchmarking target that should be investigated to improve efficiency.
• DEA can simultaneously consider various input and output factors with different units of measurement.

To measure efficiency using DEA, DMUs must be set. DEA presupposes the homogeneity of the analysis target, requiring individual DMUs to perform similar activities to produce products that can be compared [35]. Furthermore, similar resources or capital must be input in all DMUs, and performance must not be influenced by external factors. Accordingly, the DMUs in this study, consisted of shipyards focusing on new shipbuilding. However, shipyards for which output variable data (new ship construction) could not be secured were excluded from the analysis.

The selection of the input and output variables is important for ensuring the reliability of the DEA results. That is, it is necessary to select input and output variables that can accurately reflect the objectives, targets, and production environment of the shipyard. Our objective was to verify whether the SSML assessment result is an indicator of shipyard efficiency. As such, a DEA model was built using the number of employees and docks as inputs and the new ship construction as the output.

Furthermore, as described above, the DEA model was categorized into a CCR model (for Charnes, Cooper, and Rhodes [33]) and BCC model (for Banker, Charnes, and Cooper [30]) based on the assumption of the effect of scale, and also categorized into an input-oriented (minimum input) model and output-oriented (maximum output) model, depending on the purpose of measuring efficiency. Regarding the selection of the model, since new ship construction is greatly affected by external factors, the input-oriented (minimum input) model was selected, which minimizes the input for a fixed output. The commercial software Frontier Analyst was used for the DEA.

Next, two models were defined as shown in Table 4, to perform the DEA. However, as the input and output of the DEA model are proportionally correlated, for the SSML score (the input variable), the proposed model uses the value of the perfect score (5.0) minus the score of each factor. This was a given correlation as an input factor of the DEA model, in which the lower the maturity level score, the closer it is to the advanced level. That is, a high maturity level indicates a relatively small input factor value, meaning that the output can be achieved with less effort. Thus, the maturity level was analyzed by subtracting the projection value derived via DEA from the perfect score (5.0).

Table 4. Input and output variables definition of each target data envelopment analysis (DEA) model.

Model	Category	Items	Remark
Traditional model	Input variables	X1: Employee number X2: Number of dock and berth
	Output variables	• Y1: New shipbuilding volume (CGT)
Proposed model	Input variables	X3: Process X4: System and automation	Smart shipyard maturity level assessment
	Output variables	• Y1: New shipbuilding volume (CGT)

Owing to insufficient data, DMUs could not be performed for all companies in Table 3. However, a DEA was conducted for shipyards with available data on output Y1 (L1–L5, M2, M3, and M4) common to the two models and X1 and X2 of the traditional model in Table 4.

There are considerations concerning the timing of the input and output data. As shipyard construction volume greatly varies with the time period, large gaps in the potential production volume of a shipyard may arise when using the proposed model’s input data X1–X2 and Y1 of when the maturity level assessment was performed in 2009. Moreover, data on the shipyard’s production capacity are not explicitly disclosed or available. Accordingly, for the data of X1–X2 and Y1, we considered the largest volume of new ship construction during the last 10 years and the number of employees and docks/berths in that year. Consequently, the 2019 data were selected. Although applying these conditions will lead to mismatched timing in the output and input data, we ignored it because our objective was to compare the results of DEA, which uses physical conditions (workforce, resources, etc.) as input, with the results of the proposed method, which uses the maturity level as input, rather than present quantitative information on each shipyard through precise assessment results. Furthermore, as the speed of change in shipyards is slow, it was judged that the level assessed in 2019 had not dramatically changed over the last 10 years.

Before describing the analysis, the following assumptions were made: If the objective reliability of the relationship between the input and outputs considered in Table 4 and the DEA method is secured, then the level (or efficiency) of each shipyard will be determined by the DEA results. However, we aimed to identify which input/output and DEA model best expressed the phenomenon (shipyard level); the judgment criteria for the suitability of the analysis results were defined as the shipyard level, which were generally known as follows. For the assumptions of the relative levels of L1–L5 and M2–M4, they are classified based on the global ranking of each shipyard. First, L1–L3 are within the top five shipyards worldwide, with little change in ranking over the past 20 years. Thus, their scale, as indicated by sales and orders, and also their overall production level is among the best worldwide. Next, L4 and L5 are within the top 10 shipyards worldwide; although, the scale is somewhat smaller than L1–L3, this does not necessarily indicate a larger difference in production level. Next, M2 and M3 are shipyards in the top 10–50 worldwide, with relatively high-ranking volatility. Finally, M4 corresponds to shipyards ranked 50–100. However, as the difference in order and construction volume decreases at lower ranks, the difference in SSML does not increase as much as the difference in rank. Accordingly, L1–L3 is defined as the large group, L4 and L5 as the mid-sized-large group, M2 and M3 as the mid-sized group, and M4 as the small group, which are assumed to be the guidelines for analyzing the DEA results. According to the previous study, if the skill level of a large shipyard is set to 100, the level of a medium-sized shipyard is 75–85, and that of a small shipyard is approximately 50 [36]. Therefore, this quantitative guideline is used as comparative data to examine the efficiency feasibility of each shipyard through DEA analysis.

5. Results of the Data Envelopment Analysis

DEA was performed for Case 1 of a single model consisting of L1–L5 and M2–M4 as DMUs and Case 2 of split models with L1–L5 and M2–M4 separated.

5.1. Case 1

In Case 1, L1–L5 and M2–M4 were configured as one model. Figure 6 shows the efficiency and the (SE) results derived by applying the CCR and BCC analysis methods concerning the traditional model and proposed model. In Figure 6 (raw data is Table A8), SE values below 1 were derived. If the SE is less than 1, the DMUs are in a state of increasing or decreasing returns to scale states, signifying that there is inefficiency owing to scale. Next, the results were derived from the CCR and BCC methods using the traditional and proposed models.

Figure 6. Efficiency and scaled efficiency of (a) traditional model and (b) proposed model (Case 1).

Figure 6 shows the analysis results of the BCC and CCR methods, under traditional analysis conditions using the number of employees and docks as input variables. First, concerning the BCC method, the efficiencies of all shipyards, excluding M2 in BCC and M2 and M4 in CCR, were 100% or close to 100%, confirming that there was no discriminatory power between the shipyards. Furthermore, concerning both BCC and CCR, L1 and M2 was lower than M3 and M4, although it ranked higher than both in reality; hence, they did not meet the guidelines assumed in Section 4.

Next, Figure 6b shows the analysis results from BCC and CCR, under the analysis condition using two previously analyzed SSML categories as the input variables; specifically, the average of the perfect score (5.0) of each module minus the score of each factor.

In Figure 6b, the CCR analysis results under the proposed model show efficiencies of 21.56%, 7.95%, and 4.57% for M2–M4, respectively, thus exhibiting differentiation with L1–L5. However, the difference became excessively large, resulting in practically impossible values for the projections of process and system and automation, which are inputs. Therefore, the analysis results are unreasonable. On the contrary, the BCC results, in Figure 6b, show the analysis results from the BCC method under the proposed model. All shipyard efficiency results using the BCC method under the proposed model satisfied the assumed guidelines in Section 4. L1–L3 was at 100% and was located in the best practice line, followed by L4 and L5 at approximately 80%, showing a difference of approximately 20% with the large group. Next, M2 and M3 showed values in the upper 70%, slightly less than the mid-sized and large groups; and M4, belonging to the small group, was approximately 60%. Thus, in the proposed model, the BCC analysis presents appropriate efficiency and projection values considering the qualitative level of each shipyard. Concerning the construction volume projections (as marked in red in Table A11), however, the values of M3 and M4 were unrealistically overestimated. This problem likely occurred because the difference in construction values from other shipyards was too large, despite being an input-oriented model. Accordingly, in the DEA analysis of Case 2, L1–L5 (large group) and M2–M4 (small and mid-sized group) were divided into separate models.

5.2. Case 2

Owing to the large difference in scale between the large-, small-, and mid-sized groups, the construction volume projections calculated for the small- and mid-sized shipyards were overestimated. Hence, in Case 2, L1–L5 (large group) and M2–M4 (small and mid-sized groups) were divided into separate models for the analysis. Since BCC showed more reasonable results than CCR in the analysis of Case 1, only the results under the proposed model were addressed in Case 2. For reference, Table A14 and Table A15 in Appendix D show the detailed results under the traditional model for Case 2.

Figure 7 summarizes the efficiency and SE results. According to the results under the proposed model of Figure 7b,d, there was no change in L1–L5, which belongs to the large group, with only a change in M2–M4. That is, since the model for the small group was separated considering the difference in construction size, the large group showed no change from the existing results; and M2–M4, which were separated from the construction of large shipyards into a group of small shipyards, were calculated independently, thus adjusting the efficiency.

Figure 7. Efficiency and scaled efficiency of traditional model and proposed model (Case 2).

Comparing the efficiency of CCR and BCC using the proposed model in the M2–M4 analysis, BCC (100%, 100%, and 80.22%) better reflects the shipyard guidelines than CCR (100%, 40.55%, and 21.20%). In other words, since M2 and M3 correspond to mid-sized shipyards with similar scale and technology level and M4 corresponds to small shipyards, BCC exhibited more reasonable results than CCR.

Additionally, Table A16 in Appendix D shows the DEA results of proposed approach and explains the detailed analysis results from the BCC methods. As shown in Table A16, the process and system and automation levels of L4 and L5 (large shipyards) must be improved by approximately 11–16% to reach the levels of L1–L3. As for M2–M4, whereas M2 and M3 are at 100% efficiency within that group, M4 shipyards must improve their process level by 20% and system and automation by 27%.

As a result of DEA analysis with respect to various factors and cases as compared with the existing DEA technique which uses physical elements such as the number of employees and docks as input, the proposed method, which uses the SSML as input, derived more reasonable results for the efficiency and projection of actual shipyards.

6. Conclusions

To develop a framework for diagnosing the smart maturity level of the shipbuilding industry, this study analyzed existing research on smart manufacturing, smart factories, and maturity models. Technology demand surveys were also performed to reflect the characteristics of the shipbuilding industry, and a technical roadmap for realizing smart shipyards that reflected the opinions of an expert group was proposed. The SSML assessment framework developed through this process defined five maturity levels for each of the following keywords: connectivity, automation, and intelligence. Furthermore, based on the defined levels, a diagnostic tool comprising 61 items for four criteria and 10 modules was developed. While the structure presented in prior research was used for the criteria and modules of the diagnostic tool, the 61 detailed inquiry items were reconfigured reflecting the characteristics of shipyards. This developed framework was used to diagnose large- and medium-sized shipyards and subcontractors in South Korea, after which the results were analyzed. Automation was assessed the lowest in the maturity level of shipbuilding-related companies, and it was confirmed that those companies could be divided into groups through a bubble chart analysis using process and system/automation criteria as the two axes.

Next, the assessment of SSML was used to conduct a DEA, which was capable of quantitative analysis. Using the SSML as the input variable, DEA can derive the efficiency levels of the subject companies and confirm the level of improvement required to reach 100% efficiency for each module through quantitative indicators. In addition, for the same companies, we performed a comparative analysis between the traditional DEA model, which used the number of employees, scale of facilities, etc. as input variables; and the other DEA model, which used the SSML as the input variable. The results demonstrated that the model using the maturity level as the input variable derived more reliable results that were well matched with a previous survey [36].

However, for DEA to be more pertinent, the number of DMUs must be at least two to three times greater than the sum of the number of input and output variables [37]. As the assessment was based on a rather insufficient number of DMUs (companies to be diagnosed), further investigations with more DMUs are strongly recommended, including not only South Korean shipyards but also shipyards of similar scales in China, Japan, and Europe.