Next Article in Journal
Applying Roadmapping and Conceptual Modeling to the Energy Transition: A Local Case Study
Next Article in Special Issue
Implementation of a Recycled Polypropylene Homopolymer Material for Use in Additive Manufacturing
Previous Article in Journal
Spatiotemporal Analysis of Urbanization Using GIS and Remote Sensing in Developing Countries
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
Sustainability
Volume 13
Issue 7
10.3390/su13073682
sustainability-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Evaluation of eMaintenance Application Based on the New Version of the EFQM Model

by
Renata Turisová
1,
Hana Pačaiová
1,*,
Zuzana Kotianová
1,
Anna Nagyová
1,
Michal Hovanec
2 and
Peter Korba
2
1
Department of Safety and Quality Production, Faculty of Mechanical Engineering, Technical University of Kosice, Letná 1/9, 042 00 Košice-Sever, Slovakia
2
Department of Aviation Engineering, Faculty of Aeronautics, Technical University of Kosice, Rampová 7, 041 21 Kosice, Slovakia
*
Author to whom correspondence should be addressed.
Sustainability 2021, 13(7), 3682; https://doi.org/10.3390/su13073682
Submission received: 25 February 2021 / Revised: 23 March 2021 / Accepted: 24 March 2021 / Published: 26 March 2021
(This article belongs to the Special Issue Sustainable Manufacturing and Supply Chain in the Context of Industry 4.0: Opportunities and Challenges)
Browse Figures
Versions Notes

Abstract

:
Maintenance management is connected with two opposing aspects, management costs and operational efficiency. With the implementation of new technology within the Industry 4.0 (I4.0) concept, new technical solutions are being created. These solutions (mainly robotic workplaces) must reach a maximum performance rate, production quality, and, of course, high availability. Their operation, during the whole life cycle, is expected to be absolutely safe with minimum maintenance costs. These trends, even though they seem to be optimistic, face a lot of problems. The conducted research follows up on the results of previous research aimed at the initial assessment Slovak industrial company readiness status for the I4.0 conception between 2017 and 2019. The aim of the ongoing research was to assess the readiness status in more than 70 industrial organizations in the selected area for the new concept of maintenance management (eMaintenance) and its relation to machinery integrated safety. The research was carried out by questioning, with the structure of individual questions and closed answers stemmed from the self-evaluation according to the new European Foundation for Quality Management (EFQM) Excellence Model (2020). The results of the research were presented to managements of questioned organizations and confirmed the assumptions about a low level of maintenance management transformation to eMaintenance.

1. Introduction

The Industry 4.0 concept, since it was first named in Germany in 2011, has undergone various stages of both development and implementation, mainly in the countries with developed industries [1,2,3]. In the Slovak republic, the share of the automotive industry accounts for almost 50% of the entire industry [4], so digitalization (as the main pillar of I4.0) is one of the core priorities of the Slovak industry and society transformation. The nation-wide initiative, called ‘The concept of intelligent industry for Slovakia’ [5], the aim of which is to transform, strengthen, and maintain the competitiveness of the Slovak industry, with the help of the latest technological development, creates space for the research in the readiness of Slovak companies for the implementation of the I4.0 concept. The aim of this research [6,7] is, by means of analyses, to reveal gaps in the entrepreneurial environment, mitigate mistrust of managers, and create space for the understanding how cyber-physical systems support the innovation of production and services and the increase of competitiveness [8]. Many current studies emphasize that just overestimated expectations or management fears can be an obstruction of the development of I4.0 [9,10].
As Madsen [9] clearly summarized in his paper, the concepts of modern management have evolved significantly in the last few years (e.g., Agile Management, Big Data analysis, Customer Relationship Management). As he said, I4.0 can be understood as a highly normative management concept that improves the performance of the organization, respectively: flexibility, productivity, and quality of processes or production.
The versatility of this concept is that it includes many partial management concepts, such as the Internet of Things (IoT), Big Data, and digitalization [9,11,12]. On the other hand, this concept is not strictly limited because it can be implemented as an integrated part of many other (already established) management concepts.
According to [11], I4.0 represents the creation of a dynamic technological, but primarily communication, environment (Internet of Things), enabling fast feedback. Its implementation changes the relationship between suppliers and producers, between producers and customers, between a human and a machine, between machines themselves, or even between a material and a machine [12]. An important change occurs in the requirements for machinery safety (integrated machinery safety) and machinery maintenance. Reduction of machine energy losses, reducing waste, and accident prevention are vital for sustainable manufacturing in the I4.0 environment [12,13].
These requirements (for maintenance and machinery integrated safety) have been developed in the long-term [14,15,16]. EU directives (e.g., Directive 2006/42/EC and 2009/104/EC) defined a framework for machinery integrated safety and maintenance management during their entire life cycle [17,18]. The machine manufacturer must, already in the phase of its conception, consider the future use of the designed machinery and its safety and, by ‘setting’ maintenance requirements, support this safety [19,20]. Besides the concern about the machine’s reliability during operation, they must, in the designing stage, take account of future specific maintenance activities and, by implementing safety measures, reduce health and safety risks of maintenance workers when performing these activities [21].
A machine operator tends to maintain its maximum availability for a long time. However, the machinery condition itself changes, as do requirements for its maintenance. Implementation of various conceptions and tools for maintenance management support, such as Total Productive Maintenance (TPM), mainly in the automotive industry [22,23], Reliability Centered Maintenance (RCM) [24,25], risk-based inspection (RBI) [26,27], in the nuclear, metallurgic, and petrochemical industries, aims to support business organization objectives.
These tools serve to decrease so-called blind periodic inspection [25] and use the “know-how” of maintenance staff to create a dynamic maintenance program. In the last decade, the monitoring and evaluation of machinery status has become fundamental for effective maintenance management [28,29]. The predictive maintenance strategy is a proven method (or complex of methods) that increases the availability and safety, makes it possible to flexibly plan machinery shutdowns based on their status, increases the knowledge of possible causes of malfunctions (root cause analysis), enables the optimization of the need for spare parts, and generally improves the health condition of the core production technology [30,31,32].

2. Literature Review

With the arrival of I4.0, the position of predictive maintenance becomes even stronger [33]. Its nine basic pillars, as described by Vaidya, S. [34] or according to Cruz-Machado, A.V. are building blocks [35]: Internet of Things (IoT); augmented reality; clouds; big data; advanced simulation; cyber security; autonomous systems (robots); additive manufacture (3D printing); and System integration (horizontal/vertical), which are a base for monitoring and processing data not only about the production process but also about machine and machinery status (asset) [36]. Yet the mutual management and interconnection of these pillars describing a complex of applied technologies within I4.0 is not strictly clear [37]. Integration of management systems (IMS) based on risks, in organizations with previously implemented management systems, is a relatively new task (e.g., ISO 9001:2015, ISO 45001:2018, ISO 14001:2015). Additionally, at present, complexly managing the requirements of interested parties is not a simple duty for the management [38]. The uncertainty of the global business environment [39] increases the pressure on correct decision making and the application of preventive tools to reduce business risk [40,41].
Maintenance management (MM) in a I4.0 factory (also Smart factory) plays an important role in risk prevention, using online monitoring and machinery status evaluation and predicting its remaining lifetime (Predictive Maintenance Programs (PMPs)) [30,42]. With the arrival of I4.0, several definitions appeared in an attempt to describe the new position and requirements for MM, e.g., the eMaintenance (also e-Maintenance (eM)) can be described as the area of maintenance where technology is used to provide decision support for operations and maintenance through the application of advanced information technology [43] or, according to [44], eM can be considered as a philosophy supporting the move from fail and fix maintenance practices to predict and prevent strategies, which is a proactive concept. Maintenance Management 4.0 (MM4.0) contains a holistic view on data sources, ways of connection, ways of collecting, and ways of analysis, with the aim to predict future functional failures/asset efficiency and prescribe the most effective preventive measures [43,45].
Authors A. Cachada et al. [46] regard Maintenance I4.0 (M4.0) as a type of architecture of an intelligent and predictive maintenance system, aligned with Industry 4.0 principles, that considers advanced and online analysis of the collected data for the early detection of the occurrence of possible machine failures and supports technicians during the maintenance interventions by providing a guided intelligent decision support.
According to [47], Smart Maintenance (SM) manages the maintenance of manufacturing plants in environments with pervasive digital technologies. Smart Maintenance, according to [48], represents a decision support system (SMDSS) that provides additional automation for a predictive maintenance plan for equipment.
Mainly in industrial organizations, I4.0 implementation status or requirements for MM4.0 (eM, SM) implementation is a considerably difficult issue related not only to the availability of I4.0 technologies (9 pillars [34]) but also to priorities, possibilities, and management’s ability to apply suitable tools and technologies for the monitoring of the status of core or critical machinery [49]. Not all current construction solutions, whether robotic workplaces or integrated manufacturing systems, meet the conditions for complex online monitoring. Design and development of such machinery is still in the phase of development, and the communication between the operator and manufacturer is on an individual basis, where the operator is presumed to have an idea about the need for processing and evaluating data, i.e., eM solutions [50].
The conducted research between 2017 and 2019 at the authors’ workplace [7], focused on the assessment of the I4.0 implementation level in Slovak companies, has brought answers to some questions regarding the readiness to manage I4.0 type companies in Slovakia, yet, as for eM, it has just scratched the surface. For this reason, a further research aim was to assess the status in MM and its level, with the view of eM requirements and machinery integrated safety (IS) [51].
IS and eM areas, in the most general meaning, play significant roles in the era of the fourth industrial revolution. Therefore, it is important for the top managers, also in these areas, to create suitable assumptions [30]. The success of I4.0 basic principle implementation is conditioned by the overall readiness of particular companies to absorb the mentioned principles and ideas and use them in their further development [52].
For this reason, for the purposes of ongoing research, focusing on eM and IS areas, the EFQM Excellence model was used again but as its new version, the EFQM Excellence Model 2020 [53]. The questionnaire sent to 70 respondents was made in such a way that only the assumption part of the new EFQM Excellence Model (EFQM 2020) was used, consisting of Direction and Execution parts. We also used the proportionality of point evaluation, where, by a simple transformation of assigned points, we created a new point evaluation, where the maximum number of points possible to be reached was 500 for IS and 500 for eM. The questionnaire concept (see Appendix A) stemmed from the total maximum evaluation, so an organization could reach a 100% under conditions that 50% were assigned to IS evaluation and another 50% to eM evaluation.

3. Materials and Methods

Efficiency measuring by means of the EFQM Excellence model (only EFQM) has a history spanning several years, during which the experience from practice with the implementation of the mentioned model resulted in many changes and improvements [54,55]. The EFQM model makes it possible to evaluate the total efficiency of an organization [56]. It also enables the evaluation of specific areas or approaches [57,58].
In 2020, this model was completely changed [53]. The innovated EFQM 2020 is designed to be a practical and objective tool enabling organizations to obtain a complete overview of their current exceptionality level [53,54]. It is also focused on the support at priority identification and motivation to constant improvement, but it also serves as a decision-making support. The advantage of the EFQM 2020 model is that it is universal [54,57], which means it can be applied for any organization (its part or a subsystem), regardless of its size or focus. In order for an organization to be successful, it must be clearly formulated and have ad share direction by all stakeholders, i.e., a clear strategic intention and management able to make the right decisions at the right time. It needs to develop and improve the abilities of its employees, partnerships, and processes, with the purpose of the execution of its intentions and acquiring added value of the product for its customers. The fulfilment of basic assumptions of organization development, to which the first two parts of a three-part EFQM 2020 are devoted, i.e., Direction and Execution [53], is a necessary condition of efficiency improvement, prosperity, and successful reaching of its own goals [59]. That is why, within the conducted research into proving organizations’ for I4.0 (in IS and eM areas), we were oriented exactly on these two mentioned parts of the EFQM 2020 efficiency evaluation model.

3.1. Exceptionality Criteria

EFQM 2020 represents a framework based on seven criteria [53], by means of which it is possible to perform a thorough evaluation of the exceptionality level of any organization as a whole or its particular areas. The criteria are divided into three parts. The first five criteria found in the first two parts are labelled as criteria describing the basic assumptions of evaluated object exceptionality. They describe and evaluate what the organization is doing and how it is doing it. The remaining two criteria, belonging to the third part, evaluate the obtained results (see Figure 1).
As it was not our goal to evaluate the organization as a whole, but the perception of assumptions for a successful IS and eM implementation in the organization by its management, we inspired ourselves only by asking questions from the assumption part of the model.
As has already been mentioned, the assumption-oriented part of the model is structured in two independent parts: Direction and Execution. In Direction, two basic assumption criteria were created: Purpose, vision, and strategy (PV&S); Organizational culture and leadership (OC&L), and in Execution, three basic assumption criteria: Engaging stakeholders (ES); Creating sustainable value (CSV); Driving performance and transformation (DP&T) (see Figure 2).
The form and structure of questioning (see Appendix A) stemmed from the criteria focused on the description of organization exceptionality within the EFQM model. We assigned each criterion one question, which was directed to the IS area, as well as the eM area. The structure of particular criteria within EFQM 2020 is described in Table 1.
Unlike in the structure described in Table 1, which is oriented on the organization as a whole, we created a “narrowing” only to IS and eM areas. However, we regarded both these areas as subsystems and mapped all criteria within the model after a simple reformulation, also by number of points from the EFQM 2020 criterion.

3.2. Organization Readiness Model

The extent of a questionnaire (see Appendix A) focused on organizational readiness in IS and eM areas cannot be anywhere near as complex as what EFQM 2020 offers. Yet it is possible to use the same structure, be inspired by specific criteria, and modify them into questions, as well as maintain the proportionality of numeral evaluation of individual criteria. EFQM 2020 stems from the self-evaluation of an organization but it is subsequently modified by an experienced external evaluator. In our questioning, we used only subjective estimations from top managers of a questioned organization. Thus, the framework of this questioning did not allow us to use a wide range of questions, as it was with the self-evaluation using the EFQM 2020 exceptionality model. On the other hand, it was not a question of complex evaluation of an organization but only evaluation of the perception of the questioned organization exceptionality assumption fulfilment level in the area of machinery integrated safety (IS) and eMaintenance (eM) from their top managers’ viewpoints. Therefore, not only did we reduce the number of questions, we also modified the method of evaluation. From the EFQM 2020 model, within the questioning, we overtook its assumption part structure as well as the RADAR card logic [56,57,59] for the quantification of individual answers.
The basic EFQM 2020 conception was narrowed in such a way that it reflects the readiness level of questioned organizations in two selected areas from the Industry 4.0 conception viewpoint. Particularly, the first part of this questionnaire was focused on respondents’ perceptions from the viewpoint of degrees and method of machinery integrated safety (IS). The second part of the questionnaire was focused on maintenance (eM), according to I4.0. Both areas could reach the point evaluation from 0 to 500 points (i.e., from 0 to 50%). Point assignment to particular questions stemmed from the evaluation of individual criteria within the EFQM 2020 model, yet it transformed in a way that maintained proportionality, with the maximum number of points the organization may get being 1000 (expressed in per cent: 100%). The assignment of maximum point values to specific questions from particular IS and eM areas is presented in Table 2.
The selection of point evaluation in Table 2 was determined by the fact that, in individual organizations, within questioning, not only did we evaluate particular areas but also common evaluation of both areas, which were interconnected.
For each criterion, in terms of the model structure, closed questions were created. A respondent could choose one of six answers from “a” to “f”, formulated to create direct answers to a particular question. The respondent did not state any numeral values, they simply picked one of the answers. The first answer, always marked in order as “a”, was always negative. In the evaluation, we assigned it the value 0. The last answer to the question, marked “f”, corresponded with maximum agreement. Satisfaction was the highest evaluation. This answer was always assigned the maximum number of points. The other answers were assigned point values, according to which criterion the relevant questions were assigned. The specific numeral values for individual answers and question types are described in Figure 3.
The questioning was done electronically by means of information-communication technologies (IKT). The respondent sample consisted of a group of randomly chosen top managers from selected Slovak organizations. From each of the following areas: automotive, mechanical engineering and services, 50 top managers were addressed. From the other areas—business, construction, electrical engineering, energy and metallurgy—20 top managers were addressed. The total return of questionnaires was 51.43%.
We saw the highest return in the automotive area (32 questionnaires, 89%) and the lowest within the area of metallurgy (4 questionnaires, 11%).

4. Results

The average score of gained points within the mentioned questioning was, for integrated safety (IS), approximately 90 points (9%); higher than the average score recorded for the overall evaluation in the maintenance area (eM). Figure 4 shows a higher variability of respondents’ answers to the questions aimed at maintenance (eM) than to questions aimed at integrated safety (IS).
It is also interesting that, in the case of integrated safety (IS), the average was significantly lower than the median, whereas, with the maintenance evaluation (eM), the average was significantly higher than the median. This may be because, statistically, individual respondents generally perceive the readiness of their organization in the area of integrated safety (IS) more positively than in the area of maintenance (eM). While the evaluation median in the maintenance area (eM) reached a value slightly over 150, the evaluation median for integrated safety (IS) approached almost double of the given evaluation. One of the possible explanations of the given situation was given by one of the respondents, who commented on the results obtained from the research as follows (we note that it is an individual, statistically insignificant opinion of one single respondent):
“Under the term of integrated safety, managers usually imagine something abstract, i.e., a sort of a system with its ties oriented on safety and integrated in all company management systems. Therefore, the readiness for IS change within I4.0 requirements seems to them considerably higher than it is in the case of maintenance. They imagine maintenance as a system closely bound to specific technical and technological equipment”.
Next, we focused our attention on the differences of reached scores for individual criteria. Figure 5 presents a boxplot analysis for the mentioned criteria.
It is obvious from this analysis that, for both integrated safety (IS) and maintenance (eM), approximately the same evaluation was reached only with Cthe Creating sustainable value (CSV) criterion. With the Purpose vision, and strategy (PV&S) criterion evaluation, pthe maintenance was evaluated statistically significantly higher with most respondents; however, we must note that there was a higher stability of evaluations. In the remaining three areas, the evaluation of machinery integrated safety (IS) was considerably higher than the evaluation of maintenance (eM).
Better evaluation with the PV&S criterion in the maintenance area (eM) probably results from a traditional long-term positive perception of its purpose and a subsequently appropriately shared strategy and vision in Slovak organizations. Statistically, the greatest differences were in the driving performance and transformation (DP&T) area, where the readiness for transformation was perceived to be much higher in the machinery integrated safety (IS) area than in maintenance (eM). The same applies to Engaging stakeholders (ES), as well as Organizational culture and leadership (OC&L) criteria.

5. Discussion

As we have mentioned, randomly addressed respondents were a part of the top management of Slovak organization in eight sections: automotive, business, construction, electrical engineering, energy, mechanical engineering, metallurgy, and services.
Figure 6 shows a Boxplot analysis for individual sections. We would like to note that, apart from three sections (automotive, mechanical engineering, and services), the telling force of analyses from other sections was lower due to the small sample of respondents. In all three mentioned sections, better values were observed with integrated safety (IS) evaluation than with maintenance (eM) evaluation. It is apparent that the number of respondents was highest in the automotive section.
The corresponding boxplots for the automotive section are very similar to those being reached with the evaluation of all respondents. For machinery integrated safety (IS), in the areas of both services, as well as mechanical engineering, a lower quartile range was reached than in the case of the automotive section.
In recent years, maintenance has become more understood: “as a combination of all technical, administrative, and managerial actions during the life cycle of an item intended to retain it in or restore it to a state in which it can perform the required function [60]”. However, maintenance management requires binding property care [61,62] with the business objectives of organizations. The most efficient strategy is predictive maintenance, the aim of which is to optimize maintenance plans based on the measurement of parameters correlated with equipment conditions [63,64]. This strategy requires sufficient resources but mainly specialists’ knowledge and skills [65].
As it follows from research results, focused on two important areas from the I4.0 viewpoint, it has been confirmed that, where predictive maintenance is being introduced, the managers perceive eM implementation as a challenge [9,66]. From the viewpoint of long-term strategy and vision, they understand eM as an important aspect of business sustainability and a rise of competitiveness. The safety level (IS) is standardly put in place within organization management, backed up by legal regulations emphasizing risk reduction already in the phase of machine design until its disposal [67]. Therefore, the evaluation of higher readiness for the transformation in the area of IS is a result of managers’ experience. The area of maintenance management relates to the overall level of organization management, willingness to invest in maintenance and repairs improvement, and collection and evaluation of information, so the lower level of Organizational culture and leadership (OC&L) criterion evaluation comes as no surprise at all.
Yet a question remains unanswered: how, when perceiving integrated safety, the managers can take account of the requirement for security [68], mainly during the creation and management of integrated safety systems and robotic workplaces [11]. The machinery safety directive [17,67] does not deal with this area, which creates space for the research in the field of design and new technical solution of machines with regard to the provision of prevention against external threats.

6. Conclusions

Whatever the interpretation of analyzed results may be, the fact is that the empirical research on a randomly selected sample of top managers of selected Slovak organizations has shown that they perceive the readiness of their organizations in the area of machinery integrated safety (IS) on a higher level (average score 9%) than the readiness of maintenance (eM) to be transformed in terms of I4.0 requirements. A higher variability of point evaluations within the maintenance area (eM) represents a higher diversity of the perception of its readiness. As the subsequent Boxplot analysis, carried out for individual criteria, has shown, in the maintenance area (eM), higher attention should be paid to Organizational culture and leadership, Engaging stakeholders, and Driving performance and transformation criteria. By focusing higher attention to the issue characterized by mentioned criteria, it is possible, based on empirically performed research, to achieve a faster and more efficient maintenance (eM) transformation in terms of I4.0. The issue of sharing purpose, vision, and strategy should be the subject of interest of managers in order to improve integrated safety (IS) in organizations. The stratification of individual respondents, according to sections derived from the focus of organizations that the respondents come from, has not brought any statistically significant deviations in the three main sections, which statistically had s significant number of respondents. Boxplot graphs from such stratified data approximately correspond to boxplots acquired from the whole sample of respondents. This condition (except for the mentioned variability or statistically insignificant number of respondents in the selected sections) may be interpreted as approximately the same level of readiness of Slovak organizations for I4.0 requirements, regardless of their classification into sections.
Using the new EFQM 2020 model to analyze the readiness of companies for I4.0 in a specified area (in our case IS and eM) represents a simple tool for improvement of organization. The evaluation of direction and execution helps to identify weaknesses in relation to the requirements of stakeholders, performance, and sustainability, the achievement of own objectives in the analyzed areas, and their overall impact on the level of I4.0 implementation in the organization.
Although the use of the EFQM 2020 model for assessing the readiness of organizations to implement I4.0 represents a limitation [54], its results may be the basis for further analyses and methods to examine the effectiveness of digitization (I4.0 pillars) in Slovak companies. One of the challenges for future research is the evaluation of stakeholders’ perceptions after I4.0 is implemented in the company and their comparison before and after implementation.

Author Contributions

Application of statistical, mathematical techniques, writing and final review of the paper, R.T. and H.P.; management and validation, H.P. and A.N.; development of methodology, R.T. and Z.K.; research, data collection, and final review of the paper, P.K., Z.K. and M.H. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Not applicable.

Acknowledgments

This contribution is the result of the projects implementation: Ministry of Education, Science, Research, and Sport of the Slovak Republic APVV No. 19-0367; APVV No. 15-0527 and KEGA No. 015TUKE-4/2019; and Research & Development Operational Program funded by the ERDF ITMS: 26220220182.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table
Table A1. Questionnaire items based on the modified EFQM methodology.
Table A1. Questionnaire items based on the modified EFQM methodology.
CriterionQuestionAnalysis of Integration Safety (IS) and Maintenance (eM)
PV&S1In your organization, is there a clearly defined purpose/target for IS/eM areas (e.g., vision of safety, asset management, its maintenance and I4.0)?
PV&S2Does your organization look into the needs of involved parties for IS/eM areas (e.g., requirements of legislation, customers and employees in safety area, safety devices that must be implemented on ‘your’ machines and their maintenance)?
PV&S3Does the organization collect information and seek opportunities or challenges for IS/eM areas (e.g., requirements for a Logout/Tagout system (LOTO) and supplied machines, additional safety devices, or news, so that high safety culture in the organization is reached)?
PV&S4Is the strategy of your organization focused on seeking adequate answers to challenges for IS/eM areas? (e.g., is there a preference for predictive maintenance and its diagnostics for critical machines? Is there a high integrated safety level required, e.g., are the best solutions searched for?)
PV&S5Have you implemented a system of reporting in order to correctly check the services and find process effectivity for IS/eM areas? (have you implemented key performance indicators (KPIs) for safety and maintenance? Are they measurable? Is their regular evaluation clearly defined in the system? Is customer satisfaction evaluated: customers and also maintenance, e.g., operational manager and occupational health and safety (OHS) manager)
OC&L6Does your organization define its values and derive specific steps from them for IS/eM areas? (are (key performance indicators (KPIs) in maintenance in accordance with the goals of both the organization and OHS? Is there anyone responsible and does their regular evaluation have an effect on the improvement)
OC&L7Is interaction with involved parties at IS/eM area development supported? (are concerned employees addressed, e.g., when choosing protective devices and LOTO devices?)
OC&L8Do you feel support from the part of your organization for employees who want to make changes up to the level of radical transformation in IS/eM area?
OC&L9Does the organization collect feedback from involved parties about activities in IS/eM areas (e.g., the opinion of the operator, maintenance manager, operational manager, customer, OHS manager, etc.)?
ES10Does the organization use communication channels enabling customers to communicate with the organization easily and continuously in IS/eM areas? (Are they also online, …, top level?)
ES11Does the organization enable its employees to actively participate in the value creation process in IS/eM areas? (mainly with regard to I4.0 trends, does it support LOTO training, diagnostic methods, new safety software and devices?)
ES12Do business or regulation oriented involved parties join in with the strategy creation within IS/eM areas (customers and their satisfaction, inspectors, etc.)?
ES13Does the organization actively use all opportunities for its development in IS/eM areas (trainings, seminars, ministry calls for projects, customers, and their requirements, etc.)?
ES14Does the organization join its partners and suppliers in their business activities in such a way that it creates a high degree of transparency and trust, as for the aims related to IS and eM areas (e.g.,: requirements for spare parts suppliers, for safety devices, LOTO solutions, safe behavior, etc.)?
CSV15Does the organization develop its product portfolio in order to meet current and future expectations of customer groups in IS and eM areas?
CSV16Does an external company provide you with activities in the organization for IS/eM areas?
CSV17Does the organization minimize the negative ecological and social effects of its own activities for IS and eM areas?
CSV18Is your organization oriented on building customer experience in IS and eM area (e.g., the feedback work, failures, accidents, etc.)?
DP&T19Does your organization identify, analyze, and subsequently manage risks or take account of perspectives in IS/eM areas (risk assessment and management, e.g.,: criticality of machinery, FMEA, (Failure mode and effects analysis) etc., risk assessment during changes on a machine, hazardous energy identification, etc.)?
DP&T20Is the organizational structure of your organization compatible with the current strategy? Is it regularly updated for IS/eM areas?
DP&T21Does the organization introduce new effective technologies for IS/eM areas? (Who is responsible for new trends?)
DP&T22Does the organization use its own knowledge and access knowledge for third parties to cooperate with other involved parties to improve value creation in IS/eM areas (e.g., with research institutions)?
DP&T23Does the organization sufficiently use its financial sources for IS/eM areas?
Note: Integrated safety (IS); eMaintenance (eM); purpose, vision, and strategy (PV&S); organizational culture, and leadership (OC&L); engaging stakeholders (ES); creating sustainable value (CSV); driving performance and transformation (DP&T).

References

  1. Ejsmont, K. The Impact of Industry 4.0 on Employees—Insights from Australia. Sustainability 2021, 13, 3095. [Google Scholar] [CrossRef]
  2. Miśkiewicz, R.; Wolniak, R. Practical Application of the Industry 4.0 Concept in a Steel Company. Sustainability 2020, 12, 5776. [Google Scholar] [CrossRef]
  3. Castelo-Branco, I.; Cruz-Jesus, F.; Oliveira, T. Assessing Industry 4.0 readiness in manufacturing: Evidence for the European Union. Comput. Ind. 2019, 107, 22–32. [Google Scholar] [CrossRef]
  4. Rekordné Číslo. Slovensko Ostáva Lídrom vo Výrobe Áut na Obyvateľa. Available online: https://ekonomika.sme.sk/c/22299052/slovensko-ostava-lidrom-v-produkcii-automobilov-na-obyvatela.html (accessed on 20 January 2021).
  5. Koncepcia Inteligentného Priemyslu pre Slovensko. Available online: https://www.mhsr.sk/inovacie/strategie-a-politiky/smart-industry (accessed on 15 March 2021).
  6. Grenčíková, A.; Kordoš, M.; Sokol, J. The Approach to Industry 4.0 within the Slovak Business Environment. Soc. Sci. 2019, 8, 104. [Google Scholar] [CrossRef] [Green Version]
  7. Turisová, R.; Sinay, J.; Pačaiová, H.; Kotianová, Z.; Glatz, J. Application of the EFQM Model to Assess the Readiness and Sustainability of the Implementation of I4.0 in Slovakian Companies. Sustainability 2020, 12, 5591. [Google Scholar] [CrossRef]
  8. Radanliev, P.; De Roure, D.; Van Kleek, M.; Santos, O.; Ani, U. Artificial intelligence in cyber physical systems. AI Soc. 2020. [Google Scholar] [CrossRef] [PubMed]
  9. Piccarozzi, M.; Aquilani, B.; Gatti, C. Industry 4.0 in Management Studies: A Systematic Literature Review. Sustainability 2018, 10, 3821. [Google Scholar] [CrossRef] [Green Version]
  10. Madsen, D.Ø. The Emergence and Rise of Industry 4.0 Viewed through the Lens of Management Fashion Theory. Adm. Sci. 2019, 9, 71. [Google Scholar] [CrossRef] [Green Version]
  11. Zheng, T.; Ardolino, M.; Bacchetti, A.; Perona, M. The applications of Industry 4.0 technologies in manufacturing context: A systematic literature review. Int. J. Prod. Res. 2020, 1–20. [Google Scholar] [CrossRef]
  12. Felsberger, A.; Reiner, G. Sustainable Industry 4.0 in Production and Operations Management: A Systematic Literature Review. Sustainability 2020, 12, 7982. [Google Scholar] [CrossRef]
  13. Sartal, A.; Bellas, R.; Mejías, A.M.; García-Collado, A. The sustainable manufacturing concept, evolution and opportunities within Industry 4.0: A literature review. Adv. Mech. Eng. 2020, 12, 1–17. [Google Scholar] [CrossRef]
  14. Galvez, N.; Marsot, J.; Martin, P.; Siadat, A.; Etienne, A. EZID: A new approach to hazard identification during the design process by analysing energy transfers. Saf. Sci. 2017, 95, 1–14. [Google Scholar] [CrossRef] [Green Version]
  15. Etherton, J.R.; Myers, M.L. Machine safety research at NIOSH and the future directions. Int. J. Ind. Ergon. 1990, 6, 163–174. [Google Scholar] [CrossRef]
  16. Gardner, D.; Carlopio, J.; Fonteyn, P.N.; Cross, J.A. Mechanical Equipment Injuries in Small Manufacturing Businesses. Knowledge, Behavioral, and Management Issues. Int. J. Occup. Saf. Ergon. 1999, 5, 59–71. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  17. European Parliament and of the Council. Directive 2006/42/EC of the European Parliament and of the Council of 17 May 2006 on Machinery, and Amending Directive 95/16/EC (Recast); European Parliament and of the Council: Brussels, Belgium, 2006. [Google Scholar]
  18. European Parliament and of the Council. Directive 2009/104/EC of the European Parliament and of the Council of 16 September 2009 Concerning the Minimum Safety and Health Requirements for the Use of Work Equipment by Workers at Work (Second Individual Directive Within the Meaning of Article 16(1) of Directive 89/391/EEC); European Parliament and of the Council: Brussels, Belgium, 2009. [Google Scholar]
  19. Calabrese, F.; Regattieri, A.; Botti, L.; Mora, C.; Galizia, F.G. Unsupervised Fault Detection and Prediction of Remaining Useful Life for Online Prognostic Health Management of Mechanical Systems. Appl. Sci. 2020, 10, 4120. [Google Scholar] [CrossRef]
  20. Mwanza, B.G.; Mbohwa, C. Safety in Maintenance: An Improvement Framework. Procedia Manuf. 2017, 8, 657–664. [Google Scholar] [CrossRef]
  21. Aust, J.; Pons, D. Bowtie Methodology for Risk Analysis of Visual Borescope Inspection during Aircraft Engine Maintenance. Aerospace 2019, 6, 110. [Google Scholar] [CrossRef] [Green Version]
  22. Gligorijevic, J.; Gajic, D.; Brkovic, A.; Savic-Gajic, I.; Georgieva, O.; Di Gennaro, S. Online Condition Monitoring of Bearings to Support Total Productive Maintenance in the Packaging Materials Industry. Sensors 2016, 16, 316. [Google Scholar] [CrossRef] [Green Version]
  23. Xiang, Z.T.; Feng, C.J. Implementing total productive maintenance in a manufacturing small or medium-sized enterprise. J. Ind. Eng. Manag. 2021, 14, 1–24. [Google Scholar] [CrossRef]
  24. Igba, J.; Alemzadeh, K.; Anyanwu-Ebo, I.; Gibbons, P.; Friis, J. A Systems Approach Towards Reliability-Centred Maintenance (RCM) of Wind Turbines. Procedia Comput. Sci. 2013, 16, 814–823. [Google Scholar] [CrossRef] [Green Version]
  25. Carretero, J.; Pérez, M.J.; García-Carballeira, F.; Calderón, A.; Fernández, J.; García, J.D.; Lozano, A.; Cardona, L.; Cotaina, N.; Prete, P. Applying RCM in large scale systems: A case study with railway networks. Reliab. Eng. Syst. Saf. 2003, 82, 257–273. [Google Scholar] [CrossRef]
  26. Ambühl, S.; Dalsgaard Sørensen, J. Sensitivity of Risk-Based Maintenance Planning of Offshore Wind Turbine Farms. Energies 2017, 10, 505. [Google Scholar] [CrossRef] [Green Version]
  27. Tan, Z.; Li, J.; Wu, Z.; Zheng, J.; He, W. An evaluation of maintenance strategy using risk based inspection. Saf. Sci. 2011, 49, 852–860. [Google Scholar] [CrossRef]
  28. Ugechi, C.I.; Ogbonnaya, E.A.; Lilly, M.T.; Ogaji, S.O.T.; Probert, S.D. Condition-Based Diagnostic Approach for Predicting the Maintenance Requirements of Machinery. Engineering 2009, 1, 177–187. [Google Scholar] [CrossRef] [Green Version]
  29. Štancel, M.; Pačaiová, H.; Chovancová, E. Maintenance performance audit software. In Proceedings of the 2017 IEEE 15th International Symposium on Applied Machine Intelligence and Informatics (SAMI), Herl’any, Slovakia, 26–28 January 2017; pp. 455–460. [Google Scholar] [CrossRef]
  30. Carnero, M.C. Selection of diagnostic techniques and instrumentation in a predictive maintenance program. A case study. Decis. Support Syst. 2005, 38, 539–555. [Google Scholar] [CrossRef]
  31. Neslušan, M.; Minárik, P.; Grenčík, J.; Trojan, K.; Zgútová, K. Non-destructive evaluation of the railway wheel surface damage after long-term operation via Barkhausen noise technique. Wear 2019, 420–421, 195–206. [Google Scholar] [CrossRef]
  32. Stejskal, T.; Dovica, M.; Svetlik, J.; Demec, P. Experimental assessment of the static stiffness of machine parts and structures by changing the magnitude of the hysteresis as a function of loading. Open Eng. 2019, 9, 655–659. [Google Scholar] [CrossRef]
  33. Hovanec, M. Digital factory as a prerequisite for successful application in the area of ergonomics and human factor. Theor. Issues Ergon. Sci. 2017, 18, 35–45. [Google Scholar] [CrossRef]
  34. Vaidya, S.; Ambad, P.; Bhosle, S. Industry 4.0—A Glimpse. Procedia Manuf. 2018, 20, 233–238. [Google Scholar] [CrossRef]
  35. Alcácer, V.; Cruz-Machado, V. Scanning the Industry 4.0: A Literature Review on Technologies for Manufacturing Systems. Eng. Sci. Technol. Int. J. 2019, 22, 899–919. [Google Scholar] [CrossRef]
  36. Hovanec, M.; Pačaiová, H.; Hrozek, F.; Varga, M. Proactive Ergonomics Based on Digitalization Using 3D Scanning and Workplace Modeling in Texnomatix Jack with Augmented Reality. Naše More 2014, 61, 22–26. [Google Scholar]
  37. Culot, G.; Nassimbeni, G.; Orzes, G.; Sartor, M. Behind the definition of Industry 4.0: Analysis and open questions. Int. J. Prod. Econ. 2020, 226, 1–15. [Google Scholar] [CrossRef]
  38. Talapatra, S.; Santos, G.; Uddin, K.; Carvalho, F. Main benefits of integrated management systems through literature review. Int. J. Qual. Res. 2019, 13, 1037–1054. [Google Scholar] [CrossRef]
  39. Bennett, N.; Lemoine, G.J. What a difference a word makes: Understanding threats to performance in a VUCA world. Bus. Horiz. 2014, 57, 311–317. [Google Scholar] [CrossRef]
  40. Pačaiová, H.; Nagyová, A. Risk Based Thinking—New Approach for Modern Enterprises’ Management. In Proceedings of the Advances in Human Factors, Business Management and Society, Orlando, FL, USA, 21–25 July 2018; Springer: Cham, Switzerland, 2018; pp. 524–536. [Google Scholar] [CrossRef]
  41. Chiarini, A. Risk-based thinking according to ISO 9001:2015 standard and the risk sources European manufacturing SMEs intend to manage. TQM J. 2017, 29, 310–323. [Google Scholar] [CrossRef]
  42. Pexa, M.; Hladík, T.; Ales, Z.; Legát, V.; Müller, M.; Valášek, P.; Havl, V. Reliability and risk treatment centered maintenance. J. Mech. Sci. Technol. 2014, 28, 3963–3970. [Google Scholar] [CrossRef]
  43. Johansson, N.; Roth, E.; Reim, W. Smart and Sustainable eMaintenance: Capabilities for Digitalization of Maintenance. Sustainability 2019, 11, 3553. [Google Scholar] [CrossRef] [Green Version]
  44. Iunga, B.; Levrata, E.; Marquezb, A.C.; Erbec, H. Conceptual framework for e-Maintenance: Illustration by e-Maintenance technologies and platforms. Annu. Rev. Control. 2009, 33, 220–229. [Google Scholar] [CrossRef]
  45. Jasiulewicz-Kaczmarek, M.; Gola, A. Maintenance 4.0 Technologies for Sustainable Manufacturing—an Overview. IFAC PapersOnLine 2019, 52, 91–96. [Google Scholar] [CrossRef]
  46. Cachada, A.; Barbosa, J.; Leitño, P.; Gcraldcs, G.; Deusdado, L.; Costa, J.; Teixeira, C.; Teixeira, J.; Moreira, A.H.J.; Moreira, P.M.; et al. Maintenance 4.0: Intelligent and Predictive Maintenance System Architecture. In Proceedings of the 2018 IEEE 23rd International Conference on Emerging Technologies and Factory Automation (ETFA), Turin, Italy, 4–7 September 2018. [Google Scholar] [CrossRef]
  47. Bokrantz, J.; Skoogh, A.; Berlin, C.; Wuest, T.; Stahre, J. Smart Maintenance: A research agenda for industrial maintenance management. Int. J. Prod. Econ. 2020, 224, 1–14. [Google Scholar] [CrossRef]
  48. Bumblauskas, D.; Gemmill, D.; Igou, A.; Anzengruber, J. Smart Maintenance Decision Support Systems (SMDSS) based on corporate big data analytics. Expert Syst. Appl. 2017, 90, 303–317. [Google Scholar] [CrossRef]
  49. Pačaiová, H.; Glatz, J.; Kacvinský, Š. Positive and negative aspect in application of maintenance management philosophy. J. Appl. Eng. Sci. 2012, 10, 99–105. [Google Scholar] [CrossRef] [Green Version]
  50. Kužma, D.; Korba, P.; Hovanec, M.; Dulina, L. The Use of CAX Systems as a Tool for Modeling Construction Element in the Aviation Industry. Naše More 2016, 63, 134–139. [Google Scholar] [CrossRef]
  51. Racz, S.G.; Breaz, R.E.; Cioca, L.I. Evaluating Safety Systems for Machine Tools with Computer Numerical Control using Analytic Hierarchy Process. Safety 2019, 5, 14. [Google Scholar] [CrossRef] [Green Version]
  52. Pacana, A.; Siwiec, D.; Bednárová, L. Method of Choice: A Fluorescent Penetrant Taking into Account Sustainability Criteria. Sustainability 2020, 12, 5854. [Google Scholar] [CrossRef]
  53. EFQM Model. The EFQM Excellence Model; EFQM: Brussels, Belgium, 2020; Available online: http://www.efqm.org (accessed on 25 February 2021).
  54. Nenadál, J. The new EFQM model: What is really new and could be considered as a suitable tool with respect to quality 4.0 concept? Qual. Innov. Prosper. 2020, 24, 17–28. [Google Scholar] [CrossRef] [Green Version]
  55. Suárez, E.; Calvo-Mora, A.; Roldan, J.R.; Perianez-Cristobal, R. Quantitative research on the EFQM excellence model: A systematic literature review (1991–2015). Eur. Res. Manag. Bus. Econ. 2017, 23, 147–156. [Google Scholar] [CrossRef]
  56. Wang, G.; Liu, H.; Li, H.; Luo, X.; Liu, J. A Building Project-Based Industrialized Construction Maturity Model Involving Organizational Enablers: A Multi-Case Study in China. Sustainability 2020, 12, 4029. [Google Scholar] [CrossRef]
  57. Leonard, D. Leveraging the Benefits of the EFQM Innovation Lens and ISO 56002 Innovation Management Systems Guidance. Qual. Manag. Forum 2020, 46, 1–7. [Google Scholar]
  58. Fonseca, L.; Amaral, A.; Oliveira, J. Quality 4.0: The EFQM 2020 Model and Industry 4.0 Relationships and Implications. Sustainability 2021, 13, 3107. [Google Scholar] [CrossRef]
  59. Ghasemi, A.; Nadiri, M. Performance assessment of Iranian petrochemical companies using sustainable excellence model. Saf. Sci. 2016, 87, 280–291. [Google Scholar] [CrossRef]
  60. Cavalcante, C.A.V.; Lopes, R.S. Multi-criteria model to support the definition of opportunistic maintenance policy: A study in a cogeneration system. Energy 2015, 80, 32–40. [Google Scholar] [CrossRef]
  61. Ghobakhloo, M. Industry 4.0, digitization, and opportunities for sustainability. J. Clean. Prod. 2020, 252, 1–21. [Google Scholar] [CrossRef]
  62. Maletič, D.; Maletič, M.; Al-Najjar, B.; Gomišček, B. An Analysis of Physical Asset Management Core Practices and Their Influence on Operational Performance. Sustainability 2020, 12, 9097. [Google Scholar] [CrossRef]
  63. Opocenska, H.; Nahodil, P.; Hammer, M. Use of multiparametric diagnostics in predictive maintenance. MM Sci. 2017, 2017, 2090–2093. [Google Scholar] [CrossRef]
  64. Baron, P.; Kočiško, M.; Dobránsky, J.; Pollák, M.; Telišková, M. Research and application of methods of technical diagnostics for the verification of the design node. Measurement 2016, 94, 245–253. [Google Scholar] [CrossRef]
  65. Tureková, I.; Depešová, J.; Bagalová, T. Machinery Risk Analysis Application in the System of Employee Training. Appl. Mech. Mater. 2014, 635–637, 439–442. [Google Scholar] [CrossRef]
  66. Zgodavova, K.; Kisela, M.; Sutoova, A. Intelligent approaches for an organisation’s management system change. TQM J. 2016, 28, 760–773. [Google Scholar] [CrossRef]
  67. Pačaiová, H.; Andrejiová, M.; Balažiková, M.; Tomašková, M.; Gazda, T.; Chomová, K.; Hijj, J.; Salaj, L. Methodology for Complex Efficiency Evaluation of Machinery Safety Measures in a Production Organization. Appl. Sci. 2021, 11, 453. [Google Scholar] [CrossRef]
  68. Tubis, A.A.; Werbińska-Wojciechowska, S.; Góralczyk, M.; Wróblewski, A.; Ziętek, B. Cyber-Attacks Risk Analysis Method for Different Levels of Automation of Mining Processes in Mines Based on Fuzzy Theory Use. Sensors 2020, 20, 7210. [Google Scholar] [CrossRef]
Sustainability 13 03682 g001 550
Figure 1. Innovative EFQM 2020 model [53].
Figure 1. Innovative EFQM 2020 model [53].
Sustainability 13 03682 g001
Sustainability 13 03682 g002 550
Figure 2. Division of the EFQM 2020 model on the assumptions and results part [53].
Figure 2. Division of the EFQM 2020 model on the assumptions and results part [53].
Sustainability 13 03682 g002
Sustainability 13 03682 g003 550
Figure 3. Point evaluation of answers to questions from particular criteria (Source: own research).
Figure 3. Point evaluation of answers to questions from particular criteria (Source: own research).
Sustainability 13 03682 g003
Sustainability 13 03682 g004 550
Figure 4. Total evaluation of research results by means of a boxplot analysis (Source: own research).
Figure 4. Total evaluation of research results by means of a boxplot analysis (Source: own research).
Sustainability 13 03682 g004
Sustainability 13 03682 g005 550
Figure 5. Boxplot analysis of research results according to criteria (Source: own research).
Figure 5. Boxplot analysis of research results according to criteria (Source: own research).
Sustainability 13 03682 g005
Sustainability 13 03682 g006 550
Figure 6. Boxplot analysis of research results according to selected sections (Source: own research).
Figure 6. Boxplot analysis of research results according to selected sections (Source: own research).
Sustainability 13 03682 g006
Table
Table 1. Assumption criteria.
Table 1. Assumption criteria.
Assumption Criteria Description
Purpose, vision, and strategy
(PV&S)		An excellent organization is defined by inspiring purpose and vision, developing an effective strategy aimed at the creation of sustainable advantages and creation, and implementation of an organization management system and efficiency management system.
Organizational culture and leadership
(OC&L)		Organizational culture is conditioned by values and standards of organization behavior that are shared by its employees in the organization and which form their behavior, as well as their behavior towards involved parties that are important from the view of purpose, vision, and strategy.
Engaging stakeholders
(ES)		An excellent organization identifies individual types and categories within involved parties (customers, employees, company, partners, suppliers) that are important for purpose, vision, and strategy. It creates a sustainable value, evaluates its own performance, and takes appropriate measures to ensure its own future.
Creating sustainable value (CSV)An excellent organization understands that creating sustainable values is crucial for long-term development and financial force. A clearly defined purpose, together with a strategy, determine for whom the organization creates sustainable advantages.
Driving performance and transformation
(DP&T)		In order for an organization to be excellent today and in the future, it must be able to successfully manage and improve everyday business and manage changes that constantly happen inside and outside the organization.
Table
Table 2. Point evaluation of integrated safety (IS) and e-Maintenance (eM) areas.
Table 2. Point evaluation of integrated safety (IS) and e-Maintenance (eM) areas.
Criterion.ISeMTotal
Number of QuestionsMax. PointsTotal PointsNumber of QuestionsMax. PointsTotal Points
PV&S5168051680160
OC&L4208042080160
ES5168051680160
CSV445180445180360
DP&T5168051680160
Total23-50023-5001000
Note: Integrated safety (IS); eMaintenance (eM); purpose, vision, and strategy (PV&S); organizational culture and leadership (OC&L); engaging stakeholders (ES); creating sustainable value (CSV); driving performance and transformation (DP&T).
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Turisová, R.; Pačaiová, H.; Kotianová, Z.; Nagyová, A.; Hovanec, M.; Korba, P. Evaluation of eMaintenance Application Based on the New Version of the EFQM Model. Sustainability 2021, 13, 3682. https://doi.org/10.3390/su13073682

AMA Style

Turisová R, Pačaiová H, Kotianová Z, Nagyová A, Hovanec M, Korba P. Evaluation of eMaintenance Application Based on the New Version of the EFQM Model. Sustainability. 2021; 13(7):3682. https://doi.org/10.3390/su13073682

Chicago/Turabian Style

Turisová, Renata, Hana Pačaiová, Zuzana Kotianová, Anna Nagyová, Michal Hovanec, and Peter Korba. 2021. "Evaluation of eMaintenance Application Based on the New Version of the EFQM Model" Sustainability 13, no. 7: 3682. https://doi.org/10.3390/su13073682

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop