An Intelligent Framework for the Evaluation of Compliance with the Requirements of ISO 9001:2015

Abstract

A quality management system (QMS) allows an organization to analyze the needs of its customers and to define the appropriate processes for efficient production and service delivery. Customer satisfaction is an important part of this system, with a significant impact on its corporate sustainability. The purpose of this research is a proposal for assessing whether the QMS of an organization meets the requirements defined in the standard ISO 9001:2015. The development of the proposal follows a methodology based on the paradigm of design science research where we identify the problem and the motivation and the objectives are defined as a solution to the problem, and as a consequence, we developed an expert system based on a previously identified ISO 9001:2015 process map and transferred it to a knowledge base. The system allows the determination of the degree of compliance with the requirements of the standard: this provides valuable information to organizations, allowing them to implement improvement measures. The expert system is limited in terms of the set of identified inputs, outputs, and acceptance criteria, but its effectiveness was demonstrated through the evaluation of the system.

1. Introduction

Nowadays, companies need to adapt to an increasingly competitive market due to the globalization of the economy. In this situation, quality in products and processes is one of the main strategies adopted by organizations [1]. Porter [2] considered that a product differentiated by quality and design guarantees the loyalty of the buyer, while one poorly differentiated would motivate possible substitutions with others of equal or low cost.

In addition, achieving customer satisfaction is one of the fundamental objectives of organizations today, so more and more organizations have focused their efforts on improving the relationship with stakeholders such as customers, suppliers, workers, owners of an organization, investors, competitors, legislators, public bodies, and society in general.

Therefore, stakeholder engagement is key to achieving these goals and an important component of good corporate governance. This requires the participation of stakeholders in the sustainable use and management of resources, training, communication, and awareness of the problems. This is one of the reasons why organizations often implement quality management systems (QMSs).

Organizations are aware of how happy customers tend to be loyal, while unsatisfied customers may cause indirect losses of 10 additional customers [3,4]. Consistent with this idea, Hill and Alexander [5] quantified the number of customers an average organization loses as between 10 and 30 each year. In these cases, the problem is not only to lose them as customers but also to know the reason why they decide to stop buying.

Martínez-Perales [6] shows a clear relationship between the management system standards and sustainability, considering ISO 9001:2015 [7] as one of the standards. Customer satisfaction (Section 9.1.2 ISO 9001:2015) is one of the aspects related to the economic dimension of the set of identified sustainability variables (economic, environmental, and social). Strenitzerová et al. [8] indicated that the success of customer satisfaction is closely related to a company’s quality management and has a significant impact on the company’s future and its customer-based corporate sustainability.

Although the standard ISO 9001:2015 indicates the minimum requirements to be met by the organization, in some cases, it is necessary to develop or clarify how each of them is carried out. Furthermore, this type of quality standard helps organizations to reduce the loss of knowledge as they encourage the documentation of the different processes and procedures. However, this is not enough in many cases, as they are only recommendations.

The implementation of a QMS in an organization represents an important investment. In recent years, although local and national public administrations have economically supported the adoption of QMS standards in Spain [9,10], the number of certificates remained stable, despite the tough economic crisis suffered in the country. In fact, the number of organizations certified according to ISO 9001 and ISO 14000 is, respectively, around 32,000 and 13,500 [11]. Moreover, we should also consider that many organizations are in the process of adapting their QMS, following the publication of ISO 9001:2015 in 2015 and the subsequent cancellation of ISO 9001:2008.

An additional aspect is the constant integration and modernization of information and communication technologies (ICTs) in organizations. This may allegedly represent a change of mentality in the search for a sustainable environment through the optimization of material and human resources in Spanish organizations; however, this approach remains poorly implemented. A report [12] from 2018 determined the use of ICTs in Spanish organizations with a synthetic indicator that measures the digitalization of organizations of ten sectors, and information and communication organizations appeared as the most digitized in all the categories, namely, in the area of environmental management systems for SMEs and large organizations (52 points out of 100) and also in the area of microenterprises (34.7 points out of 100) [12,13].

The government has launched different plans in order to improve this situation. Some good examples are Avanza and Avanza 2 [14] and the actions of the Digital Agency for the Development of the Information Society. One of the main objectives of these initiatives is the implementation of business solutions based on the use of ICTs to increase competitiveness and productivity of organizations [15].

According to [16], applications based on artificial intelligence (AI) increase the benefits of sustainability design and commercialization and also directly enhance sustainable growth and performance. The positive effects of applications based on AI on sustainable growth and performance represent a clear impulse for sustainability design and commercialization. The integration of an “expert” component in the structure of information systems for sustainable development is represented by [17]. The expert systems are an example of sustainable environmental management.

A series of reports by Accenture [18] indicates that 84% of executives believe they will not achieve their growth objectives unless they scale up AI. Moreover, 75% of them believe they risk going out of business in five years if they do not scale up AI, and 76% of them struggle with how to scale up AI across the business. Gartner’s report [19] recommends that vendors use the simplest approach that can do the job on state-of-the-art artificial intelligence techniques. The report presented by Capgemini [20] showed that 78% of organizations implementing AI increase operational efficiency by more than 10%. As a conclusion of these studies, it is clear that most companies will choose artificial intelligence to improve their income in the future.

The purpose of this research is to analyze and propose an intelligent framework that allows the evaluation of compliance of a part of an organization or company with the normative requirements of ISO 9001:2015. It will allow the detection of deficiencies in an efficient way (sustainability), such as the optimization of material and human resources.

This intelligent framework is represented by an expert system considered as an expression of the knowledge-based systems that constitute the field of study of AI. The simpler the AI, the easier it will be to be adapted. As a consequence, we have chosen an ES for our research due to its simplicity, low cost, and maintenance, which will then positively impact the organization, thanks to its easy adaptation and use that will enhance sustainability.

The importance of this work resides in two related concepts: one is the assessment of compliance with the requirements, and the other is the improvement in quality, which can provide a competitive advantage over the organization’s competitors [21]. With this tool, organizations will be able to evaluate the degree of compliance with the different requirements demanded by the standard ISO 9001:2015. Once the requirements have been evaluated, the organization will be able to adopt a series of decisions to correct the detected deviations and attenuate the loss of knowledge with the use of AI, supported by the tasks of education and extraction of knowledge, which will be transformed into rules.

This document is organized as follows: Section 2 analyzes the related work, Section 3 describes the methodology, Section 4 presents the design and development of the artefact, Section 5 describes the validation and evaluation of the system, Section 6 includes a discussion and, finally, Section 7 presents the conclusions.

2. Concepts and Related Works

2.1. Overview of ISO 9001:2015

The ISO standards are a set of standards aimed at organizing the management of an organization in its various activities (from manufactured products to quality). These standards are defined by the International Organization for Standardization (ISO), which is focused on the creation of international standards to facilitate world trade.

The ISO 9000 family of standards is a set of quality and quality management standards that specify the requirements for a QMS. These standards apply to any type of organization regardless of its size or sector of activity.

In addition, they include the minimum content of QMSs, such as guides and specific implementation tools [22]. These standards are divided into

• ISO 9000. Quality Management Systems—Basic Principles and Vocabulary.
• ISO 9001. Quality Management Systems—Requirements.
• ISO 9004. Quality Management Systems—Guidelines for Improving Performance.

The revision of the ISO 9001 standards happens every 5 years [22]. In 2012, a revision was made by adding some improvements. The edition of 2015 included 10 clauses instead of 8, but they did not impact the standard requirements and did not imply changes in its management system. The structure of the standard is distributed in clauses as follows: (1) Object and field of application, (2) Normative references, (3) Terms and definitions, (4) Context of the organization, (5) Leadership, (6) Planning, (7) Support, (8) Operation, (9) Performance evaluation, and (10). Improvement.

The ISO 9001:2015 [7] considers the activities of an organization, regardless of its size and industry. This standard aims at increasing customer satisfaction by the conformity of provided products and services. It is one of the most recognized standards in the world, with more than one million organizations already certified [23].

According to ISO 9001:2015, the adoption of a QMS should be considered as “a strategic decision for an organization that can help it improve its overall performance and provide a solid foundation for sustainable development initiatives”.

ISO 9001 involves benefits related to the efficient use of resources, process improvement, and increased customer satisfaction. Fonseca et al. [24] performed a bibliometric study of the benefits of management systems certification (MSC), identifying the works that support those benefits. These benefits can be

• Internal: benefits such as improved product quality, better process performance, short delivery times, cost reductions, improved system documentation, higher quality awareness.
• External: benefits such as improved customer satisfaction, better market image, and stronger competitive position.

The results of the study supported the view that the research of MSC benefits is indeed a topic of high interest for both academia and practitioners. The most valued standards were ISO 9001 Quality Management Systems Certification (QMSC) and ISO 14001 Environment Management Systems Certification (EMSC).

Zimon and Zimon [25] confirmed that the implementation of the QMS, according to ISO 9001, improves the processes related to the management of supplier responsibilities. This efficiency translated into high financial liquidity, which was confirmed by a strong correlation between the ratio of short-term liabilities to financial liquidity.

Domingues et al. [26] concluded that the selection of auditors with the appropriate knowledge and experience in ISO 9001:2015 to evaluate the QMS of ISO 9001:2015 contributed to the business and processes, as well as performance improvement and sustainable outcomes.

The version ISO 9001:2008 [27] was updated to ISO 9001:2015 [7], promoting risk-based thinking. It provides a process-oriented approach to documenting and reviewing the structure, responsibilities, and procedures necessary to achieve effective quality management in an organization, including strong customer orientation, motivation and involvement of top management, and continuous improvement.

In this version, the concept of “product” is replaced by “products and services”. Its approach is based on risks, and the concept of preventive actions disappears. The commitment to quality is reinforced through strong leadership.

With the new revision of the ISO 9001 standards, the basic principles of quality management have been reduced to seven [28]: (1) Customer focus, (2) Leadership, (3) Engagement of people, (4) Process approach, (5) Improvement, (6) Evidence-based decision-making, and (7) Relationship management.

The most significant changes introduced in this version of the standard, extracted from different sources [23,29], are detailed below:

• Leadership: ISO 9001:2015 reinforces the commitment to quality through strong leadership and eliminates the figure of the management representative. It extends leadership to all levels of the organization.
• Risk-Based Thinking is also one of the main changes in the new version of ISO 9001. The standard does not include the concept of preventive action and is replaced by a way of thinking, adding some systematic assessment of potential and actual issues with the aim of making processes more robust and capable.
• Interested parts: The identification of the needs and expectations of interested parts (stakeholders) is one of the new requirements of ISO 9001 2015, in point 4.2. Stakeholders are, among others, internal and external customers, shareholders, owners, employees, suppliers, and partners, as well as society.
• Change control: Point 6.3 clearly details the need for change planning and control, something not very clear in the current version of ISO, and that in an environment like the current one (where change is a constant), it is essential.
• Strategic Directions: This requirement has been added to the management review to try to meld business and quality systems.
• Knowledge Management: It requires organizations to consider knowledge to ensure the quality of the goods and services produced. These requirements, referring to organizational knowledge, are included in order to avoid loss of knowledge or mistakes in knowledge capture and distribution. They can stimulate the acquisition of knowledge by the organization, for example, through, experiential learning, mentoring, or benchmarking.

Customer satisfaction is one of the objectives to be achieved by organizations, and one of the options to achieve this is to implement a QMS meeting the requirements of ISO-9001. Therefore, it is important to carry out a detailed investigation of this standard, which will facilitate the extraction of knowledge from it.

2.2. Expert Systems

An expert system (ES), one of the branches of AI, can be defined as a computer system that simulates human experts in a given area [30]. These systems can be used by nonexpert people to improve their problem-solving skills.

The ES provides adequate technology to automate reasoning processes to solve problems where more traditional computational solutions are not adequate. They are used in different fields of application: medicine, chemistry, geology, or computer science. They help inexperienced people to solve problems that require “specialized formal knowledge”, getting frequently faster conclusions than human experts.

The basic structure of an ES has the following components [31]:

• Knowledge Base: It contains the knowledge necessary to understand, formulate, and solve problems.
• Inference Engine: A rules interpreter that determines which rules are applicable at any given moment and that oversees their execution.
• Fact Base: work memory with the facts about a problem.
• Knowledge Acquisition: It allows the acquisition of knowledge and stores it in its knowledge base in an automated way.
• User Interface: It allows interaction between the ES and the user, presenting and facilitating the management of information.
• Explanation Subsystem: explains the lines of action followed by the ES.

The knowledge of an ES can be represented in several ways. A common method of representing knowledge, as a very simple representation of human reasoning, is in the form of type of rules like

IF <Premise> THEN <Conclusion>

As seen in this section, we have determined the most important aspects considered to develop an ES as the chosen artefact to support the solution to the problem. The choice of this artefact is conditioned by its simplicity, thus promoting the use of simple and sustainable methods that facilitate their implementation in companies. The basic components are presented. They will serve as a reference in the development of the ES, which, in turn, will provide a solution to the problem of identifying the points to be improved to achieve the final objective: checking the compliance with the requirements of ISO 9001.

2.3. Expert Systems Applied to Quality Systems

There are a large number and variety of publications on ES in general, but few contributions (see

Table 1. Expert systems applied to quality systems.

Authors	Year Pub.	Soundest Conclusions
Gipe and Jasinski [32]	1986	An ES was developed to solve problems in quality systems, such as the detection of significant variations in process production, the diagnosis of out-of-control events using a rule base, and the analysis of undiagnosed out-of-control events using additional process data or process knowledge.
Evans and Lindsay [33]	1987	An ES of statistics applied to quality control was developed. This system not only selects the experience in the control table, but it rather offers interpretations of such graphs, providing conclusions about the control process.
Elshennavy and Hosni [34]	1988	A knowledge-based quality control system was developed that is suitable for variable-specific inspection procedures and for the selection of control charts.
Pfeifer [35]	1989	The successful results of an ES applied to the detection of defects during the production process in Germany were described.
Ntuen, et al. [36]	1989	An ES was developed and, by applying pattern recognition, it was able to inspect manufacturing processes.
Lee, et al. [37]	1989	The development of an ES to evaluate quality controls was reported.
Crawford and Eyada [38]	1989	This ES was developed for the purpose of planning the allocation of resources for the quality assurance program.
Eyada [39]	1990	This ES was developed for the evaluation of procedures within the quality control audit, considering the process for suppliers and products.
Brink and Mahalingam [40]	1990	This ES evaluates the quality of production level, making it possible to detect and correct defects that are produced during the production process.
Fard and Sabuncuoglu [41]	1990	An ES was developed to select, through attribute sampling, the most appropriate type of sampling for each case: single, double, or multiple.
Ohta and Kanagawa [42]	1990	A project of sampling by attributes based on fuzzy logic was proposed.
Allen and Kathawala [43]	1992	It presents a summary of several current ES used in quality management. It also shows a framework for selecting suitable application candidates and a guide for integrating ES technology into these applications. Following the framework will significantly improve the chances of a successful ES implementation.
Rehbein et al. [44]	1992	Different applications in the process control industry are analyzed. It also includes several case studies that discuss the justification, development, and implementation of ES applications in this field.
Moore [45]	1995	An ES for process control at a general level was developed.
Paladini [46]	2000	The development of an ES for quality inspection evaluation was published.
Liao et al. [47]	2004	An expert advisory system (in the shape of an ES) for ISO 9001 was proposed. This system integrated the ISO 9001 quality system guidelines and an evaluation approach based on the criteria of the Malcolm Baldrige National Quality Award (MBNQA) into a knowledge-based ES.
Reffat and Harkness [48]	2001	It proposes the development of an ES to assess the effects on the environmental quality of proposed modifications to an office building following a post-occupancy assessment. This model was called the Expert System for the Evaluation of Environmental Quality (ESEQE).
Liukkonen et al. [49]	2011	It presents works focused on controlling production rather than managing it. It also explains how a system can exploit actual production data and use them to diagnose and optimize manufacturing processes.
Ling-Zhong [50]	2010	It explains how an ES uses fuzzy quality and entropy function deployment logic to determine the intensity of criteria at trade fairs to develop a fuzzy decision support system. The resulting fuzzy values can be used to analyze the variance and importance of criteria at trade fairs more effectively compared to sharp values.
Behbahani et al. [51]	2012	The use of a knowledge-based system (KBS) for statistical process control (SPC) was proposed to organize this area of knowledge using a case study through case-based reasoning (CBR).
Andrade et al. [52]	2011	It presents an ES oriented towards the ISO 9001, which facilitates audit organization processes while preparing the organization for possible audits.
Bewoor et al. [53]	2012	It presents another solution to advise on compliance by providing preventive and corrective actions to the organization.
Herghiligiu et al. [54]	2019	It presents the modeling with fuzzy logic of the influence of the integration of an environmental management system (EMS), which can support sustainable development and value creation.

) have focused on ES applied to quality management.

Some of these ES are based on previous versions of ISO 9001:2015 and have mainly focused on the improvement of individual processes, such as the detection of errors in the production process and the control of the supplier process. The ES proposed in our research focuses on the most recent version of the standard and will allow checking the degree of fulfillment of requirements by an organization to comply with the ISO 9001 standard in its QMS.

2.4. Ontologies

An ontology is an explicit specification of a conceptualization [55]. Gruber [56] used this concept in the context of knowledge sharing, where an ontology is a description of the concepts and relationships that can exist for an agent or a community of agents.

According to [57], the idea is to transform information into knowledge using formalized knowledge structures (ontologies) that reference the data through metadata, under a common standardized scheme on some knowledge domain. The benefits of using ontologies proposed by [57] can be summarized as follows:

• Provides a way to represent and share knowledge using a common vocabulary.
• Allows the use of a knowledge-sharing format.
• Provides a specific communication protocol.

The authors of [58] presented a requirements elicitation tool called ElicitO, aimed at empowering requirements analysts with a knowledge repository that helps in the process of capturing precise nonfunctional requirement specifications during elicitation interviews. This tool helps to automate the process of identifying nonfunctional requirements (NFRs) relevant to a certain domain. It also helps the requirements analysts in the process of requirement elicitations to obtain a rapid understanding of all relevant functional and nonfunctional requirements of a given domain.

Additionally, [59] provides guidelines for specifying ontologies at the knowledge level, particularly on how to deal with the specification for conceptualization. The proposed methodology is based on the work by Fernandez [60], which combines a set of stages and strategies to build ontologies with the following steps:

• Step 1—Specification. The goal is a document describing the ontology.
• Step 2—Conceptualization. It is aimed at organizing and structuring the acquired knowledge using an external representation language that is independent of implementation languages and environments.
• Step 3—Implementation. It is focused on implementing the conceptual model into a formal language like Ontolingua, Resource Description Framework Schema (RDF/S), or Web Ontology Language (OWL).
• Step 4—Evaluation. It is basically a technical judgment of the ontology, its software environment, and the documentation regarding a frame of reference.

3. Methodology

We followed the methodology proposed by Peffers [61], based on the paradigm of design science research (DSR), which has its roots in the science and engineering aspects of AI. Design research uses design as a research method [62], with the aim of creating “solutions to specific classes of relevant problems through a rigorous process of construction and evaluation” [63].

The entry point was a problem-centered approach based on previous studies by Porter, where a product that is not differentiated by quality and design will affect the loyalty of the buyer, possibly resulting in replacement with other products.

The proposed method [61] describes the design process in six steps (shown in Figure 1):

• Identify Problem and Motivate: The researcher must identify the problem and the motivation to understand the context of the problem and thus define the objectives to be achieved. Following the proposed methodology, the first step is identifying the problem and motivation reflected in the introduction. Since a deficient QMS can affect customer satisfaction, the knowledge of the degree of compliance with the requirements demanded by ISO 9001 can improve this aspect—this is our objective and motivation.
• Define Objectives of a Solution: The researcher must consult several sources, both from industry and from academia, to suggest a design to achieve and test the proposed objectives. In Section 2, we consulted a set of sources to contextualize the problem, choosing the design of an ES. This will be the artefact used to help the organization to verify the degree of compliance with the requirements proposed by ISO 9001, providing information on the set of processes and on each of the processes individually.
• Design and Development: The researcher must develop an artefact for the suggested identified solution that best adapts to the business domain. The details of this step are presented in Section 4, where ISO 9001:2015 is discussed in detail (Section 4.1). The basis for developing the ES is presented (Section 4.2), explaining how knowledge of the standard is extracted by generating a formalized data structure. The architecture of the ES is explained, detailing how the user interfaces. It also specifies the programming language, as well as the different libraries and tools (Section 4.3).
• Demonstration: The researcher must demonstrate the use of the artefact to solve one or more instances of the problem. Section 4.3 explains how the system is verified.
• Evaluation: The researcher must conduct an evaluation, which may be based on a formal evaluation, test activities, simulations, usability studies, and/or a case study. Details of this step can be found in Section 5.
• Communication: Finally, the researcher must provide a conclusion related to the evaluation of the results of the process by communicating the results. This information is detailed in Section 6.

4. Design and Development of the Artefact

Continuing with the third step of the methodology, the following phases in the design and development of the System have been followed.

4.1. System Analysis

An analysis of the different processes contained in ISO 9001:2015 has been carried out, obtaining a map of the process of four levels:

• Planning Processes. These are those that are directly related to the responsibilities of senior management and that establish the objectives and quality policy to be carried out by the organization.
• Resource management processes. These provide, manage, and maintain the necessary resources for the production of the product or service within the organization.
• Product realization processes. They oversee the production or provision of the service of the organization.
• Measurement and analysis processes. They are responsible for monitoring and evaluating the rest of the processes, measuring, and suggesting improvements.

These four levels have been broken down into other sublevels, as shown in Figure 2. A total of 19 processes have been identified, which have been traced to the requirements established by the standard (see Figure 2). The identification of the processes has been carried out by an expert in the field, based on his experience and knowledge of the regulations.

4.2. System Development

The following phases have been followed for the development of the ES:

Phase 1

—Knowledge Base. Constructed through 7 steps. It explains (1) how the process map is obtained from the standard, (2) how traceability of the processes and subprocesses is generated, (3) how to identify the set of requirements applicable to each process that will later be evaluated by the ES, (4) how to verify that all the requirements are perfectly identified, (5) how these requirements are transformed into checklists, (6) details of how the set of rules is generated from all this information, which forms the knowledge base, and (7) how the knowledge structure is transformed into a standardized format that will be used by the ES.

Phase 2

—System Design. Proposing an architecture to support the ES, showing how each process defined by the standard is going to be evaluated, indicating what information needs to be entered, and what is going to result as output. It also shows the most important aspects to configure the ES to evaluate the whole defined map of processes. Finally, it summarizes all the premises to develop the system, such as the use of technologies, type of architecture, authentication requirements, and profile management.

Phase 3

—Design Interface. Defining the graphic aspect of the system, acting as a human–machine interface (HMI).

The next subsection shows the details of these phases.

4.2.1. Phase 1—Knowledge Base

The objective of this research is not to develop an ontology but to present a possible basis for future research to continue this work. The first step is a preliminary study to deduct the most important aspects of the use of an ontology: this will serve as a method to develop a standardized structure of knowledge in the design phase. We only need a standardized knowledge structure to develop our ES, so the idea is to use the basis of a possible ontology but the ontology is limited to the knowledge structure.

At this stage, all the information required for the knowledge base is already generated. This information must be transformed into data (the knowledge base of the system) that an ES can handle to evaluate the processes. The ES will be designed to evaluate processes independently of how those are integrated into a life cycle. The method focuses on the inputs and outputs of each process and the acceptance criteria.

One key element in the definition of processes is preconditions. They are low-level details that the user needs to know at the beginning of the evaluation of the QMS. They allow us to break down and define the scope of specific requirements. This information has been collected from a human expert (domain expert in the quality area) through systematic interviews by the knowledge engineer.

With these first data, the large majority of the requirements are already defined, and we can get other additional information such as

• Each of the processes of the QMS. All these processes are reviewed by the domain expert.
• Each of the requirements for each of the processes. All these requirements are reviewed by the domain expert.
• The input and output items required by each process. The domain expert defined, according to his experience and knowledge, a series of specific inputs and outputs for each process, a total of 69. The knowledge engineer, through different interviews with the expert, created a map showing each defined element, the process to which it corresponds, and whether it is an output or an input.
• The transition criteria for each process defined by the domain expert. This includes the entry criteria and exit criteria for each process. With this information, the human expert verifies that each of the proposed processes is achieved. At the end of the evaluation, if the end-user expects to meet all the requirements, all the transition criteria of each phase must be satisfied.

All these details are analyzed by the domain expert and put into context with each of the processes. Figure 3 provides a schematic representation of any process of ISO 9001:2015 and shows the interaction of its elements.

The ES has been designed to work with process units (Figure 4). These process units can be combined with each other to shape the whole life cycle of the system.

Each process unit identifies

• Inputs (IN): a series of documents or feedback from other processes. The inputs will be processed by a process or activity.
• Activities: provides traceability between processes. They are the requirements that the process has to comply with.
• Outputs (OUT): the expected or intended result of a process, e.g., documents or feedback to other processes.
• Acceptance Criteria (AC): used to determine if a process can be considered completed or requires rework and new review.

Both figures share common elements so that the adaptation of each element of a process from ISO 9001:2015 to the ES can be carried out in a simple way. Each of these processes can be combined with each other through configuration files. Configuration files are used to adapt each one of the process units defined above.

Table 2. Example configuration file.

ID	Name	Global Process	Prev. Process	Next. Process
1	Identification and context of the organization interested parties and the QMS	1	0	0
2	Determination and commitment of senior management	1	0	0
3	Establishment of the quality policy	1	0	0
4	Evaluation of risks and opportunities	1	0	0
5	Identification and traceability	1	0	0
6	Documented Information	1	0	0
7	Communication	1	0	0
8	Determination of objectives	1	0	0
9	Design and development	1	0	10
10	Projects control	0	9	11
11	Sales	0	10	12
12	Shopping	0	11	13
13	Production	0	12	14
14	Packing, packaging, and shipping	0	13	0
15	Human Resources	1	0	0
16	Infrastructure management and maintenance	1	0	0
17	Monitoring and evaluation	1	0	0
18	Nonconformity, corrective action and improvement	1	0	0
19	Internal audit	1	0	0

ID: identify each of the 19 proposed processes (1 to 19) with the next information; Name: indicates the name of the process; Global Process: a “1” indicates it is a global process that ensures the correctness and control of, and confidence in, the product realization processes and their outputs. These processes are the planning processes, the resource management processes, and the measurement and analysis processes; Previous Process: indicates which process is the previous process (1 to 19, except the process itself). 0 means there is no prior process; Next Process: indicates the subsequent process (1 to 19, except the process itself), so, generally, the outputs of this process are the inputs of the next one. 0 means the process does not precede any other.

shows an example of how the different processes of ISO 9001:2015 are linked together. There are processes that precede other processes or others that are previous or are integral processes that work along the life cycle.

The structure of this file allows it to be used by the ES to evaluate other types of standards with a basic configuration process.

The ES distinguishes between two types of configuration files needed to assess the compliance of ISO 9001:2015:

• A global configuration file defining the 19 processes identified by the standard and intended to be evaluated, as well as the inputs and outputs of each of these processes.
• A configuration file for each process, specifying the rules to be applied by the inference engine, and the variables to be handled by these rules depending on the phase of the process being evaluated (e.g., preconditions, activities, transition criteria).

During the import of these configuration files, the system processes all the information and stores it in its knowledge base.

The outputs of the application correspond to a series of reports generated in real-time by the system during the evaluation of the processes. The flexibility of these outputs is founded on the fact that, regardless of the standard to evaluate compliance with regulatory requirements, the user will get a graph to monitor the status of each of the processes and will be able to generate reports containing all the recommendations and amendments necessary to optimize the processes, which depend on the rules that have been fulfilled during the evaluation.

The following steps have been taken to transform the content of the ISO 9001 into a formalized knowledge structure that will form part of the knowledge base of the ES:

Step 1.

Obtaining the ISO 9001 process map. The identification of the process map is explained in Section 4.1 “System Analysis”.

Step 2.

Traceability of the corresponding subprocesses (clauses of the standard) to the previously identified processes. This traceability is reviewed by the domain expert based on the process map he has previously identified. All subprocesses have been tagged in a specific order from P01_xx to P019_xx and are mapped to each specific section of the regulation.

Example:

• P01_xx; “Identification and context of the organization interested parties and the QMS” -> chapter 4.
• P08_xx; “Determination of objectives” -> Chapter 6.2, where XX will include the number of requirements mapped.

Step 3.

Identification of the set of requirements applicable to each subprocess. The standard requires the compliance of all the requirements mentioned in the standard when they are applicable. It is not possible to exclude any requirement unless it is not applicable to the QMS and you have an acceptable justification. For example, if you are not purchasing any services or goods from an external provider, there is no need to develop quality management tools and controls for those areas [64]. With the help of the knowledge engineer, the domain expert checks that all those requirements that are identified from the standard are completely represented, and, in those cases where it is necessary to qualify or expand, he decides based on his experience. This way, the data extracted from the standard are transformed into knowledge.

The requirements are usually expressed as “shall” statements. The recommendations are express as “should”, “may” indicates a permission, and “can” indicates a possibility or a capability. Hence, all requirements using the “shall” statement is mandatory. Chapters 4 to 10 of the standard ISO 9001:2015 include all possible requirements. All requirements have been identified, captured, labeled, and distributed among all identified processes (Figure 2). In the case of this data structure, all statements expressed as “shall” have been captured as mandatory requirements. This is the first parameter to define the structure of an ontology.

All the requirements have been captured through a requirements management tool, in our case, we used “Requisite Pro” [65], and they can be exported to a spreadsheet file (XLS). All the chapters from 4 to 10 have been captured and labeled according to subprocesses, thus representing another parameter of the ontology.

Guidelines have been followed to correctly identify all requirements and then capture the rules. We have considered the indications in [66] as specific guidelines for the generation of requirements. The advantage is that all the requirements are already included in the ISO 9001:2015, and they only need to be identified.

Example 4.1.ISO-9001:2015: The organization shall determine external and internal issues that are relevant to its purpose and its strategic direction and that affect its ability to achieve the intended result(s) of its quality management system. The organization shall monitor and review information about these external and internal issues. NOTE 1 Issues can include positive and negative factors or conditions for consideration. NOTE 2 Understanding the external context can be facilitated by considering issues arising from legal, technological, competitive, market, cultural, social and economic environments, whether international, national, regional or local. NOTE 3 Understanding the internal context can be facilitated by considering issues related to values, culture, knowledge and performance of the organization.

From this text, two specific requirements were specified by the requirements capture guidelines:

• REQ_01:The organization shall determine external and internal issues that are relevant to its purpose and its strategic direction and that affect its ability to achieve the intended result(s) of its quality management system.
• REQ_02:The organization shall monitor and review information about these external and internal issues.

The requirements will be identified and labeled as P01_REQ_01 and P01_REQ_02.

Step 4.

Verification of requirements to ensure that all the requirements have been correctly identified, labeled, and mapped to a specific process. Following the requirements capture guidelines, these will be unambiguous, comprehensible, internally consistent, feasible, valid, and verifiable.

Step 5.

Generation of the checklist to evaluate compliance with each of these requirements. Checklists are a formal inspection technique, one of the most popular techniques for supporting the inspection techniques [67]. We followed several guidelines [68,69] during the process.

All the requirements of the standard ISO 9001:2015 have been identified and a corresponding set of checklists has been created for checking compliance, covering all the processes. These checklists, along with other aspects of the standards, have been transformed into rules for the ES. We have considered the material included in [64] as a good reference since all the processes of ISO 9001:2015 are very detailed, and it is a good complement to have a solid knowledge database.

Step 6.

Generation of the set of rules for the knowledge base. Both the requirements and the checklists have become a set of rules for the ES when evaluating the fulfillment of each process, and therefore, of the life cycle. The process for transforming these checklists into production rules is the following one:

• Each of the requirements of the ISO 9001:2015 is assigned to a checklist. That, together with the domain expert, has allowed us to obtain the rules of the system. For each checklist, a set of at least three rules are defined:

  ○

  Rule 1: Indicate to the user what to do based on initial conditions.

  ○

  Rule 2: If the user has not performed the action, it shows “how to” perform it.

  ○

  Rule 3: If the user has performed the action, it shows “how to evidence it”.

We show below an example of how a checklist is created from a requirement of the standard in the shape of a set of rules:

Requirement: 4.1.ISO-9001:2015. Checklists: Has the organization determined the external and internal issues that are relevant to its purpose and strategic direction?

This checklist is represented in the fact P01_Checklist_001 (Identification and context of the organization interested parties and the QMS Process) Rules:

○

RULE 1: IF P01_Requirement_1=YES THEN The organization shall determine external and internal issues that are relevant to its purpose and its strategic direction and that affect its ability to achieve the intended result(s) of its quality management system, according to the 4.1. ISO 9001:2015 paragraph.

○

RULE 2: IF P01_Requirement_1=YES AND P01_Checklist_001=NO THEN If we have not done so, redetermine internal and external issues.

○

RULE 3: IF P01_Requirement_1=YES AND P01_Checklist_001=YES THEN If we have done so, document internal and external issues.

The ES allows us to show, for each conclusion of each rule, a help message in text that helps us to understand the requirement through an explanation module. All this information, again, is collected from the domain expert through different interviews and meetings.

Step 7.

Generation of a standardized data structure that later becomes an XMLS.

Following this process, the content of the standard is transformed into a standardized data structure that feeds the knowledge base. The data is included in a spreadsheet file (xls), which is transformed into XML (eXtensible Markup Language) through a process that is transparent to the user: this XML file is loaded into the ES. This allows users with little AI knowledge to add new rules in an easier format instead of directly updating the XML file. At the end of this process, the data set has been transformed into knowledge.

4.2.2. Phase 2—System Design

The architecture is based on a client-server model. It has a degree of flexibility that allows the application of different regulations of varied nature to the same project. Knowledge is entered into the system through configuration files in an xlsx format of Microsoft Excel (see Figure 5).

The tools that have been used in the development of ES are the following ones:

• Java has been selected as the programming language, adding specific libraries for implementation.
• JruleEngine [70], as the inference engine based on Java Specification Request 94, version 1.111, is considered as a sophisticated IF/ELSE sentence interpreter, which acts on input objects (facts) to produce output objects (conclusions or inferences). It is based on a direct chaining algorithm using XMLS syntax. The set of rules has been adapted to an XMLS format that allows the use of the knowledge base by JruleEngine.
• JfreeChart [71] offers different tools to implement all types of diagrams and graphics elements that are very useful for the preparation of reports and dashboards.
• MySQL [72] is the selected database management system. Workbench has been selected for its capacity for the management of tables and the application of tables.

The inference engine applies the rules of the knowledge base to the facts of the working memory. JRuleEngine is based on the forward chain algorithm. The forward-chaining starts with some facts (data in the working memory) that are the available data. The inference engine checks (matches) the data in the working memory with the background of the rules (IF clause) and triggers them until some objective is satisfied or until no rule is applicable. The data is fact or evidence, and knowledge is the set of rules stored in the knowledge base. The rules are objects of class org.jruleengine.rule.RuleImpl, which are loaded from the database. Additionally, these can be loaded by an XML file or JRuleEngine APIs so that rules can be stored externally into a database. The input object in the JRuleEngine is called the fact, and the output object is called the conclusion. A method of the class can be called directly from the rule.

Other features considered in the development of the system are

• Use of free software technologies. The system has been developed with nonproprietary development environments. In addition, the system components which are intrinsic to an ES, such as the knowledge base and the inference engine, have also been developed with free software technologies.
• The system architecture will be restricted to a local/business network environment, not allowing any external connection to the application for security reasons.
• The system has an authentication module and a hierarchy of roles so that each type of user in the application has a series of specific permissions to perform specific actions and to evaluate a series of processes according to their area of knowledge.
• The application has two distinct user profiles: the administrator and the standard user. The administrator profile can do maintenance and system management operations, such as updating the knowledge base and registering new users and projects. The standard user profile can do the evaluation of the processes of a project and the monitoring of its evolution through different graphical tools of analysis.

4.2.3. Phase 3—Design Interface

In this stage, the application has already been developed, including the windows that interact with the user, allowing users to enter values and to show the results. The graphic interface allows users to create profiles (users and projects) and customize their access through specific roles (e.g., project manager, development manager, quality manager). The system allows them to individually update the set of rules of each process of the system in conjunction with the configuration file that contains the definition of each process, which shape the knowledge base. During the rules loading process, the updated information of each process is shown as they are processed through the configuration files so that the user will have a detailed summary of all changes produced along the import process. Once the set of configuration files has been loaded, the system creates a project and the corresponding users (with username and password). The user can specify which of the processes that are part of ISO 9001:2015 will be evaluated. This window shows the degree of compliance of each evaluated process (Figure 6).

The outputs of the application correspond with a series of reports generated in real-time by the system during the operation of validation of the project processes. The user will show a pie chart to monitor the state of each one of the processes and will be able to generate text reports to show all the recommendations and necessary changes to optimize the process (Figure 7). The label “yes” shows the percentage of requirements that are satisfied, the label “no” shows the percentage of those requirements that are not yet fulfilled while the label “N/A” shows the percentage that is not applicable. This last concept is defined thanks to the preconditions. All the checklists are labeled (activity column) together with the level of compliance (values).

The user can enter the values of the preconditions and, using the checklists, specify whether the organization complies or not with the requirements of each process. At the same time, the user can check if the inputs and outputs of each process are satisfied (Figure 8).

As the user indicates to the system the degree of compliance with each requirement, the system indicates the corrective actions to be taken in the event of noncompliance and even how to provide evidence of compliance. At any time, the user can generate a report showing the overall degree of compliance with each of the requirements. This can be done for all processes.

4.3. Testing

As part of the fourth step of the methodology, we verified that the domain of knowledge is sufficiently identified in the knowledge base by checking that all the requirements identified for each process are traced to rules and checklists. The consistency and integrity of the rules of the knowledge base have been verified with the following checks: redundant rules, conflict between rules, circular rules, contradictory rules, missing rules, and unnecessary rules. Once verified that the system is logically consistent, the system was validated to ensure that the results obtained are the same as those achieved by an expert. An expert in this area of knowledge helped to validate each of the processes, completing the demonstration of the system.

5. Validation and Evaluation of the System

Continuing with the fifth step of the proposed methodology (evaluation), we evaluated the performance of the system using several performance metrics on accuracy, sensitivity, and specificity. We used a confusion matrix to describe the performance of the ES from test data where positive (i.e., true) values are known [73]. The basic attributes of a confusion matrix are, respectively, true-positive, true-negatives, false-positives, and false-negatives. The accuracy of the classification, sensitivity, specificity, positive predictive value, and negative predictive value can be defined using the elements of the confounding matrix as

• Accuracy = (TP+TN)/(TN+TP+FN+FP)
• Positive Predictive Value = TP/(TP+FP)
• Negative Predictive Value = TN/(FN+TN)
• Sensitivity = TP/(TP+FN)
• Specificity = TN/(TP+VN)

where

Accuracy is calculated as the number of all correct predictions divided by the total number of the dataset. The best accuracy is 1.0 (100%), whereas the worst is zero.

Sensitivity is calculated as the number of correct positive predictions divided by the total number of positives. The ideal value of sensitivity is 1.0 (100%), whereas the worst is zero.

Specificity is calculated as the number of correct negative predictions divided by the total number of negatives. The ideal value of specificity is 1.0 (100%), whereas the worst is zero.

Positive predictive value is calculated as the number of true-positive predictions divided by the total number of positive predictions. The ideal value is 1 (100%), whereas the worst is zero.

Negative predictive value is calculated as the number of true-negative predictions divided by the total number of negatives. The ideal value is 1.0 (100%), whereas the worst is zero.

True-Positive (TP). A requirement, precondition, or transition criteria marked as positive that was correctly classified as positive by the model.

True-Negative (TN). A requirement, precondition, or transition criteria marked as negative that were correctly classified as negative by the model.

False-Positive (FP). A requirement, precondition, or transition criterion marked as negative that was incorrectly classified as positive.

False-Negative (FN). A requirement, precondition, or transition criterion marked as positive that was incorrectly classified as negative.

The expert evaluated all the results collected during the tests: he confirmed that either the results are correct or if there was an error and we needed to amend something in the system.

Table 3. Confusion matrix with the value of each measurement—first set of tests.

Actual	Prediction—First Test
	Positives	Negatives	Total
Positives	True-Positive (TP) = 338	False-Negative (FN) = 62	400
Negatives	False-Positive (FP) = 0	True-Negative (TN) = 159	165
Total	344	221	565

and

Table 4. Confusion matrix with the value of each measurement—after 5 series of tests.

Actual	Prediction—Last Set of Tests
	Positives	Negatives	Total
Positives	True-Positive (TP) = 338	False-Negative (FN) = 0	388
Negatives	False-Positive (FP) = 0	True-Negative (TN) = 177	177
Total	388	177	565

show the results we got from the first tests and the last set of tests, with the corrected errors detected during the process.

• Accuracy (%) = 87.96%
• Sensitivity (%) = 84.50%
• Specificity (%) = 96.36%
• Positive predictive value (%) = 98.26%
• Negative predictive value (%) = 71.95%

The first tests identified syntax errors caused by the incorrect introduction of rules or facts. Semantic errors were also detected because the knowledge was not well communicated between the expert and the knowledge engineer. Therefore, a series of corrections had to be implemented.

• Accuracy (%) = 100%
• Sensitivity (%) = 100%
• Specificity (%) = 100%
• Positive predictive value (%) = 100%
• Negative predictive value (%) = 100%

Once all improvements had been made, our system achieved 100% accuracy on a set of 565 input and output parameters. Sensitivity had a high value, which shows the system’s ability to correctly perform the requirements ratio, preconditions, transition criteria, and elements. In the first test, the specificity had a lower value. This can be explained by the fact that our system contained some errors in the initial ruleset, but once corrected, the value went up.

One important measure for our system is the true predictive value. With a value of 100%, it is a very promising result. The results show that in the first validation tests, the system included some errors, but after several changes and tests, they were corrected and the expected results were achieved.

6. Discussion

The data were extracted from the standard, following the established guidelines, and then they were transformed into knowledge with the help of an expert human in quality management, who contributed with his experience and validated that the final extracted knowledge was consistent with the domain. Once the knowledge base was generated, it was loaded into the ES, and then the human expert evaluated the system and its results, confirming the validation of the system.

The system is effective because it provides guidance to an organization to achieve compliance with the requirements of the standard. The results show that accuracy, sensitivity, and specificity are very high, reaching their ideal value, so the results provided by the system fulfill the expectations.

The ES is limited in terms of the set of identified inputs, outputs, and acceptance criteria. Although they generally represent all possible combinations, this can be improved. The same applies to the interpretation made to the identified processes. This mapping can be extended or improved with user feedback, and thanks to the flexible architecture of the ES, the required changes can be quickly implemented using the appropriate configuration files.

The explanatory texts that are part of the explanation module can be extended with new annotations, which may facilitate the understanding of the reasoning of the system by the user. The explanatory texts have been deducted from the standard itself and from the references used to extract the knowledge.

The analyzed systems in related literature are mostly based on ES. Some of them are based on fuzzy logic solutions, while others, in knowledge-based systems (KBS). In general, quality control processes are evaluated through the search, detection, and correction of defects, mostly in production processes. Some specific ones are oriented towards ISO 9001, although all of them used the previous version ISO 9001:2008. Others facilitate the processes of audit or quality inspections. All of them are aimed at improving the quality control processes of an organization. Our system shares the final objective of all these systems, i.e., improving the quality of processes, but from a different perspective, focused on helping organizations to implement and check whether their QMS meets the requirements proposed by ISO 9001:2015. We used an ES for this purpose. Another difference is that the rest of solutions are stand-alone environments, which clearly limit the possibility of adaptation to evaluate other regulations or expanding the knowledge base.

The ultimate goal of our system is to establish a mechanism to assist organizations in the implementation and certification of ISO 9001. The results confirm that the final objective is achieved, but also that we provide a methodology to extract knowledge that can be connected to a system capable of evaluating other processes. This is not included in other approaches. In this way, the developed artefact can be used to evaluate other regulations and standards from other domains, with minimal configuration changes, a good feature that increases its sustainability and maintenance.

7. Conclusions

The implementation and certification of ISO 9001 is a difficult decision for organizations since it requires a relevant allocation of financial and material resources. However, the implementation of ISO 9001 implies important benefits, both internal (through better product quality, better process performance, short delivery times, cost reductions, better system documentation, and greater internal knowledge of the processes of the organization) and external (through better customer satisfaction, a better market image, and a stronger competitive position).

The findings of this study have important practical implications. The proposed ES provides an overview of the degree of compliance with the requirements of ISO 9001: it provides information on what actions the organization must adopt to improve its processes, thus facilitating the achievement of adequate quality of products and services.

After performing an ISO 9001:2015 analysis, we generated 20 files. The first one referred to all ISO 9001:2015 processes, establishing a relationship between them and identifying what the inputs and outputs of each of them are. The remaining 19 files represent a compilation of the sections of the standard and apply a set of rules to see if the organization meets the minimum requirements to get the ISO 9001:2015 certification. The knowledge has been extracted manually, and it has been verified as complete with all the relevant information of the standard.

We want to highlight that we decided to apply a series of rules to ensure that the processes meet the requirements, and, in the negative case, the ES will detect the possible weaknesses of each process, providing very useful information for the organization which can update or implement improvements in processes.

Moreover, we want to remark that the ES and the methodology are not only designed to observe whether an organization meets the minimum requirements for ISO 9001 certification. It also provides patterns to observe the weaknesses of organizations against the implementation of a QMS, encouraging improvement and the adoption of new activities to meet the requirements of the standard. This leads to improvements in products and services.

The developed ES provides a quick, cheap, and easily accessible solution to know, at first sight, what requirements of ISO 9001:2015 the organization already meets with its QMS. Therefore, this information can guide the actions to adapt the noncompliant aspects. Our results have important managerial implications since the organization’s quality manager will get a global vision of the organization in an easy and systematic way, and thus be able to detect any kind of deviation after using this system. This method could also be used in purchasing processes or in supplier evaluation. It is first necessary to see what requirements are being fulfilled and then to analyze each one of them in order to be able to implement the system. Sustainability is enhanced as deficiencies can be detected in an efficient way.

The results of this work open new further actions since the process can be improved with the use of ontologies. The ontologies provide a common shared vocabulary (concepts) to represent the information included in the documents (contents). This will allow the precise extraction of the desired information. The design and architecture of the ES also promote the implementation of evaluations of other regulations with similar structures, i.e., organized in processes and with the fulfilment of specific requirements for them. In fact, we are already working on extending the system to other standards such as ISO 14001 or the European Foundation Quality Management (EFQM) model.