Collaborative Ontology Engineering Methodologies for the Development of Decision Support Systems: Case Studies in the Healthcare Domain

Abstract

New models and technological advances are driving the digital transformation of healthcare systems. Ontologies and Semantic Web have been recognized among the most valuable solutions to manage the massive, various, and complex healthcare data deriving from different sources, thus acting as backbones for ontology-based Decision Support Systems (DSSs). Several contributions in the literature propose Ontology engineering methodologies (OEMs) to assist the formalization and development of ontologies, by providing guidelines on tasks, activities, and stakeholders’ participation. Nevertheless, existing OEMs differ widely according to their approach, and often lack of sufficient details to support ontology engineers. This paper performs a meta-review of the main criteria adopted for assessing OEMs, and major issues and shortcomings identified in existing methodologies. The key issues requiring specific attention (i.e., the delivery of a feasibility study, the introduction of project management processes, the support for reuse, and the involvement of stakeholders) are then explored into three use cases of semantic-based DSS in health-related fields. Results contribute to the literature on OEMs by providing insights on specific tools and approaches to be used when tackling these issues in the development of collaborative OEMs supporting DSS.

1. Introduction

Healthcare sectors and services such as assisted care are gathering more attention from both scholars and practitioners, with the call for research and development of new models, new technological advances and more demand-oriented healthcare systems [1]. The digital transformation of healthcare services is leading to a further need in eliciting the knowledge of both users and stakeholders, and to manage and use the massive, various and complex healthcare data [2]. These are dispersedly obtained from distributed devices (including tablet computers and personal digital assistants) and cannot be used for further analyzing and decision supporting if they are collected and organized in a weak-semantic manner [3].

Ontologies—shared, explicit and formal conceptualizations of domains of knowledge [4]—are one of the cornerstones of the Semantic Web and have been adopted in several fields to represent concepts and the relationships occurring among them. Ontologies can provide a shared and formal vocabulary for different concepts, as well managing information deriving from different sources and fostering data interoperability among different applications. Moreover, leveraging Description Logic-based languages to formalize information, ontologies can enable semantic reasoning processes, which allows inferring new knowledge from the information retained in the ontology [5]. With the emergence of Internet of Things (IoT) and the development of IoT applications in many fields, the ontology has been used in those contexts where experts’ knowledge covers a pivotal importance: this include areas in which the digital transformation is triggering relevant societal challenges.

In the field of Healthcare, Semantic Web can enable solutions providing both a shared domain-related vocabulary of concepts, and a backbone for ontology-based Decision Support Systems (DSSs). A DSS is an information artifact that can provide support to organizations by easing the process of decision-making at several levels and in different fields. The term became widely adopted in the 1980s (although research in this area can be traced back to 1960s with the term “management information system”) and the possibilities provided by the union between Artificial Intelligence techniques and DSS was evident since the beginning. Semantic Web technologies share some similarities with DSSs, in that they offer the possibility to interpret data in a precise way, thus enabling the delivery of relevant and reliable information to systems’ end users. In particular, knowledge-based DSSs (in this case, ontology-based DSSs) exploit ontological representation(s) of domain(s) of knowledge to suggest end users actions or providing them with support in decision-making [6]. Leveraging on reasoning processes that generate new pieces of information through inference, ontologies not only offer a shared vocabulary, but are expected to enable Expert System-like functionalities through semantic reasoning.

The application of semantic models in healthcare DSSs can be appreciated in:

• customization of smart environments for frailest segments of population [7,8];
• automatic recognition of human indoor activities in different environments [9,10];
• appliances and smart devices’ tailored actuation [11];
• monitoring of indoor safety [12,13];
• adapting comfort metrics [14,15], modeling energy information for buildings [16];
• fostering Electronic Health Record interoperability [17].

Finally, ontologies are applied to combine the possibility to rely on shared conceptualizations and leverage the results from reasoning processes, as in the cases of physical exercises customization [7] and rehabilitation [18].

However, the process that brings to the development of an ontology is long and difficult, as it usually requires domain experts to be involved in the definition of the knowledge to be modeled. For this reason, since the beginning of Semantic Web, researchers have proposed several process methodologies for the development of ontologies, and a dedicated research field—ontology engineering [19]—emerged: these consist of structured processes that divide ontology engineering into less complex and manageable tasks. Ontology engineering methodologies (OEMs) usually identify the goals of the ontological model, select development languages, enable the identification of the roles, activities, and responsibilities of process participants, and in general is expected to enhance the quality of the output ontology (see Section 2).

Although there exists a significant number of different OEMs, there is not a single and universally recognized methodology standing out. In fact, since the early 2000s it was noted that the process of ontology building resembled a craft, rather than an engineering activity [20]: many OEMs derive from researchers’ direct experience in research projects and there is no clear insight on the specific impacts that activities, tasks and development choices had in the development process. Methodologies differ widely according to their development approach, stakeholders’ participation to the development process, details in defining the tasks and support provided in the various stages.

Based on the literature review, an empirical investigation of how ontology engineers are dealing with the main issues and challenges that entail the ontology building process, even if with the adoption of well-known OEMs, is requested. This study aims to explore the decisional elements entailed in the OEM application process when it is developed as a base for a DSS, with a focus on the main shortcomings and challenges identified in previous analyses.

The reminder of this paper is organized as follows: Section 2 briefly presents the state of the art in the field of Ontology Engineering and reports a classic taxonomy of OEMs, together with insights from a meta-review of literature on the topic. Section 3 introduces three use cases of ontology engineering process for the development of DSSs in the health domain, delving into details regarding the selection and application of different OEMs. Section 4 discusses the main outcomes from the use cases and the meta-review, trying to summarize the authors’ development experiences with three different OEMs. Finally, Conclusions summarize the main contributions of this work.

2. Ontology Engineering Methodologies: A Brief State of the Art

Ontology Engineering is the discipline that ranges from methodologies for ontology development to methods dedicated to support models’ modification and evolution through their life cycle. Scientific interest in ontologies rose in the early 1990s and a few years later the need for guidelines for the development emerged. The main activities to be considered in the different stages, the available classifications of types of OEMs, and a review of the existing criteria to evaluate them, are described below.

2.1. Stages and Key Activities in OEMs

The list of the fundamental activities covered by the majority of OEMs can be summarized according to Simperl et al.’s survey [21] as:

• Feasibility study: a task dedicated to investigating whether the use of a domain ontology or the adoption of an ontology-based application can contribute to solve the problem at hand.
• Domain analysis: investigating the domains of knowledge involved in the to-be-developed ontology.
• Conceptualization: identifying the main concepts and their relations of the domain ontology.
• Implementation: actual implementation (through formal representation languages) of the domain ontology.
• Maintenance: a task taking place after the development of the ontology that consists of adapting the model to new requirements.
• Use: after the development, this task foresees the use of the ontology in applications and its possible alignment with other models (i.e., the activity of identifying corresponding concepts between different ontologies).

During the tasks of domain analysis, conceptualization, and implementation some OEMs stress the importance of knowledge acquisition, reuse of existing ontologies, evaluation of the ontology, and documentation of the development process.

The activities and in general the engineering process are divided into three main stages (Figure 1):

• Ontology management, which addresses all the activities involved in the preparation before development, such as feasibility studies, cost-benefits analysis, preliminary identification of the type of ontology to be developed, etc.;
• Ontology development and support, which collects the core development activities, including those related to knowledge elicitation and formalization, development (including authoring) and documentation of the ontology and its engineering process;
• Ontology use, grouping those activities taking place after the ontology is developed and needs to be maintained and updated, as well as used in application.

Methodologies have evolved together with the adoption of ontologies in many fields, underlining different perspectives and aspects considered in their engineering. As a result, there exists many OEMs—some of them share many similarities, others differ greatly. Even the momentum of the Semantic Web as a promising tool to represent the various “Things” of IoT did not lead to the development of a unified OEM—even though some researchers pointed out this need [21]. In fact, each OEM stresses different aspects of the ontology engineering process: some of them are mostly focused on domain analysis, while others further investigate the development phase. Moreover, some OEMs are specifically dedicated to ontology authoring (i.e., the part of ontology engineering concerned with the actual formalization and development of the ontology, thus including formalization aspects, conceptual choices, and axiomatization), while some others are mainly thought for collaborative development.

2.2. Classifications of the Methodologies

Beyond the identification of key stages, OEMs can focus on developing ontologies from scratch or offer support in the identification of ontological (and non-ontological) resources to be reused, or can stress some particular aspects of the development process.

According to [22], OEMs can be divided in three broad categories, depending on the types of actors involved in the ontology’s engineering process:

• Non-collaborative OEMs: this category encompasses those OEMs that provide in a systematic and formal way phases, tasks descriptions and lists, and workflows that are necessary to develop an ontology. These methodologies do not stress the cooperation among stakeholders but are rather focused on the activities that need to be tackled to conduct ontology engineering and development.
• Collaborative OEMs: the methodologies belonging to this category also define in a systematic and formal way the steps (phases, tasks, and workflows) necessary to develop the ontology, but emphasize a relevant involvement of ontology experts, knowledge engineers, domain experts throughout the activities. A continuous cooperation toward a commonly agreed knowledge (and its formalization) is pivotal for the success of these OEMs.
• Custom OEMs: more recently, there emerged some OEMs defined “custom”, which do not necessarily define engineering activities in a formal way, although stressing the importance of the involvement of stakeholders and communities of practice. OEMs in this category are oriented toward a decentralized (and most of the time collaborative) development of models [22]. This classification allows highlighting the two fundamental elements for an OEM: a formal definition of the tasks required for developing ontologies, and the possibility to have different professionals (domain experts, knowledge engineers, ontology experts, etc.) participating in the development process.

Another OEMs categorization, focused on the type of approach underlying the methodology and inspired by software development methodologies, is proposed by [23] and results in three main categories:

• Waterfall: the OEMs belonging to this category foresee a list of activities (as depicted in Figure 1) that guide the developer from the basics to the maintenance activities. These methodologies are designed as a linear sequence of steps, culminating with the delivery of the ontology once the linear sequence is completed. The ontology developer or knowledge engineer is expected to lead the process and to control the interaction among domain experts, whose role is passive [24]. In general, waterfall OEMs may start with a feasibility study, followed by five main activities: (1) Specification: states why the ontology is necessary to solve the problem at hand, how and who is going to use the model; (2) Conceptualization: the knowledge composing a domain is retrieved and represented in an informal way to obtain a “conceptual model” of the domain(s); (3) Formalization: the experts’ conceptual model of the domain is transformed into a formal or semi-computable model; (4) Implementation: an ontology language (or more) is selected and the output of the previous point is represented by means of the selected ontology language; (5) Maintenance: after the development of the ontology, the model can be updated, amended or enhanced according to the modifications occurring to the domain of knowledge or due to the emergence of new requirements. Prominent examples of OEMs belonging to this category can be traced back to early 1990s (e.g., Uschold and King’s methodology [25] followed by its upgrade by Uschold and Gruninger [26]; TOVE methodology [27]; DOGMA, which splits the specification of ontology’s concepts from its axioms [28].
• Lifecycles: contrary to Waterfall OEMs, this category does not foresee a prefixed list of activities to be performed in a specific order: instead, an ontology is thought as a product that goes through a lifecycle, during which it may or may not undergo certain activities [29]. The lifecycles approach divides the various activities into phases (Requirements development, Ontological analysis, Ontology design, System design, Ontology development and reuse, System development and integration, Deployment, Operation and maintenance), each one groups activities around specific inputs, outputs and goals. Some phases share invariant dependencies: for example, the identification of requirements impacts on the development phase, but the development phase may vary considerably between two ontologies. An ontology is expected to go through each of the phases more than once during its life, but some of these phases may occur in sequence or in parallel; in this scenario, the ontology is constantly evolving. Each phase is detailed with a list of questions, whose answers allow identifying the main phase’s output—which can be used in the following phase. Lifecycle OEMs see the ontologies as the result of an iterative engineering process and as part of complex systems, which include people, processes, hardware, software, and data processing. The most relevant OEMs pertaining to this category are METHONTOLOGY [30], DILIGENT [31], On-To-Knowledge [32], NeOn [33].
• Agile: perhaps the most recent approach to ontology engineering, the methodologies belonging to this category are inspired by agile software engineering methodologies. Generally, agile methodologies present some fundamental intervention areas (Pre-development, which may include the identification of the problem, goals, requirements, knowledge elicitation; Development, which can encompass conceptualization of the domain, and its subsequent formalization with ontological languages; Post-development, an intervention area dedicated to maintenance, update and evaluation of the ontology). These areas can be reiterated until the final version of the ontology is reached, and they foresee the participation of different stakeholders in the development process (also reducing the role of ontology engineers to the final area, those dedicated to formalization with ontological languages). These OEMs recognize that building ontology is a time-consuming process that require skilled human resources, while agile approaches best fit the needs of supporting rapidly changing requirements and models evolution [34]. Moreover, these methodologies highlight the fact that ontology building should be a collaborative process involving communities of stakeholders—and, in some cases, a process that should be accessible also to stakeholders who do not possess a strong knowledge of ontology engineering [35]. Agile methodologies can also draw from extreme programming principles and tackle more than one modeling problem at a time by suggesting the reuse of Ontology Design Patterns (ODPs) (e.g., eXtreme Design [36,37]), which are a set of design and development principles deriving from other (foundational) ontologies.

2.3. Criteria of Analysis for OEMs: A Meta-Review

The choice of the OEM to be adopted requires considering a trade-off between several features. For example, Kotis et al. [22] argue that collaborative OEMs allow ontology engineers to achieve more immediately their engineering goals and requirements, but they require the choice of widely known tools throughout the lifetime of the ontology, in order to maximize the evolution and reuse opportunities.

Along with this line, several contributions in the literature analyzed and compared a set of well-known and representative OEMs against different criteria. They evaluated if existing OEMs include (or not) the practical aspects to be considered in the ontology engineering process, in terms of both activities and recommendations on specific features of ontologies.

Table 1. Criteria traced in in reviews and surveys of OEMs, organized according to the stages of ontology engineering and other features considered (references in table are temporally ordered).

Activities and Other Features of OEMs Analyzed in Literature Reviews and Surveys	Fernández-López et al. [20]	Corcho et al. [40]	Simperl et al. [21]	Iqbal et al. [38]	Al-Baltah et al. [41]	Simperl et al. [42]	Kotis et al. [22]	Sattar et al. [39]
Stage of Ontology management
Feasibility study	X
OEM/ontology and development type				X			X
Stage of Ontology development and support
Design and specification (purpose, scope, users)	X	X			X			X
Domain experts/Knowledge workers/community involvement/roles/needs						X	X	X
Domain analysis			X				X	X
Knowledge acquisition	X	X	X		X	X		X
(Strategy for) level of application engagement/dependency				X		X
Conceptualization (including arguments/discussions for identifying concepts)		X	X	X	X	X	X	X
Knowledge elicitation
Knowledge formalization					X	X
Ontology reuse, compare, merge, evolution				X		X	X	X
Implementation	X	X	X		X			X
Evaluation/Validation/Exploitation		X	X		X		X	X
Documentation/Detailed versioning		X			X		X	X
Stage of Ontology use
Maintenance	X	X			X			X
Use/operation	X	X				X
Support for interoperability				X				X
Other features
Inheritance from knowledge engineering/rooted in well-established methodologies						X		X
Level of detail of the methodology				X		X		X
Recommended tools, methods techniques support						X	X
Collaborative engineering processing/construction	X			X		X	X	X
Recommended lifecycle	X			X		X
Project management processes (IEEE standard)	X	X				X		X

summarizes the main criteria adopted in these contributions. Specifically, the activities considered in the analysis are grouped by the three stages of ontology engineering identified by Simperl et al. [21]: Ontology management, Ontology development and support, and Ontology use. Other relevant criteria that were cited consistently in some surveys [20,38,39] do not strictly belong to a single stage, therefore we grouped them in Others features of OEMs. For example, the feature of “Support for interoperability” considers if the ontologies developed using the selected OEM share the same skeleton or high level concepts of other models [38,39], thus it pertains to the overall methodology and should be considered to be well in its selection.

The analysis of the main methodologies conducted by the above-mentioned reviews and survey allows drawing some interesting considerations regarding the state of the art Ontology Engineering as a discipline.

Ontology management. Interestingly, no review directly addresses the feasibility study as part of the pre-development process, the way it is performed or supported by specific tools and actor involvement. This is particularly important as often ontology engineers underestimate the efforts to be undertaken in the following phases and the importance of a methodological framework, as recognized in early years [21]. Even if competencies, requirements of the ontology and scenarios in which the ontology should be used are generally specified in all OEMs, the feasibility study is only cited by Fernández-López and Gómez-Pérez [20] (and also in a methodology proposed by Sensuse et al. [43]). Some authors refer to the existence of general “preliminary tasks” as the definition of a strategy for building the ontology, including the level of expertise of available domain experts and the type of ontology—however, not even these preliminary tasks are analyzed in detail. It is generally acknowledged that in the early stages, the aim and scope of the targeted ontology should be carefully specified, but it is suggested to have a not too specific focus to facilitate the exploitation of the ontology from a wider community [22]. Some other aspects are rarely mentioned, such as estimating the costs of the ontology building process, which requires also estimating the human resources, such as domain experts, ontology engineers, and API developers that are needed to accomplish the engineering task [39].

Ontology development and support. Reviews argue that most of the OEMs describe the process model to be followed in this stage [42]. Activities of design and specification, knowledge acquisition, conceptualization, implementation, and evaluation are presented and described in the majority of OEMs. Indeed, development activities are primarily performed by domain experts [21]. Specification of domain and acquisition of knowledge should be supported by training seminars, important for understanding the terminology of the core of ontologies for stakeholders who are unfamiliar or have a controversial understanding [39]. At the same time, it is fundamental to dedicate attention to the design (and planning) phase, including related documentation, as the decisions and activities taken have major influence on the success (or failure) of the ontologies developed [20,38,43]. Finally, existing reviews show that the stage of evaluation is well-supported by means of reference frameworks or formal logics [22], but it is important that is not developed as the ending of the process of building ontology, rather than a continuous process providing feedback for other related activities [43]. All these insights further highlight the importance of developing ontologies that should be primarily aimed to be kept alive, and to adopt OEMs that increase their reuse potential, as it is the case of some collaborative OEMs reviewed in [22].

Ontology use. Recent reviews show that several OEMs do not offer suitable support for maintenance, documentation, integration and interoperability [39]. Ontology needs to change over time, thus modification and maintenance are mandatory, but the aspects for ensuring them (including collaborative processes) is observed to be often neglected also in less recent surveys (e.g., [38]). Some authors proposed indeed their own methodologies starting from the observation that very few OEMs clearly state the methods and techniques suggested in the methodology (for instance, [24]).

Other features of OEMs. Some reviews include also the comparison with the IEEE standard for software development as its processes include several aspects and activities that are useful also in ontology engineering, but their results show that very few OEMs include management aspects (e.g., [20]). Furthermore, criteria as the roles and needs of stakeholders (including domain experts, knowledge workers—persons working with their knowledge and know-hows to solve problems—communities), the focus on collaborative engineering processing or construction, and the recommended lifecycle, have been mainly considered in the more recent reviews or surveys [22,38,39,42,43]. Kotis et al. [22] argue that a key trend in OEMs is the adoption of collaborative methodologies, supported by already available and well-known tools. The increasing complexity of the requirements from the real world, and the need to keep ontologies live, evolved and increase their reuse potential, further lead to the need for collaborative OEMs [22,39]. Nevertheless, the analysis by Sattar et al. [39] shows that researchers still pay little attention to the collaborative construction aspect of domain ontology, and the availability of domain experts should also be considered. In addition, ontologies should be made open and linked to other ontologies reaching different communities of domain experts (e.g., by reuse of classes or properties), in order to increasingly enable their evolution and sustainability. This is confirmed also by the results by Simperl et al. [21], showing that the latest developments in the Semantic Web area call for user-friendly development environments, use of valuable amounts of publicly available data, a stronger focus on data- and reuse-driven ontology development (and less development from scratch), and the combination of manual and automatic ontology engineering activities [42].

The choice of an effective methodology that has been used for building ontologies is argued to be a difficult task [41]. From the one side, proposed OEM are not unified and each group applies its own approach, even if they do not prevent them from being used with other methodologies [40]. From the other side, real-world scenarios require customizable methods, rather than pre-defined workflows as proposed by the majority of methodologies [21]. In general, reviews agree on the fact that no OEM is shown to completely satisfy all the criteria, neither provide details on description of ontology building sessions, activities and employed methods and techniques [24,38,39]. Some important activities and techniques are also missing in all methodologies, despite their level of maturity, as for the METHONTOLOGY cited in [20]. Thus, it is important to understand the features guiding the choices of OEM (and its management) that are aligned with the requirements elicited with the stakeholders in terms of feasibility, roles and expertise, and possible scenarios of application and reusability. Indeed, OEMs should be framed in order to support all involved stakeholders, including domain experts, knowledge workers and ontology engineers, during all phases of the lifecycle [22]. Moreover, the insights gained from comprehensive case study findings and domain-specific guidelines for decision-making would help the OEM field in obtaining more elaborated and precise description of the criteria and features to be considered [42].

3. Development of Ontology-Based Decision Support Systems in Healthcare Domains

The empirical investigation takes advantage of three use cases aimed at the development of semantic-based DSSs in health-related fields. Ontologies in this domain are both domain and/or task specific, and application oriented: in this way, the models can satisfy the possibility of being highly re-usable and easily extensible to other pertinent domains [24]. Collaborative construction, reusability and lifecycle recommendations are crucial issues in ontology development [38], and this is true especially in this field. The three use cases derive from research projects in which the adoption of ontologies was deemed as necessary to both deliver a shared definition of the domains involved and to generate meaningful inferences.

Considering the cooperative nature of all the involved research projects and the presence of domain experts in very specific fields of healthcare, together with the necessity to validate the inferences produced by the ontology-based DSSs, the selection of the OEMs fell into the collaborative ones. Specifically, collaborative OEMs were implemented in the three cases with different approaches, i.e., waterfalls, lifecycle, agile. This allowed us to capture different levels of complexity in ontology development, independently from the level of detail of each OEM.

Table 2. Overview of the three cases.

Cases	Health-Related Field	Collaborative OEM	Involvement of Domain Experts
Case 1	Ambient Assisted Living	NeOn (lifecycle)	Multiple kinds of domain experts
Case 2	Tailored indoor comfort	Uschold (waterfall)	One kind of domain experts
Case 3	Health Tourism	UponLite (agile)	Multiple kinds of domain experts

below summarizes some key features of the three illustrative cases.

The three use cases are all related to the healthcare domain and foresee the development of a DSS:

• Case 1 tackles the issue of developing an ontology-based DSS aimed at reconfiguring the domestic environments for elderlies who face some limitations in daily life activities; the role of the DSS is, in this use case, to suggest architects and designers a list of suitable options to reconfigure the set of domestic appliances and to identify some sensors that could help monitor the dweller in his/her environment. The result of the development process is fully described in [44].
• Case 2 faces the possibility of providing tailored indoor comfort in cruise cabins for passengers characterized by specific necessities. The purpose of the DSS developed within this context is to provide suggestions of (or trigger automatic regulation for) specific indoor comfort metrics that can impact on the passengers’ onboard experience [45,46].
• Finally, Case 3 aims at providing a digital and knowledge-based DSS to support the development of health tourism services. A considerable amount of information pertaining different domains—and involving several stakeholders—is considered to provide tourism destination managers with tailored suggestions on how to enhance tourists’ experience and which services develop [47].

In this Section, the use cases are introduced with the following content: the problem at hand is presented, then the selected OEMs is stated (and briefly described), finally the ontology—output of the application of the selected methodology—is discussed, with evidence of shortcomings and adopted tool and practices to overcome them.

3.1. Reconfiguring Domestic Environments

Problem at hand. The issue of reconfiguring the domestic environments for individuals characterized by impairments is the first use case: it comes from an Italian national research project in the field of Ambient Assisted Living (AAL) and Ambient Intelligence. In a context of an aging population, characterized by different age-related impairments, the possibility of reconfiguring the domestic environment with a set of appliances that can support inhabitants in coping with their limitations can play a pivotal role in granting residents an independent lifestyle. In fact, some appliances are more easily adopted by older residents because of their characteristics (e.g., a simple and intuitive graphic user interface, the possibility to provide multi-sensorial feedbacks, simplified doors or knobs, etc.), while common indoor sensors (e.g., thermo-hygrometer, CO₂ concentration, illuminance sensors, etc.) can trigger simple but effective actuations that can enhance the indoor life of residents. The use of ontology is fostered to develop a DSS able to help architects and designers in the reconfiguration process (Figure 2). The project involved academics from different from five different national institutes (four health professionals, three designers, four automation and appliances experts), three national companies (one from the ICT domain, two from the appliances and sensors industry).

Selecting the methodology. The OEM selected for developing the ontology underlying the DSS was the NeOn Methodology [33]. The main reasons for the selection of NeOn were the fact that the ontology underlying this application was expected to evolve progressively, as more requirements and domains were discovered and new information (often already formalized somehow) needed to be added to the ontology: NeOn—with its lifecycle approach to ontology engineering—was selected as the best candidate to help the consortium in delivering the ontology. In fact, this methodology does not provide a rigid procedure, instead it identifies nine scenarios for the development of ontologies (single ontologies, a set of ontologies linked by some kind of domain-dependent relations, ontology networks). The scenarios cover a variety of situations occurring when facing ontology engineering, and to provide a unique definition of the processes and activities that can be faced during the engineering process, a glossary is provided. The nine scenarios can be summarized as: (1) From specification to implementation, encompassing all the core activities related to engineering; (2) Reusing and reengineering non-ontological resources; (3) Reusing ontological resources; (4) Reusing and reengineering ontological resources; (5) Reusing and merging ontological resources; (6) Reusing, merging and reengineering ontological resources; (7) Reusing ontology design patterns (ODPs); (8) Restructuring ontological resources; (9) Localizing ontological resources. While Scenario (2) is explicitly dedicated to the reuse of resources not formalized into ontologies, Scenarios from (3) to (6) and (8) are dedicated to highlight different ways in which identified ontological resources can be reused as part of the ontology that is being developed: the selection of one (or more) scenario depends on the case at hand and from the number of candidate reusable ontologies. Scenario (7) is focused on the possibility of reusing (or adapting) existing ODPs to model the problem at hand, while Scenario (9) deals with the translation (localization) of the ontology in a language different from the one used during the conceptualization of the model. Due to the possible combinations of its scenarios, the NeOn methodology proposes two ontology life-cycle model, waterfall and iterative: in other words, the methodology is appropriate for both waterfall engineering and lifecycle engineering.

NeOn was selected for the development of the ontologies in the project because of the possibility to use it as lifecycle methodology, which allows reiterating the different phases, and also because the extent of the domains at hand was not clear from the beginning. In the conceptual phase, the project foresaw the participation of clinical personnel, automation experts, domestic environments experts, and designers. In this context, domain knowledge was elicited through general discussions until consensus on the main domains involved was reached. It is worth noticing that to increase participants’ commitment to the ontology development phases, participants were acknowledged of what is an ontology and what is its main purpose (within the project), so that they could all contribute to the conceptualization phase (Figure 3).

Development of ontology. The domains elicited following NeOn were mainly four: the person and his/her personal information; the person’s health conditions (which according to clinical personnel, was needed to capture the functioning of an individual); the domestic environment (namely the rooms involved in the ontology); the set(s) of appliances populating the domestic environment. Following Scenario (1) of the NeOn methodology, it was possible to draft the Ontology Requirements Specification Document (using the guidelines provided in [48]), identifying and sharing the purpose, scope and intended end users and uses of the model, and highlighting a list of preliminary terms deriving from the Competency Questions. The intended use of the ontology was clearly stated in matching each individual—characterized by his/her specific health condition—with a set of domestic appliances that are recognized as a suitable option to help him/her to overcome the limitations of his/her impairment. Compiling the ORSD also allowed to identify the necessity to properly categorize the appliances according to concepts shared by all the partners (i.e., to provide a consistent taxonomy for classifying appliances according to the benefits their features can grant to users).

Some ontological resources were found suitable candidates to describe the persons and their personal information, and the persons and their health conditions from a functional point of view: the Friend Of A Friend vocabulary (FOAF) [49] and the International Classification of Functioning, Disability and Health (ICF) [50], while for the other domains the stakeholders decided not to reuse any of the existing ontologies, focusing instead on developing the ontologies from scratch. Therefore, the NeOn methodology’s scenarios adopted for the project were 1 (from the identification of requirements to the development of the ontologies), 8 (for the reuse and partial reengineering of FOAF and ICF ontologies, which where respectively enriched of some properties and pruned of unnecessary branches), and 2 (the classification of appliances was conducted following the requirements of the Americans with Disability Act [51], which indicates some features that appliances designed for people with disabilities should comply to). Resource Description Framework (RDF) [52], Ontology Web Language (OWL) [53], Semantic Web Rule Language (SWRL) [54] and SPARQL [55] were selected as development languages. The development phase was conducted using the Protégé ontology editor [56], since it provides an easy development environment accessible also to non-experts in ontology engineering, and followed an iterative process, incrementing and/or modifying the ontology after each meeting with the stakeholders: while modeling the persons, their data and their health conditions required less iterations, modeling the classification of appliances required a few more iterations until consensus was reached—a fact that was connected to stakeholders’ different perceptions of the specific domain of knowledge. An example of some of the concepts modeled in the ontology is provided in Figure 4.

The final versions of the ontologies were developed in about 11 months and in English (because all of the models reused were already in English and to increase its possibilities of being reused), and the test for this ontology was performed using the Pellet reasoner [57], while retrieval of stated and inferred triples was performed after uploading the semantic model on the Stardog semantic repository [58]. The DSS was tested with a set of use cases provided by clinical personnel and architects (as documented in [44]).

3.2. Tailored Indoor Comfort for Cruise Cabins

Problem at hand. The second use case is drawn from an industrial research project oriented at providing a new and IoT-enriched cruise cabin environment, able to provide tailored comfort for passengers of cruise cabins through a DSS (Figure 5). The list of comfort metrics that can be personalized was limited to indoor lighting (illuminance, light tone, direction of lights, and flash blindness), temperature, humidity rate. One of the project’s aim was also to consider passengers characterized by mild impairments that may be common for people aged 55 or more (such as light sensitivity impairment, asthma, sensitivity to indoor temperature due to other conditions). The ontology aimed at providing a formal representation of the domains involved and general rules to actuate indoor comfort according to the passengers’ health status and preferences, and was expected to be part of a wider application (as described in [45]).

Selecting the methodology. The methodology adopted for this use case was Uschold and King’s method [25], a waterfall methodology composed of four main activities: (1) Identify the purpose of the ontology, including its users and the possibility to rely on Competency Questions; (2) Building the ontology, an activity that is divided in three sub-phases; (3) Evaluation of the developed ontology in the context for which it was developed; (4) Documentation of the ontology to ease its reuse and to make explicit the assumptions made by ontology engineers. The “core activities” of knowledge elicitation and development are included in activity (2), which is divided in: (2a) Ontology capture, which consists of the identification of the key concepts and the relationships existing in the domain of interest. In this sub-activity, a comprehensive list of the concepts and the relationships is expected as output, together with a textual definition of all of them. The identification of concept and relationships covers a pivotal role in this sub-activity, therefore the methodology provides three strategies to facilitate knowledge elicitation [26]. (2b) Coding, grouping development and conceptualization—output of the previous sub-activity—with a formal language. (2c) Integrating existing ontologies, a sub-activity that foresees the possibility to reuse existing ontological resources.

Considering the straightforward approach envisaged by the client, a waterfall methodology was selected due to its rigid structure. In this particular use case, the client provided the general knowledge and the desiderata, while the definition of the details was performed with client’s representatives (three domain experts with significant background in designing, building and deployment of cruise cabin units): in fact, while the first of the four phases was conducted in direct collaboration with the client, the second activity (2a) was further elaborated with some representatives of the client, who participated in some aspects of the definition of the domain and its main concepts and relationships; another relevant reason that suggested the adoption of a rigid and well-established OEM was the discontinuity of the cooperation between ontology engineers and client’s representatives: although on regular basis, the collaboration between domain experts and ontology engineers was not continuous, as in the previous use case.

Development of ontology. The domains found to be relevant to develop the ontology were the passenger and his/her data, passengers’ health status, cabin environment, a set of activities passengers can perform in the cabin, sensors and devices deployed within the cruise cabin, comfort metrics to be monitored and personalized (illuminance, temperature and humidity rate). The output of this activity resulted in a series of UML diagrams connecting the different domains and providing shared definitions of the concepts and relationships. The development of the ontology (performed with Protégé and with the same languages adopted in Case 2 resulted in a model composed of five modules (Passenger and health condition, comfort module, activities in the cabin, cabin environment, devices). To provide a description of passengers’ health status, a subset of the ICF ontology was reused (limited to describe the functional impairments of those conditions identified during the phase Identify the purpose with the client), while for the definition of indoor comfort metrics in the cabin it was decided to model relevant parts of the information included in standards and relevant literature (e.g., UNI 12464-1, ASHRAE Standard 55, Guide for Passenger Comfort on Ships). The list of activities and the representation of the devices were modeled from scratch, in accordance with client’s specifications. Leveraging reasoning capabilities, a set of rules (SWRL) [54] for general inferences—connecting the activities that can be performed within a cabin and the adaptation of some indoor comfort metrics—were developed. The consistency of the ontology was performed leveraging Stardog semantic repository reasoning system (which also allows the use of SWRL). The prototypical ontology-based DSS was validated with scenarios provided by the client, and a full documentation of the ontology, its development process and the inferences it can draw was produced [46]. The ontology was developed following all the phases of Uschold and King’s methodology once, with periodical checks of the outputs with the client at the beginning and end of each phase, in about nine months: this resulted in the performing of a single reiteration of the OEM to deliver the first ontology prototype.

3.3. Alpine Health Tourism KPIs

Problem at hand. In the context of a European research project supporting a sustainable development of the health tourism sector in specific regions, a DSS aimed at helping decision makers in discovery the potential of local and/or regional natural resources was developed (Figure 6). Local authorities, tourism offices, local and regional policy makers should be made aware of the potentialities that natural resources can have on specific target users, as forests, moderate and high altitudes of mountains, and blue spaces (lakes, rivers, streams, ponds and waterfalls) may significantly impact on an individual’s health status and can, therefore, be exploited to develop new nature-based tourism products [59]. The role of the ontology-based DSS is twofold: on the one hand, it must provide a shared representation of the touristic locations and their nature-based resources; from the other hand, it should help the stakeholders to identify which target groups (i.e., sets of individuals grouped according to a health-related issue) could benefit mostly from the peculiar natural resources of a location.

The project involved different core partners: researchers from the fields of tourism, eco-medicine, geography, together with SMEs and representatives from nature-based touristic locations (national parks and protected areas) in different regions—for a total of 11 institutions. Moreover, considering the collaborative approach of the project, different stakeholders (regional departments for tourism, associations promoting the use of natural resources in health tourism, regional development agencies, etc.) from some of the European countries involved in the project were periodically involved in some decision-making processes via moderated meetings (a total of 90 stakeholders from Austria, Germany, Switzerland, Italy, France and Slovenia). Core partners, together with the outputs elicited from the stakeholders meeting, were responsible for providing the domain knowledge necessary for modeling the ontology.

Selecting the methodology. Considering the variety and the number of stakeholders involved, and the gradual elicitation of the domain knowledge, the engineering methodology selected for the development of the ontology and the DSS was the agile UponLite [35]. Before addressing the activities composing the methodology, core partners discussed and elaborated a feasibility study to understand the advantages of an ontological approach to this project: a set of tools derived from management (such as decisional diagrams and SWOT analysis) were adopted to highlight risks and opportunities of developing a knowledge-based DSS. This OEM fosters the active role of stakeholders in knowledge identification and definition, using widely known tools (e.g., diagrams, spreadsheets, etc.): in this way, all the stakeholders could contribute to the elicitation of relevant pieces of information to be formalized in the ontology. The role of the domain experts is primary in gathering information, defining the domain(s) and conceptualizing them, while the role of ontology engineers is marginalized, as they intervene almost at the end of the process to produce the formal model. UponLite is composed of six steps, each of which produces an output that is enriched and refined in the following step: (1) domain terminology, to identify the main terms of the domains at hand; (2) domain glossary, which defines the terms; (3) Taxonomy, organizing terms into a hierarchy; (4) Predication, terms representing properties are connected to the entities they characterize; (5) Meronymy, identifies complex entities and their components; (6) Ontology, produces the ontology in a formal language. The methodology does not foresee a sequence, as steps from 3 to 5 can happen in parallel, while the output(s) of some steps can be used as a feedback to improve previous steps, thus enabling an incremental and cyclic development process. Moreover, some of the steps can be skipped, according to the modeling problem and to the degree of formalization of the domains (e.g., a well-structured domain with a defined terminology and taxonomy would not require domain experts to go through steps from 1 to 3). UponLite stresses the importance of social and collaborative aspects of ontology engineering, enabling the domain experts and the stakeholders involved in the engineering process to actively participate in the development of the model using common tools (e.g., conceptual maps, spreadsheets, diagrams, etc.).

Development of ontology. Using this agile methodology, core partners and stakeholders were able to identify the main natural resources and services they wanted to be involved in the description of a health tourism location (e.g., for the natural resources: forest, lake, waterfall, etc.; for the services: physiotherapy, spa treatments, nutritional advice, etc.). Experts also defined a set of target groups to classify tourists according to their health issues (e.g., obesity, back pain, allergies, exhaustion, etc.). An excerpt of the main concepts for this ontology is provided in Figure 7.

Domain experts also provided some rules (in natural language) to assess the impact that specific natural resources may have on target groups. The process took advantage of spreadsheets and conceptual maps to identify and link the main concepts, which were later defined and refined by undergoing a process of revision during the stakeholder conferences. Completing the five steps of UponLite required 12 months (steps where iterated many times in six national stakeholder meetings and one international stakeholder meeting), also due to the necessity to have steps’ output validated during the several meetings. The result consisted of a set of spreadsheets containing the defined terminology, a set of properties and a set of preliminary rules to link terms, entities, and properties together. The engineering process involving ontology experts started when the results from the previous steps were considered solid enough to be modeled, while acknowledging that significant modifications to the elicited and conceptualized knowledge could have happened according to core partners’ opinions. The ontology was developed in W3C-endorsed languages RDF and OWL, while rules identified during the meetings were translated with SWRL to generate inferences. The resulting DSS was able to score specific attributes of destinations and to infer the most suitable target group for each tourist destination. The consistency check of the ontological model was performed using Pellet reasoner.

4. Discussion

This section provides the results from the cross-case analysis on the main features, phases and shortcomings of OEMs adopted in the development of the related DSS. Issues in stakeholders’ involvement, development stages and other activities requiring decisional steps beyond to the simple execution of the OEM, are discussed.

4.1. Preliminary Activities and Feasibility Study

In all the three use cases, the development of a digital tool to support decision-making in the related healthcare field was foreseen in the project planning. The problems identified in the operational field were, respectively, a knowledge base supporting reconfiguration of domestic environments, the design of a tool to support indoor comfort in cruise cabins, and the exploitation of natural resources in health tourism. At the beginning of two of the three project (namely Cases 1 and 3) projects, the potentialities of ontologies and Semantic Web and the efforts required in the engineering and development of an ontology, were not known to all project partners. A preliminary activity consisted of the presentation of Semantic Web and ontologies, their uses as DSSs and how they are built. Thus, a feasibility study was considered strictly necessary to justify the choice of an ontology as the most suitable technology for the needs of the projects. A shared assessment of the validity and potential economic benefits of the ontology-driven approach to solve the contextual problems [21,23] was thus essential for the following development phases. Specifically, in Cases 1 and 2 the feasibility study was introduced after project initiation (before the beginning of ontology development activities), while in Case 3 it was planned and duly reported as it was necessary to assess and demonstrate the validity of the semantic-based DSS for a wider number of stakeholders. In Case 2 this phase involved only the ontology engineers and the client and its representatives that identified the potential solutions by brainstorming and use of decisional diagrams. In Case 3 the ontology engineers involved two different groups of researchers in two different meetings, adopting the focus group methodology and a set of tools as decisional diagrams and the SWOT analysis. In Case 1 the preliminary study did not include a structured approach and was performed internally by the ontology engineers, once identified the opportunity of reuse some ontological resources to reduce development efforts.

We can thus argue that the feasibility study represents a fundamental activity for assessing the validity of the ontology-based application before presenting it to the final users (i.e., the decision makers that will leverage on the outputs of the DSS based on the ontology). It should be carried out with other specialists and R&D experts (e.g., researchers) in order to understand possible shortcomings in the ability to solve the problem under study and to address the stakeholders’ needs. Moreover, the results of feasibility studies provided further insights in guiding the choice of the OEM and to identify the best approach to tackle the domain at hand. Finally, the phase of feasibility study can be enriched by supporting tools, often used in brainstorming phases in innovation processes, especially if there is a collaborative environment from the beginning.

4.2. Project Management Processes

A key shortcoming highlighted in the literature is the low attention paid in OEMs to aspects related to management [20]. In the three use cases, project management processes as foreseen in IEEE standard, i.e., project initiation, project monitoring and control, and ontology quality management, were considered important. The approaches adopted in the wider project context stimulated the mind-set also of ontology engineers in carrying out OEMs that allowed to deliver the final tool, by accomplishing the objectives of time, cost, and quality. In Case 1 it was fundamental the project initiation, with a training of participants on what is an ontology and what is its main purpose. In the following phases, all kind of stakeholders were involved in planned meetings in order to collect feedbacks at the moment of release of each version of the ontology. The Deming or Plan-Do-Check-Act cycle was adopted in the most important issues (e.g., the identification of all possible health conditions to be included in the semantic model) in order to guarantee the final expected quality, and allowed to prevent possible problems in the subsequent phase. In Case 2 the project officially started with an official meeting between the ontology engineer and the client, in which main deadlines and human resources efforts were estimated in each phase. Periodical checks on the outputs of each development phase were planned and strictly respected according to client requests. The ontology quality management process was more difficult, mainly due to low involvement of other technical experts, but the systematic feedbacks from domain experts on a regular basis ensured the alignment with client’s expectations. In Case 3, the aim of defining domains led to the organization of a kick-of meeting entirely dedicated to the ontology engineering process. This allowed to assign the responsibilities to each involved stakeholder and better ensure their engagement along the overall process. Moreover, a brief introductory presentation allowed to improve the initial level of knowledge of domain experts on ontologies, confirming the importance of training for understanding the core terminology of ontologies for stakeholders with less expertise [39]. The agile methodology was selected also to guarantee a gradual elicitation of the domain knowledge, thus more frequent meetings with flexible schedules were organized between the different kinds of stakeholders, both among core partners and domain experts and stakeholders. Project monitoring was ensured by the release of shared reports after each meeting, including also the tasks to be performed before the subsequent meeting in order to be aligned with project macro-deadlines. This process included also the monitoring of quality in a perspective of continuous improvement. On the one hand side, the quality of the output, in terms of decision makers’ requirements, was ensured by the direct involvement of all kinds of stakeholders into knowledge acquisition, domain analysis and conceptualization: to ease the knowledge elicitation process, according to Agile methodologies guidelines, stakeholder adopted familiar and effective tools as conceptual maps and graphical diagrams to sum up all advancements. On the other hand, the quality of the overall process was guaranteed by the identification of a moderator and the constant involvement of technical experts.

Basing on the cross-case analysis, we can thus argue that project management processes can be more easily enhanced in ontology engineering when the output is an ontology developed within a wider project context. The project monitoring can be ensured through strictly planned meetings with main users of the ontology itself, when the OEM is based on sequential phases (i.e., waterfalls or lifecycles), while periodical meetings with more flexible schedules are more suitable for agile and collaborative OEMs. Useful tools are the release of periodical reports on both advancements in ontology engineering (as already foreseen in the Documentation phase) and eventual deviations from deadlines in project progress, in addition to the feedbacks from involved stakeholders. In this case, the ontology quality is better enhanced, as far as the involvement of a moderator and more technical experts is ensured.

4.3. Support for Reuse of Existing Ontologies

Another key challenge in OEMs, and specifically collaborative ones, is the ability to maximize the evolution and reuse opportunities [22]. Results of the use cases confirm that adopting a methodology that is more focused on reuse enables ontology engineers to search for and leverage more easily on available sources of knowledge. This is the example of NeOn, adopted in Case 1: the stakeholders jointly selected the ontological resources available, while for some domains the cost of reengineering existing model was judged higher than a development from scratch. Nevertheless, the requirements of stakeholders can strictly limit the flexibility of ontology engineer, as verified in Case 2, where the reuse of subsets of existing ontologies was partially reduced. Case 3 and in general agile methodologies could limit the opportunities of reuse, as major work in domain analysis and conceptualization is left to stakeholders. In fact, agile methodologies empower the role of domain experts and stakeholders participating in the key phases of the engineering process (from knowledge elicitation to conceptualization), but this results in the development from scratch of many domains already formalized in shared ontologies (e.g., the identification of target groups in Case 3 could have reused parts of the ICF ontology or parts of the International Classification of Diseases ontology [60]). Thus, agile methodologies without the guide of ontology experts throughout the phases may require a higher effort in aligning the developed ontologies with existing model. This also confirms that one of the main shortcoming in existing OEMs is the lack of details on tools and techniques supporting reuse [22].

Nevertheless, the high availability of reports delivered in each iteration and the necessity to have activities’ output validated during each meeting is increasing the quantity of ontology modules to be openly shared and reused in other contexts and domains. In general, we can argue that ontology engineers should look for a trade-off in the reuse that take into account the stakeholders’ requirements, the possible extendibility of the domain and the number of tasks demanded to them.

4.4. Involvement of Stakeholders

Among the key issues recognized in particular in more recent literature reviews, many OEMs do not sufficiently take into account the impacts in the total development of the roles, previous experiences, needs and motivations of stakeholders, including ontology engineers, domain experts and decision makers that will use the semantic-based DSS [42]. Indeed, most of the available OEMs do not consider the continued involvement of domain experts (a fact also noted by authors of [24]). The three use cases show that the involvement of stakeholders should vary according to the approach of the OEM, the desired level of ontology reusability and the expected use of the final output, i.e., the semantic-based DSS. In the Case 2, the more attention toward the implementation stages and the requirements on the usability of the final ontology resulted in the definition of two different kinds of involvement: the client, i.e., the decision maker that will use the ontology-based DSS, in the project initiation phase and in the final delivery stage, while the domain experts (in this case the representatives that are mainly involved in operational tasks) collaborated in all development phases with the ontology engineers, but not in a continuous collaboration. The lifecycle OEM adopted in Case 1 allowed for defining also an iterative involvement of the experts participating to the intermediate and final meetings planned in each phase. This involvement was especially important to overcome the shortcomings of the OEM in terms of supportive tools. In Case 3, the involvement of stakeholders is continuous: the domain experts, who include the decision makers, had a primary role in gathering information, defining the domain(s) and conceptualizing them. A moderator was identified to coordinate the efforts of different stakeholders, with different needs and expertise, in the several meetings organized to maximize opportunities of collaboration and development outputs.

Table 3. Main challenges deriving from OEMs’ shortcomings in the three use cases.

Use Cases	Feasibility Study	Project Management Processes	Support for Reuse	Involvement of Stakeholders
Case 1	Not foreseen in planning, after introduced as essential; Preliminary study, no direct involvement of stakeholders	Project initiation: meeting to plan resources and timings of each phase; Project monitoring: periodical checks; Ontology quality management: feedbacks on output quality	Partially reuse of available subsets of existing ontologies, limited by decision makers	Differentiated: Customer (decision maker) in the first and final phase; Focused collaboration with domain experts in development sages
Case 2	Not foreseen in planning, introduced as essential; Involvement of technical experts in preliminary meeting; Use of graphical tool (decisional diagrams)	Project initiation: training of stakeholders; Project monitoring: periodical meetings; Ontology quality management: PDCA cycle at the end of each phase	Choice of OEM and scenario to maximize reuse; Choice of reuse for part of the domains; Follow-up report on reuse and adaptability of ontology	Iterative: planned meetings at every stage, with repeated involvement of domain experts and technical experts in subsequent steps
Case 3	Foreseen in project planning and duly reported; Involvement of two groups of stakeholders in structured meetings; Use of more tools (decisional diagrams, SWOT analysis)	Project initiation: dedicated kick-off meeting with assignment of responsibilities; Project monitoring: intermediate reports; Ontology quality management: continuous improvement for quality of both output and process	Less opportunities for reuse of available ontologies; Large availability of material on developed ontology, thus increase	Continuous and moderated: Definition of roles in the kick-of meeting; Coordination by a moderator

summarizes the main points discussed in this section.

5. Conclusions

This study aims to contribute to the growing debate on new methods and technological developments to be considered in the digital transformation of healthcare systems. Both research and practitioners highlight the pivotal importance of technologies that enable them to tackle with the availability of massive, various, and complex healthcare data, and the need to elicit the knowledge of both users and stakeholders. In this sense, ontologies and Semantic Web can enable a shared formal vocabulary for different concepts and act as backbones for ontology-based DSSs, thus allowing to manage the information deriving from different sources and to foster data interoperability among different applications. Despite the valuable contribution, the process that brings to the development of an ontology is long and difficult. As a consequence, several OEMs and related classifications have been proposed in the literature to assist the formalization and development of ontologies.

This paper investigates some of the main shortcomings in existing OEMs, related decisional elements, and challenges to be considered to ensure that the ontology meets the quality design criteria, and thus contributes to the building of a DSS that satisfies both users and stakeholders’ needs. Firstly, a key contribution is the meta-review of previous contributions analyzing and comparing a set of well-known and representative OEMs against different criteria, thus providing a reference on the major features and shortcomings considered in the assessment of OEMs. These include: the delivery of a feasibility study, the introduction of project management process, the support for reuse of existing ontologies, the involvement of stakeholders. Secondly, the identified issues have been empirically explored in three use cases, specifically three projects in health-related fields where a collaborative OEM was selected to act as a base of a DSS. The three use cases were selected in different healthcare domains (ambient assisted living, comfort of cruise cabins, and health tourism), with the need to elicit knowledge from different stakeholders, and applying different approaches in OEMs (i.e., lifecycle, waterfall, agile). Results from the use cases show that the issues have been tackled in the three OEMs with different approaches and tools. These vary according to the type of approach, the trade-off between stakeholders’ requirements and the tasks demanded to them, and the management processes to maximize opportunities of collaboration and development outputs. Shortcomings and advantages of adopted OEMs are also highlighted.

Practical contributions of this work entail the growing awareness of the pivotal importance of ontologies and Semantic Web to tackle the challenges of digital transformation of healthcare. The study provides some reference insights both for ontology engineers, with the identification of key issues to be considered in the choice of OEM (especially collaborative OEMs), and for stakeholders, who can find some suggestions on possible tools and decisional elements requiring collaborative efforts with technical experts.