Skip to Content
MDPIMDPI

Advanced

Applied SciencesApplied Sciences
PDF
  • Article
  • Open Access

14 September 2026

26 Pages

Federated Data Fabric for the Cloud-Edge-IoT Continuum

,
,
,
,
,
,
,
and
1
Department of Mathematics, Physics and Informatics, University of Gdańsk, 80-308 Gdańsk, Poland
2
Systems Research Institute, Polish Academy of Sciences, 01-447 Warsaw, Poland
3
Department of Communications, Universitat Politècnica de València, 46022 Valencia, Spain
4
Telefónica Research and Development, 28050 Madrid, Spain
Appl. Sci.2026, 16(18), 9102;https://doi.org/10.3390/app16189102
This article belongs to the Section Computing and Artificial Intelligence
Version Notes
Order Reprints

Abstract

In recent years, further rapid growth of the number of available data sources has been observed. Among them, solutions based on the Internet of Things (IoT) start to play an increasingly important role, forming the basis of the Cloud-Edge-IoT continuum. Because of the large volumes and wide variety of data that IoT devices offer, efficient and flexible methods must be developed for data sharing, processing, and analysis (both within and outside of IoT). In this context, the concept of the architecture of the Federated Data Fabric, which focuses on manageability and flexibility of data processing, enabling distribution and federation of data processing components across the Cloud-Edge-IoT Continuum is presented. It provides all essential components, including (1) data catalog, (2) data-as-a-product management, and (3) data processing pipelines. Furthermore, the proposed architecture is capable of seamlessly combining heterogeneous data models and efficiently handling large volumes of batch and streaming data by offering efficient semantic annotation and translation capabilities. The proposed architecture is presented in the context of the EU-funded projects aerOS, where it was first implemented and empirically validated, and O-CEI, where it is currently under active development.

1. Introduction

Data and its processing have always been essential components that provide value to computer systems. At the end of the last century, the importance of utilizing data became apparent, with the emergence of the data-as-a-product [1] concept. Generally, a data product can be understood as a specific data resource, such as a dataset or a data stream, designed to meet particular business objectives and intended for specific groups of clients or users. A data owner manages and exposes the data product. In addition to the raw data, data products include relevant metadata, to facilitate use by potential recipients (e.g., customers). To better understand this concept, a number of “data architectures” have been proposed, the most common of which are: data warehouse [2,3], data lake [4], data lakehouse [5], data mesh [6,7,8], and data fabric [9,10,11]. It is noteworthy that most resulting, actual, deployments follow a centralized approach. This can be attributed to the timeline during which these architectures were first considered (the end of the last and the beginning of this century), when centralized approaches (with a “cloud in the center”) were the most adequate (and popularized) solutions for the enterprise-level data management. In light of this, the subsequent discussion will briefly discuss the intricacies of each data management approach.
Data warehouse is designed to facilitate swift, actionable querying of structured data for analytics purposes. It typically offers pre-built functionalities (e.g., Online Analytical Processing, in-memory processing) and robust SQL support. A data lake is an architecture designed to handle structured, semi-structured and unstructured (mostly raw) data. This makes data lakes more flexible than data warehouses because they can accommodate a wider range of data types. While data warehouses interpret incoming data (schema on write), data lakes interpret them during access (schema on read). Data lakes are often used for streaming, machine learning, and data science applications. A data lakehouse provides a blend of elements from data lake (e.g., storage) and data warehouse (e.g., management and performance) architectures. It offers support for both structured and unstructured data storage and attempts to eliminate the disadvantages of both architectures while retaining their best features. A data mesh emphasizes decentralization and distribution in data management [7,12]. It is also characterized by the utilization of data-as-a-product principle and by federated computational governance. Finally, a data fabric [13,14,15,16,17] provides a unified collection of services that offer data assets based on a variety of data sources. While data mesh focuses on data ownership and data management, abstracting from underlying technologies, data fabric focuses on data access and the communication infrastructure. These approaches can complement each other. It should be noted that among these architectures, data mesh and data fabric allow for and target decentralized implementations.
Recently, processing massive amounts of data, generated by artifacts in the ever-evolving Cloud-Edge-IoT (CEI) computing continuum [18,19], has become crucial to solve more advanced business use cases and to enable prescriptive and autonomous analytics on, often combined, data. However, the enormous diversity of these artifacts and, consequently, the resulting vast heterogeneity of the data they offer pose significant challenges. Additionally, since most data in the CEI continuum is generated and transferred in a streaming fashion, processing mechanisms must efficiently and flexibly handle data streams. Because the (streaming) data is both structurally and semantically diverse, this necessitates the provision of versatile methods for representing and mapping metadata (such as metric units, context, ownership, provenance, …, etc.) and data models. Additionally, data within the computing continuum is often offered at multiple organizational levels and/or by multiple stakeholders (multi-tenancy). For instance, measurement data, from devices used in a manufacturer’s industrial plants, may be fully utilized at the “internal” level. At the same time, it may be partially exposed “externally” (to business partner(s)), possibly after appropriate transformation(s). Therefore, the data infrastructure architecture, for the CEI continuum, must enable multi-level configuration and management, as well as the combination and effective transformation mechanisms, for the data offered.
The following work proposes a novel Federated Data Fabric (FDF) to address the aforementioned challenges. The FDF is a data-mesh-based architecture that manages a distributed federation of independently governed nodes. Each node is responsible for creating, securing, and exposing individual data products. The FDF offers and integrates components for (1) a data catalog, (2) data-as-a-product management, and (3) data processing pipelines. It efficiently handles the multi-level creation and management of heterogeneous streams of semantically demarcated data within the CEI continuum environment.
In this context, the following subsections briefly discuss the novel data management approaches introduced in the literature and compares them to our innovative FDF architecture.

1.1. Orchestration Platforms to Manage the Cloud-Edge-IoT Continuum

The concepts of data warehouse, data lake, data lakehouse, data mesh, and data fabric provide the foundations for modern data management architectures. However, the emergence of the Cloud-Edge-IoT continuum has introduced new requirements that go beyond the scope of these classical paradigms. Data are now generated and consumed across highly distributed and heterogeneous environments, spanning resource-constrained IoT devices, edge infrastructures, and cloud platforms, often involving multiple administrative domains and stakeholders. Such scenarios require not only scalable computation and communication mechanisms, but also flexible approaches to data management, interoperability, governance, and domain federation.
In recent years, a growing number of platforms and architectures have been proposed to address these challenges. Furthermore, Meta-Operating System (Meta-OS) initiatives and edge orchestration frameworks aim to provide unified management of distributed computational resources and applications across the continuum (see Table 1). On the other hand, federated data infrastructures and Data Space initiatives focus on enabling trusted data sharing, sovereignty, and interoperability across organizational boundaries. At the same time, recent research efforts [11,17,20] have explored novel data fabric architectures and semantic data management approaches tailored to distributed environments.
Table 1. Representative Meta-OS initiatives and edge orchestration frameworks for the Cloud-Edge-IoT continuum.
Table 1 summarizes the characteristics of representative Meta-OS initiatives and edge orchestration frameworks targeting the CEI continuum. These platforms primarily focus on the management of distributed computational resources, service deployment, and execution across heterogeneous infrastructures, typically relying on containerization and Kubernetes-based orchestration technologies to enable portable and scalable deployment across the continuum. More recent Meta-OS proposals have also started to incorporate data-aware capabilities, including mechanisms for data collection, abstraction, and sharing, recognizing the increasing importance of data-centric services in CEI environments.
Nevertheless, the management of data throughout its complete life-cycle remains largely a supporting capability rather than the primary objective of these platforms. A key distinction emerges between Meta-OS solutions and traditional edge orchestration frameworks. In Meta-OS architectures, data management functionalities are increasingly designed as tightly integrated components of the platform itself, directly interacting with the orchestration and service management layers. In contrast, edge orchestration frameworks usually consider data management as an external concern. They rely on independent services or add-on components that exploit their deployment and execution capabilities but remain only loosely coupled to the orchestration core. Consequently, advanced capabilities such as data-as-a-product management, semantic annotation and translation, federated data governance, and the autonomous orchestration of data processing pipelines are generally absent or delegated to external subsystems. In the case of aerOS [29] and O-CEI [30], these capabilities are natively integrated through the FDF, which constitutes the dedicated data management layer of the Meta-OS architecture. It is closely coupled with the architecture’s core orchestration and service management mechanisms, rather than being deployed as an independent add-on.
While Meta-OS and edge orchestration frameworks address the infrastructure and execution dimensions of the CEI continuum, the challenge of enabling secure, sovereign, and interoperable data exchange across organizational boundaries has motivated the emergence of federated data infrastructures and Data Space initiatives. These approaches, discussed next, provide complementary capabilities centered on trusted data sharing, policy enforcement, and data sovereignty, which can be integrated with dedicated data management solutions such as the proposed FDF.

1.2. Federated Data Spaces and Data Fabric Architectures

Recent literature describes data spaces (DS) as federated infrastructures for sovereign cross-organizational data exchange, typically realized through platforms that combine connectors, participant identity, metadata catalogs, policy enforcement, and governance services [31,32]. Their main technical novelty is that interoperability is coupled with sovereignty, so data can be discovered and exchanged across organizational boundaries without giving up control over usage constraints, contracts, or provenance. Data-spaces-based approaches operationalize this through connector-mediated exchange and a separation between control-plane functions such as policy and negotiation, and the data plane used to transfer governed assets [32,33].
This design is especially relevant in distributed environments, because it enables policy-aware exchange without assuming centralized ownership of data or infrastructure [31,32,33]. Recent work connecting aerOS with data spaces further shows that these mechanisms can be embedded into Meta-OS style continuum platforms, making trusted and distributed access to data-space products technically viable in Cloud-Edge-IoT settings [34]. However, the data-space literature still tends to focus on sovereign exchange and federation services more than on semantic translation, CEI-specific heterogeneous data mediation, or unified handling of streaming and batch data in one architecture.
A complementary line is data mesh, whose recent reference-architecture work makes the paradigm technically concrete through three main viewpoints: domain architecture, self-serve data platform, and federated governance [8]. Its novelty lies in elevating data-as-a-product to an operational architectural principle, so domains publish governed and discoverable data products rather than merely exposing raw datasets. Data fabric addresses a different layer of the problem by using metadata-driven and semantics-aware integration to create a logical fabric across heterogeneous sources and formats [35]. Its innovation is strongest in active metadata, catalogs, and semantic integration, but generic proposals usually under-specify cross-organizational federation and CEI-oriented deployment.
Table 2 provides a qualitative, literature-based feature mapping at the architectural level, based on the capabilities explicitly described in the cited reference specifications and literature [8,31,32,33,34,35]. In the table, ✓ denotes a capability that is explicitly and natively addressed by the considered architecture; (✓) denotes partial, optional, indirect, or ecosystem-dependent support, and × indicates that the capability is not explicitly addressed in the considered sources. Since the compared approaches operate at different levels of abstraction, the comparison aims to characterize their architectural scope and emphasis rather than to establish an absolute ranking or implementation-level performance comparison. In particular, the considered data-space approaches emphasize sovereign federation and policy-controlled exchange [31,32,33,34], data mesh emphasizes domain ownership and data-as-a-product [8], while semantic data-fabric approaches focus on semantic and metadata-driven integration across heterogeneous data sources [35]. Against this background, FDF is designed to bring these complementary concerns together within a single CEI-oriented architectural framework, addressing federation, data-as-a-product, semantic interoperability and translation, catalog-driven discovery, policy-based access control, multi-tenancy, FAIR alignment, provenance, streaming support, edge-awareness, and modular deployability. Therefore, the full coverage reported for FDF reflects the intended architectural scope of the proposed framework rather than its superiority to the other compared approaches in terms of the maturity, performance, or completeness of each individual capability.
Table 2. Federated data spaces and data fabric architectures.

2. Federated Data Fabric

The Federated Data Fabric (FDF), proposed in this contribution, is based on data mesh principles [36] and supports a distributed architecture that offers flexible data-as-a-product management. The FDF concept and architecture offer an extension and generalization of the work undertaken and continuing as a part of the aerOS and O-CEI Horizon Europe projects. In the FDF, a data product is a combination of data, metadata, and software that aligns with the FAIR (Findable, Accessible, Interoperable, Reusable) [37] principles. For data to be Findable (discoverable), the FDF must keep track of its location and availability, allowing to openly expose an unequivocal identification of all pertinent data in the system. Accessible data is uniformly available across the system via the FDF, whereas access is restricted to the authorized users. The data must also be easily Interoperable through the use of commonly agreed-upon semantic data models, or interoperability mechanisms (e.g., supported by syntactic and/or semantic translation). Finally, to be Reusable, it must support openness and provide mechanisms that enable it to function across different use cases without requiring ad-hoc integrations.
The FDF architecture has been designed to support the proposed characteristics of a data product and is composed of FDF nodes. Each node is equipped with an instance of the Orion-LD NGSI-LD Context Broker (CB) [38], which supports the formation and orchestration of the federation and synchronizes the necessary metadata, expressed using the aerOS Continuum ontology [39]. In particular, synchronization applies to the metadata provided by individual Data Catalog instances. Thanks to this synchronization and unified access/management capabilities, the FDF architecture offers an integrated view of the data products offered, thereby justifying the term “data fabric” used in its name. As depicted in Figure 1, each FDF node, provides the components responsible for creating, securing and managing the offered data products. The following section will provide a more detailed description of the core components of the FDF.
Figure 1. High-level overview of the FDF node architecture.
Data Security Manager, Access Control: these components manage and enforce access control policies for consuming data products offered by the FDF node. They may rely on multiple cybersecurity tools, to manage the authentication and authorization of users of the FDF. On the architectural level, the Data Security Manager is required to provide interoperable security mechanisms that allow all nodes of the FDF to create, share, and keep up-to-date (i.e., synchronize) all data about users, groups, privileges and access logs. However, it should be stressed that there is no requirement for every node to use exactly the same security-related software solution. As a matter of fact, due to the modular design, each deployment of FDF may involve different security and authorization software stack. Moreover, the same is true for every part of the FDF, as long as critical components, such as Access Control, are synchronized across nodes. That said, for instance, the aerOS implementation [40] incorporates robust authentication, authorization and access control (AAA) mechanisms across multiple components through integrations with Keycloak [41], KrakenD [42], and new technologies such as Apache APISIX [43] and Open Policy Agent [44] (OPA).
Data Product Manager: a component providing an interface that enables registered data product owners to define and offer data products and data sources within the FDF (see, Section 3). Specifically, the Data Product Manager allows data product owners to build and deploy data products by selecting from a predefined list of available, deployable data processing components that offer functionality in one of the data product pipeline stages (see, Figure 2). Next, owners connect these components to a selected data source and define how the output of the configured data product will be exposed to the data consumers. The Data Product Manager coordinates with the Data Catalog and the Data Security Manager components to oversee data product governance.
Figure 2. High-level overview of the Data Product Pipeline architecture.
Data Catalog: a registry of data products available through FDF nodes. It includes metadata, such as ownership information, to support data governance capabilities, as well as access-related information for the data. The Data Catalog serves as a knowledge base for all data integrated into the FDF nodes’ data infrastructure. It contains information about data offered by data products managed by the Data Product Manager, without necessarily exposing the structure of the data product itself (see, Section 3). The Data Catalog focuses on describing the output of a data product and any relevant provenance information, such as (1) who produced the data, (2) how often it is updated, (3) what is its scope, temporal or geographical relevance, as well as “units of measurement” (if applicable). It is assumed that Data Catalog uses a specific, extendable ontology to describe its contents. One such ontology was prepared for the aerOS project and is used in its implementation of the FDF. In the more general context, popular standards and ontologies for data catalogs include: schema.org, Dublin Core, DCAT (W3C Data Catalog Vocabulary), PROV-O (W3C Provenance Ontology), SKOS (W3C Simple Knowledge Organization System), ISO 11179 standard [45] for metadata registries. Again, selection of a specific semantic knowledge representation belongs to the FDF facilitators and different deployments may use different ontologies/knowledge graphs.
Data Product Pipeline: represents the space where data products accept, process, and output data. This space is supported by common transport, storage, and access mechanisms. The subsequent section will provide a more thorough elaboration on this pertinent concept.

3. Data Product Pipeline

In order to address the heterogeneity of data processing and transportation needs, as well as to enable the implementation of various solutions where data flows are processed and presented exactly as required in a specific use case, the FDF does not propose a unified solution with a closed set of standards and data models. Instead, it provides a supporting framework and a set of building blocks that can be configured dynamically within the framework. This architecture, illustrated in Figure 2, comprises of two primary components that form the base framework of the data product pipeline: (i) the transport layer and (ii) the data ingestion/processing/storage mechanisms.
Adhering to the principles of good design for the IoT systems, the transport layer is decentralized and reactive (in the sense of the Reactive Manifesto [46]), supporting both batch and streaming modes, including persistent (“infinite”) streams. The aerOS/O-CEI implementation of the transport layer uses a publish-subscribe broker [47] that exposes a Kafka-compatible protocol for streaming data and a set of connectors that are to be used by data components. It is important to note that, in the architectural sense, Kafka is “central” in that it allows broker nodes to be distributed and provides redundancy mechanisms so that the implementation is not reliant on a single point of failure. Depending on the configuration of a particular data product, data can be read directly from the transport layer. In such cases, the endpoint (or, in the case of a streaming broker, a specific topic) in the transport layer that serves as the output of a data product is exposed through the access control component, allowing data consumers to connect directly to it. Direct connection to the transport layer is useful for applications that consume streaming data and want the fewest processing stages possible. However, this requires implementing the transport-layer protocol directly on the data consumer side.

Data Product Pipeline Building Components

Data products are built from data components that can be deployed through the Data Product Manager, over the data product pipeline. Data products can consume any number of specific data sources and produce a specified output. Each data component has a precisely defined function within the data product that can be assigned to a given stage. Once a data product is deployed, the data flows linearly from the specific data source into the selected components of the ingestion, processing, and storage stages (in that order), before being exposed to the authorized consumers.
Although data components may have very different functionalities, there are a few technical requirements that all of them must adhere to. First, they must communicate with the transport layer, rather than directly with other data components. This is required and ensures that (1) data flow is controlled, (2) security assurances are maintained, and (3) any errors, congestion, or misconfiguration are properly logged and reported. Second, to preserve the distributed nature of the entire FDF, the data components must be individually orchestrable so the Data Product Manager can deploy them, monitor their deployment status, and restart, or redeploy, them if necessary. To achieve this, software virtualization and containerization techniques are used for every FDF component, aligning with the pursued modularization principles.
In the ingesting stage, data components are responsible for reading data from the data sources and pushing it to the transport layer for consumption by the processing stage(s). Separating the ingesting stage from other stages serves the following purposes. First, due to the distributed nature of data products, the data components, in the ingesting stage, can be deployed in any compatible environment, selected by the data product owner. This includes the environment in which the data source is located. Thus, the architecture enables data product owners to maintain the data sources and any necessary access credentials within a secure environment under their control. Such assurances are particularly important in the case of sensitive data that must not leak outside a given scope, and that may need to be (partially) redacted before being sent to the FDF transport layer. In this way, the ingesting stage addresses the critical privacy and security concerns at the earliest possible point in the data flow.
The primary function of the ingesting stage is to connect a specified data source to the transport layer. For this reason, the components in this stage are called “connectors”. Each connector must be able to read from a data source and write to the transport layer. Their role is to adapt any data transport paradigm to the one used by the transport layer. The architecture assumes that the transport layer implements the Reactive Streams paradigm [48]. Therefore, reactive and streaming data sources require minimal integration. Separate connectors are needed for data sources that, for instance, use relational databases, expose data via a REST API, or implement a streaming protocol that is not directly compatible with the transport layer. In such cases, a connector could periodically pull data from a database, call specific endpoints, perform decoding or decompression, or any other necessary operation. Here, any implicit metadata (e.g., provenance) that cannot be inferred from the data itself should be added during the ingesting stage. When connecting to a relational database, for example, the connector may need to periodically query it, push data to the transport layer only if the previously pushed data is outdated, and annotate the data with information such as the database’s text encoding and/or appropriate timestamps.
Once the bridge between a data source and the FDF is set up in the ingesting stage, the data can optionally be processed in the processing stage. While the ingesting stage focused on the needs and requirements of the data producer, the processing stage addresses the utility of the data, from the consumer’s point of view.
Any changes to the format, contents, or metadata can be performed in this stage. For example, encoding of text data may be changed, images may be re-encoded, metadata added or removed, etc. The only requirements for the data processing component are to connect its input and output to the transport layer. In a data product, the data processing components are chained together to pass data from one component to the next in a linear fashion. It is important to note, that even if the access control is properly integrated into a processing component, dependency on external services may introduce an additional potential point of failure that may cause errors or delays.
Even though components, in this stage, are usually prepared to address specific data processing needs, they can often be generalized to resolve a family of similar issues. For example, a component for re-encoding images may be designed to be configurable, to support many commonly used image formats. Similarly, changing the text encoding does not require re-implementing a data processing component for every text-based data format. However, the question of balance between the simplicity and generalization remains open and needs to be addressed on a case-by-case basis.
Note that data components can be instantiated and configured independently in different data product deployments. This flexibility means the same component can address the needs of different consumers when deployed in distinct data products. For example, if two consumers require data from the same source in different formats, two data products would be deployed with equivalent ingestion stages and different processing components. The multiple layers of separation and distribution that the FDF provides mean the processed data is provided to consumers independently and in a decoupled fashion. Neither the data products nor the consumers need to depend on or be aware of each other. However, keep in mind that the data product manager must have access privileges to the data source. Ultimately, therefore, the owners of the data sources decide which data products can access their data.
The processing stage may also perform advanced operations on data. For example, selective redaction of data (available in the ingesting stage) may also be performed here. This would be the case if the level of redaction depends on consumers of a particular data product. If only some consumers are allowed to view full data, two distinct data products would be deployed, with only one producing the unredacted data. The data products would then use the same ingesting stage but different access-control configurations, and only one would process the data to redact parts of it.
Two families of operations on the data that are of particular importance in the IoT domain are semantic annotation and semantic translation. Together, they enable lifting data to the semantic level, adding or changing metadata, and manipulating its meaning to align with the semantics of multiple (different) systems, providing high levels of semantic interoperability. Those mechanisms are described in detail in Section 4.
Finally, the storage stage provides a way to preserve and query the data. Unlike the ingestion and processing stages, storage stage components are shared between data products. This architectural solution allows for the collection and fusion of data from multiple sources, paving the way for advanced queries, aggregation, reasoning, clustering, and other necessary operations to gain insight into collective data. Note that this component is optional, depending on the individual use case.
Even though the high-level architecture of the FDF is quite permissive when it comes to every stage of the data product pipeline, there are a number of practical (restricting) considerations. Robust storage solutions (i.e., ones that provide security, redundancy, high availability, support multiple paradigms and data schemas, etc.) require a considerable amount of resources, especially compared with components in the ingesting and processing stages. They are also, in principle, more difficult to re-orchestrate in resource-constrained environments. Similar to the ingesting stage and the data sources, each storage component requires implementation of a connection to the transport layer. Here, the schematic design includes this component as a functional resort. Deployment-wise, it would be up to every data product manager, or the FDF owner, to decide the specific strategy for federated/decentralized storage (which implies considerations from multiple angles such as volume or persistence history, among others). Moreover, for the reasons stated above, it is not recommended to use multiple storage components in a single FDF deployment. The suggested alternative is to choose a single robust and feature-rich component that covers the data querying needs of the client(s). For example, Orion-LD Context Broker [38] is used in both, aerOS and O-CEI projects. It allows the collection of data in the NGSI-LD format that supports any ontology, and may be queried for specific parts of data objects.
In summary, the data product pipeline provides the structure and the technological backbone for the configuration and deployment of data products. In each deployment of a data product, the ingesting stage reads data from a data source, and optionally processes and stores it in later stages. Communication between components is supported by the transport layer.

4. Semantic Data Processing

In heterogeneous environments, the semantic treatment of data is essential, because it enables the representation of the entities and their explicit relationships, providing mechanisms for data understanding, integration, and enhanced discovery. The Data Product Pipeline can be used to transform raw, or already semantically demarcated, data into data represented with a selected semantics used by the FDF node. Here, data transformations may be required if the raw data uses different syntax or semantics. The tools described in the next two sections form the foundation of the semantic data products that can be deployed in the Data Product Pipeline.
Note that selecting appropriate data models is another important part of designing any data infrastructure. This selection concerns more the use case than the architectural level. Specifically, it determines the requirements for any necessary data transformations. The proposed approach assumes that, internally, the FDF is semantics-based, as semantic data representation enables flexible and adaptable data modeling and processing. This is especially important in the vastly heterogeneous CEI continuum environment.
The architectural components of the FDF nodes provide the necessary building blocks for designing a distributed, scalable data infrastructure. Here, in order to add a new data product to an FDF node, the product must first be created using the Data Product Pipeline mechanisms and then registered via the Data Product Manager. The complexity of the pipeline depends on the nature of the data source. Fortunately, the semantic data processing tools, proposed for use in the pipeline, are efficient and highly configurable.

4.1. Semantic Annotation

First, semantic annotation aims at ingesting raw data from external data sources and enriching it with appropriate semantic information to enable further processing.
Currently, there are several existing open-source tools that can be accommodated and applied for this purpose. One of them is the Semantic Annotator, developed in the ASSIST-IoT project [49] and further enhanced in the aerOS project. It can annotate arbitrary JSON, XML, and CSV data, based on transformation rules expressed in CARML [50] (see Figure 3). The tool can perform single-message annotation tasks using the REST API, as well as streaming annotation, interacting with message brokers such as Kafka or MQTT.
Figure 3. Semantic data annotation component.
The Semantic Annotator is designed around the annotation of messages in persistent streams of data, organized into channels. Each channel is independently configured with an appropriate CARML transformation and connects to Kafka or MQTT message brokers to ingest and output data, as well as channel monitoring messages and optional errors. REST API is exposed for the purposes of channel management. The annotator can connect to any number of brokers to consume and produce data, so cross-broker annotation channels are also possible. Figure 4 shows an overview of the main Semantic Annotator’s building blocks and their interaction. The main component, the annotation core,—is responsible for performing the annotation task, based on a set of CARML-expressed rules, and using JSONPath, XPath, or CSV selectors. The communication manager is an intermediary between message brokers and the rest of the Semantic Annotator. It is responsible for passing messages to and from the brokers, and channel manager, which in turn uses the annotation core to perform actual message annotations. The channel manager is used to configure independent annotation channels, each with their own CARML transformations, and connections to one or more streaming brokers. The storage component stores channel status and configuration, so that it may be restored after restart. Information about channel status and configuration (including available CARML annotations) is exposed via the REST manager.
Figure 4. Semantic data annotation component architecture.
CARML annotations are expressed in RDF, in a format derived from RML [51], which allows for the definition of multiple input data sources (e.g., files). Additionally, CARML supports streaming sources, making it ideal for processing (potentially) infinite data streams.
The CARML input source specification (see Figure 5) defines the format of data flowing through the stream and the language used to select parts of the data (JSONPath for JSON, XPath for XML, or CSV selectors for CSV). Following that, the shape and contents of the output RDF are defined as a series of mappings, using the R2RML ontology [52]. This can be done either directly, using static values, like a hard-coded string or number, or by selecting a value from the input stream and pasting it in a predefined slot in a set of RDF triples.
Figure 5. Example CARML annotation—definition of streaming CSV source (abridged).
Figure 6 and Figure 7 present an abridged example of a CSV input and an excerpt of a mapping that reads the distance parameter from the input and expresses it in terms of the “distanceCollision” RDF predicate. The result of the abridged mapping is presented in Figure 8. Note that CSV selectors are simple (see example in Figure 7 under the “rml:reference” property). They define only either a column number or column name (in case of CSV files with headers). JSONPath and XPath selectors can be more complicated. Additionally, CARML supports the declaration of custom functions using the Function Ontology (FnO) [53] that can perform various data manipulations, such as string or mathematical operations.
Figure 6. Sample CSV input data for a CARML annotation.
Figure 7. Example CARML annotation—definition of CSV data mapping (abridged).
Figure 8. Example CARML annotation—resulting RDF (abridged).

4.2. Semantic Translation

When constructing a Data Product Pipeline, in addition to annotating raw data, it may be necessary to adjust data that has already been semantically annotated using an alternative ontology. This requires the use of a tool that offers semantic translation. The O-CEI Semantic Translator component is based on Inter-Platform Semantic Mediator (IPSM), a generic streaming semantic translation software developed during INTER-IoT [54], and further enhanced within the ASSIST-IoT [49] and aerOS [29] European projects. The rules for translation used by the tool are expressed in a dedicated Alignment Format [55] (IPSM-AF), based on parametrized, highly readable Turtle RDF syntax. The workflow processing diagram of the Semantic Translator, including the relevant components, is depicted in Figure 9.
Figure 9. Semantic data translation through central ontology.
RDF producer is an arbitrary source of RDF data. Semantic Translator can handle multiple RDF producers concurrently. Using translation rules contained in the input alignment, represented in the IPSM-AF, it expresses the input data in terms of the “central semantics”. The fact that data semantics are expressed in the terminology of the central ontology allows for their flexible use. When creating translation channels, data consumers apply the appropriate alignments, again expressed in IPSM-AF, allowing them to adapt the semantic model to their preferred model.
Semantic translation, represented by an IPSM-AF alignment, can use any number of translation steps defined as applications of alignment cells. When performing the translation, these steps are executed sequentially, resulting in the corresponding transformations on the input RDF graph. In the alignment definition, each cell specifies the expected input pattern for the data (selecting an RDF subgraph to be transformed) and the output pattern defining the result of applying the transformation. As a result of applying the cell, RDF graph elements that are not matched remain unchanged, while the matched portion is replaced with the result of the translation.
A sample input pattern (in the Turtle RDF notation) using a hypothetical “port ontology” is presented in Figure 10. Elements marked with the “var” namespace (such as “var:elem” or “var:name”) represent variables used in the translation process. In the case of the input pattern, they prepare data that can then be used/substituted via the output pattern and define the cell translation result.
Figure 10. Alignment cell—input pattern specification in Turtle RDF syntax.
The output pattern in the alignment cell definition, shown in Figure 11, refers to variables defined by the input pattern (“var:id” and “var:name”), but also to the variables “var:station” and “var:geopos”, which did not appear in the input pattern. Such variables can be defined within alignment cells using a predefined/preloaded set of functions. For example, the variable “var:geopos” can be assigned the value “Point(var:lat,var:long)”, expressed in the geoSPARQL [56] WKT (Well-Known Text) format, using the string concatenation function. The output pattern also uses the Generic Ontology for IoT Platforms [57] and the Semantic Sensor Network Ontology [58].
Figure 11. Alignment cell—output pattern specification in Turtle RDF syntax.
The main components of the Semantic Translator architecture are shown in Figure 12. The tool provides two types of interfaces: REST and reactive streaming (see [48]). In addition to serving configurational purposes, the REST interface offers a quick, one-message-at-a-time translation service. The message broker-based interface, on the other hand, is designed to handle large, asynchronous data streams and supports both Kafka and MQTT message brokers. In both cases, translation is performed transactionally, using rules defined in IPSM-AF alignment files. Once uploaded to the translator’s alignment repository, these files and their translation rules are compiled into applicators for use in the translation process.
Figure 12. Semantic data translation component architecture.
The reactive stream processing rests on semantic translation channels and a central ontology. Technically, a translation channel is a one-way stream that accepts messages described with a given ontology and translates them into a different ontology. The translation is performed “on the fly”, as messages pass the channel, and is configured by two one-way alignments per channel—mediated by the central ontology (see Figure 9). The Semantic Translator can work with an arbitrary central ontology. However, based on the original design of the proposed approach, it is stipulated that it should be modular [59,60]. Modularity makes creating translation rules much easier because it reduces the number of choices that the author of the alignment must make. Moreover, it facilitates FDF design flexibility as, for instance, adding (or removing) certain translation(s) requires only adding (or removing) selected modules. There are also non-technical guidelines for the central ontology that render some ontologies more suitable for this role than others. In particular, a good candidate should have a “broad enough scope” and represent the domain of interest well.
The performance of the Semantic Translator’s key mechanisms was thoroughly investigated based on an early, yet fully functional, version of the tool [61]. The test environments used at that time were comparable to contemporary Raspberry Pi systems. The results demonstrated that the Semantic Translator provides high performance that scales well with increased computing power from the runtime environment. Translation channels can also be “parallelized,” allowing for better utilization of available hardware resources.
Table 3, for example, illustrates the outcomes of an experiment involving the processing of 40,000 randomly generated messages requiring non-trivial semantic translation on a dual-core machine with a total of four “logical” cores. A single translation channel was used, with the degree of parallelism in message processing increased gradually. As can be seen, the “performance gain” (represented by the reduction in total processing time for all messages) achieves its maximum value at a channel parallelism factor of 4; any further increase results in either a substantial increase in the median or a failure to achieve a sufficient reduction in total processing time. In other words, 4× in-channel parallelism exhausted the resources of the hardware used in the experiment.
Table 3. Semantic translation—channel parallelism vs performance.
The performance of other components, such as the Semantic Annotator, was evaluated in practice through several aerOS pilot deployments (e.g., in automotive IoT scenarios). However, it should be noted that deploying complete FDF node requires more hardware and system resources than a single Raspberry Pi machine can provide. Although FDF components can operate individually in a distributed manner and utilize various computing and hardware resources, their integration necessitates a containerized-enabled environment. Experiments over such an integrated system are envisioned to be conducted in the future.

5. Federation and Governance in the FDF

Two complementary aspects of the FDF architecture are essential in order to substantiate its federated nature. The first relates to the associated security, privacy and governance considerations, including threat assumptions, trust boundaries, access control, security concerns and auditability. The second relates to the federation primitives that govern how nodes interoperate. These cover trust establishment, catalog synchronization, identity and access management interoperability, policy propagation, provenance exchange, and failure handling.

5.1. Security, Privacy and Access Control

The first governance-related consideration included by the FDF is identity trust. Since each node in the FDF may behave differently when inserting (Data Product Owner, cf. Figure 13) or retrieving (Data Consumer, cf. Figure 14) data from the fabric, establishing an authorization flow to ensure trustworthiness is a first-class concern. First, users, roles and organizations (along with their associated credentials) are managed semantically using FOAF- and ORG-aligned entities [62] in an LDAP directory on the entrypoint domain. This directory is hosted and controlled centrally by the creator of the original federated ecosystem. Every node then authenticates through a local Keycloak instance. Cross-domain interactions are mediated by signed OAuth access tokens implemented as JSON Web Tokens (JWT), whose signatures are verified at the receiving domain. For example, Open Policy Agent (OPA) checks the token against the issuing Keycloak’s public key before any Context Broker or Data Product Manager operation is permitted. In this way, trust is not uniformly assumed across the federation. Rather, trust requires a bilateral relationship specific to the role and scope encoded in the verified token, which is established on a per-request basis.
Figure 13. Authorization workflow for Data Product Owners in FDF.
Figure 14. Authorization workflow for Data Consumers in FDF.
However, governance does not stop at authorization. Access control requires fine-grained policies that specify who can access what. In this case, the FDF commissions Data Product Owners to express their policies as Rego rules [63], which are enforced at the API gateway layer by KrakenD/Apache APISIX acting as the policy enforcement point in front of the Orion-LD Context Broker and Data Product Manager. Policy conflicts across federated domains (e.g., two nodes issuing contradictory access rules for a shared data product and certain users) are resolved via a precedence order that prioritizes the rule created in the data owner’s authoritative originary domain. This results in the override of other rules.
In terms of reliability and node trustworthiness, the FDF relies on an additional layer of the aerOS Meta-OS [64]. This module computes a “Trust Score” (a floating-point value ranging from 0 to 1) that combines sub-scores for reliability, security, and reputation sub-scores, with a penalty term triggered by self-healing alerts. Scores are propagated across the federation via an IOTA Tangle ledger. Since this value is accessible across domains (see Federation in Section 5.2), a degradation threshold can be established in the score (e.g., showcasing failure numbers) so that a domain can be disregarded for data sharing without requiring a centralized authority. Leveraging the setup of a DLT immutable record enables provenance and auditability in the Federated Data Fabric. IOTA exposes an actionable API that allows users to define messages for inserting relevant events that can be inspected in the tamper-resistant provenance log.
Regarding regulatory alignment, the FDF defers an in-depth study to future work. However, it establishes a solid, minimal governance posture grounded in concrete technical mechanisms. For the Data Act, the DCAT-aligned Data Catalog and standardized NGSI-LD API ensure that data products are discoverable and portable across the federation, eliminating the need for owner-specific integration efforts. Regarding the Data Governance Act, the Data Product Manager enables each data product owner to create and revoke their own Rego access policies, maintaining the Context Broker’s role as a neutral enforcement point rather than a de facto controller. Communications are protected in transit via TLS-secured REST APIs, which provide a baseline confidentiality guarantee for data exchanged across federation boundaries. Regarding the GDPR, the LDAP-backed identity model establishes a traceable link between a person’s identity and any granted access (which is already covered by the access restriction mentioned above), and LDAP’s native support for record rectification and deletion facilitates honoring data subject rights. Finally, the immutable, IOTA-backed trust records lay the groundwork for a future incident-reporting workflow under NIS2, even though a formal process has not yet been implemented.

5.2. Federation and Data-Synchronization Mechanisms

The aerOS Meta-OS architecture takes a clear federated approach to achieving interoperability across heterogeneous environments, including public and private clouds, edge clusters, and mission-specific networks. This enables independently governed operators to securely collaborate without surrendering sovereignty over their data or infrastructure. Federation applies to three main elements in the continuum: (i) user services, (ii) contextual data, and (iii) service orchestration. The aerOS distributed state repository mainly enables these federation mechanisms by allowing the sharing of key continuum information, including FDF data product descriptions and their associated metadata. This data is stored locally in each domain but is available universally in a collaborative manner without centralization or duplication of data [65].
In practice, the distributed state of aerOS is managed by several instances of the Orion-LD Context Broker. These instances form a Distributed State Network of Brokers (DSNB). Contextual information is represented as NGSI-LD entities and shared among the brokers through the automatic creation of inclusive Context Source Registrations (CSRs). This NGSI-LD-defined mechanism allows a broker to transparently redirect or distribute queries to the broker that maintains the requested entities. During normal operation, metadata changes are propagated through an event-driven mechanism based on NGSI-LD subscriptions and notifications. An initial pull-based synchronization is performed when a node joins or reconnects to the federation.
To maintain and automate this federation process across the CEI continuum domains and FDF nodes, an additional component is required. The aerOS Federator has a distributed architecture and is deployed with each Orion-LD instance. It is responsible for maintaining the CSRs in its associated Orion-LD instance by performing the necessary CRUD operations. During onboarding, a new domain’s Federator establishes a trusted relationship with an existing continuum Federator. Therefore, joining is subject to admission control and cannot be performed merely by discovering an accessible Context Broker. The joining node must authenticate itself and provide credentials and endpoint information that can be validated against the federation’s identity and governance mechanisms. This identity and access management interoperability across domains builds on the mechanisms described in Section 5.1. Once admitted, the Federator informs the existing participants of the new CB and synchronizes it with the current metadata view of the distributed state repository.
Through this process, the Federators and Context Brokers establish a trusted, continuously updated peer-to-peer network. Each Federator monitors the availability of its associated Broker, as well as the validity of the federation relationships represented by its CSRs. During a network partition, an isolated FDF node can continue to serve authorized local consumers and register and manage locally governed data products. However, remote products or services may become temporarily unavailable, and information obtained from other domains should be treated as the last synchronized view rather than the current state. Metadata changes produced during the disconnection are retained locally and are either propagated or reconciled after connectivity is restored. When a Context Broker fails, a domain leaves the federation, or an FDF node is classified as untrusted, the corresponding CSRs are disabled, updated, or removed so that requests are not forwarded to unavailable or unauthorized nodes. Thus, the federation supports degraded local operation, eventual recovery, and decentralized corrective actions, independent of a single centralized authority.

6. Illustrative Example

To highlight the scope of issues addressed by the FDF, it is instructive to consider an example from the domain of energy production and management. Although the scenario is inspired by the real-world pilot deployments of the aerOS and O-CEI projects, it does not follow either of them directly. First, note that energy management is a complex field that includes traditional and renewable energy sources with variable power outputs, energy storage systems, and producers, consumers, and prosumers. These elements are intermingled in a grid driven by variable production, demand, and prices dictated by an energy market with a large number of participants.
As demonstrated by projects such as Concept Grid [66], a modern energy grid includes a multitude of actors, stakeholders, devices and technologies. The example focuses on the data aspect and involves multiple federation levels that control how much data is shared internally and published. The end goal of the proposed system is to establish real-time or near-real-time datasets that drive energy production and demand prediction algorithms. These datasets would also allow for quick reactions to changing grid conditions in the event of outages, surges, or other unexpected energy delivery outliers. Therefore, what follows outlines the main aspects of an FDF that could be instantiated for such an energy community.

6.1. Simple and Communal Prosumers

Given the growing importance of individual prosumers [67], let’s start from the perspective of a residential home. A popular approach for prosumers living in single-family homes is to install solar panels and store the generated energy in a battery. If possible, such person can also sell the excess energy to the electric grid (see Figure 15). Within the household, this energy powers old-fashioned electric appliances, such as refrigerators, computers, and TVs, as well as air conditioners, electric heaters, and electric vehicle (EV) charging stations, among others. A growing number of homes, particularly in countries such as France (see [68]) already have smart energy meters (IoT devices) that capture data about energy production and consumption within a household. Sometimes, these IoT devices allow the collection of data that can be used for FDF implementations, such as the one proposed in this contribution.
Figure 15. Sample homestead-to-operator data fabric overview.
Details about energy usage could be useful for homeowners, their communities (e.g., buildings, smart energy neighborhoods, villages, etc.), appliance manufacturers, energy producers, and governing bodies. This information can be used for market research to inform decisions. However, for privacy reasons, homeowners may not want to publicly share all energy consumption data. It is likely that they would prefer to make informed decisions and provide selected data to certain entities (e.g., public administrations) while withholding it from others (e.g., the appliance manufacturers).
In FDF terms, let’s assume the energy company selling or buying energy from the prosumer has access to the household’s energy meter. This allows the company to determine how much to charge or pay the homeowner for consumed or produced energy. The energy meter is a smart device with its own interface, and is connected to the FDF via its own ingesting stage. The meter’s data is encrypted and contains a client ID and a time series with multiple power and energy parameters. These parameters may include active, reactive, and apparent power; current; phase voltage; and more for L1, L2, and L3 phases. For an example of an energy meter data schema, see Energy Models from the Smart Data Models Initiative [69]. To prevent data leaks, the ingestion stage is deployed on the energy company’s hardware within its network. There, the data is decrypted and forwarded. During the processing stage, the client ID is matched against information from internal databases. This data is semantically annotated with location, homeowner’s name, etc., then passed to the storage stage. Note that this data product is configured individually for each energy meter and client to provide the ingesting stage with the necessary credentials to access the energy meter. It is also deployed separately for each individual client. The energy company controls every such data product entirely. Each deployment of a data product supplies the storage stage with more data. Eventually, this provides insights about the clients. This can lead to the optimization of pricing for each prosumer. For instance, a company may pay less for energy from a prosumer who consistently produces a surplus, then sell that energy to a neighbor at a higher price.
The same forces that drive capitalistic optimization can motivate consumers to organize “against” companies and form energy communities. This approach has recently gained popularity in community-driven renewable energy integration research [70,71]. One way to express this idea is to extend the prosumer identity from a single home to a community of homes, each with electric appliances, power sources, and batteries. In simple terms, this allows people to use their neighbor’s battery to store their excess energy, based on the assumptions of benevolence and reciprocity.
To this end, a federation can be set up (see Figure 16). It will use the same ingesting stage as the energy company but be deployed locally. This means that every prosumer in the neighborhood can join a federation of prosumers who share energy usage and production statistics, which benefits the community. The FDF enables the creation of a neighborhood dataset, which can result in a fairer pricing and a more stable power grid. Because the data products for the neighborhood federation are independent, neighbors may share less data with each other than with the company. This, in turn supports privacy preservation. For example, within the neighborhood federation, it may be sufficient to share a homestead’s total active and reactive energy import and export, without disclosing details relevant to a power company, such as phase voltage or power factor. The practicality of these solutions depends on market conditions and the pressure that the citizens can exert on governing bodies. Therefore, this is beyond the scope of this paper.
Figure 16. Sample communal data fabric overview.

6.2. Urban Energy Management

In the context of governance, municipal bodies are often interested in accurate energy information within their jurisdictions. To obtain an accurate overview of a city’s needs and status, data from multiple heterogeneous sources must be collated. While a single homestead or neighborhood may be served by a single energy company, this scenario likely involves the need to address data interoperability. Specifically, when collecting data from an entire city, one must assemble data from single-family homes, residential blocks, commercial buildings, city lights, traffic infrastructure, and more. These data sources have different energy profiles, and are likely operated by different providers, called “operators.” Furthermore, they are highly likely to have a varying degree of smart readiness (see Smart Readiness Indicator [72]).
Nevertheless, it is possible to use the FDF to create a coherent dataset. In this case, each device type requires its own ingestion stage (see Figure 17). Whether it is a building block retrofitted with solar panels, a public EV charging station, smart traffic lights, or a shopping center, each energy meter interface (i.e., the data source) must be addressed individually. Additionally, data from Non-Intrusive Load Monitoring (NILM) devices, installed across the city, must be included. The scope of NILM data may differ greatly, targeting a given location in a grid, rather than a single device with a known set of properties or a single user. Even when well-known standards are followed at the municipality level, crucial differences in the format and content of the data will exist. The semantic processing stage is critical here. After ingesting the data, some sources may require simple conversions of power or energy units or adjustments to timestamps. Others will require a full semantic annotation. For instance, this would involve converting raw CSV data to formats with explicit semantics. Implementing semantic annotation and translation requires an appropriate level of familiarity with both the initial data produced by devices and the data model expected by the city administration IT systems.
Figure 17. Sample urban data fabric overview.
The requirements of a public contract may force individual operators to set up and maintain their own storage stage so city officials can query each data repository separately using a common query language. For example, SPARQL allows you to query multiple servers with a single query. This scenario plays to the strengths of semantic translation. Assuming each “operator” has its own data model, two data products must be deployed: one for the operators and one for the city. The first writes the full data to a database on the operator’s premises and keeps it confidential so only the data owner may access it. The latter is open to the city administrators and may be public, either as a whole or in part. Both data products share the same ingestion stage. During processing, the same semantic annotation process is likely applied, followed by different semantic translations. In the city-dedicated data product, the role of semantic translation is to select the parts that can be shared with the city and align the data with the city’s ontology. Although parts of the data products are shared, the deployments are handled individually by the Access Control mechanisms. For municipal data dissemination, operators and city officials are connected to the FDF as data producers and consumers, respectively.
This hypothetical example illustrates the distributed nature of FDF and the independence of its parts. For a more centralized example of a practical, citywide energy market, see the CityxChange project [73].

6.3. Reflections Arising from the Examples

The distributed nature of the FDF architecture is the key when it comes to answering complex interconnections present in any energy management scenario. As the complexity of the considered example grows, the ability to granularly control access to data products, orchestrate their components, and keep them independent of each other becomes not just a gratuitous set of features, but a necessity.
It is important to note that energy management, previously mentioned as an illustration of FDF usability, serves as a more general enabler for numerous applications across the CEI continuum. For example, energy management in Combined Heat and Power (CHP) units permits advanced load scheduling for appliances, like electric water/space heaters. Intelligent BEV (Battery Electric Vehicle) charging strategies for charging stations enable reductions in energy consumption and emissions in vehicle fleet operations. There are also more specific benefits of smart energy management (and they are being addressed by the O-CEI project), in addition to the already-mentioned encouragement of renewable use, enhanced grid stability, reduced energy waste, better management of energy prices, etc.
The FDF architecture enables all these benefits by virtue of high manageability and orchestrability, supporting computations on the far edge, in the cloud, or anywhere in between, at the level of a data product, or even (depending on implementation) selected data components.

7. Empirical Validation

The FDF architecture has been implemented and empirically validated as part of the aerOS project. One of the aerOS pilot deployments [74] (Pilot 5), in which FDF played a central role, involved the development of a management system for an energy-efficient, healthy, sustainable smart building. The system under consideration comprises five primary components: an IoT system, a forecasting system, a recommender, and a graphical user interface (GUI).
The IoT system consists of a network of sensors and actuators that are responsible for controlling and adjusting the environmental conditions within the smart building. The sensors utilized in this study encompassed a range of detectors, including binary state (e.g., open/closed), presence and motion sensors, along with meters for measuring power, temperature, humidity, pressure, concentrations of CO, CO2, NH4, and various particulate matter (PM) levels.
The actuator system is responsible for receiving the environmental parameters from the forecasting system. The actuators subsequently employ these parameters to regulate devices such as the heating, ventilation, and air conditioning (HVAC) system or dehumidifier. For instance, when the levels of CO2 or PM2.5 are elevated the air purifier is activated, when the humidity is high, the dehumidifier is turned on, and if the temperature is excessively high or low, the HVAC system adjusts accordingly. In circumstances where the forecast indicates that environmental conditions will remain within acceptable parameters, actuators receive a signal to deactivate environmental devices. This ensures energy efficiency and maintains a comfortable indoor environment.
The recommender is a software component that generates seat recommendations based on the building’s energy usage and employees’ preferences. It receives building room information, desk availability, and “health scores” generated by a dedicated AI-based component. It also takes employees’ preferences into account, which are provided through the GUI application. The GUI also offers the recommender’s output to employees. The system has the capacity to interact with the AI component and the GUI via MQTT topics exposed by FDF.
In the context of the smart building pilot deployment, a total of eleven data products were defined and rigorously validated. The results of this process have been thoroughly reported in accordance with the established KPIs (key performance indicators) [75]. The KPIs considered encompassed, among other metrics, edge processing performance, service availability, and service scalability. Notably, FDF was employed in two additional aerOS project pilots. However, its role in these deployments was more restricted than the application outlined in the context of the smart building management system. Due to the innovative features and functionality of the FDF architecture and its initial implementation, the tool has been incorporated into the EU Innovation Radar collection [76].

8. Conclusions and Future Work

IoT-based systems grow in size and complexity and require continuous, real-time data transfer and interoperability on multiple levels. This contribution presents the Federated Data Fabric (FDF), a hybrid data infrastructure architecture grounded in the tenets of data fabric and data mesh. The FDF addresses the needs of large distributed IoT ecosystems by providing robust support for persistent streams and semantic processing, thereby facilitating semantic and syntactic interoperability of both data and metadata. The FDF principally emphasizes the flexibility and manageability of deploying data processing artifacts, otherwise known as data products. Each data product is managed autonomously by its respective owner and is divided into stages that can be orchestrated individually. This enables the distribution of data processing across the Cloud-Edge-IoT continuum. Additionally, access control mechanisms empower users to establish federations in which each participant voluntarily agrees to share specific data with others. Accordingly, individual stakeholders are capable of enforcing their respective “requirements” and “policies” independently across the strata of the continuum.
The concept, as well as the initial implementation (https://github.com/aerOS-Project (accessed on 28 August 2026)) of the FDF architecture described in this paper, were developed as part of the aerOS project and empirically validated through several pilot deployments. In addition to the fundamental structure—comprised of the Data Catalog, Access Control, and Data Product Manager—the implementation provides components for the semantic annotation and translation of streaming and batch data. In accordance with contemporary software deployment trends, the architecture is predicated on containerized software orchestration mechanisms proffered by the aerOS Meta-OS.
The proposed approach responds to the prevailing tendency of Meta-OS architectures to evolve beyond conventional edge orchestration frameworks by incorporating data management as an inherent capability. In the aerOS Meta-OS, the FDF fulfills this role by providing data-aware orchestration across the CEI computing continuum. Unlike existing data space, data mesh or data fabric approaches, which only address subsets of the problem, the FDF combines federation, interoperability, governance and data management capabilities within a single, CEI-oriented architecture.
As part of the O-CEI project, work is underway to isolate and unify the communication mechanisms used by semantic tools (the annotator and translator), as well as to expand and refine the FDF federation mechanisms. The final result will be verified by applying the enhanced FDF implementation to large-scale O-CEI pilots in several domains, including energy management, agriculture, and vehicle fleet management. This will enable the performance and scalability of the proposed architecture and its updated implementation to be thoroughly verified. The use of FDF in the O-CEI pilot projects will also help identify any potential technological, technical, or architectural limitations of the solution described in this article. The results obtained through this process will be utilized in practice and disseminated in the future.

Author Contributions

Conceptualization, W.P. and P.S.; software, P.S., W.P., I.D.M.-C., I.L. and R.V.; writing—original draft preparation, W.P. and P.S.; writing—review and editing, W.P., P.S., I.L., R.V., Z.W., M.P., M.G., and K.W.-M.; project administration, I.L. and M.G.; funding acquisition, I.L., M.G. and M.P. All authors have read and agreed to the published version of the manuscript.

Funding

This research was funded by the EU Horizon Europe grants: “Autonomous, scalablE, tRustworthy, intelligent European meta Operating System for the IoT edge-cloud continuum”—aerOS (ID 101069732) and “Open CloudEdgeIoT Platform Uptake in Large Scale Cross-Domain Pilots”—O-CEI (ID 101189589).

Institutional Review Board Statement

Not applicable.

Data Availability Statement

The complete source code for the software presented in this study is openly available on GitHub at https://github.com/aerOS-Project (accessed on 28 August 2026).

Conflicts of Interest

Author Ignacio D. Martinez-Casanueva, was employed by the company Telefónica Research and Development, Madrid, Spain. The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as potential conflicts of interest.

References

  1. Wang, R.Y.; Lee, Y.W.; Pipino, L.L.; Strong, D.M. Manage your information as a product. Sloan Manag. Rev. 1998, 39, 95–105. [Google Scholar]
  2. Inmon, W.H. Building the Data Warehouse; Wiley: Indianapolis, IN, USA, 2005. [Google Scholar]
  3. Kimball, R.; Ross, M. The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling, 3rd ed.; John Wiley & Sons: Indianapolis, IN, USA, 2013. [Google Scholar]
  4. Azzabi, S.; Alfughi, Z.; Ouda, A. Data Lakes: A Survey of Concepts and Architectures. Computers 2024, 13, 183. [Google Scholar] [CrossRef] [Scilit]
  5. Schneider, J.; Gröger, C.; Lutsch, A.; Schwarz, H.; Mitschang, B. The Lakehouse: State of the Art on Concepts and Technologies. SN Comput. Sci. 2024, 5, 449. [Google Scholar] [CrossRef] [Scilit]
  6. Machado, I.A.; Costa, C.; Santos, M.Y. Data Mesh: Concepts and Principles of a Paradigm Shift in Data Architectures. Procedia Comput. Sci. 2022, 196, 263–271. [Google Scholar] [CrossRef] [Scilit]
  7. Goedegebuure, A.; Kumara, I.; Driessen, S.; Van Den Heuvel, W.J.; Monsieur, G.; Tamburri, D.A.; Nucci, D.D. Data Mesh: A Systematic Gray Literature Review. ACM Comput. Surv. 2024, 57, 11. [Google Scholar] [CrossRef] [Scilit]
  8. van der Werf, D.; Moreira, J.; Piest, J.P.S. Towards a Data Mesh Reference Architecture. In Enterprise Design, Operations, and Computing. EDOC 2024 Workshops; Lecture Notes in Business Information Processing; Springer: Cham, Switzerland, 2025; Volume 537, pp. 339–353. [Google Scholar] [CrossRef] [Scilit]
  9. Priebe, T.; Neumaier, S.; Markus, S. Finding Your Way Through the Jungle of Big Data Architectures. In Proceedings of the 2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA, 15–18 December 2021; IEEE: New York, NY, USA, 2021; pp. 5994–5996. [Google Scholar] [CrossRef] [Scilit]
  10. Blohm, I.; Wortmann, F.; Legner, C.; Köbler, F. Data products, data mesh, and data fabric. Bus. Inf. Syst. Eng. 2024, 66, 643–652. [Google Scholar] [CrossRef] [Scilit]
  11. Karlapalem, K.; Krishna, P.R.; Valluri, S.R. Data Fabric Technologies and Applications. In Proceedings of the 8th International Conference on Data Science and Management of Data (12th ACM IKDD CODS and 30th COMAD); Association for Computing Machinery: New York, NY, USA, 2025; pp. 362–365. [Google Scholar] [CrossRef] [Scilit]
  12. Bode, J.; Kühl, N.; Kreuzberger, D.; Holtmann, C. Toward Avoiding the Data Mess: Industry Insights from Data Mesh Implementations. IEEE Access 2024, 12, 95402–95416. [Google Scholar] [CrossRef] [Scilit]
  13. Gartner IT Glossary: Data Fabric. 2025. Available online: https://www.gartner.com/en/data-analytics/topics/data-fabric (accessed on 28 August 2026).
  14. LaPlante, A. Data Fabric as Modern Data Architecture; O’Reilly Media, Inc.: Sebastopol, CA, USA, 2021. [Google Scholar]
  15. Gkonis, P.; Giannopoulos, A.; Trakadas, P.; Masip-Bruin, X.; D’Andria, F. A Survey on IoT-Edge-Cloud Continuum Systems: Status, Challenges, Use Cases, and Open Issues. Future Internet 2023, 15, 383. [Google Scholar] [CrossRef] [Scilit]
  16. Parimi, S. Data Mesh vs. Data Fabric: The Future of Data Management. Int. J. Sci. Technol. 2025, 16, 1–23. [Google Scholar] [CrossRef] [Scilit]
  17. Gieß, A.; Hutterer, A. The Future of Data Management: A Delimitation of Data Platforms, Data Spaces, DataMeshes, and Data Fabrics. Inf. Syst. e-Bus. Manag. 2025, 23, 971–997. [Google Scholar] [CrossRef] [Scilit]
  18. Judvaitis, J.; Blumbergs, E.; Arzovs, A.; Mackus, A.I.; Balass, R.; Selavo, L. A Set of Tools and Data Management Framework for the IoT–Edge–Cloud Continuum. Appl. Syst. Innov. 2024, 7, 130. [Google Scholar] [CrossRef] [Scilit]
  19. Hasan, M.M.; Sultana, T.; Hossain, M.D.; Mandal, A.K.; Ngo, T.T.; Lee, G.W.; Huh, E.N. The journey to cloud as a continuum: Opportunities, challenges, and research directions. ICT Express 2025, 11, 666–689. [Google Scholar] [CrossRef] [Scilit]
  20. Hoseini, S.; Theissen-Lipp, J.; Quix, C. A survey on semantic data management as intersection of ontology-based data access, semantic modeling and data lakes. J. Web Semant. 2024, 81, 100819. [Google Scholar] [CrossRef] [Scilit]
  21. KubeEdge: A Kubernetes Native Edge Computing Framework. Available online: https://kubeede.io/en/ (accessed on 28 August 2026).
  22. OpenYurt: An Open Platform That Extends Upstream Kubernetes to Edge. Available online: https://openyurt.io/ (accessed on 28 August 2026).
  23. Baetyl: Baetyl, Extend Cloud Computing, Data and Service Seamlessly to Edge Devices. Available online: https://baetyl.io/en/ (accessed on 28 August 2026).
  24. EdgeX Foundry: The Enabled Open Software Platform. Available online: https://www.edgexfoundry.org/ (accessed on 28 August 2026).
  25. aerOS Project. aerOS Architecture Definition (2). aerOS Deliverable 2.7. 2024. Available online: https://aeros-project.eu/wp-content/uploads/2024/07/D2.7.pdf (accessed on 28 August 2026).
  26. ICOS Project. ICOS Architectural Design. ICOS Deliverable 2.4. 2024. Available online: https://www.icos-project.eu/files/deliverables/D2.4-architectural-design-it2.pdf (accessed on 28 August 2026).
  27. NEMO Project. NEMO Meta-Architecture, Components and Benchmarking. Final Version. NEMO Deliverable 1.3. 2024. Available online: https://meta-os.eu/wp-content/uploads/2024/09/NEMO_D1.3-NEMO-meta-architecture-components-andbenchmarking.Final-version_V1.1.pdf (accessed on 28 August 2026).
  28. NebulOuS Project. Prototype of Monitoring & Fog/IoT Data Streams Management in Cloud Computing Continuum. NebulOuS Deliverable 5.1. 2024. Available online: https://nebulouscloud.eu/wp-content/uploads/2026/02/D5.1-Prototype-of-Monitoring-FogIoT-Data-Streams-Management-in-Cloud-Computing-Continuum.pdf (accessed on 28 August 2026).
  29. aerOS Project. 2025. Available online: https://aeros-project.eu/ (accessed on 28 August 2026).
  30. O-CEI Project. 2026. Available online: https://o-cei.eu/ (accessed on 28 August 2026).
  31. Bacco, M.; Kocian, A.; Chessa, S.; Crivello, A.; Barsocchi, P. What are data spaces? Systematic survey and future outlook. Data Brief 2024, 57, 110969. [Google Scholar] [CrossRef] [Scilit]
  32. Bader, S.; Pullmann, J.; Mader, C.; Tramp, S.; Quix, C.; Müller, A.W.; Akyürek, H.; Böckmann, M.; Imbusch, B.T.; Lipp, J.; et al. The International Data Spaces Information Model—An Ontology for Sovereign Exchange of Digital Content. In The Semantic Web—ISWC 2020; Lecture Notes in Computer Science; Springer: Cham, Switzerland, 2020; Volume 12507, pp. 176–192. [Google Scholar] [CrossRef] [Scilit]
  33. Arnold, B.T.; Lange, C.; Gillmann, C.; Decker, S. A Service Architecture for Dataspaces. arXiv 2025, arXiv:2507.07979. [Google Scholar] [CrossRef] [Scilit]
  34. Julian, M.; Esbri, M.A.; Lacalle, I.; Cabanillas, L.; Vaño, R.; Pallau, C.E. Distributed and Trusted Access to Data Spaces’ Products Employing Sovity and Data Fabric. In GIECS 2025; Communications in Computer and Information Science; Springer: Cham, Switzerland, 2026; Volume 2719, pp. 93–108. [Google Scholar] [CrossRef] [Scilit]
  35. Rieyan, S.A.; News, M.R.K.; Rahman, A.M.; Khan, S.A.; Zaarif, S.T.J.; Alam, M.G.R.; Hassan, M.M.; Ianni, M.; Fortino, G. An advanced data fabric architecture leveraging homomorphic encryption and federated learning. Inf. Fusion 2024, 102, 102004. [Google Scholar] [CrossRef] [Scilit]
  36. Supramanian, A.V. Data Mesh Architecture: Revolutionizing Enterprise Data Management through Decentralization. Int. J. Sci. Res. Comput. Sci. Eng. Inf. Technol. 2025, 11, 63–71. [Google Scholar] [CrossRef] [Scilit]
  37. FAIR Principles. Available online: https://www.go-fair.org/fair-principles/ (accessed on 28 August 2026).
  38. Orion-LD Context Broker. Available online: https://github.com/FIWARE/context.Orion-LD (accessed on 28 August 2026).
  39. aerOS Continuum Ontology. 2025. Available online: https://github.com/aerOS-Project/continuum-ontology (accessed on 28 August 2026).
  40. Software for Delivering Intelligence at the Edge—Final Release. aerOS Deliverable 4.3. 2025. Available online: https://aeros-project.eu/wp-content/uploads/2025/11/D4.3.pdf (accessed on 28 August 2026).
  41. Keycloak: Open Source Identity and Access Management. Available online: https://www.keycloak.org/ (accessed on 28 August 2026).
  42. KrakenD: The Fastest API Gateway. Available online: https://www.krakend.io/ (accessed on 28 August 2026).
  43. Apache APISIX: API Gateway & AI Gateway for APIs and AI Agents. Available online: https://apisix.apache.org/ (accessed on 28 August 2026).
  44. Open Policy Agent. Available online: https://www.openpolicyagent.org/ (accessed on 28 August 2026).
  45. ISO 11179; Information Technology—Metadata Registries (MDR)—Part 1: Framework. IEC: Geneva, Switzerland, 2015.
  46. The Reactive Manifesto. 2017. Available online: http://www.reactivemanifesto.org (accessed on 28 August 2026).
  47. Redpanda: High-perf Agentic Data Plane & Streaming. Available online: https://www.redpanda.com/ (accessed on 28 August 2026).
  48. Reactive Streams. 2022. Available online: https://www.reactive-streams.org/ (accessed on 28 August 2026).
  49. ASSIST-IoT—EU H2020 ICT-56-2020 Project: Enablers Documentation. 2023. Available online: https://assist-iot-enablers-documentation.readthedocs.io/en/latest/ (accessed on 28 August 2026).
  50. A Pretty Sweet RML Engine. Available online: https://github.com/carml/carml (accessed on 28 August 2026).
  51. RDF Mapping Language (RML). Available online: https://rml.io/specs/rml/ (accessed on 28 August 2026).
  52. R2RML: RDB to RDF Mapping Language Schema. Available online: https://www.w3.org/ns/r2rml (accessed on 28 August 2026).
  53. The Function Ontology. Available online: https://fno.io/ (accessed on 28 August 2026).
  54. Palau, C.E.; Fortino, G.; Montesinos, M.; Exarchakos, G.; Giménez, P.; Markarian, G.; Castay, M.V.; Fuart, F.; Pawłowski, W.; Mortara, M.; et al. (Eds.) Interoperability of Heterogeneous IoT Platforms; Internet of Things; Springer: Cham, Switzerland, 2021. [Google Scholar] [CrossRef] [Scilit]
  55. Szmeja, P.; Pawłowski, W.; Ganzha, M.; Paprzycki, M.; Wasielewska Michniewska, K. Alignment format for semantic translation. In Proceedings of the International Conference on Sustainable Technology and Advanced Computing in Electrical Engineering (ICSTACE 2021), Surat, India, 11–12 November 2021; Lecture Notes in Electrical Engineering; Springer: Singapore, 2022; Volume 939, pp. 651–671. [Google Scholar]
  56. geoSPARQL. Available online: https://www.opengeospatial.org/standards/geosparql (accessed on 28 August 2026).
  57. Generic Ontology for IoT Platforms. Available online: https://inter-iot.github.io/ontology/ (accessed on 28 August 2026).
  58. Semantic Sensor Network Ontology. Available online: https://w3c.github.io/sdw-sosa-ssn/ssn/ (accessed on 28 August 2026).
  59. Stuckenschmidt, H.; Parent, C.; Spaccapietra, S. (Eds.) Modular Ontologies. Concepts, Theories and Techniques for Knowledge Modularization; State-of-the-Art Survey, LLNCS; Springer: Berlin/Heidelberg, Germany, 2009; Volume 5445. [Google Scholar]
  60. Shimizu, C.; Hammar, K.; Hitzler, P. Modular Ontology Modelling. Semant. Web J. 2022, 14, 459–489. [Google Scholar] [CrossRef] [Scilit]
  61. Ganzha, M.; Paprzycki, M.; Pawłowski, W.; Szmeja, P.; Wasielewska, K.; Solarz-Niesłuchowski, B.; de Puga García, J.S. Towards High Throughput Semantic Translation. In Interoperability, Safety and Security in IoT; LNICST; Springer: Cham, Switzerland, 2018; Volume 242, pp. 67–74. [Google Scholar] [CrossRef] [Scilit]
  62. W3C Government Linked Data Working Group. The Organization Ontology. W3C Recommendation, 16 January 2014. 2014. Available online: https://www.w3.org/TR/vocab-org/ (accessed on 28 August 2026).
  63. Open Policy Agent Contributors. Policy Language. Open Policy Agent Documentation. 2026. Available online: https://www.openpolicyagent.org/docs/policy-language (accessed on 28 August 2026).
  64. aerOS Project Consortium. Trustworthiness. aerOS Documentation. 2026. Available online: https://docs.aeros-project.eu/en/latest/aaa_trust/trust/ (accessed on 28 August 2026).
  65. Vaño, R.; Lacalle, I.; Palau, C.E. Federation of distributed domains in the Cloud-Edge-IoT Continuum. In Proceedings of the 2nd International Workshop on MetaOS for the Cloud-Edge-IoT Continuum, MECC’25; Association for Computing Machinery: New York, NY, USA, 2025; pp. 14–19. [Google Scholar] [CrossRef] [Scilit]
  66. EDF Power Networks Lab. Concept Grid. Available online: https://www.edf.fr/en/the-edf-group/inventing-the-future-of-energy/r-d-global-expertise/our-offers/edf-power-networks-lab/our-testing-facilities/concept-grid (accessed on 28 August 2026).
  67. European Environment Agency. Energy Prosumers in Europe—Citizen Participation in the Energy Transition. Available online: https://www.eea.europa.eu/en/analysis/publications/the-role-of-prosumers-of (accessed on 28 August 2026).
  68. Connectivity Standards Alliance. Linky Smart Meters + Zigbee: Smarter Energy Management for France. 2025. Available online: https://csa-iot.org/newsroom/linky-smart-meters-zigbee-smarter-energy-management-for-france/ (accessed on 28 August 2026).
  69. Smart Data Models. Smart Data Models Energy Data Models—AC Measurement. Available online: https://github.com/smart-data-models/dataModel.Energy/blob/master/ACMeasurement/ (accessed on 28 August 2026).
  70. Ekechukwu, D.E.; Simpa, P. A comprehensive review of renewable energy integration for climate resilience. Eng. Sci. Technol. J. 2024, 5, 1884–1908. [Google Scholar] [CrossRef] [Scilit]
  71. De Simone, E.; Rochira, A.; Mannarini, T. Individual and community catalysts for Renewable Energy Communities (RECs) development. Curr. Opin. Psychol. 2025, 62, 101987. [Google Scholar] [CrossRef] [Scilit]
  72. Smart Readiness Indicator. Available online: https://energy.ec.europa.eu/topics/energy-efficiency/energy-performance-buildings/smart-readiness-indicator_en (accessed on 28 August 2026).
  73. CityxChange. Available online: https://cityxchange.eu/ (accessed on 28 August 2026).
  74. aerOS Project Deliverable D5.4. Available online: https://aeros-project.eu/wp-content/uploads/2025/11/D5.4.pdf (accessed on 28 August 2026).
  75. aerOS Project Deliverable D5.6. Available online: https://aeros-project.eu/wp-content/uploads/2025/11/D5.6.pdf (accessed on 28 August 2026).
  76. Federated Data Fabric Enabling Seamless Usage of Information Anywhere and Anytime. Available online: https://innovation-radar.ec.europa.eu/innovation/66360 (accessed on 28 August 2026).
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Article Metrics

Citations

Article Access Statistics

Journal Statistics
Multiple requests from the same IP address are counted as one view.
Data-Driven Reduction of Rail Pad Dynamic Stiffness Testing: Benchmarking a cGAN Against Gaussian Process Kriging
Previous
A Flux-Vector Direct Torque Control Strategy Based on d,q-Axis Flux Closed-Loop Regulation for Permanent Magnet Synchronous Motor Drives
Next