AI and Data Democratisation for Intelligent Energy Management

Abstract

Despite the large number of technology-intensive organisations, their corporate know-how and underlying workforce skill are not mature enough for a successful rollout of Artificial Intelligence (AI) services in the near-term. However, things have started to change, owing to the increased adoption of data democratisation processes, and the capability offered by emerging technologies for data sharing while respecting privacy, protection, and security, as well as appropriate learning-based modelling capabilities for non-expert end-users. This is particularly evident in the energy sector. In this context, the aim of this paper is to analyse AI and data democratisation, in order to explore the strengths and challenges in terms of data access problems and data sharing, algorithmic bias, AI transparency, privacy and other regulatory constraints for AI-based decisions, as well as novel applications in different domains, giving particular emphasis on the energy sector. A data democratisation framework for intelligent energy management is presented. In doing so, it highlights the need for the democratisation of data and analytics in the energy sector, toward making data available for the right people at the right time, allowing them to make the right decisions, and eventually facilitating the adoption of decentralised, decarbonised, and democratised energy business models.

1. Introduction

Currently, there is increasing debate over the importance of data democracy [1,2]; press pieces, academic articles, and debates over mainstream media expand or redefine the term and continuously reshape its meaning and framework. The growth in data volumes (i.e., big data [3]), the expansion in data types/formats, and the plethora of data analysis and process methodologies (e.g., data mining [4,5] and machine learning [6,7,8] procedures), combined with the constantly increasing momentum of Artificial Intelligence (AI) [9,10], indicate that the 21st century marks the era of data science [11,12,13,14]: everybody imprints data somewhere (on the Web, Internet of Things (IoT), etc.) almost about everything.

AI-based analytics can identify hidden, potentially useful information patterns within large datasets that can be transformed into actionable outcomes and knowledge to support improved decision making [15]. However, in most cases, few possess these data.

This is particularly evident in the energy sector. The energy system is going through an unprecedented transformation that can be summarised in two main factors: supply side changes mainly orienting on the installation of renewable energy sources [16] and demand-side solutions orbiting around behavioural change and further highlighting a new, more active role for consumers [17]. Such a transformation is significantly affecting the business structure and determining the creation of completely new concepts of utility. In such a context, one of the main factors is that we are straying away from the concept of delivering a commodity and instead entering a new economy of services.

1.1. Barriers to the Diffusion of AI and Data Democratisation in Energy Sector

Energy utilities tend to believe that AI will have a big impact on their business: AI spreading is perceived to eventually allow them to align their business processes to novel optimal energy management operations [18], within a context of rising share of deployed renewable energy [19]. However, they are far from integrating AI in key business processes [20]. This is mostly due to energy utilities—despite having high technical skill in power systems—very often lacking on their workforce Information and Communication Technologies (ICT) background to fully exploit the AI potential in their business, and hence progress toward integrating AI applications or services within their processes [21].

In addition to poor-ICT-skilled stakeholders, a high number of leading-edge Small and Mid-size Enterprises (SMEs) are emerging to provide a vast number of smart energy management services and technologies to energy stakeholders. However, the fuzzy and often unclear regulation on access to data has been instead hindering the potential of sustainable business cases and return on investment for SMEs.

This includes the General Data Protection Regulation (GDPR), which is in force in the European Union (EU) and the European Economic Area (EEA). The GDPR is liable for keeping energy consumer data, as well as fragmented regulation on data access and cross-country harmonised or standardised requirements for services provisioning to energy stakeholders. The GDPR also addresses the transfer of personal data outside the EU and EEA areas, while one of its main aims is to simplify the regulatory environment for international business by applying a unified regulatory framework within the EEA. Despite the plethora of such regulations worldwide, there are steps forward to align different legal frameworks among countries in the context of data sharing. It is worth mentioning that the GDPR became a role model for many national laws outside the EEA (e.g., in Chile, Japan, Brazil, South Korea, Argentina and Kenya). Moreover, the California Consumer Privacy Act (CCPA) and the Personal Information Protection and Electronic Documents Act (PIPEDA) of Canada also share many features with the GDPR.

Additionally, the presence of consolidated functional/organisational silos, combined with the lack of semantic and business interoperability across data stream providers, has too been hindering the exploitation of the high potential of AI-based analytics in the energy sector. While most value chains cross the boundaries of individual sector, existing big data and IoT applications in Business to Business (B2B) energy settings tend to be “siloed”: they do not provide easy ways for sharing, reusing, and repurposing data assets, cognitive models, and AI services across domains/stakeholders. It becomes evident that combined data from different domains (energy, climate, economics, and financing; user behaviour and societal trends; regulations; and many more) can open a wide spectrum of unprecedented AI energy analytics services [22], if effectively tackled with appropriate technological solutions.

1.2. Overarching Objective of This Research

The aim of this paper is to analyse AI and data democratisation, in order to explore the strengths and challenges in terms of data access problems and data sharing, algorithmic bias, AI transparency, privacy, and other regulatory constraints for AI decisions, as well as to review novel applications in different domains, giving particular emphasis on the energy sector. In this context, a data democratisation framework for intelligent energy management is presented, integrating seven components for: (1) data discovery and interoperability, (2) data quality compliance, (3) data privacy and sharing, (4) AI-based library, (5) analytics service builder and visualisation, (6) AI energy analytics services, and (7) security. In doing so, it highlights the need for the democratisation of data and analytics in the energy sector, toward making data available for the right people at the right time, allowing them to make the right decision, and eventually facilitating the adoption of decentralised, decarbonised, and democratised energy business models.

The rest of the paper is organised as follows. In Section 2, the progress toward democratisation of data in the era of data science is analysed. Section 3 presents the growing importance of AI and data democratisation through different applications for insights-driven decisions. A data democratisation framework is presented in Section 4 that enables AI-based cross-sector analytic tools for intelligent energy management, based on seamless data information and knowledge exchange under respective sovereignty and regulatory principles. Finally, the last section summarises the key issues that have arisen in this paper.

2. Data Democracy in the Era of Data Science

Thousands of terabytes per day [23], containing insightful information and knowledge, may be lost, or stored without being processed properly. Moreover, few individuals or institutions in the world possess critical amounts of data [24,25], thereby having advantage over others; to them, data democracy is far from a ‘lucrative’ concept. However, there exist more barriers than the latter’s reluctance to share their data. It seems that the concept of data open access follows the footprints of a past, yet insistent, controversy: open-source versus commercial software applications and operational systems. The difficulties and corresponding lessons learned from their coexistence could pave the way for establishing data democracy.

Open-source software inspirators sought to break the barriers established by commercial software monopolies and to make software accessible to all. However, people still use commercial software on a large scale, and IoT is not yet framed with open-source products. Similarly, smart devices, like smartphones and tablets, are exclusively using operating systems of oligopoly companies, which use but do not give the collected data back to society. Moreover, it is controversial to see many open-source software products being utilised by tech and consultancy giants to eliminate their product cost and increase their profits, without them giving at least credits to the developers of these products, or the corresponding communities.

These limitations hindering the expansion of open-source software resemble the barriers that data democratisation is facing: complexity, limited education or training from non-experts, lack of innovation and product evolution, and limited product support post-installation. In other words, the voluntary work of open-source communities is not enough in complex societies, with limited spare time, and large global division of labour. In essence, progress toward data democracy dictates that the public and private sectors alike be convinced for its importance and that stakeholders (including civil society and institutions) be transformed, rather than build up antagonistic institutions/communities to break into the existing regime.

2.1. Conceptualising Data Democratisation

Considering the above, people tend to define data democracy as something more than data open access [1]. Interestingly, data democracy is all about data open access. Extended definitions only seek to further elaborate its concept trying to establish it against real-world barriers. In brief, data democratisation is the process of bringing digital information to the average, non-expert end user. By doing so, the latter would be able to gather, process, and analyse data to reach critical conclusions (i.e., make decisions) without requiring outside help. The end goal is to transform people or employees into well-educated ‘data science citizens’ (or ‘lay data scientists’).

Education is not the only prerequisite, however. Data must be easily accessed from safe, secure, organised repositories at any time by anyone—with respect to legal confidentiality and privacy issues. Hence, the quality of the provided data is of paramount importance. It cultivates the necessary trust among the end users, who want to use unquestionably (the content of) data platforms implementing the data democracy concepts and principles. In that context, data democracy requires extra personnel (engineers, developers, business analysts, legal staff, etc.) to guarantee all the above. Even if the motivation lies in bypassing current data gatekeepers (IT personnel), data democracy may indirectly empower other sectors (data science, data mining, data analytics, cloud computing, legal advisors, and so on).

Therefore, data democratisation should not be viewed as a stand-alone procedure that simply reduces labour cost and mitigates the barriers of an old-fashioned division of labour. It is rather an entirely new concept, which restructures the economy (i.e., production, logistics, management, etc.) and the social structure in terms of efficiency. This signifies a novel point of view regarding the relationship and power balance among citizens, institutions, governments and policymakers, employers, technocrats, and many more.

Data democracy seems to bring to life an economic structure that encompass the benefits, and transcends the boundaries, of both ‘free market’ and ‘monopolistic/oligopolistic’ economic models, in terms of access to data. The former resembles a decentralised system. The latter a hierarchical, strictly vertically organised system. A decentralised system is closer to the widespread globalisation economic process. However, the decentralised system lacks a fundamental assumption of the ‘free-market’ model: the absolute knowledge of all stakeholders over what is going on in the entire market (i.e., nobody has benefit over the others). This assumption though is only (theoretically) realistic in the ‘monopolistic/oligopolistic’ system. All production and related information/data are concentrated to the few vertically structured enterprises. The flexibility of the decentralised global economy could be combined with fuller/holistic access to data and knowledge, which is yet only conceivable in the ‘monopolistic/oligopolistic’ systems. In that context, stakeholders can have a holistic, fully informed point of view over different economic sectors.

2.2. Data Democracy in the Public Sector

Yet, the progress toward democratisation of data is slow, with data distribution remaining non-homogeneous along the global public sphere. The private sector is, broadly speaking, more reluctant to carry out radical reforms in this direction. On the other hand, Open Government Data (OGD) is gradually gaining acceptance [26,27,28]; see for examples the United States [29], India [30], and the United Kingdom [31]. OGD is data produced by governments or government-controlled entities and can be freely used and reused or redistributed by anyone. OGD promote transparency as well as accountability of governments (and their affiliated entities) to citizens, and can support scientists/researchers, politicians, and a diversity of other stakeholders to elaborate evidence-based policies and encourage citizens to take better-informed personal decisions.

The importance of the latter is ever evident during the COVID-19 virus pandemic, and it will be in its aftermath. Intuitively, it is considered that OGD will result in economic growth: companies are expected to make use of OGD, thereby promoting business creation and innovative, citizen-centric services. For example, the European Commission (EC) in 2011 expected that its Open Data Strategy for Europe would have improved the European economy by offering an annual 40-billion-euro boost [32]. However, this implied that companies would invest in the utilisation of this strategy (and open data in general) and reform their structural organisation accordingly [33]. The motivation, however, behind such an adaptation would be increased efficiency and profits.

2.3. Data Democracy in the Private Sector

Companies should be convinced that data democracy is worth the trouble. Despite the barriers, though, some companies have shown considerable interest in open data and some novel business models have emerged [34,35]. When it comes to data democracy in the private sector, the concept is significantly different than in the public sector: while OGD is freely available to all members of society (including the private sector), this is not the case in the private sector, where data democracy is perceived as the act of opening organisational data to as many employees as possible [36]. This limitation (i.e., this intra-organisational definition of open data) is restrictive for companies and citizens/consumers alike. In a decentralised way of production, where too many different companies are involved in the production of final products and services, there must be inter organisational implementations (i.e., cloud platforms), which share open data. This would benefit the enterprises belonging to the same sector and/or chain of production and would likely provide consumers with the capacity to provide feedback about the end products.

2.4. Data and AI Democracy and the EU’s Strategy to Overcome Related Barriers

The process of transforming non-experts into ‘lay data scientists’ must be discussed in parallel with the democratisation of data [37,38,39] and AI/machine learning [40,41,42,43,44]. Open data can be used to feed predictive models allowing non-experts to reach crucial data-driven decisions. Understanding the algorithms behind AI models nowadays, however, requires a basic grasp of computer science and a solid understanding of mathematics or statistics [45].

The lack of appropriate tools for capturing the real-time dynamics, the scarcity of and competition for AI experts, the need for knowledge transfer to new contexts and accordingly for training new AI for each different context, have started being addressed by the EC, through:

• The recent initiative to boost AI [46] to support everyday life, while creating a fertile ground for EU entrepreneurs and SMEs to create innovative AI-based services.
• The AI4EU platform [47], which has been funded by the European Union (EU) with a view to set up and manage a one-stop-shop environment where know-how resources, algorithms, solutions, and services related to AI are channelled.

However, the success of the above EU-level AI strategy strongly depends on the successful implementation of the European Strategy for Data [48], since AI solutions/services could be successfully adopted to the extent of facilitated and increased EC-level data sharing, while fully respecting privacy and other regulatory constraints.

Another critical barrier to be solved for AI to be expanded, which follows the selection of an AI algorithm for a given application, lies in the requirement of advanced programming skills for its utilisation in most commercial AI frameworks. Therefore, democratising AI requires that the corresponding frameworks, and especially their Graphic User Interfaces (GUI), be upgraded. In that sense, AI platforms that require minimum coding, if any, are also a much-needed step toward data democracy. Besides, it was only after mitigating the complexity of interaction between end users and computing machines that computing science (and its products and services) became popular, proliferated, and widely used by non-experts.

3. The Growing Importance of AI and Data Democratisation

AI and data democratisation is gaining ground and acceptance, yet the pace has hitherto been slow. This underexplored area has significant potential in both theory and practice. In that context, and to our knowledge, there are only a few press pieces and academic articles in this field of study.

3.1. A Brief Literature Overview

In the housing market, data democracy provides both sellers and consumers with insightful data and the corresponding indices to make vital decisions over future alternatives [49,50,51]. In the domain of healthcare, data democracy means more than decision support [52,53]: shared, big data repositories are perceived to strengthen synergies among research teams, laboratories, and the broader science community, facilitating cross-validation of scientific conclusions. In this case, however, data democracy should encompass responsible, legitimate, and secure sharing of personal, sensitive information of patients [54], as well as tracing medical conclusions to both data and expert opinion, to avoid rendering exaggeration or irresponsible advice authoritative (e.g., during the COVID-19 pandemic [55]).

Several applications of data democratisation can also be found in the field of energy resource, including data democratisation in offshore drilling [56], oil and gas operations [57,58], as well as earth [59] and climate [60] sciences. These research activities are orbiting around efforts to gain useful insights into—and improve—resource and sectoral efficiency (e.g., in digital agriculture [61]).

More specifically, in climate change science and policy, there are increasing calls for ‘unboxing’ the ‘black box’ of the complex and diverse [62] quantitative systems modelling activities [63], which are predominantly used to inform decisions [64], and for opening them to non-experts [65]. This entails making science comprehensible and digestible [66], which goes beyond developing open-source tools [67] and instead aims to promote science co-production [68], inclusivity [69], and pluralism [70].

3.2. Data Democracy in the Energy Sector

Despite the energy sector being generally considered as rather conservative investment-wise, the advent of AI is boosted by the rising availability of much larger datasets being generated at an unprecedented rate at the edge of the grid. Grid-owned assets, such as Intelligent Electronic Devices (IEDs), transformers, feeders, Phasor Measurement Units (PMUs), as well as non-grid-owned assets, such as decentralised generation and smart buildings, flexible industrial and residential building-level or community-level aggregated loads, and model-based (simulated) data, are opening up to more distributed architecture, which incorporates more local-level data distributed integration, control, and applications, thus creating new innovation challenges.

Moreover, off-domain data, such as Energy Performance Certificates, building stock auditing, Energy Performance Contracts, weather conditions, climate data, geographical imagery, multimedia unstructured data sources, financial data on energy efficiency investments, socioeconomic data, social media, energy end-user’s characteristics and comfort levels, etc., may allow novel energy analytics to provide energy stakeholders with more robust actionable insights, if suitably integrated [22,71]. This creates the need for disruptive, cutting-edge approaches towards a trustful and legally binding data sharing culture, where data value is shared and exchanged amongst the members of the ecosystem, following a fair distribution of resources, hence facilitating a seamless integrated energy value chain and more efficient business processes [72].

Dealing with cross-domain data requires a precise understanding of their nature and radically new digital technologies, such as AI, IoT, as well as cloud and big data technologies, to efficiently process large quantities of data within tolerable elapsed times, with safety and security [73]. The lack of energy-tailored data ownership and sovereignty management policies has prevented in many cases from making available and sharing data while respecting privacy, protection and security, and appropriate learning-based modelling capabilities.

Distributed Ledgers and Blockchain technologies are ideal candidate technologies to implement decentralised privacy- and regulation-aware data governance and sovereignty, due to their intrinsic support for data immutability, transparency, accountability, and traceability, enabling full data provenance tracking (e.g., [74,75]). However, such technologies do not currently offer the necessary scalability to address the short latency requirements typical of the smart energy grids environments [76,77].

Finally, much like energy, climate, and biophysical limits, big data and data democracy have also been studied and/or promoted as concepts in the context of transitioning from a linear to a circular economic model [78,79,80], where user-friendly data can provide insights for improving decision-making in terms of efficient resource use. Democratised AI tools can help predict trends of future supply and demand, weather fluctuations, demographics, biophysical limitations, as well as rates of reducing waste/externalities, reusing materials, and recycling products.

It is evident that extensive innovative approaches are needed to assess the knowledge and tools developed over the years and extend them to a holistic, AI-powered, human-centric framework, aimed to optimise business processes and ensure the long-term competitiveness across the energy sector [81] and intertwined domains (including environment, society, and economy) [82].

4. A Data Democratisation Framework for Intelligent Energy Management

In this section, we propose a framework that enables AI-based analytic tools for intelligent energy management, based on seamless data/information and knowledge exchange, under respective sovereignty and regulatory principles. It aims at evolving and scaling up innovative AI energy analytics services, which significantly contribute to achieve techno-economically optimal management of the energy value chain, especially for SMEs and non-tech end-users.

The data democratisation process for intelligent energy management can take several forms. More specifically, the framework consists of seven main components: (1) data discovery and interoperability, (2) data quality compliance, (3) data privacy and sharing, (4) an AI-based library, (5) analytics service builder and visualisation, (6) AI energy analytics services, and (7) security (Figure 1).

AI energy analytics services address the needs of a large number and diversity of energy stakeholders, including Transmission System Operators (TSOs), Distribution Service Operator (DSOs), energy suppliers, Distributed Energy Resource (DER) operators, aggregators and/or energy cooperatives, Energy Service Companies (ESCOs), building operators, funding and energy agencies, policymakers, off-grid data/solution providers, and other third parties. These AI-based services are grouped into the following categories:

• Increasing the efficiency and reliability of the electricity network.
• Optimising the management of DER assets connected to the grid.
• De-risking investments in energy efficiency and increasing the efficiency and comfort of buildings.

4.1. Data Discovery and Interoperability

This component deals with the discovery and interoperability of the various data from different sources and/or platforms. It provides the necessary functionalities to support end users (including data producers and/or data consumers, as well as non-expert end-users) to join the platform and search for and discover services matching their goals and business needs. A field for orchestrating semantic interoperability (e.g., based on the International Data Spaces [83] for data sharing) describes the data exchange interfaces for read/write access and import/export of available data and output analytics.

Moreover, this component provides standardised data gates relying on existing or promising standardised data models. More specifically, it is based on open standards, open Application Programming Interfaces (APIs) like NGSI-LD CIM APIs, and open data models, such as FIWARE [84], BIM [85], SAREF/SAREF4Building [86], BRICK [87], and HAYSTACK [88]. This component includes connectors (plugins and IT interfaces) for already International Electrotechnical Commission (IEC) Common Information Model (CIM) standardised data and/or data platforms, as well as connectors to non-electrical/non-grid assets (gas, water, etc.), non-energy external data sources (sensing systems, weather data, etc.), and market data. Connection to the specific data hubs/systems is also driven.

4.2. Data Quality Compliance

The data quality compliance component is responsible for converting primary data collected from different sources into a format that renders them available for specific uses and suitable for generating new information and supporting decision-making. It allows integration, pre-processing, and querying of heterogeneous data. The ingestion of the datasets is followed by a series of data pre-processing actions. Data cleansing and curation focuses on the research, design, and development of the optimal data pre-processing pipelines.

4.3. Data Privacy and Sharing

Data sharing is an important step for democratising data. In this respect, this component deals with the integration of blockchain, enabling a secure, trusted, and flexible data/model sharing for B2B cross-stakeholders. This component handles smart energy cross-domain heterogeneous data and metadata after a process of harmonisation and integration. The data are maintained off-chain, ensuring privacy/GDPR compliance, and data sovereignty. Its core functionalities offer trusted mechanisms of data sharing and re-using.

A full governance cycle is implemented in order to support the creation, specialisation, update, storage, and maintenance of the Data Access Policies allowing also to dynamically derive access control policy rules based on contextual information (e.g., volume or frequency of data access, location of request, etc.). The resulting connector enables and facilitates fully interoperable, cross-stakeholder data access and sharing, while allowing data owners to maintain full control and sovereignty of their data, and fulfilling regulatory, privacy, business, and security constraints.

4.4. AI-Based Library

A library of reusable AI-based machine learning models is made available with a view to promote quick adaptation and re-use of machine learning models along different contexts. This component elaborates Automated Machine Learning (AutoML) methods and processes that render ML available to non-experts, as well as provide documentation and tutorials for each algorithm. In that context, various comprehensible metrics indicating, for example, the degree of accuracy, bias, and/or fit of a predictive model to training data, as well as robustly measuring the sensitivity of an AI model to these, can be important steps toward AI democratisation, and vital for the end-user to understand which algorithm best fits their needs.

This component also leverages on privacy-preserving federated learning, where data models are shared at the level of a subset of training models, hence yielding significant reduction of communication costs, and fully addressing data privacy concerns. Moreover, when partial data sharing is allowed, or data providers are willing to share their own data, the component combines unsupervised deep learning/reinforcement learning models for cross-context transfer learning (e.g., cross-context artificial intelligence by design) with inline adaptive self-learning to improve the resulting ML models.

4.5. Analytics Service Builder and Visualisation

The aim of this component is to develop and validate descriptive, predictive, and prescriptive/anticipatory data analytics along a variety of significant applications, by making use of the AI-based library. It incorporates the analytic service builder, in order to enable reuse of business logic capabilities on the selected datasets. It offers an environment to create customised services, based on a set of available analytical features, implementing different statistical and analytics algorithms, along with the functionality to perform custom queries on the available datasets.

Moreover, the visualisation engine, where the end user may have the opportunity to select one of more visual presentation modes, is enriched by innovative multi-dimensional visual analytics. These visualisations vary from simple statistical charts to map views and more, aiming to deliver useful understanding and insights coming from the data to the end users.

4.6. AI Energy Analytics Services

The AI energy analytics services component spans from analytics-based applications, tailoring power network improved decision-making to improve its reliability, to applications tailored to commercial aggregators/suppliers aimed to optimise management of flexibility assets, to applications addressing building-scale, energy-efficient comfort management, and dynamic renewable investment risk assessment.

4.6.1. Increasing the Efficiency and Reliability of the Electricity Network

The ability to analyse and manage vast volumes of data is increasingly important to improving operational efficiency and reliability of the electricity network. Proactive cross-functional, cross-domain analytics services for integrated grid-owned asset and grid operation management, tailored to power network operators, namely TSOs and DSOs, include:

• Predictive and prescriptive TSO/DSO grid-owned asset maintenance to facilitate and support grid optimal operation and/or planning, by trading off maintenance cost against accelerated asset ageing due to network overloads, and by combining and integrating grid endogenous and exogenous context-based information, such as Light Detection and Ranging (LIDAR), weather, and geographic.
• Edge-level network load and renewable energy generation prediction, by leveraging and integrating heterogeneous data from network assets, consumers and DERs smart meters, weather forecasting, geographical information, with a view to providing actionable insight to enable optimised grid operation and planning.

4.6.2. Optimising the Management of DER Assets Connected to the Grid

In the last two decades, the energy sector has undergone a significant transition. Government incentives and advances in technology have resulted in a proliferation of distributed renewable energy resources, particularly solar and wind generation. In addition, plug-in electric vehicles are now penetrating the market. As a result, the power load has an entirely different character and, in the resulting distributed grid, the energy and information flows are bidirectional. Therefore, it is critical to leverage available data to better understand energy demand, and align this to energy generation and distribution, to maximise operational efficiency. In particular, DER-level predictive analytics services include predictive analytics for DER/prosumer flexibility potential forecasting for: (i) optimal DER coordination and operation, (ii) optimal cross-energy (including electric vehicles) or cross-infrastructure (water network) management, (iii) voltage peak-to-peak (VPP) portfolio optimised management, and (iv) optimised power market operation.

4.6.3. De-Risking Investments in Energy Efficiency and Increasing the Efficiency and Comfort of Buildings

The built environment is responsible for 40% of final energy consumption [89]; therefore, it is important to increase the energy efficiency and comfort of buildings. Furthermore, there is a need to increase investments in energy efficiency measures, which can be achieved by de-risking investments, in particular by reliably predicting and monitoring energy savings. Predictive analytics for enhanced reliability and reduced risks of energy efficiency investments, tailored to ESCOs and financing institutions, are aimed at performing finer-grained prediction for building comforts. This is done by integrating a variety of historical data on energy efficiency investments, with near-real-time metered energy consumption, thereby contributing to better defining energy performance contract conditions.

4.7. Security

This last component is responsible for researching, maintaining, and reinforcing the legal and security policies and mechanisms employed (authentication, authorisation, auditing, policy-based management, and data encryption) including but limited to proactive vulnerabilities/risk assessment, secure session management, cross-site scripting protection, command injection flaws, buffer overflows protection, input sanitisation, and protection against logging breaches. For every new functionality of the platform, a validation process against existing data protection requirements takes place to ensure compliance with EU and national regulations and data protection acts. This component also ensures that the data transferred to the cloud for processing meet the security constraints—i.e., they are encrypted and anonymised where necessary.

5. Discussion

Despite the large number of technology-intensive organisations, their corporate know-how and underlying workforce skill are not mature enough for a successful rollout of AI services in the near future. However, things have begun changing, owing to the increased adoption of data democratisation processes, and the capability offered by emerging technologies for data sharing, while respecting privacy, protection, and security, as well as appropriate learning-based modelling capabilities for non-expert end-users.

In the energy sector, the constantly increasing momentum of data technologies and the growing trend toward a data economy, which leverages on exploiting the untapped economic value of legacy data assets, represents an unprecedented opportunity for energy stakeholders. These trends, combined with AI, constitute a catalyst toward conceptualising and generating innovative energy management services and applications.

They can gain significant improvements in the reliability and efficiency of electricity grid operation and planning; enable better and more efficient management of the grid-owned, behind-the-meter assets connected to the grid; while at the same time enabling a more reliable assessment of energy efficiency and renewable energy financial investments, taking into due consideration operational performance monitoring and comfort of citizens.

At the same time, such a trend paves the way for Energy Data Spaces, as the energy service data-driven collaborative ecosystem of the future.

However, energy stakeholders (including related SMEs and non-tech industries) are facing challenges in adopting AI, which may reduce the speed with which AI is adopted and thus limit its economic potential, with major challenges being represented by access to AI enablers, access to human capital, and AI-skilled talented workforce.

These challenges create the need for the democratisation of data and analytics in the energy sector, to make data available for the right people at the right time, enabling them to make the right decisions, thereby facilitating a seamless integrated energy value chain and more efficient business processes. Moreover, it is necessary to provide the appropriate framework toward a trustful and legally binding data sharing culture in the energy sector, where data value is shared and exchanged among the members of the ecosystem participants, following fair distribution of resources.

6. Conclusions

The data democratisation framework presented enables cross-sector analytic tools for intelligent energy management, based on seamless data/information and knowledge exchange under respective sovereignty and regulatory principles. These capabilities render AI as a new factor of production that can drive growth in at least three directions, those of energy commodities networks, DERs (renewable energy generation, buildings, districts, and communities), as well as synergies with and implications on other energy and non-energy domains (e.g., heat, transport, water, personal safety, and finance).

AI-based applications range from optimal risk assessment for energy efficiency investments planning, to optimised management of grid- and non-grid-owned assets, improved efficiency and reliability of electricity networks operation, as well as new socially and environmentally sustainable business models. These applications include a large number of energy stakeholders, including TSOs, DSOs, DERs operator, aggregators, ESCOs, building operators, and off-grid data/solution providers.

New, energy-based social service might enable pairing of public electric vehicle charging stations with adjacent renewable energy generation assets, and providing incentives for drivers to charge their vehicles in time intervals with surplus renewable energy capacity. It may also provide incentives to alleviate energy poverty at the local level, while offsetting for example energy surplus credits from free delivery to poor people from the energy bills in a peer-to-peer fashion, or targeting optimal trade-off among personal comfort and optimal energy management, or otherwise deliver personal safety/security and ambient assisted living services.