Analysis of Variability in Electric Power Consumption: A Methodology for Setting Time-Differentiated Tariffs

Abstract

The increasing concern for environmental conservation has spurred government initiatives towards energy efficiency. One of the key research areas in this regard is demand response, particularly focusing on differential pricing initiatives such as Time-of-Use (ToU). Differential tariffs are typically designed based on mathematical or statistical models analyzing historical electricity price and consumption data. This study proposes a methodology for identifying time intervals suitable for implementing ToU energy tariffs, achieved by analyzing electric power demand variability to estimate demand flexibility potential. The methodology transforms consumption data into variation via the coefficient of variation and, then, employs k-means data analysis techniques and the a priori algorithm. Tested with real data from smart meters in the Colombian electrical system, the methodology successfully identified time intervals with potential for establishing ToU tariffs. Additionally, no direct relationship was found between external variables such as socioeconomic level, user type, climate, and consumption variability. Finally, it was observed that user behavior concerning consumption variability could be categorized into two types of days: weekdays and non-working days.

1. Introduction

The growing concern for the care and conservation of the planet has driven significant global changes in the approach to environmental issues. From the United Nations Framework Convention on Climate Change in 1992 to the historic Paris Agreement in 2015, international commitments have been established to reduce greenhouse gas emissions and combat climate change [1,2].

In response to these commitments, regional organizations and governments have implemented policies and guidelines to achieve specific goals in environmental care and emission reduction. Examples include the European Green Deal, which aims to reduce greenhouse gas emissions by 55% by 2030, and Colombia’s CONPES 4075, which sets the goal of reducing CO₂ emissions by 51% by 2030 and achieving carbon neutrality by 2050 [3,4].

In this context, the transition to renewable energy sources, such as solar and wind, has accelerated. However, power generation from these sources presents challenges due to their high variability and dependence on environmental conditions. This poses challenges in accurately predicting energy generation and in the stability of supply [5,6].

In light of this situation, smart grids have emerged as a promising solution to integrate renewable energy generation with the traditional electrical grid [7]. These grids require real-time information provided by smart meters, which allow for measuring electrical energy consumption and other parameters. The installation of smart meters has experienced significant growth worldwide, driven by governmental policies and regulations. By 2024, the Asia-Pacific, Europe, and the United States are expected to lead the market with millions of meters installed in each region [7].

The use of smart meters has generated a large amount of data, as they can collect information at intervals of seconds, minutes, or hours [8]. This data availability has spurred research focused on improving energy efficiency and reducing consumption in both residential and industrial contexts. One such field of research is demand response (DR), which seeks to change the end-user’s consumption pattern to smooth consumption peaks [9,10]. DR integrates price incentive programs to change end-users’ consumption patterns, achieving the stability and balance of energy resources in addition to providing economic efficiency to network stakeholders. Thus, methods like Real-Time Pricing (RTP), Time-of-Use pricing (ToU), and Critical Peak Pricing (CPP) have been proposed, with the ToU and RTP being the most-widely used. TOU pricing is a rate where the energy price varies in time intervals ranging from hours to days or weeks. The latter is preferred by both network operators and consumers [11].

Various methodologies for determining tariff intervals leverage statistical tools, utilizing historical energy price data and aggregated energy consumption [12,13]. For instance, Ref. [12] proposes a methodology employing a Gaussian Mixture Model based on historical pricing data. Other methodologies incorporate mathematical models to abstract user preferences and the dynamics of energy prices and consumption [14]. Notably, some approaches, like the one proposed by [15], employ quadratically constrained quadratic programming and stochastic optimization techniques to determine the most-suitable time intervals for implementing differential tariffs. This article introduces a novel approach for identifying time intervals for differential electricity tariffs, particularly focusing on the Time-of-Use pricing (ToU) strategy. It proposes a method that uses data analysis techniques from information collected by smart meters. This method focuses on establishing those time intervals where the flexibility of electric energy demand is most notable and, therefore, strategic for tariff implementation. The method’s development is based on the premise that a user, showing high variability in his/her consumption over a specific time interval, is more likely to exhibit flexibility in his/her energy consumption during the same period.

The structure of this document is as follows: Section 2 explores two fundamental concepts for this research: demand response and demand flexibility. Section 3 presents a literature review of various demand response tariff types and explores user flexibility through load monitoring. In Section 4, the methodology proposed in this article is presented, aimed at identifying time intervals based on the variability of user consumption. Section 5 discusses the results obtained from applying the proposed methodology to a case study involving a dataset of energy consumption from 73,794 users of the Colombian electrical system. Section 6 offers discussions and the limitations of this study. The conclusions are drawn in Section 7, and finally, Section 8 outlines future work.

2. Conceptual Framework

In this section, we delve into two pivotal concepts of our research identified in the literature review: demand response and electricity demand flexibility. Demand response underscores the proactive engagement of consumers in adjusting their electrical consumption in response to external signals, thereby facilitating more-effective energy management. Conversely, demand flexibility encompasses the capacity of users to alter their consumption patterns within a specified time frame, whether through augmentation, reduction, or redistribution of their electrical energy usage.

2.1. Demand Response

Demand response involves adjustments in the electrical consumption of users in reaction to changes in electricity prices, economic incentives, or guidelines from network operators. This practice transforms consumers from passive entities to active agents in energy management, enabling them to economize energy and contribute to the stability of the electrical grid through load flexibility. Currently, DR is evolving towards a more-inclusive modality, integrated demand response (IDR), encompassing various energy sources such as electricity, thermal, and natural gas. This evolution allows users not only to modify their electrical consumption, but also to choose between different energy sources. DR is part of an approach towards smart energy management, aiming to maximize the aggregate benefits of smart energy systems and the daily gains of electric companies [16,17].

2.2. Electric Demand Flexibility

Flexibility in electrical systems is gaining importance due to the transition to renewable generation sources like solar and wind. These energies, despite their significance in the energy transition, pose challenges due to their variable and uncertain nature. Flexibility, in this context, refers to the ability of the electrical system to adapt reliably and cost-effectively to variations in electrical generation and demand across all timescales. Traditionally focused on generation, flexibility now also encompasses areas such as demand response, energy storage, and dispatchable distributed generation. It is fundamental for the effective integration of renewable energies, impacting the reliability and costs of the system and contributing to its resilience [18].

In the context of buildings, demand flexibility manifests in the ability to adapt energy demand and generation to user needs, the electrical grid, and climatic conditions. It is achieved through the modification of load profiles, such as demand reduction or shifting, and can be of two types: implicit, related to the adaptation of consumer behavior to price signals, and explicit, which requires the intervention of an aggregator and is marketable in energy markets. These strategies allow more-efficient management of consumption, adapting to the requirements of the grid and reducing the operational costs of buildings [19].

In the context of demand response, the flexibility of users is leveraged to induce adjustments in the load profile of the electrical grid. This flexibility is implicit and depends on the behavior and preferences of consumers regarding their energy use. Understanding user flexibility involves a detailed analysis of their consumption patterns and their willingness to modify them. In this study, we propose a method to estimate user flexibility by examining the variability in their electrical energy consumption. The premise underlying our research is that greater variability in consumption may indicate a higher degree of user load flexibility.

3. Literature Review

The primary objective of this research is to establish a potential time interval for implementing differential pricing by leveraging user flexibility. Flexibility, as defined in the context of demand response (DR), is the capability of users to modify their electricity usage patterns in response to external signals such as price changes or grid needs. This modification can include shifting or reducing consumption during specific periods. Various differential pricing types exist, each tailored to leverage different aspects of user behavior and grid requirements:

• Real-Time Pricing (RTP): This pricing strategy is characterized by frequent changes in electricity prices, often on an hourly basis, reflecting the real-time cost of energy production and supply–demand imbalances. It encourages users to adjust their consumption in response to these price signals, thus contributing to grid stability [17,20].
• Time-of-Use pricing (ToU): Under ToU pricing, electricity rates vary at predetermined time blocks, typically set as peak, off-peak, and sometimes, shoulder periods. This pricing model is less dynamic than RTP, but offers more predictability to consumers, allowing them to plan their energy usage around these different rate periods [17,20].
• Critical Peak Pricing (CPP): CPP involves significantly higher prices during critical peak events, which are typically rare, but can occur during extreme demand situations or when grid reliability is at risk. This model aims to drastically reduce consumption during these critical periods [17,20].

Each of these pricing strategies, with RTP and ToU being the most-commonly used, plays a crucial role in the broader scope of demand response, which is an integral part of advanced metering infrastructure (AMI) and smart grid operations. By employing these differential pricing mechanisms, grid operators can incentivize consumers to modify their energy usage, contributing to grid stability and resilience, efficient resource utilization, and enhanced economic efficiency for all stakeholders involved [11,17,21].

Building on these concepts, Ref. [11] introduced a methodology utilizing a demand response aggregator (DRA) to create ToU tariffs based on next-day discounts (DB-ToU). This strategy aims to smooth energy consumption curves by leveraging the flexible demand of residential users. The DRA analyzes customer behavior in response to incentive policies, implementing daily pricing schemes and communicating price signals to manage demand and reduce peak load. This approach includes collecting consumption profiles, calculating rewards, and formulating future policies, employing reinforcement learning to optimize DRA performance. Notably, this method prioritizes user privacy and has effectively improved the quality of the aggregated load profile without significantly impacting the aggregator’s revenues.

Expanding upon these themes, Ref. [21] developed a framework for efficient demand control in the smart grid, utilizing data from the AMI network. This framework comprises four main components: energy distribution companies; a module for predicting demand and pricing rates based on demand response; AMI metering infrastructure; and demand-side energy management. The energy management component recommends using the Binary Particle Swarm Optimization (BPSO) algorithm to regulate internal consumption in response to network operator incentives. Additionally, the prediction module employs a multilayer perceptron trained with historical consumption data, encompassing residential, commercial, and industrial consumption patterns. These data are crucial for price-based demand response programs like day-ahead pricing (DA), ToU, and CPP over a year. The model has shown potential in reducing electricity costs, minimizing user discomfort, and lowering carbon emissions, thereby enhancing consumption efficiency.

Further exploring demand response strategies, Ref. [22] focused on developing a demand response model based on genetic algorithms for residential users. This model is designed to optimize load shifting in response to demand response events, taking into account consumer preferences and constraints. It employs a realistic low-voltage distribution network consisting of 236 buses to demonstrate the application of the model. The study successfully showed that this approach can lead to considerable energy cost savings for consumers and improvements in voltage profiles, highlighting the effectiveness of genetic algorithms in managing demand response in smart grids.

Furthermore, Ref. [13] explored the design of Critical Peak Pricing (CPP) tariffs in the context of Great Britain’s electricity market. Their investigation focused on using market index prices and volumes to analyze the price and demand distribution, which are essential for establishing an effective CPP tariff. The study aims to enhance demand response, optimize energy consumption patterns during peak periods and balance the cost of electricity supply with retail market revenue. Their methodology involves a detailed statistical analysis of market data, leading to the design of a CPP tariff structure that could influence consumer behavior and contribute to grid stability.

In a similar vein, Ref. [15] presented a study aimed at optimizing the total economic welfare, considering the fluctuating nature of electricity demand and price sensitivities. This is achieved through the design of Time-of-Use (ToU) tariffs. The study particularly addresses challenges arising from uncertainties in the price elasticities of electricity demand. The authors employed an innovative methodology using quadratically constrained programming within a stochastic optimization framework. Similarly, Ref. [12] developed ToU tariffs based on existing flat rates using clustering techniques. Employing the Gaussian Mixture Model innovatively, they categorized half-hour interval flat-rate tariffs into clusters, laying the groundwork for ToU tariffs. This approach aims to mirror the variations in energy prices and system load demands in tariff design.

The initial aim of this section is to develop a methodology for establishing differential tariffs based on the electrical consumption flexibility of users. A key approach to determine this flexibility involves studying the users’ load and estimating their adaptability potential, a process known as load monitoring (LM).

LM is divided into three main categories: intrusive load monitoring (ILM), non-intrusive load monitoring (NILM), and other methods. The ILM methodology, which requires a meter for each appliance, can be costly and face connectivity issues. Hence, the NILM methodology has been researched, estimating individual appliance consumption through algorithms that disaggregate the total consumption [23]. NILM is further subdivided into machine learning (ML) techniques, Pattern Matching (PM), and single-channel Source Separation (SS) [24].

In the research conducted by [25], a load-monitoring (LM) methodology was presented using the non-intrusive load monitoring (NILM) technique to optimize energy buying and selling operations in a microgrid. The methodology focuses on disaggregating high-energy-demand appliances, employing a multitask sequence-to-now learning structure with convolutional neural networks. Additionally, a method based on the hidden Markov model is proposed to support the microgrid’s bidding decisions in the energy market, enabling reduction in consumption and costs and generating additional revenue by selling surplus energy.

In a study by [26], a Pattern Matching (PM) methodology for electrical load disaggregation was introduced using the Affinity Propagation (AP) clustering algorithm. The method involves creating appliance consumption templates and a time-segmented state probability (TSSP) matrix for each appliance to improve load disaggregation accuracy and speed. Tested on the AMPds2 database, the method showed a high success rate in load state identification and accuracy in load decomposition.

Additionally. Ref. [27] proposed a NILM approach integrating Appliance Usage Patterns (AUPs) for improved active load identification and prediction. The method starts with learning AUPs using NILM algorithms based on spectral decomposition, adjusting the a priori probabilities through a fuzzy system. The effectiveness of this approach was demonstrated on standard databases, showing notable improvements in active load estimation and forecasting total active energy consumption with promising results.

This section has explored various types of differential tariffs (RTP, ToU, CPP) and related research highlighting their integration and response to proposed tariffs. Some studies showcase an interaction between the designers of differential tariffs and the decision-makers’ choice to adopt them. Notably, certain studies designing differential tariffs, by analyzing past energy prices and demand/consumption volume, utilize statistical tools, stochastic mathematical models, and in some cases, machine learning techniques. However, no research was found that utilizes demand variability to determine the time intervals for setting differential tariffs. Moreover, the estimation of flexibility was further explored through load monitoring, distinguishing between intrusive and non-intrusive methods. These approaches employ statistical and computational (machine learning) mathematical tools. A notable limitation in these studies is the requirement for significant data volume, meaning user consumption data in minute intervals, from 5 to 20 min, which could complicate processing and increase the complexity of the methodology due to the large amount of information. A privacy concern identified is the potential exposure of users’ appliance usage and preferences, which raises questions about user consent.

4. Methodology

After reviewing the literature, this study embarks on an unexplored path by primarily focusing on the flexibility of electricity demand as a key variable to estimate tariff intervals. Proposing to assess user flexibility based on their consumption variability, this approach diverges from traditional load monitoring techniques, which often raise privacy concerns by revealing specific appliance usage in homes. The developed methodology evaluates electric energy demand flexibility, particularly using the coefficient of variation, which measures consumption variability as the standard deviation divided by the mean of hourly consumption. This innovative method seeks to identify optimal time intervals for implementing a differentiated Time-of-Use (ToU) tariff structure, considering various exogenous variables such as consumers’ socioeconomic level, climatic conditions, altitude, types of users, and geographical location.

The proposed methodology of this study is articulated in an eight-phase process designed to find time intervals where differential pricing can be established, as depicted in Figure 1. Initially, data standardization and amalgamation are conducted to streamline the segmentation by users, enhancing data processing efficiency. Subsequent phases involve calculating consumption variability through the coefficient of variation for each user. This is followed by the application of clustering techniques to identify representative user groups. The most-promising clusters are then scrutinized to select users with the highest potential for adopting a differential tariff scheme. Clusters exhibiting low consumption variability are excluded from the analysis. Finally, clustering and analysis methods are reapplied to ultimately determine appropriate rate ranges. Each of the phases is explained below.

4.1. Data Combination and Standardization

The initial phase of the proposed methodology adheres to the Extraction, Transformation, and Loading (“ETL”) framework, involving the meticulous collection, cleansing, and structuring of hourly electrical consumption data from smart meters. This “Extraction” phase includes the removal of noise and the correction of incomplete or inconsistent records. Additionally, integration with relevant databases—such as socioeconomic levels, climate, and location—occurs in the “Transformation” phase to enable a comprehensive analysis. Transformation techniques like one-hot encoding for non-ordinal categorical variables and numerical scaling for ordinal ones prepare the data for the subsequent “Loading” stage and subsequent optimization in machine learning models, as per the ETL best practices [28,29]. This process ensures data quality and utility for further analysis, depicted in Figure 2.

4.2. Consumption Segmentation by Customer

In this phase, consumption data for each user are aggregated into individual files, resulting in a single file per user that encompasses all available records. This method facilitates efficient data management, substantially optimizing processing time in subsequent stages by obviating the need to traverse the entire database for information on a specific user. Moreover, a rigorous selection of consumption profiles is conducted, excluding users whose data do not represent at least one full year, equating to a minimum of 8760 hourly consumption readings. This refinement ensures that the consumption variability can be accurately assessed for each consumer included in the analysis. This procedure, as depicted in Figure 3, processes each of the files through a cluster of virtual machines (VMs).

Figure 3. Consumption segmentation by customer.

Figure 3. Consumption segmentation by customer.

4.3. Calculation of the Coefficient of Variation

In this phase of the study, the coefficient of variation for each user is computed on an hourly basis, spanning from the first to the twenty-third hour of the day, distinguishing between weekdays, Saturdays, and Sundays or holidays, as depicted in Figure 4. It is important to note that, at this stage, the coefficient calculation is conducted using only electrical consumption data, without considering external variables such as consumers’ socioeconomic level, weather conditions, altitude, types of users, or geographical location.

The coefficient of variation is defined as the ratio of the standard deviation to the mean of the hourly electrical consumption, normalizing consumption variability across different users. By standardizing consumption variability, equitable comparisons between users with diverse consumption patterns are enabled [30]. Analyzing this coefficient is vital for identifying energy usage trends, which is key to determining optimal times for applying differential tariffs, ensuring that the tariff structure aligns with the inherent fluctuations in electrical energy consumption.

Figure 4. Calculation of the coefficient of variation.

Figure 4. Calculation of the coefficient of variation.

4.4. Clustering by the Coefficient of Variation

The fourth stage of the methodology is devoted to the categorization of consumers based on the variations in their electrical energy consumption, specifically their CV values for each hour of the day. To this end, various clustering techniques were assessed, including k-means, k-medoids, hierarchical clustering, and DBSCAN. Each technique was selected based on the specific characteristics of the dataset, such as type, distribution, size, the presence of outliers, and clustering preferences, as outlined by [31]. An in-depth analysis of the dataset in this study indicated that the k-means method is most-suited for this context. Utilizing k-means facilitates the identification of clusters based on hourly consumption variability, represented by coefficients of variation from $C{V}_0$ to $C{V}_{23}$, covering all 24 h. The elbow method and the silhouette coefficient were employed to determine the optimal number of clusters.

As a result of this process, illustrated in Figure 5, various user conglomerates were identified. These conglomerates reflect distinct electric consumption profiles differentiated for each type of day: weekday, Saturday, and Sunday/holiday.

Figure 5. Clustering by the coefficient of variation.

Figure 5. Clustering by the coefficient of variation.

4.5. Generation of Association Rules

The current phase aims to investigate how exogenous variables such as socioeconomic status, geographical location, and climatic conditions influence the variability of electrical energy consumption. Building upon the analysis by [32], which identified correlations between household characteristics and energy consumption patterns, this stage seeks to establish relationships between these characteristics and consumption variability. The a priori algorithm was employed, focusing on cluster labels determined by the coefficient of variation and household features like socioeconomic level, city, and local climate. The intricacies of this analysis are illustrated in Figure 6.

As a result, detailed rules were generated that elucidated how exogenous variables impact the variability of energy consumption. This insight is invaluable for crafting public policies and implementing differentiated tariff systems that effectively reflect the variability and specific needs of users.

Figure 6. Generation of association rules and classification.

Figure 6. Generation of association rules and classification.

4.6. Selection of Clusters and Analysis of Time Intervals

From the results derived from the clustering process, those clusters that stand out for their magnitude and the density of users contained in them were identified and prioritized, taking as the main reference the variability in electric energy consumption. Subsequently, time intervals were analyzed, where significant variability in user consumption was perceived. These intervals are essential, as they suggest opportunities for the application of differential tariffs, guiding electric supply entities towards billing strategies more suited to the consumption habits of their customers.

4.7. Removal of Clusters with Low Variability

This stage of the methodology aims to identify and discard those clusters that exhibit low variability in their electricity consumption. This filtering process is essential for directing the study towards users with high variability consumption patterns. To determine the relevance of each cluster, an analysis of the coefficient of variation was carried out. Only those groups with an average variability above 70% were retained; otherwise, they were excluded. This decision was based on the premise that users with low variability in their electricity consumption are unlikely to benefit from a differential tariff scheme due to their limited flexibility in consumption. Therefore, the study focused on users susceptible to adapting to such a scheme. With the adjusted dataset, the stages of grouping by the coefficient of variation, the generation of association rules and classification, and the selection and analysis of time intervals were repeated, now focusing on users with marked variability in their electricity consumption.

4.8. Determination of Tariff Intervals

The final phase involved the concrete definition of the time intervals where the differentiated tariffs will be applied. It is imperative that this phase be adaptable according to the specificities of each electric company and its respective market. The selection of time slots should seek a balance that benefits both the electric companies and the users, based on the available energy sources and other factors pertinent to the implementation.

5. Results

The methodology outlined in the previous section was applied to a dataset provided by a network operator in Colombia, under a confidentiality agreement that protects their identity. This dataset spans from the year 2018 to 2021, comprising 2,470,092,041 records, and reflects the hourly electricity consumption of 73,794 users. With a size of 157 GB, the repository provides a detailed overview of the fluctuations and trends in energy demand across different hours and years.

To analyze user behavior, it was decided to apply the proposed methodology to three different categories of days: weekdays, Saturdays, and Sundays/holidays. This approach affords a deeper understanding of the variations in energy consumption on these diverse types of days. In the following sub-sections, the results obtained for each of these categories will be presented, adhering to the stages described in the methodology.

5.1. Data Combination and Standardization

The initial dataset supplied by the network operator was in csv format. These were transformed into files with a .parquet extension, taking advantage of the inherent benefits of the Parquet format in managing large volumes of information. Due to its columnar design, Parquet optimizes compression and facilitates rapid queries by reading only the columns required for a specific query, thereby minimizing input/output operations. Before proceeding with the analysis, several filters were established to ensure data integrity. These filters ensured the exclusion of inconsistent consumption records, whether due to missing data, anomalies, or other similar issues.

Subsequently, this database was enriched by incorporating additional user information, such as user type, voltage level, socioeconomic level, city, and climate. This resulted in an expanded database that offers a more-holistic view of the consumers.

5.2. Consumption Segmentation by Customer

During this stage, a consolidation of the electrical consumption data was achieved. The information corresponding to each consumer was stored in individual files, improving the efficiency of subsequent analyses. This procedure eliminates the need to traverse the entire database to access the records of a specific user. In addition, a selection threshold was established, preserving only those users who had at least 8760 hourly consumption records, equivalent to a year of measurements. This criterion reduced the sample from 73,794 initial users to 44,574, 44,529, and 44,516 for each type of day. Such an approach allowed a significant reduction in the storage requirements, from 19 to 4.3 GB, optimizing not only information management, but also the scalability and utilization of computational resources for future analyses. Figure 7 illustrates the flow of records from the previous and current stages.

5.3. Calculation of the Coefficient of Variation

In this stage of the analysis, the focus was on calculating the CV for each hour of the day, corresponding to those users with more than one year of electrical consumption records. As detailed in Figure 4, this coefficient is the ratio between the standard deviation and the mean consumption, providing an index of relative variability used to compare different energy consumption patterns.

To obtain the CV, the calculation was carried out for each hour of the day, starting with all the user’s consumption records in the first hour. The mean and standard deviation were computed, and these values were used to calculate the coefficient. This process was repeated sequentially for the 24 h, generating a dataset where each column represents the CV of a specific hour of the day.

The analysis was tailored to three distinct categories of days: weekdays, Saturdays, and Sundays or holidays, leading to the creation of three separate datasets. These datasets comprised 44,574, 44,529, and 44,516 entries, respectively, corresponding to the number of users analyzed for each category. This methodical approach not only streamlined the data-management process, but also achieved a substantial reduction in storage requirements. The original dataset size of 4.3 GB was efficiently condensed to an aggregated volume of merely 85.6 MB.

5.4. Removal of Clusters with Low Variability

The results of this phase are shown first due to a key finding that emerged during the “formation of groups by coefficient of variation (CV)”, “generation of association rules and classification”, and “selection of clusters and analysis of time intervals”. A group of users within the clusters with low variability in their electrical consumption was identified.

From the analysis of Figure 8, Figure 9 and Figure 10, and Table 1, Table 2 and Table 3, it can be concluded that, regardless of the day type, there is a common pattern: each category includes a cluster with low variability, specifically the clusters labeled as 0, which show an average coefficient of variation close to 70%. These clusters not only include the largest number of users on all three types of days, but also encompass those with the least variability in their consumption. According to the previously described methodology, users belonging to these low-variability clusters (clusters labeled as “0”) were excluded from subsequent analyses. Our focus is on generating hourly tariffs adapted to users with greater flexibility in their electrical consumption.

In another aspect, to determine which exogenous variables influence the variability of electrical consumption, the a priori association rule algorithm was applied. The results, presented in Table 4, highlight the rules with the highest support and confidence. Notably, cluster 0—the one with the lowest coefficient of variation—is featured as a consequent in all these rules. This indicates that a significant proportion of users with low variability in their consumption belong to lower and middle socioeconomic level (specifically, level 2 and 3) and are primarily residential users. This group of users in Colombia is usually composed of working-class individuals, generally characterized by having fixed work schedules.

• Weekday

Table 1. Number of users per cluster for the weekday type.

Cluster	0	1	2	3	4	5
Number of Users	32,826	9762	253	1220	367	146
Average $C{V}_{0-23}\left(\frac{Wh}{Wh}\right)\%$	70.17	152.37	878.46	415.48	866.97	1811.93

Table 1. Number of users per cluster for the weekday type.

Cluster	0	1	2	3	4	5
Number of Users	32,826	9762	253	1220	367	146
Average $C{V}_{0-23}\left(\frac{Wh}{Wh}\right)\%$	70.17	152.37	878.46	415.48	866.97	1811.93

Figure 8. Average coefficient of variation of electrical energy consumption for the weekday type.

Figure 8. Average coefficient of variation of electrical energy consumption for the weekday type.

• Saturday

Table 2. Number of users per cluster for the Saturday type.

Cluster	0	1	2	3	4	5
Number of Users	26,087	14,637	315	2565	673	252
Average $C{V}_{0-23}\left(\frac{Wh}{Wh}\right)\%$	63.71	119.45	466.33	233.78	467.18	832.06

Table 2. Number of users per cluster for the Saturday type.

Cluster	0	1	2	3	4	5
Number of Users	26,087	14,637	315	2565	673	252
Average $C{V}_{0-23}\left(\frac{Wh}{Wh}\right)\%$	63.71	119.45	466.33	233.78	467.18	832.06

Figure 9. Average coefficient of variation of electrical energy consumption for the Saturday type.

Figure 9. Average coefficient of variation of electrical energy consumption for the Saturday type.

• Sunday/Holiday

Table 3. Number of users per cluster for the Sunday/holiday type.

Cluster	0	1	2	3	4	5
Number of Users	26,866	14,179	220	2335	654	262
Average $C{V}_{0-23}\left(\frac{Wh}{Wh}\right)\%$	65.08	123.54	599.97	257.31	515.71	954.16

Table 3. Number of users per cluster for the Sunday/holiday type.

Cluster	0	1	2	3	4	5
Number of Users	26,866	14,179	220	2335	654	262
Average $C{V}_{0-23}\left(\frac{Wh}{Wh}\right)\%$	65.08	123.54	599.97	257.31	515.71	954.16

Figure 10. Average standard deviations of electrical energy consumption for the Sunday/holiday type.

Figure 10. Average standard deviations of electrical energy consumption for the Sunday/holiday type.

Table 4. Association rules by day type for all user.

Day Type	Antecedent	Consequent	Support	Confidence
Weekday	“socio_eco_level_3”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.40	0.77
	“socio_eco_level_2”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.15	0.69
Saturday	“socio_eco_level_3”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.32	0.62
	“socio_eco_level_2”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.11	0.50
Sunday	“socio_eco_level_3”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.33	0.64
	“socio_eco_level_2”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.11	0.51

Table 4. Association rules by day type for all user.

Day Type	Antecedent	Consequent	Support	Confidence
Weekday	“socio_eco_level_3”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.40	0.77
	“socio_eco_level_2”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.15	0.69
Saturday	“socio_eco_level_3”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.32	0.62
	“socio_eco_level_2”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.11	0.50
Sunday	“socio_eco_level_3”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.33	0.64
	“socio_eco_level_2”, “cold_climate”, “regulated_residential_user”	cluster_label_0	0.11	0.51

5.5. Clustering by the Coefficient of Variation

From the filtered data (excluding users with low variability), the k-means clustering technique was reapplied, resulting in the distinction of six different clusters for each day category. The elbow method and the silhouette coefficient were employed to determine the optimal number of clusters.

Next, graphs illustrating the average coefficient of variation corresponding to each cluster are shown, classified according to the type of day. These graphs are accompanied by detailed information about the characteristics of each group.

• Weekday

This subsection details the findings from clustering the variability of electrical consumption during weekday.

The analysis of Figure 11 and Table 5 and Table 6 reveals interesting patterns in the distribution of users among the clusters. It is notable that clusters labeled as “0” and “1” accommodate the largest number of users. In particular, cluster “0” is characterized by a coefficient of variation that remains constant throughout the day, averaging around 150%. On the other hand, the behavior in cluster “1” differs significantly: it shows pronounced variability during the morning hours, from dawn until approximately 10 h. Subsequently, its coefficient of variation stabilizes, remaining constant until 20 h, when a new increase in variability is observed.

Table 5. Cluster information for weekday by socioeconomic level.

Cluster	Number of Users	Socioeconomic Level
		0	1	2	3	4	5	6
0	9604	734	818	2673	4539	561	214	65
1	1243	222	79	263	532	123	12	12
2	348	27	23	57	177	58	5	1
3	305	208	5	34	55	1	0	2
4	127	84	5	9	28	1	0	0
5	121	24	5	26	57	7	0	2

Table 5. Cluster information for weekday by socioeconomic level.

Cluster	Number of Users	Socioeconomic Level
		0	1	2	3	4	5	6
0	9604	734	818	2673	4539	561	214	65
1	1243	222	79	263	532	123	12	12
2	348	27	23	57	177	58	5	1
3	305	208	5	34	55	1	0	2
4	127	84	5	9	28	1	0	0
5	121	24	5	26	57	7	0	2

Table 6. Cluster information for weekday by user type.

Cluster	User Type
	Official	Residential	Industrial	Commercial
0	4	8870	68	662
1	2	1021	22	198
2	1	321	1	25
3	0	97	26	182
4	0	43	3	81
5	1	97	0	23

Table 6. Cluster information for weekday by user type.

Cluster	User Type
	Official	Residential	Industrial	Commercial
0	4	8870	68	662
1	2	1021	22	198
2	1	321	1	25
3	0	97	26	182
4	0	43	3	81
5	1	97	0	23

Figure 11. Average coefficient of variation of electrical energy consumption for weekday.

Figure 11. Average coefficient of variation of electrical energy consumption for weekday.

• Saturday

The results of the clustering applied to the variability of electrical energy consumption on Saturday are presented in this subsection.

The analysis of Figure 12 and Table 7 and Table 8 reveals that the majority of users are concentrated in clusters labeled “0” and “1”. In cluster “0”, a coefficient of variation that remains relatively constant is observed, although with a slight elevation in the early morning hours, specifically around 4 hours, and a slight decrease in the afternoon, near 6 hours, averaging around 119%. On the other hand, cluster “1” also shows a fairly stable coefficient of variation throughout the day, but with a notably higher average, approximately 226%.

Table 7. Cluster information for Saturday by socioeconomic level.

Cluster	Number of Users	Socioeconomic Level
		0	1	2	3	4	5	6
0	14,455	643	1186	4046	7330	884	299	67
1	2666	482	177	613	1110	205	59	20
2	276	212	5	18	41	0	0	0
3	161	20	10	17	87	25	1	1
4	631	81	46	128	301	59	9	7
5	253	43	16	55	112	21	0	6

Table 7. Cluster information for Saturday by socioeconomic level.

Cluster	Number of Users	Socioeconomic Level
		0	1	2	3	4	5	6
0	14,455	643	1186	4046	7330	884	299	67
1	2666	482	177	613	1110	205	59	20
2	276	212	5	18	41	0	0	0
3	161	20	10	17	87	25	1	1
4	631	81	46	128	301	59	9	7
5	253	43	16	55	112	21	0	6

Table 8. Cluster information for Saturday by user type.

Cluster	User Type
	Official	Residential	Industrial	Commercial
0	5	13,812	54	584
1	2	2184	54	426
2	0	64	26	186
3	0	141	0	20
4	0	550	8	73
5	1	210	2	40

Table 8. Cluster information for Saturday by user type.

Cluster	User Type
	Official	Residential	Industrial	Commercial
0	5	13,812	54	584
1	2	2184	54	426
2	0	64	26	186
3	0	141	0	20
4	0	550	8	73
5	1	210	2	40

Figure 12. Average coefficient of variation of electrical energy consumption for Saturday.

Figure 12. Average coefficient of variation of electrical energy consumption for Saturday.

• Sunday/Holiday

In this subsection, the results derived from the application of clustering techniques to the variability of electrical energy consumption on days categorized as Sunday/holiday are presented.

The examination of Figure 13 and Table 9 and Table 10 reveals that the majority of users are primarily concentrated in clusters “0” and “2”. It is notable that cluster “0”, which houses the largest number of users, exhibits a coefficient of variation that remains constant throughout the day, hovering around 123%. In contrast, the coefficient of variation for cluster 2 fluctuates around a value close to 253%. It is important to highlight that the consumption patterns observed on Saturdays and Sundays show a notable similarity.

Table 9. Cluster information for Sunday/holiday by socioeconomic level.

Cluster	Number of Users	Socioeconomic Level
		0	1	2	3	4	5	6
0	14,114	691	1173	4066	6947	865	298	74
1	154	40	8	15	65	26	0	0
2	2336	469	145	521	962	182	45	12
3	612	154	40	108	257	40	6	7
4	243	69	16	41	96	16	0	5
5	191	142	2	19	27	1	0	0

Table 9. Cluster information for Sunday/holiday by socioeconomic level.

Cluster	Number of Users	Socioeconomic Level
		0	1	2	3	4	5	6
0	14,114	691	1173	4066	6947	865	298	74
1	154	40	8	15	65	26	0	0
2	2336	469	145	521	962	182	45	12
3	612	154	40	108	257	40	6	7
4	243	69	16	41	96	16	0	5
5	191	142	2	19	27	1	0	0

Table 10. Cluster information for Sunday/holiday by user type.

Cluster	User Type
	Official	Residential	Industrial	Commercial
0	5	13,423	61	625
1	0	114	1	39
2	3	1867	46	420
3	0	458	20	134
4	1	174	7	61
5	0	49	14	128

Table 10. Cluster information for Sunday/holiday by user type.

Cluster	User Type
	Official	Residential	Industrial	Commercial
0	5	13,423	61	625
1	0	114	1	39
2	3	1867	46	420
3	0	458	20	134
4	1	174	7	61
5	0	49	14	128

Figure 13. Average standard deviation of electrical energy consumption for Sunday/holiday.

Figure 13. Average standard deviation of electrical energy consumption for Sunday/holiday.

5.6. Selection of Clusters and Analysis of Time Intervals

Among the obtained results, it stands out that the majority of users are grouped into clusters labeled as “0”, followed by those identified with the label “1” for weekday and Saturday types. For Sunday and holiday types, the clusters labeled as “0” and “2” prevail. To further understand the behavior of these groups, which house a larger number of users, a box plot was developed for each of these clusters. To compare the coefficient of variation for different types of days and clusters, it was decided to graph an additional line in each diagram, which has a value of 1.2-times the average electric consumption relative to the standard deviation of the electrical energy consumption of users. This graphical representation facilitates the identification of patterns and anomalies in consumption variability, providing a visual reference to assess the data dispersion in relation to the mean.

• Weekday

The analysis of the box plot in Figure 14 for cluster number “0” reveals an interesting dynamic in consumption variability. A notable increase in variation is observed between 5 and 7 h. Subsequently, this variability decreases, remaining stable until approximately 18 h, at which point, it begins to steadily decrease. Despite these changes, the average coefficient of variation remains above 150%.

On the other hand, Figure 15 shows that, in cluster number “1”, the variability is quite constant during the morning, extending until noon. A slight increase in variability around noon was detected, followed by a decrease during the afternoon. From 18 h, a slight increase is observed, which intensifies between 21 and 23 h. It is important to note that the average variation in cluster “1” is significantly higher compared to that of cluster “0”.

Figure 14. Box plot for cluster 0 for the weekday type.

Figure 14. Box plot for cluster 0 for the weekday type.

Figure 15. Box plot for cluster 1 for the weekday type.

Figure 15. Box plot for cluster 1 for the weekday type.

• Saturday

For clusters “0” and “1”, which encompass the largest number of users, a detailed analysis was carried out using box plots, the results of which are presented in Figure 16 and Figure 17, respectively. The analysis of Figure 16, corresponding to cluster “0”, shows that the coefficient of variation experiences a notable increase between 6 and 7 h, followed by a slight decrease and a period of stability until approximately 17 h, when a slight decreasing trend is observed during the night. In contrast, Figure 17, representing cluster “1”, indicates that the coefficient of variation remains relatively constant throughout the day, albeit with slight increases in the hours between 6 and 9 h.

• Sunday/Holiday

Figure 18 and Figure 19 present the box plots for clusters “0” and “2”, which are those with the largest number of users. For cluster “0”, we observed an increase in the coefficient of variation from 6 until about 8 h, followed by a phase of stability until noon. During the afternoon hours, specifically from 12 to 15 h, a slight peak in variability is detected, which then gradually decreases until 17 h. On the other hand, Figure 19, corresponding to cluster “2”, shows a different pattern. Here, the coefficient of variation remains constant, with a slight increase between 6 and 8 hours. Subsequently, it remains stable until the night, where a significant increase in variability is observed from 19 to 23 h.

5.7. Generation of Association Rules and Classification

Table 11. Association rules by day type, for the filtered data.

Day Type	Antecedent	Consequent	Support	Confidence
Weekday	“cold_climate”, “socio_eco_level_3”, “regulated_residential_user”	cluster_label_0	0.38	0.85
	“cold_climate”, “socio_eco_level_2”, “regulated_residential_user”	cluster_label_0	0.22	0.87
Saturday	“cold_climate”, “socio_eco_level_3”, “regulated_residential_user”	cluster_label_0	0.39	0.82
	“cold_climate”, “socio_eco_level_2”, “regulated_residential_user”	cluster_label_0	0.21	0.83
Sunday	“cold_climate”, “socio_eco_level_3”, “regulated_residential_user”	cluster_label_0	0.39	0.84
	“cold_climate”, “socio_eco_level_2”, “regulated_residential_user”	cluster_label_0	0.23	0.85

illustrates the association rules generated using the a priori algorithm, applied to different types of days from the filtered dataset, which excludes users with low variability removed in previous stages of the methodology. Revealingly, all the resulting rules indicate that the users belong to cluster “0”, which includes the majority of users for each type of day. Predominantly, these users are in socioeconomic strata 2 and 3 and are the residential type. When contrasting these rules with those obtained before removing users with higher variability (Table 4), where the antecedents were also residential users from socioeconomic level 2 and 3, it was deduced that the variability of consumption or the coefficient of variation of electrical energy is not directly related to exogenous variables such as climate, socioeconomic level, or user type. This suggests that users with the mentioned characteristics can present both low and high variability in their electrical energy consumption.

5.8. Determination of Time-of-Use Tariff Intervals

The analysis carried out in the previous stages of this methodology revealed that certain time intervals exhibit notable flexibility in energy consumption. This conclusion is based on the box plots of Figure 14, Figure 15, Figure 16, Figure 17, Figure 18 and Figure 19, which illustrate the coefficient of variation for the clusters with the largest number of users. During weekdays and Saturdays, the most-representative cluster is 1, while on Sundays and holidays, it is cluster 2. Both clusters present an average coefficient of variation exceeding 120%, suggesting a considerable ability of these users to adapt to various time-based tariff schemes. In contrast, users belonging to cluster 0, for all types of days and at certain times, record a coefficient of variation below 120%:

• During weekdays, it is observed that the coefficient of variation from 5 to 18 h exceeds 120%, with a notable peak between 5 and 7 h.
• On Saturdays, the coefficient of variation remains above 120% from 6 to 17 h, experiencing a peak between 6 and 7 h.
• On Sundays, the coefficient of variation exceeds 120% from 6 to 17 h, with a significant increase between 6 and 9 h and a slight rise from 19 to 23 h.

However, it is crucial to integrate this information with the financial and generation data of the network operator. This combination will allow the determination of time intervals that not only promote savings and energy efficiency for consumers, but also optimize the operational and economic benefits for electric companies. The implementation of differentiated tariffs in these time intervals could translate into more-balanced and -sustainable energy consumption, as well as more-efficient operation of the electrical grid.

6. Discussion and Limitations

This study successfully developed and validated a methodology to identify optimal time intervals for implementing differential tariff schemes, based on the variability of electrical energy consumption. The phases that required the most time and computational resources were “consumption segmentation by customer” and “calculation of the coefficient of variation”. Processing the data through these two stages reduced the dataset size, facilitating subsequent analysis and processing. By utilizing hourly data instead of minute-by-minute data, this methodology significantly reduced the data volume, thereby simplifying the complexity. The proposed methodology can be implemented in other settings with hourly energy consumption data from a group of electric system users, to identify time intervals where users have the highest potential to implement differential tariffs.

While studies such as [32] have found a direct relationship between external variables (consumers’ socioeconomic level, climatic conditions, altitude, types of users, and geographical location) and energy consumption, this study did not find a direct correlation between external variables and consumption variability. This may be due to an imbalance in the dataset, which is predominantly from users in one of Colombia’s major cities with a working-class population. Future research should employ a balanced database to verify a relationship between these two variables.

This study did not consider energy pricing as a variable in determining tariff intervals. Pricing is a crucial factor in the decision-making processes of electric companies and households. Therefore, it is recommended that the methodology proposed in this study be integrated with others that consider energy pricing.

When a tariff interval is proposed, the next step is to assess its effectiveness and the percentage of users willing to alter their consumption patterns. Precisely estimating the number of users who would adopt a differential tariff remains challenging [33,34]. User willingness to adopt a differential tariff depends on multiple variables, with no clear consensus on accurate determination [33]. Acceptance rates could vary between 1% and 43% [33]. Some advocate for a mixed approach that combines social and economic analysis techniques to estimate user adoption rates [34]. Others suggest direct surveys to gauge users’ willingness to accept such tariffs [35,36]. It is advisable that the time intervals identified in studies be confirmed through pilot testing with users or through direct surveys to validate the effectiveness of a differential tariff scheme.

7. Conclusions

The case study identified specific intervals for different types of days: on weekdays from 5 to 18 h with a peak between 5 and 7 h; on Saturdays from 6 to 17 h with a peak between 6 and 7 h; and on Sundays and holidays from 6 to 17 h with peaks between 6 and 9 h. The early morning hours, especially from 6 to 9, are most-suitable for implementing Time-of-Use tariffs due to the observed high variability peaks in clusters with the most users.

Sundays and holidays showed consumption patterns similar to Saturdays, suggesting the potential simplification of categorization into just two types of days: weekdays and non-working days.

Correlating energy consumption variability with exogenous variables such as socioeconomic level, user type, climate, and city did not reveal a direct relationship. Users from different socioeconomic levels and climates, especially in Bogotá, exhibited both low and high variabilities. This could be due to a dataset imbalance skewed towards the working-class residents of Bogotá. Therefore, applying this methodology to more-balanced datasets that include a diversity of user profiles for more-robust validation is recommended.

It is important to note that, between 60 and 73% of the users, depending on the day type, showed low variability, with a coefficient of variation close to 70%. These users were excluded from the analysis of differential tariff schemes due to their potentially low flexibility in consumption patterns.

This methodology is intended as a complementary tool in the formulation of differential tariff schemes. For effective implementation, additional information must be considered, including financial and generation data from network operators. The aim is not only to promote savings and improve efficiency for consumers, but also to optimize the operational management of electric power companies.

8. Future Work

For future research, it is proposed to explore the relationship between the flexibility and variability of electrical energy consumption. This will require conducting tests with real users under differential tariffs to study their responses. Additionally, it is suggested to adjust the methodology to integrate both the variability and the total volume of energy consumption and prices. This would allow for more-precise identification of temporal intervals and the application of association rules to discover possible links between actual consumption and external variables. Another adjustment would be to modify the methodology so that classification is performed independently of the type of day.

Given that the present study did not establish a conclusive relationship between the variability of electrical energy consumption and the mentioned exogenous variables, a future line of research could involve exploring different exogenous variables. This would help to determine whether the lack of correlation observed is specific to the dataset used or if it is a more-general trend.