C6EnPLS: A High-Performance Computing Job Dataset for the Analysis of Linear Solvers’ Power Consumption

Abstract

In recent decades, driven by global efforts towards sustainability, the priorities of HPC facilities have changed to include maximising energy efficiency besides computing performance. In this regard, a crucial open question is how to accurately predict the contribution of each parallel job to the system’s energy consumption. Accurate estimations in this sense could offer an initial insight into the overall power requirements of the system, and provide meaningful information for, e.g., power-aware scheduling, load balancing, infrastructure design, etc. While ML-based attempts employing large training datasets of past executions may suffer from the high variability of HPC workloads, a more specific knowledge of the nature of the jobs can improve prediction accuracy. In this work, we restrict our attention to the rather pervasive task of linear system resolution. We propose a methodology to build a large dataset of runs (including the measurements coming from physical sensors deployed on a large HPC cluster), and we report a statistical analysis and preliminary evaluation of the efficacy of the obtained dataset when employed to train well-established ML methods aiming to predict the energy footprint of specific software.

1. Introduction

The computing capabilities of high-performance computing (HPC) systems have impressively increased in recent decades. This remarkable growth inevitably comes with a dramatic increment in power consumption [1]. Therefore, just like sustainability is becoming a key challenge in many human activities, maximising energy efficiency—besides performance—is becoming a priority for HPC facilities too.

Several works [2] focus on hardware efficiency by proposing technologies to reduce the environmental footprints of microprocessors, chips, and devices. Some recent attempts [3,4,5,6] focus instead on the software level by studying predictive models to assess the power required by parallel algorithms. The latter set of works stems from the observation that each execution of a job on an HPC infrastructure requires the allocation of specific resources, thus contributing to the system’s total energy consumption. The ambitious goal is therefore to predict the energy consumption of an HPC system based on the energy consumed by individual jobs: a feature that could be particularly useful for many HPC tasks (e.g., energy-aware workload management, load balancing, infrastructure design, etc.). For example, if the HPC scheduler could rely on accurate prediction of the job’s energy profiles over time, it could take into account this information to allocate the jobs on the physical machines in a way that reduces the energy consumed by the whole infrastructure. More futuristics applications include energy consumption optimisations at compile time and programming tools able to suggest more energy saving ways to write the same algorithm (just as some developing tools are already able to understand the intent of the programmer as they write the code and suggest more computationally efficient solutions).

Software-level solutions can be divided into two categories: some works propose theoretical energy models [7,8,9,10] to mathematically connect the algorithm’s steps with its energy footprint, whereas other techniques [11,12,13,14,15,16,17] try to predict the job’s footprint based on historical records, usually by employing machine learning (ML) techniques. In this regard, a necessary condition to obtain accurate models is the availability of large sets of examples of past jobs’ executions. However, the high variability in the characteristics of the workloads submitted to an HPC infrastructure (e.g., the kind of the algorithm, its CPU and RAM requirements, number of accesses to disk, inter-process communication scheme, etc.) can hinder the accuracy of the predictions, even if numerous historical examples are made available [12,17,18]. In general, fine-grained knowledge about the nature of the jobs can be crucial to improve the prediction accuracy of ML techniques.

In this work, we propose a first step towards this direction by restricting the attention to a particular kind of HPC workload, the parallel resolution of large-scale linear systems, and providing the scientific community with an extensive dataset of job runs and corresponding measured performance. The choice to focus on linear solvers stems from the observation that linear systems are widely used in the scientific field and often constitute the most suitable mathematical model to represent human activities, especially in technological and industrial contexts. Furthermore, as the complexity (and duration) of the resolution process is strictly connected to the matrix rank, the workload generated when solving large systems can have a significant impact on the power consumption of the whole application.

Our analysis relies on highly accurate measurements of energy performance, obtained through the use of sensors directly installed on each node of an HPC cluster, thus overcoming the limitations [19,20] of on-chip energy monitoring frameworks such as Intel’s running average power limit (RAPL) [21].

The proposed dataset is particularly suitable for training ML techniques aiming to predict the energy consumption of HPC jobs. Nonetheless, the numerous and diverse sensor measures collected for each job, make it a good starting point for further analyses, possibly intertwining the hardware components’ utilisations with their power consumption.

The contributions of this work can be summarised as follows.

• We present a novel execution framework to automate job launches with specific constraints, in terms of both resources and scheduling.
• We describe an extensive dataset of job runs with several dimensional configurations created through the aforementioned framework and made available to the scientific community. Each job run is reported together with the information coming from a variety of sensors.
• We provide a first glance at the proposed data by analysing the distribution of energy-related targets with respect to other meaningful dataset dimensions.
• We present a preliminary experimental evaluation of the predictive capabilities of standard ML models trained based on the proposed dataset. In this regard, we underline that proposing innovative ML models to analyse the data is out of the scope of this work. The evaluation is therefore intended as a first test of the efficacy of the obtained dataset when employed to train well-established ML methods.

The paper is structured as follows. We begin by providing a review of the state-of-the-art of techniques to assess and contain the power consumption of HPC jobs (Section 2). Then, we describe our framework for automating job execution and data collection (Section 3). In Section 4, we provide a detailed breakdown of the collected dataset, delving into each specific piece of information gathered. Section 5 provides a statistical analysis of these data, while Section 6 evaluates the training of machine learning regression models given targets chosen from the entire dataset. The conclusion follows.

2. Related Work

Boosted by the need to reduce the ever-increasing environmental footprint of large datacenters, energy efficiency plays a role of growing importance in modern HPC systems [1]. Various research efforts have been devoted to this topic, tackling the problem at any architectural level, from hardware and infrastructure level to operating system, scheduling, and algorithm level.

The reduction in power consumption at the hardware level is often achieved through architectural changes and cooling methodologies [2,22]. Dynamic voltage and frequency scaling (DVFS) is one of the most used hardware-level techniques to reduce the power consumption of a processor [23,24]. Obviously, reducing the operating clock increases the duration of any running software, which may not only negatively affect the application performance [25,26,27,28], but also induce faults in the computation (making it more prone to radiation and thermal drifts) and, consequently, cause additional processing and power consumption [8,10,29]. Nonetheless, DVFS is a frequently employed energy-saving method, e.g., to improve job scheduling algorithms with power budgeting capabilities [30,31,32].

Higher-level measures to contain the power consumption of a system often rely on Intel’s RAPL [33,34], which offers a mechanism to access power consumption measures related to the system’s main hardware components. Although the reliability of these measures has been questioned in some works [19,20], RAPL is still widely employed to enforce power-aware job scheduling in HPC infrastructures [35,36] and an RAPL-alike interface is now also available for AMD’s architectures [37]. In our work, we avoid the debate about RAPL reliability by employing environmental and vendor-specific onboard sensors to measure the energy performance of the system.

Other approaches devoted to power consumption containment operate at the operating system- or job scheduling-level [4,38,39,40] by suggesting the “best” job execution order to minimise the energy requirement of the infrastructure—without any modification to hardware components nor nodes’ operational voltage/frequency. Concurrency throttling [12,41] can be classified as another system-level strategy to reduce energy consumption.

Some other attempts [3] operate at the software level and investigate the possibility of predicting the energy footprint of parallel algorithms from their characteristics (e.g., flops, memory occupation, communication scheme). Typically, these approaches can be classified into two classes.

The first class includes works [7,8,9,10,29] that propose mathematical models of power consumption based on the algorithm’s features. In particular, Choi et al. [8] develop a seminal model of the application’s power requirements mathematically combining the known features of the algorithm (i.e., operations, concurrency, and memory traffic) with those of the machine executing it (i.e., time and energy costs of each operation and each word of communication). Analogously to the approaches that could be tailored on top of the proposed dataset (which focuses just on linear solvers), Demmel et al. [42] restrict their attention to a particular kind of linear algebra task (i.e., matrix multiplication) and propose to relate energy with the algorithm’s number of flops, sent messages, and memory occupation. The work [9] proposes a simpler energy model relating the operational frequency of a multi-core machine with the algorithm’s features, and employs such a model to define an estimation methodology for the energy scalability of parallel algorithms. The work by Aply et al. [29] builds on top of Shatz and Wang [20] reliability model to design a scheduling policy that includes makespan and energy in the equation. Their idea is to allow the re-execution of some tasks in case of faults, while we propose to investigate the checkpointing and ABFT methods such as IMe.

The second class of works investigating the power consumption of parallel algorithms encompass ML techniques to predict the energy utilisation of each job from historical records of past executions [11,12,13,14,15,16,17,43]. In [12,13,14], the authors focused on typical supercomputer workloads and propose ML approaches, based on general user and application resource requests. As these features may not be enough to build a reliable predictor, Antici et al. [17] proposed to employ the natural language processing of the job’s launching options to refine the predictions of the energy consumption of a trained ML model. Hu et al. [15] designed an effective cluster service that exploits the historical information about the past executions of GPU-enabled deep learning algorithms in order to decrease the overall energy utilisation of a datacenter. See [16] for a comprehensive study on ML-based energy prediction models for HPC.

All these works (except [42], that restrict the attention to matrix multiplication) do not focus on specific algorithms but try to tackle the problem of energy prediction in a more general way, by considering any kind of job running on the HPC infrastructure, even if this can hinder the accuracy of the predictions [12,17,18]. In contrast to these approaches, our idea is to focus on a specific kind of algorithm, for which crucial features (such as the input parameters) are well known and can be included in the training set. Although we expect the models built on top of the proposed dataset to be generally more accurate in their prediction, the obvious drawback will stand precisely in their generality. Nonetheless, the knowledge of the application’s features before its execution (such as application’s tags identifying similar jobs [28,44,45], job submission information [17,44], or the kind of algorithm as in our approach) is a very important point for the accurate prediction of power utilisation in HPC systems. Our proposed dataset goes precisely in this direction by suggesting that the attention to a specific—albeit rather pervasive—task be restricted so that the analysis of the algorithm’s energy consumption can be conducted with a refined level of detail.

Other works related to our contribution are the workflow automation tools described in [46,47]. Both present interesting application-agnostic methodologies for benchmarking. While WA [47] is mainly designed for Android software and ARM architectures, JUBE [46] presents a more HPC-oriented approach that operates with Slurm and PBS schedulers.

3. Execution Framework

Jobs and measurements were carried out on ENEA’s CRESCO6 cluster [48], a modern HPC facility consisting of 434 physical nodes, each of which has two sockets of 24 cores with 2.10 GHz Intel(R) Xeon(R) Platinum 8160 processors and 192 GB RAM (i.e., a total of 20,832 cores and over 80 TB RAM). An Intel Omni-Path 100 Gb/s network interconnects the machines. More importantly, from the point of view of this work, each CRESCO6’s node is also equipped with a batch of sensors capable of making several types of measurements (e.g., power consumption, utilisation rate, temperature; each related to physical nodes, single CPUs, memory banks, cooling fans etc.) and various environmental sensors are placed in the server room to also provide information about the whole datacenter and its cooling system. See [49] for a detailed overview of the sensor infrastructure.

CRESCO6 is managed by the LSF scheduler (https://www.ibm.com/docs/en/spectrum-lsf/10.1.0?topic=lsf-session-scheduler accessed on 29 April 2025) and shared with many users. As shown in Figure 1, the system provides two login nodes on which job runs can be built and submitted. One node is designated as a monitoring node to display and retrieve the data measured by the sensors.

To generate a dataset for subsequent analysis, we performed 7200 job runs. Each job is uniquely characterised by the input parameter configurations of the considered solvers: Gaussian-elimination (as implemented in PDGESV and PSGESV routines of ScaLAPACK library [50]) and the inhibition method (IMe) (as parallelly implemented in the C-IMeFT procedure [51]). Both solvers are implemented as fault-tolerant MPI programs, but present key differences. In particular, since the ScaLAPACK routines are not fault-tolerant, this feature is achieved through checkpointing in our implementation. Differently, C-IMeFT employs the algorithm-based fault tolerance (ABFT) [52] and, therefore, it is intrinsically able to guarantee the computation from up to a configurable number of anywhere-located hard faults. See [51] for a detailed description and an evaluation of the two methods.

As regards the generation of the dataset, the manual management of the broad spectrum of possible configurations for the considered solvers would have been both impractical and inefficient. To address this challenge, we developed a dedicated framework with the primary objective of fully automating the process of generating and executing the combinatorial explosion of different job configurations. Figure 2 shows a diagram of how the submission and execution of a job take place. In the following, we provide a comprehensive exposition of the framework, underscoring its pivotal role in streamlining and optimising the process of execution and data collection.

As shown in Figure 2, the starting point of the designed framework is the Launch Step, which is responsible for the automatic generation of job submission commands. It encompasses the configurations for the various dimensions explored and the binding settings necessary to meet the resource requirements. Since the CRESCO6 cluster is used daily by many users, a simple launch of the devised jobs would have produced sensor measures highly influenced by the degree of infrastructure utilisation. To avoid this, the launch step is responsible for ensuring that the following requirements are met:

(R1)

No other user has to access/use the nodes employed by a job;

(R2)

The LSF job must use the minimum possible number of nodes;

(R3)

Jobs must be executed in sequence.

As each node consists of 48 cores, in order to meet the first requirement, the launch step always requests to LSF a number of cores equal to 48 multiplied by the desired number of nodes. The actual number of cores needed by the MPI algorithm will be defined in the MPI launch script, and if the algorithm uses fewer cores than specified, the unused processors will turn to the idle state but will not be released. In this way, we can test, for example, a job distributing its work to 48 ranks, in different settings: e.g., using all cores of one machine, using 24 cores of two different machines, or using 12 cores of four machines. Requirement R2 is needed because, by default, LSF aims to maximise the whole datacenter utilisation: if a physical node is partially used by other users, LFS may decide to assign some processes of a job to the remaining cores. Therefore, just requesting LSF a number of cores equal to 48 multiplied by the desired nodes (as stated to comply with R1) is not enough to actually guarantee that those 48 cores will belong to the same machine. The launch step is therefore responsible for specifying that each node must have 48 free cores in the LSF context of CRESCO6. Finally, the third requirement is needed to ensure the proper measurement of the job’s energy consumption through the communication mechanism with the node acquiring data from the sensors. To achieve this, the launch step relies on the dependency requirements between jobs. In particular, the specified requirement is like “ended(previous_job_name)”, indicating that the job at hand can only start after the previously submitted job has finished. The termination condition is valid independently of the termination status (error/success) of the predecessor job. This flexibility maximises the overall time efficiency of the benchmark workflow and helps to ensure continuity of operations by reducing possible delays due to individual failures.

Since LSF dynamically decides at runtime the allocation of jobs to physical nodes, we implemented a mechanism to communicate this association to the script that collects sensor data. To communicate this information to the monitoring node, options were introduced to allow a pre-execution phase to run on the first host assigned to the job by LSF. In detail, once a job is submitted, it enters the LSF queue until sufficient resources are available to satisfy the constraints specified in the submit command. When it is picked from the queue, a pre-execution script starts. The success of this script is a prerequisite for the actual execution of the job. In our context, the usefulness of the pre-execution script is related to how LSF allocates resources (i.e., nodes) based on constraint satisfaction. Since the allocated nodes may vary from job to job, the pre-execution script communicates the resource information to the measurement software. In order to limit the overhead caused by the monitoring task, our approach monitors only the nodes involved in the current job, rather than the entire system.

The third step in Figure 2 is the Execution Phase. Each machine in the CRESCO6 cluster is a Lenovo Think System SD530 equipped with vendor-specific sensors [49] that can measure various parameters, including energy consumption and node temperature. To obtain their data, during job execution, a command must be run on an isolated node in the CRESCO6 architecture (the monitoring node), which cannot be reached by user access nodes. Whenever this command is invoked, the instantaneous values sensed by the sensors on the CRESCO6’s node passed as the input parameter are returned. The collection of the measures is performed through multiple parallel instances of the same measurement script. These instances act as daemons and operate cyclically, each collecting data from a single node with configurable frequency, and populating a database. For an efficient and effective operation of the monitoring system, a locking mechanism ensures that only the necessary parallel instances are used, while the others remain idle. The parallel execution of multiple daemons allows increasing the sampling rate of the sensors’ measurements sufficiently to ensure the collection of an adequate number of measures for all executions, even for short jobs running on many cores.

At the end of all job runs, a post-execution phase takes place to stop the sampling activity of all active measurement daemons and save the collected relevant information. In particular, the monitoring data in the database are associated with the job identifier and saved in CSV format, while another file collects the job start and end timestamps in order to provide better specifications for filtering measurements. Other final summary information of the job run is saved by LSF in the job output file. A detailed account of all the information collected for each job is provided in the next section.

4. Dataset Structure

The CRESCO6 energy profiles of linear solvers (C6EnPLS) dataset, available on a GitHub repository [53], is divided into four blocks of information (corresponding to the four blue folders in Figure 2), providing a holistic overview of the executions: jobs specifications, jobs data, sensors measurements, and final results.

The jobs specifications block describes the input parameters of the routines, formed by the combinations of the input dimensions explored. The dimensions and values used are summarised in

Table 1. Description of the job specifications block fields and assigned values.

Dimension	Description	Values
Job name	Job identification name	-
Matrix size	Rank of the input matrix	5280, 10,560, 15,840, 21,120, 26,400, 31,680, 36,960, 42,240
Calculation processes	Number of processes dedicated exclusively to the calculation of the system’s solution	64, 100, 144, 256, 400, 484, 576, 768
Nodes	Number of employed physical nodes	[1,...,16]
algorithm	Considered linear solvers	IMe, ScaLAPACK
Precision	Numerical representation of real numbers	Single, double
Fault tolerance level	Number of faulty processes that can be handled	0 (no fault tolerance), 1, 2, 4, 8
Number of simulated faults	Number of faults to be simulated (and recovered)	0, maximum fault tolerance level
Rank assignment	Way to assign ranks to computing processors	Span, fill

. In particular—besides the matrix size, number of computing processors, number of physical nodes, algorithm type, and single/double floating-point precision—we included the fault tolerance level and the number of simulated faults among the dimensions to be explored. Indeed, as previously described, both the considered solvers can be adapted to tolerate up to a configurable number of hard faults (i.e., errors that invalidate all the computation on certain processors). The fault tolerance level refers to the maximum number of faults located anywhere that the procedure can tolerate. In contrast, the number of faults refers to the actual number of errors that have been simulated and recovered during the job run. Furthermore, different rank assignment modes were considered when mapping MPI ranks to computing processors. In “fill” mode, each rank is assigned a core trying to fill the physical nodes as much as possible. In “span” mode, instead, the ranks are distributed equally on all the sockets of the involved physical nodes. In both cases, the cores not assigned to any rank are left idle but not released to LSF (in order to fulfil the requirement R1 described in Section 3).

The jobs data block contains all the information collected about each job run. That is the identification number assigned by LSF to each job, the starting/ending timestamps, and the assigned physical nodes.

The sensors measurements block contains measurements collected from the sensors placed on each node in the CRESCO6 cluster. These data include power and energy consumption, ambient temperature, and cooling system data for each node involved in the job run.

Table 2. Main sensor measurements block fields and their description. Other collected sensor data are described in [49].

Field	Description
jobid	ID of LSF job
nodename	Name of the physical node. Generally, a number
timestamp_measure	Timestamp of the measure expressed in Unix time
sys_power	Total instantaneous power measurement of the computing node in watts
node_energy	Energy meter consumed by the node up to the time of reading. Useful for making differences between two readings in kWh
delta_e	Difference between previous and current measurement of energy in kWh for that node

describes a list of the physical measures that are more relevant for this work. Other sensor data collected but not used in this preliminary analysis are described in [49].

The final results block collects execution statistics including execution time, average, and total memory consumption, and any computational error discovered during the executions. Specifically, the main information in this block belongs to two different categories:

• The LSF resource usage summary, which includes: CPU time, maximum and average memory utilisation, maximum number of processes, maximum number of threads, runtime, turnaround time, etc.
• The output of the tested algorithm, which includes: norm-wise relative error of the solution, powercap energy counters, runtime of the algorithm’s subparts (i.e., initialisation and execution), etc.

Through this elaborate dataset made available to the community [53], we hope to enable the development of novel techniques devoted to understanding the energy footprint of the considered solvers. The dataset also includes Jupyter notebooks to simplify data inspection, generate summaries of relevant information about each job run, and visualise the data distributions.

5. Statistical Analysis of the Dataset

In order to provide a first glimpse of the collected data,

Table 3. Description of the dataset data.

Fields\Index	Mean	Std	Min	25%	50%	75%	Max
Matrix size	22,176	11,760.75	5280	10,560	21,120	31,680	42,240
Calculation processes	314	181.084	64	144	256	484	576
Spare processes	29.73	45.73	0	2	8	40	192
Total processes	343.73	195.41	64	148	320	492	768
Nodes	7.67	4.02	2	4	7	11	16
Fault tolerance level	3.33	2.7	0	1	2	4	8
Simulated faults	1.67	2.58	0	0	0	2	8
Processes per socket	10.99	11.11	0	0	8	23	24
ScaLAPACK checkpoint	4928	6762.64	0	0	0	10,560	21,120
ScaLAPACK blocking factor	10.38	11.63	0	0	0	24	25
Total energy (Wh)	77.73	144.39	0.73	6.88	17.69	69.4	2626.28
Peak power (W)	2131.32	1195.63	280	1120	1900	3030	5700
Average power (W)	1960.72	1064.42	304	990.10	1835.87	2813.40	4879.89
Runtime (s)	142.07	282.77	6	15	36	129.75	9481

presents a comprehensive overview of the dataset through key statistical indices for each field: minimum, maximum, and mean value, standard deviation, and 25th–50th–75th percentiles. In addition to the dimensions already presented in Table 1, the list includes the number of spare processes (i.e., the number of additional ranks necessary to provide fault tolerance), the total number of processes, the number of processes assigned to each socket (directly connected to the aforementioned “span” or “fill” policy for rank assignment), ScaLAPACK blocking factor and checkpointing rate, and four key metrics related to each job run: total energy consumption, peak power, average power, and execution time. Focusing on the latter four metrics, Table 3 suggests that, in the proposed dataset, the total energy and runtime metrics have high variability, as indicated by their large standard deviations and the extensive range between the minimum and maximum values. Also, the median values are much lower than the mean, indicating that a significant number of jobs have energy and runtime values lower than the mean, but a few jobs increase the average. On the contrary, peak and average power consumption seem more uniformly distributed, with median values more similar to the mean ones.

Figure 3 presents histograms for energy, runtime, peak, and average power consumption metrics, showing the general shape of the data distributions when considering the whole dataset samples. So, for example, the energy graph reports different ranges of energy consumed on the x axis, and the height of the bars indicates the number of samples (i.e., job runs present in the dataset) that fall into the corresponding energy range. Above the histograms, a density curve estimates the variable’s probability distribution. Given the skewed nature of energy and runtime data deducible from the values in Table 3, these two metrics are presented in a logarithmic scale to better appreciate their trends. Besides an unusual, non-strict correlation between runtime and energy consumed, the graphs suggest that the average value of the system power follows a similar trend to that of the maximum system power. However, if we restrict our attention to different dataset slices, other trends emerge.

For example, Figure 4 focuses on a subset of job runs involving just 2, 8, or 16 physical nodes. As expected, the three data series appear to have very different domains, especially for average power consumption: e.g., while most jobs running on 2 nodes have average power consumption under 1000 W, running on 8 nodes consumes between 1000 W and 3000 W, and jobs on 16 nodes generally consume between 3000 W and 5000 W. We remark that the graph presents the average power consumption for different numbers of nodes on the x axis (cumulative power, summing all nodes), and on the y axis, the number of jobs showing that power. It was built by first computing the power profiles of each job over time (summing together the contribution of different nodes if the job runs in parallel on multiple machines) and then calculating the mean value of that profile. Depending on such a value, the job contributes to increasing by one the height of the corresponding bar in the plot. Hence, as expected, the plot suggests an increase in the average power consumption linear with the number of employed nodes. Delving more into the dataset, we observe that jobs running on 2 nodes have a mean power consumption of 504 W on average, on 8 nodes 2240 W (around ×4 with respect to 2 nodes), and on 16 nodes 4271 W (around ×8 with respect to 2 nodes).

On the other hand, Figure 4 shows that the energy and runtime profiles of the three series have domains not as distinct as that of the power metrics, highlighting that some jobs running on just two nodes needed more time to complete, and, as a consequence, consumed a considerable amount of energy, comparable to that of shorter jobs involving many more nodes. This behaviour, which may appear counterintuitive, is derived from two apparently unrelated factors: (i) the jobs are not always launched saturating the cores available on the involved nodes; (ii) both IMe and ScaLAPACK parallel solvers operate on large data structures generating contention between the MPI ranks executing on different cores of the same machine in the access to the bus, cache and memory. This is especially true if the job saturates the machine’s available cores.

As a consequence, the execution of a certain job on the minimum number of physical machines may require so much more time than executing it on more machines, so that the increase in the energy derived by the employment of more nodes is actually conterbalanced by the fact that the job requires much less time to execute. The same considerations can be made looking at Figure 5, which correlates energy and runtime: jobs are ordered by runtimes on the x axis and the two lines show the energy consumed by the job (total energy) and its value normalised by the number of involved nodes (energy per node). As expected, the energy per node is linear in the runtime, while the total energy consumed by a job shows a much higher variability caused by the aforementioned resource contention.

Different trends are visible in Figure 6, where the data distributions are limited to three distinct matrix ranks (5280, 26,400, and 42,240). In general, jobs working on smaller matrices have lower runtimes and energy utilisation than solvers working on big matrices, while the distributions of average and peak power consumption seem to be less matrix-size-independent.

Figure 7 shows the distribution of the four metrics when different rank assignment methods are employed (“span” and “fill”). All in all, the trends do not highlight a significant difference in any metric between the two cases. Similar considerations can be made for Figure 8, which unveils the distributions when single or double precision is used in the solving process. However, in this case, a slight difference can be spotted in the energy (and runtime) distribution domains: double-precision jobs can consume up to 1000 Wh, while single-precision ones do not go beyond 500 Wh.

While a comparison by algorithm did not highlight significant differences, in Figure 9, the focus is on the level of fault tolerance that the solver can handle (i.e., the maximum number of processors that can incur a hard fault without invalidating the whole computation). For the sake of clarity, only two levels of fault tolerance are shown in the graph: 0 (i.e., the fault tolerance method is not implemented) and 8. Similar trends are visible for all the metrics, with only a slight difference in the shape of the histograms for average and peak power: the graph with a fault-tolerance level equal to 8 shows a longer tail to the right, corresponding to more cases with high average and peak power with respect to the no-fault-tolerance case. All in all, fault tolerance slightly influences the peak and average power without significantly impacting the overall energy consumption.

6. Regressor Evaluation and Prediction Error Analysis

In this section, we want to verify the efficacy of the created dataset as training for the ML predictors of the four previously defined target variables (i.e., total energy consumption, maximum system power, mean system power and runtime). Specifically, we consider decision tree regressor, random forest regressor, and gradient boosting decision tree regressor. We remark that this work aims to provide the scientific community with a detailed dataset of HPC job runs, useful for the further analysis of the environmental footprint of linear solvers. As such, proposing novel ML models for energy consumption prediction is out of the scope of the paper.

Table 4. Summary table of regressor prediction evaluation of the dataset.

TARGET	REGRESSOR	BEST DEPTH	RMSE	MAE	MAPE
Total Energy	Decision tree	19	0.013	0.004	0.076
Min value: 0.000470 kWh	Random forest	13	0.010	0.003	0.062
Max value: 0.938590 kWh	GBDT	5	0.010	0.003	0.119
Max power	Decision tree	10	12.676	7.966	0.024
Min value: 150.000000 W	Random forest	11	10.241	6.947	0.020
Max value: 440.000000 W	GBDT	6	10.339	7.089	0.021
Mean power	Decision tree	9	7.297	5.188	0.024
Min value: 115.000000 W	Random forest	11	6.530	4.632	0.022
Max value: 330.158420 W	GBDT	5	5.942	4.319	0.020
Runtime	Decision tree	10	22.238	7.296	0.065
Min value: 6.000000 s	Random forest	12	19.651	7.033	0.060
Max value: 2097.000000 s	GBDT	6	14.637	5.560	0.065

details the performance metrics for the chosen regression models in predicting the job’s energy performance. For each model, we used GridSearchCV to optimise hyperparameters, with negative mean absolute error as the evaluation criteria. The table also includes metrics like RMSE, MAE, and MAPE, along with the optimal hyperparameter value for depth.

When considering all the evaluation metrics, the Random Forest Regressor emerges as the best overall performer for predicting the total energy consumption and the maximum power. Instead, the gradient boosting decision tree regressor shows the strongest performance for predicting the average power and runtime.

The histograms in Figure 10 depict the distribution of prediction errors for the four chosen targets. The x axis of each graph represents the prediction error value, while the y axis represents the error frequency, that is, the number of job runs tested that returned that error value. The models were trained on 70% of the data, and the remaining 30% was used for the testing. The errors for total energy consumption and runtime show a similar trend, with a significant peak centred around zero. The errors for the power targets (maximum and average power) exhibit a wider distribution, ranging from 0 to 40 W in absolute value. However, considering the average values of the power targets (with a maximum of approximately 2130 W and 1960 W on average), this error range remains very good.

7. Conclusions and Future Work

HPC systems have witnessed explosive growth in computing power in recent decades, but this remarkable advancement comes at the cost of dramatically increased power consumption. In this paper, we reason about the use of ML techniques to predict the energy consumption of HPC workloads. In particular, we focus on the parallel resolution of large-scale linear systems and propose a large dataset of job runs equipped with information coming from physical sensors. We also describe the framework used for running the algorithms and retrieving the scattered data. Finally, we apply standard ML techniques on chosen energy-related targets and perform statistical analysis and error evaluation to assess the performance and effectiveness of such models when trained on the proposed dataset.

In future work, the challenges and goals outlined certainly include conducting analysis and evaluation for other targets already recorded in C6EnPLS, such as the utilisation and the temperature of the hardware components, the speed of the cooling fans on each machine, etc. After this, we will explore using alternative ML models, such as neural networks, which can potentially capture more complex relationships within the data by offering more accurate target predictions. Such an analysis should also help us understand which features play the most important role in determining the energy footprint of a linear solver. We also plan to enrich the dataset with further launch settings where other jobs are allowed to run on the same machine, thus making the collection more similar to real-life scenarios. However, we expect the energy consumption prediction task to be much more challenging in that setting.

Another interesting future work could involve extending the dataset and generalising the study to different solvers (e.g., indirect, iterative solvers, GPU-based implementations of IMe and Gaussian elimination, etc.), including a wider range of linear algebra problems (such as matrix multiplication and eigenvalue computation), or extending the dataset to different settings where parallel computation is supported by distributed processing engines [54,55]. Furthermore, the methodology described in this work could be easily applied to other scenarios, where the HPC environment is dedicated to the execution of parallel tasks, known to be highly energy demanding—such as meteorological computations [56] or parallel process mining workloads [57,58,59] in order to identify the features that most likely contribute to the energy footprint in those settings. In general, it would be interesting to build a dataset with mixed types of jobs in order to generalise our study to situations in which the HPC infrastructure hosts various workloads (as it is in practice). We believe that ML techniques could benefit from real-time adaptability to account for the HPC intrinsic variability. Furthermore, the exploration of the effects of power capping mechanisms could also be an interesting direction for enriching the dataset. For example, DVFS is known to be an effective way to reduce the power requirements, but its side effect of increasing the application runtime can sometimes determine higher energy consumption overall. A dataset including several power-capped job runs could also be useful to shed light on this trade-off.

Finally, the integration of the proposed framework in the JUBE system [46], enabling communication with LSF scheduler (besides Slurm and PBS) and with the CRESCO6 sensors system, represents an interesting matter of future work.