Power Estimation and Energy Efficiency of AI Accelerators on Embedded Systems

Abstract

The rapid expansion of IoT devices poses new challenges for AI-driven services, particularly in terms of energy consumption. Although cloud-based AI processing has been the dominant approach, its high energy consumption calls for more energy-efficient alternatives. Edge computing offers an approach for reducing both latency and energy consumption. In this paper, we propose a methodology for estimating the power consumption of AI accelerators on an embedded edge device. Through experimental evaluations involving GPU- and Edge TPU-based platforms, the proposed method demonstrated estimation errors below 8%. The estimation errors were partly due to unaccounted power consumption from main memory and storage access. The proposed approach provides a foundation for more reliable energy management in AI-powered edge computing systems.

1. Introduction

The exponential growth of connected sensors and Internet of Things (IoT) devices has resulted in a significant increase in data generation at the network edge. While this data-rich environment enables a broad range of AI-driven services, it also introduces significant challenges, particularly in terms of energy consumption and environmental impact. Traditional AI systems rely heavily on centralized cloud infrastructures to process and analyze vast streams of data, a model that contributes substantially to carbon emissions. Data centers alone are currently responsible for around 2% of global CO₂ emissions, and with rising demand, their energy consumption is expected to grow by 10% annually [1].

In response to these challenges, edge computing has emerged as a technically viable and energy-efficient alternative to centralized architectures. By processing and storing data closer to where it is generated—at the “edge” of the network—AI applications can reduce their dependency on power-hungry cloud servers, thereby minimizing data transmission requirements and lowering overall energy consumption. Performing AI inference locally on edge devices mitigates network dependency, minimizes latency, and contributes to substantial improvements in overall system energy efficiency. Recent advancements in low-power AI accelerators, lightweight neural network architectures, and model optimization techniques have further enhanced the feasibility of executing complex AI workloads directly on resource-constrained edge devices.

To leverage hardware accelerators, machine learning models trained using libraries must be converted into the precision and format required by the hardware accelerator used in embedded systems. Even when using the same deep learning model, differences in mapped operations between libraries or variations in performance (execution time) and energy consumption can occur, depending on the computing resources used such as CPU (Central Processing Units), GPU (Graphical Processing Units), and other accelerators. Additionally, AI hardware accelerators often employ quantization techniques to increase speed, which can impact the accuracy of the results [2]. Unlike data centers, where energy resources are comparatively abundant and scalable, edge devices often operate within limited power budgets, constrained thermal envelopes, and battery-powered contexts. In such scenarios, achieving system-wide energy efficiency depends not only on deploying low-power hardware but also on accurately estimating and managing the power consumption of AI accelerators

Estimation of power consumption is essential for several reasons. It enables system designers to configure AI accelerators effectively for a given application, balancing computational performance with energy efficiency requirements. Power estimation can be utilized for dynamic power management, such as voltage and frequency scaling, thereby extending device operational lifetime and reducing overall energy consumption. In mission-critical and energy-sensitive sectors such as energy management, healthcare, and autonomous systems, maintaining predictable and efficient power profiles is crucial for ensuring reliable system behavior and preventing operational disruptions.

This paper aims to develop a low-overhead methodology for estimating the power consumption of AI accelerators under varying workloads and operational conditions. To achieve this, we conducted experiments with AI accelerators, including GPUs and an edge TPU for embedded systems that support different precision levels of operations.

The remainder of the paper is organized as follows. Section 2 overviews previous works related to this paper. In Section 3, the experimental environment and the target device for power estimation are described, then the power estimation method and data gathering method are explained. The evaluation results for the power estimation method are presented in Section 4. Section 5 discusses the cause of errors in the power estimation. Finally, Section 6 concludes our work and suggests a future research direction to improve the estimation method.

2. Related Works

Significant resources are required to perform deep learning inference, which can be a hindrance when executing AI applications on embedded systems. Several approaches have been researched to address this challenge. Firstly, libraries that require large memory capacity can be replaced with specialized libraries designed for embedded systems, primarily targeting CPU-based inference without the use of hardware accelerators. Inference tools for embedded systems often quantize model weights or input values, reducing model size and inference time. Research by Hadidi et al. [3] compared the performance of various frameworks, including TensorFlow 2.0, PyTorch, Caffe2, and Darknet, using GPUs, TPUs (Tensor Processing Units), and VPUs (Vision Processing Units) as accelerators. However, that study did not use a single embedded board but experimented with different embedded boards, each capable of using various accelerators, making direct comparisons challenging.

Various GPU-based edge AI platforms with different computational capabilities have been released by NVIDIA under the Jetson board series [4]. Although GPUs are not inherently designed as AI-specific accelerators, their integration into embedded systems significantly enhances on-device inference performance. In addition to GPUs, several commercially available AI accelerators featuring dedicated hardware have been introduced, including Google’s Edge TPU [5] and Intel’s Neural Compute Stick (NCS) [6].

A comparative evaluation of the Edge TPU and NCS for 3D object detection tasks is presented in [7]. The work in [8] benchmarks the NVIDIA Jetson Xavier, Edge TPU, and NCS2 using AlexNet and GoogLeNet models, reporting that the NCS2 achieves higher efficiency on AlexNet, while the Edge TPU delivers higher performance with GoogLeNet. Further investigation into architectural trade-offs between computational performance and energy efficiency on Edge TPU and Cortex-A53 platforms is presented in [9], where model size was identified as a primary factor affecting Edge TPU’s performance. Subsequent research in [10] finds that the Edge TPU’s inference performance is highly dependent on the model size.

Tu et al. propose kernel-level energy predictors and scoring metrics to enhance transparency in on-device deep learning energy consumption [11]. Similarly, the MLPerf Power benchmark offers standardized frameworks for comparing energy efficiency across AI systems ranging from small edge devices to large cloud infrastructures, helping to quantify the energy/performance trade-off for edge deployments [12]. Empirical studies of embedded AI inference platforms in [13] show that real-time power profiling could allow more efficient scheduling and resource management in constrained environments. In [14], energy-aware device-level techniques, such as supply voltage scaling and quantization-aware inference, are shown to be effective in improving the energy efficiency of neural network accelerators on edge devices.

3. Estimation of Power Consumption

3.1. Target Embedded Hardware and Accelerators

The present experiments use NVIDIA’s Jetson AGX Xavier as the target board, with brief specifications shown in

Table 1. Specification of Jetson AGX Xavier.

Characteristic	Description
GPU	512 Volta GPU with Tensor cores
CPU	8-core ARM v8.2
Memory	32 GB 256-bit LPDDR4x @ 137 GB/s
Storage	32 GB eMMC 5.1

[4]. The target board features 8 power modes, allowing for dynamic adjustment of CPU and GPU clock frequencies according to the selected power mode. The minimum frequencies for CPU and GPU are 1190.4 MHz and 114.75 MHz, respectively. The maximum frequencies for power modes are summarized in

Table 2. Clock configuration of Jetson AGX Xavier.

Property	Power Mode
	MAXN	10 W	15 W	30 W	30 W	30 W	30 W	15 W
Online CPU cores	8	2	4	8	6	4	2	4
CPU maximal frequency (MHz)	2265.6	1200	1200	1200	1450	1780	2100	2188
GPU maximal frequency (MHz)	1377	520	670	900	900	900	900	670

.

We measured the power consumption of the target system in the idle state using ADpower’s HPM-300A [15], which allowed us to record power consumption at a maximum of 4 times per second. Figure 1 depicts the setup for the power measurement. The target board is connected to an AC/DC converter, at which the power meter measures the power and energy consumption of the target system. DC input is automatically adjusted by the target board according to its power mode described in Table 2. The GPU is embedded in the target board, while the TPU is connected via a USB 3.0 connection.

Table 3. Power consumption in the idle state (Watt).

Mode	10 W	15 W	30 W	30 W	30 W	30 W	15 W
Power	8.772	8.871	9.796	8.837	9.878	9.150	10.487

shows the power consumption at each mode with the lowest CPU frequency and the minimum online cores. MAXN mode is excluded because the clock speed is fixed at the maximum frequency in that mode.

The AI accelerators used in our experiments were NVIDIA GPU and Google’s Coral USB accelerator. The Coral USB accelerator is connected to the target board via USB and utilizes a Tensor Processing Unit (TPU) [5]. The precision available for the Coral USB is INT8 (Integer 8-bit) for the TPU.

3.2. Power Estimation Method

The power consumption of a CPU can be categorized into two components: dynamic power and static (or leakage) power [16]. Static power is directly influenced by the supply voltage and the leakage current, exhibiting a linear relationship with both. In contrast, dynamic power, which is also referred to as switching power, is typically modeled as being proportional to the product of the switching activity factor α, effective capacitance C, operating frequency f, and the square of the supply voltage V (P ∝ αCfV²) [17]. Although this conventional power model does not capture the full complexity of modern processors, it provides a useful framework to estimate CMOS-based processor power.

In CMOS-based designs, dynamic power consumption dominates total power usage [17]. The processor speed f is linearly proportional to the clock speed [18]. However, lowering the supply voltage increases the delay of logic circuits, which in turn requires reducing the operating frequency [19]. This relationship means that changes to operating frequency have a cubic effect on dynamic power consumption, as both frequency and voltage are interdependent. Techniques such as Dynamic Voltage and Frequency Scaling (DVFS) exploit this relationship to reduce power consumption, though they introduce a trade-off: lowering frequency and voltage improves energy efficiency at the cost of reduced processing throughput and computational performance [20].

We measured the power consumption of the CPU and the accelerators in various settings to determine the estimated power consumption based on the CMOS-based quadratic power dissipation relationship using benchmark programs. Linpack [21] for measuring the CPU power, MatrixMul from CUDA samples [22] for the GPU power, and MobileNet V1 [23] for the TPU power are used as benchmarks. In principle, any program capable of fully utilizing the computing resources can serve as a power consumption benchmark. These particular benchmarks were selected because they are widely adopted and well-recognized for testing the maximum computational workload on a processor.

To measure the CPU power consumption, the GPU clock was set at its minimum, and measurements were taken while varying the number of active CPU cores and CPU clock frequencies. First, to evaluate the effect of CPU frequency on the power consumption, the number of CPU cores was fixed while varying the frequency, and the resulting power consumption was measured. The measured power consumption includes not only the CPU but also the total power usage of the target board. Therefore, the additional power consumed as the CPU frequency increases was estimated by subtracting the idle power consumption from the total measured power. Figure 2 shows the results along with a regression analysis-derived estimation formula for the additional power consumption according to CPU frequency.

To assess the effect of the number of online CPU cores, we also measured the power consumption when varying the number of active cores at each CPU frequency level. The measurement results showed that the number of cores has a linear impact on power consumption. Since the benchmark program fully utilizes the online core at 100%, we must take core utilization rates into account when we estimate power consumption in practical situations.

The estimation model was based on the power consumption of CMOS circuits, and a linear model was derived by observing the increase in power according to the number of active cores from the idle power state. Then, the power consumption of the target board including CPU with frequency f (in GHz) and online core n is modeled as follows:

Power = Idle Power + (0.5𝑓 − 0.1) × 𝑛 × (1.666𝑓² − 4.044𝑓 + 3.309)

(1)

For the GPU power, the number of CPU cores and their clock frequencies were set to the minimum values, and power consumption was measured while varying the GPU clock. The power consumption of the GPU was estimated by subtracting the system’s power consumption without the GPU from the total power consumption. Figure 3 shows the results with an estimation formula for the additional power consumption according to GPU frequency. The resulting GPU power consumption is formulated as follows:

GPU Power = 2.78f² + 6.63f + 1.727

(2)

Since the Edge TPU operates via a USB connection, it is necessary to distinguish the power consumption of the TPU from that of the USB interface. When the Coral USB device was connected, the static power consumption was measured at an average of 1.013 W. If the model size exceeds the internal memory capacity of the TPU, it accesses the target board’s memory, resulting in additional power consumption. To ensure that power consumption from external memory access did not affect the results, we used the MobileNet V1 model, which fits within the TPU’s 8 MB internal memory. Experimental results showed that the TPU’s power consumption remained almost constant at approximately 1.65 W, regardless of the target board’s power mode. Therefore, TPU power is given as follows:

TPU Power = 1.013 + 1.65u

(3)

where u = 1 when the Edge TPU is used. Total power consumption of the target board is estimated by summing Equations (1)–(3).

3.3. Data Collection for Power Estimation

To estimate power consumption in real time, we developed a monitoring program based on the eBPF (extended Berkley Packet Filter) framework on Linux. It allows programs to monitor events at both the kernel and user application levels in an event-driven manner by attaching to hook points such as system calls, functions, kernel trace points, and network events. Since the default kernel of the Jetson Xavier does not support eBPF, it is necessary to rebuild the kernel using the source code provided by NVIDIA. Additionally, to trace kernel execution and manipulate programs, we used BCC (BPF Compiler Collection), which offers toolkits needed for working with eBPF.

Performance counters were monitored using eBPF to obtain the number of CPU cycles used per core, and this information was reflected in the number of cores in the CPU power model. We define “effective number of cores” n_e as

n_e = (sampling frequency) × (the number of CPU cycles)/(CPU frequency)

(4)

Since the CPU frequency is the maximum possible number of cycles per 1 s, (CPU frequency)/(sampling frequency) is the maximum cycles at the sampling frequency. Thus, the effective number of cores represents the actual utilization of the CPU cores with switching activities. We replace n in Equation (1) with n_e in real-time power estimation.

The other information gathered in real time includes: execution time of applications, USB data transfer for monitoring TPU usage, GPU load, and GPU frequency. For GPU information, NVIDIA’s jtop tool was used, while other information was collected directly from the kernel using the eBPF framework.

4. Evaluation of Power Estimation Method

To evaluate the presented power estimation model, we tested using CIFAR-10 image classification data [24] with the YOLO v8 model. The dataset consists of 10 categories, each containing 1000 images, for a total of 10,000 images. Data for estimating power consumption was gathered at the sampling rate 1 Hz while actual power consumption was measured using a power meter simultaneously. Figure 4 shows the experimental results with the TPU, and Figure 4 shows those with the GPU.

Total energy consumed for inference, measured with the power meter, is compared with the estimated value by the method presented in

Table 4. Total power consumption and error.

Accelerator	Energy Used	Estimation	Error
Edge TPU	910.20 J	978.72 J	68.52 J (7.53%)
GPU	950.44 J	896.29 J	−54.15 J (−5.70%)

. As shown in Figure 4 and Figure 5, the presented method overestimates the power consumption for the TPU, but under estimates that of the GPU. The error is below 8%, but the presented model suggested that inference with the TPU would consume more energy than with the GPU; however, the energy consumption was lower when using the TPU. We will discuss the estimation error in the next section.

5. Discussion

As shown in the previous section, the presented estimation method captures the overall trend of power consumption, but some inaccuracies were observed. To identify the source of the error, we investigated memory usage. As mentioned earlier, the Coral USB with Edge TPU has 8 MB of internal memory. If the model size exceeds the capacity of the internal memory, it uses the off-chip memory for streaming the model parameters. Furthermore, because the internal memory is used up, it suffers from an insufficient cache for model parameters.

Table 5. YOLO v8 internal memory usage.

On-Chip Memory	Cache	Off-Chip Memory
7.74 MB	512 B	1.73 MB

summarizes YOLO v8’s memory use.

The accesses to the main memory of the target board would affect the power consumption. However, it is difficult to measure the number of main memory accesses because the performance counter of the target board does not provide such information. Undoubtedly, loading images before inference requires a certain amount of power to perform. Currently, the presented model also ignores the access to the storage, which is also a source of estimation error. It is necessary to develop a method for obtaining information on main memory and storage usage, so that power consumption associated with their access can also be considered, and to establish a power consumption estimation method based on this information.

6. Conclusions

Developing low-overhead methodologies for estimating the power consumption of AI accelerators under varying workloads and operational conditions is a key enabler for scalable and efficient AI deployment at the network edge. Integrating such estimation frameworks into edge computing platforms could improve energy management. This study presents a methodology for estimating the power consumption of AI accelerators. The experimental results showed that while the proposed estimation model successfully captured overall power consumption trends, it exhibited certain deviations, particularly underestimating GPU consumption and overestimating that of the Edge TPU. The observed estimation errors, although within 8%, may stem from the exclusion of main memory and storage access power in the current model. As AI inference workloads involve not only computational processes but also significant memory and storage operations, future work should focus on developing mechanisms for monitoring these accesses and integrating their associated power costs into estimation models. By refining power estimation methodologies to account for these factors, more accurate and energy-efficient AI systems can be realized, enabling sustainable and reliable operation of AI services at the edge.