Benchmarking Large Language Model Inference on Limited-Resource Edge Systems

Abstract

Large language models (LLMs) are increasingly considered for deployment on edge and limited-resource systems, where local inference can reduce latency, improve privacy, and decrease dependence on cloud infrastructure. While prior studies have evaluated either task accuracy or hardware efficiency in isolation, few benchmarks combine generation-based response-quality evaluation with real-device power measurements on a representative limited-resource platform. This study addresses that gap by benchmarking twelve compact and mid-scale open-weight LLMs (sub-1B to 8B parameters), evaluating generation-based accuracy on a desktop platform and measuring deployment efficiency—throughput, power consumption, and energy use—on an NVIDIA Jetson Orin Nano Super; the accuracy–efficiency trade-off is thus established at the model-configuration level. Unlike prior Jetson-based evaluations relying solely on internal telemetry, this work pairs generation-compatible lm-eval accuracy tasks with a dual power-measurement setup that combines internal tegrastats rail readings with external board-level input power measured using a digital multimeter and explicitly compares GPU-accelerated and CPU-only inference modes. GPU-accelerated inference provided a clear advantage, increasing median throughput from 7.12 to 18.13 tok/s and improving external-meter energy efficiency from 0.453 to 0.823 tok/J, despite higher mean input power. Sub-1B models offered the best throughput and energy efficiency, whereas 7–8B models achieved stronger accuracy at a substantially higher energy cost per generated token. These results demonstrate that edge LLM deployment requires multi-objective evaluation balancing accuracy, throughput, power consumption, and energy efficiency.

1. Introduction

Large language models (LLMs) are now widely used in everyday applications, enabling conversational assistants, document understanding, and domain-specific automation. While the most capable LLM deployments have mainly relied on cloud infrastructures with abundant accelerators and memory, there is a growing push to move inference, and increasingly adaptation, closer to end users and data sources. Edge deployment can reduce network dependence, improve privacy, support operation under limited connectivity, and lower recurring cloud-computing costs. Recent surveys and reviews of edge-based LLM deployment describe this transition through fully on-device, edge-assisted, and hybrid architectures while also emphasizing application domains, security and privacy, output trustworthiness, and distributed edge-resource management [1,2,3,4,5].

Despite this interest, running LLMs on limited-resource systems remains difficult because transformer-based models combine large memory footprints, high memory-bandwidth demand, and sequential token-by-token generation. These constraints are amplified on edge devices, which typically provide less RAM, lower memory bandwidth, stricter power and thermal limits, and more heterogeneous software support than datacenter platforms. As a result, edge LLM feasibility depends not only on model size and quantization but also on runtime implementation, memory management, accelerator use, and device-level bottlenecks. Recent studies show that edge performance can be limited by memory-system behavior, paging under memory pressure, background contention, and sustained thermal effects rather than only by peak compute throughput [6,7,8,9,10,11]. These observations motivate transparent and reproducible benchmarking methods that capture both user-facing performance and the underlying resource constraints [12,13].

1.1. From Cloud-Centric LLMs to Edge-Centric Intelligence

The move from cloud-centric serving to edge-centric intelligence changes the optimization target. Cloud inference can often exploit batching, large accelerators, and highly optimized serving stacks, whereas edge inference must prioritize responsiveness, reliability, and energy use under local resource limits. This shift has been described as part of a broader transformation of the mobile AI ecosystem, where on-device language models affect software stacks, developer tools, and user experience [14,15]. Surveys of mobile edge intelligence and IoT-oriented LLM deployment similarly emphasize that local inference must be designed together with resource management, security, privacy, and application-level constraints [1,2,3,4,5].

Applied studies further show that edge LLM deployment is not a single hardware-selection problem but a systems problem shaped by the application context. Embedded robotic systems use local inference to support real-time control and privacy-sensitive interaction [16]. Malware-detection studies evaluate lightweight LLMs in terms of accuracy, latency, and operational feasibility at the edge [17]. Maritime applications consider small language models under domain-specific reliability and communication constraints [18]. Related edge-intelligence work, including wearable biomedical monitoring and deep-learning-based medical image processing, also illustrates the need to align model design with sensing, resource, and deployment conditions [19,20]. Together, these examples show that practical edge inference depends on the interaction between application requirements, acceptable quality-of-service thresholds, and platform limitations [21,22,23,24].

1.2. The Bottleneck Landscape on Limited-Resource Systems

The main constraints for edge LLM inference can be summarized as memory footprint, memory bandwidth, and power/thermal sustainability. Memory footprint determines whether a model can run without causing paging or swap activity. When RAM is insufficient, execution may become dominated by slow storage I/O rather than computation, making a device appear active while making limited progress [6]. Related studies on small and resource-constrained systems similarly show that LLM inference is highly sensitive to available memory and platform configuration [7,13,25,26].

Even when a model fits in memory, inference can remain bottlenecked by data movement. During autoregressive decoding, each generated token requires repeated access to model weights and updates to the attention key-value (KV) cache. On limited-resource systems, the limiting factor is therefore often memory bandwidth rather than arithmetic throughput alone. This view is consistent with hardware-focused analyses of LLM acceleration, which emphasize memory hierarchy and bandwidth as central determinants of inference efficiency [27]. Measurement studies on edge accelerators and embedded GPU platforms similarly argue that compute and memory behavior must be evaluated together, rather than inferred from peak compute specifications [9,11,28,29].

Power and thermal limits further determine whether observed performance can be sustained. Mobile and edge measurement studies show that heavy workloads may trigger frequency reductions over time, making short-run performance insufficient to characterize long-duration behavior. They also report that energy efficiency varies substantially with the quantization level, runtime configuration, and device class [8,10,30,31]. Energy-focused work further argues that LLM evaluation should consider not only response quality and latency but also the energy cost of inference across languages, hardware platforms, and deployment configurations [31,32].

These bottlenecks imply that meaningful edge LLM benchmarking must report more than a single headline throughput value. Measurements should separate major inference phases, account for workload settings such as prompt and output length, and include system telemetry to help identify whether performance is limited by memory pressure, bandwidth, accelerator use, or thermal behavior. This need for LLM-native and system-aware evaluation is reflected in recent benchmarking frameworks and measurement studies [6,10,33,34,35,36,37].

1.3. Making LLMs Feasible at the Edge: Compression, Quantization, and Adaptive Deployment

Several optimization strategies have been proposed to make LLMs practical on edge devices. Model compression and distillation reduce parameter count and deployment cost while aiming to retain much of the quality of larger models. Recent work also studies distillation strategies for producing multiple smaller models with different latency or size targets without retraining each model independently [38]. Reviews of on-device and small language models summarize a broader efficiency toolbox, including pruning, quantization, architectural modifications, and decoding optimizations, and discuss their suitability for mobile and embedded hardware [3,13,39,40,41,42].

Quantization is especially important for local LLM inference because it reduces the memory footprint and can improve runtime efficiency by representing weights, and sometimes activations, with fewer bits. Activation-aware methods such as AWQ aim to preserve model quality at low precision [43], while stable-rank-guided and LoRA-oriented compression methods target memory-efficient deployment in edge-computing settings [44]. At the deployment level, any-precision approaches allow applications to choose different quality–latency trade-offs, depending on device constraints [45]. System-level methods further extend this idea by selecting execution plans across heterogeneous devices, including on-device execution, offloading, and layer partitioning [46,47,48,49,50].

Recent work also explores on-device adaptation and fine-tuning. Although fine-tuning is usually more expensive than inference, new approaches reduce this cost by reusing inference engines for update procedures or combining compression with lightweight tuning strategies [44,51,52,53]. These developments suggest that edge LLM research increasingly spans both inference and adaptation, strengthening the need for benchmarks that capture end-to-end behavior under realistic device constraints [12,15].

Other approaches move beyond purely local inference and consider collaborative or edge-assisted serving. Edge-LLM proposes collaborative serving across edge resources to satisfy latency and memory constraints [54,55], while LLMEdge emphasizes localized inference and end-to-end deployment behavior on resource-constrained platforms [56]. Mobility-aware partitioning, task offloading, UAV-assisted systems, and satellite-edge settings further expand this design space by introducing additional constraints on accuracy, communication, and resource allocation [47,48,57,58]. These architectures can improve feasibility, but they also make results harder to compare unless benchmarks clearly specify network conditions, local versus offloaded execution, and workload partitioning [59,60].

1.4. Benchmarking and Measurement: Towards LLM-Native Evaluation on Edge Devices

As the edge LLM ecosystem grows, benchmarking is increasingly treated as a research problem rather than only an implementation detail. Earlier mobile and embedded AI benchmarks were often designed around vision and CNN workloads, where inputs have fixed shapes, and latency can be summarized more directly. Prior embedded-vision work illustrates this measurement tradition across FPGA-based CNN inference, real-time conveyor-belt component detection, low-power eyeglasses detection, mobile eyewear similarity retrieval, and TensorFlow Lite execution-time estimation [61,62,63,64,65]. LLM inference differs because it involves model loading, prompt processing, token-by-token generation, variable sequence lengths, and a growing KV cache. Consequently, recent work has begun to develop LLM-specific benchmarking suites, metrics, and measurement studies [10,33,34,36,66].

Empirical measurement studies show that a model that technically runs on-device may still fail to provide acceptable latency, energy use, or sustained behavior in practice [10]. Other edge-focused studies similarly report trade-offs among output quality, latency, and energy efficiency [8,30,31,32]. Evaluations across heterogeneous hardware and serving stacks show that results can vary substantially by device and software configuration, motivating standardized and transparent reporting [28,67]. IoT-oriented studies also examine the feasibility of deploying LLMs on devices and gateways, reinforcing the importance of measuring performance under realistic constraints [21,25,68].

Benchmarking frameworks and suites provide a complementary perspective. PalmBench proposes a benchmark for compressed LLMs on mobile devices, reflecting the need to compare compression strategies under on-device constraints [34]. Other studies characterize small language models on edge hardware and show that runtime behavior depends on both system configuration and workload properties, such as prompt length and generation settings [35,39]. Domain- and hardware-specific benchmarks, including object-detection evaluation on Jetson Orin NX and FPGA-based heterogeneous computing for LLM inference, further show that edge benchmarking must jointly consider accuracy, latency, throughput, energy, and device-specific constraints [69,70]. These studies help identify where current inference stacks fall short and which optimizations have the greatest impact [29,71].

Recent work also argues for more explicit evaluation philosophies and metrics for edge LLMs. New benchmarking frameworks aim to capture the factors that actually limit inference, noting that a single throughput or latency number may hide memory-bound behavior or heterogeneous-execution effects [12,33]. Other contributions propose a more principled “evaluatology”, where metrics are tied to deployment goals and stakeholder priorities [37]. This need for interpretable evaluation is consistent with broader explainable machine-learning research, where theory-driven saliency and interpretability methods are used to make model behavior more transparent in complex domains [72]. For edge LLMs, benchmarking should, therefore, not only rank configurations but also provide signals that explain why performance differs across devices and runtimes [60].

This direction aligns with broader embedded-systems research, which emphasizes practical methods for estimating and optimizing deep neural networks on embedded platforms [73]. Results from adjacent transformer workloads are also informative: studies of vision transformers on GPU-based embedded edge systems show that transformer-style models can be strongly affected by kernel efficiency and memory behavior on embedded GPUs [74]. Together, these works motivate benchmarking that is both LLM-native and platform-aware [25,28,75,76].

1.5. Motivation for Benchmarking on Limited-Resource Systems

Limited-resource systems differ widely in their constraints. Smartphones are often limited by battery life and heat dissipation, single-board computers have modest compute and memory bandwidth but can run continuously on external power, and embedded GPU modules provide acceleration while still remaining far more constrained than datacenter GPUs. Therefore, the most relevant metric depends on the use case: interactive assistants prioritize time-to-first-token and per-token latency, continuous analytics emphasizes energy per generated token and sustained throughput, and robotics or control workloads require predictable timing and low tail latency [10,31,33,36].

Recent reviews and application studies show that edge LLM deployment must also account for security, privacy, trustworthiness, reliability, and integration complexity [1,2,4,5,13,22,26,71]. On-device LLMs are promising for business and SME environments, but they introduce operational challenges when infrastructure and expertise are limited [77]. Similar constraints arise in maritime systems, UAV-assisted satellite edge computing, and embedded object-detection platforms, where model selection and resource allocation must be aligned with application-specific reliability and quality requirements [18,24,48,69].

Within this landscape, a recurring methodological gap is the lack of consensus on which metrics, workloads, and measurement practices best capture practical edge deployment quality. Edge-first work emphasizes that deployment decisions should consider models, metrics, and trade-offs together, rather than optimizing a single headline value [36]. Hardware-focused analyses similarly argue that system-level measurement is needed to identify where acceleration benefits actually come from [27]. Empirical footprint studies show that, without memory and system telemetry, benchmarks may misidentify bottlenecks or miss failure modes that appear only under tight memory limits [6]. These findings motivate the present study: a measurement-based benchmark that links response quality, throughput, power consumption, and energy efficiency to explain not only how fast a configuration runs but also what deployment trade-offs it implies on limited-resource hardware [12,25,28,59].

1.6. Scope and Contributions of This Work

Motivated by the methodological gap outlined above, this work evaluates edge LLM deployment as a joint accuracy–efficiency problem. The aim is not to propose a new model architecture, compression method, or scheduling algorithm but to provide a reproducible measurement-based analysis of how current compact and mid-scale open-weight LLMs behave on a representative limited-resource edge platform. In particular, the study links task-level response quality with hardware-level measurements of throughput, power consumption, and energy efficiency, thereby supporting model selection under practical deployment constraints.

The novelty of this work lies in combining generation-based response-quality evaluation with real-device deployment measurements on a limited-resource edge platform. Rather than reporting only task accuracy or only inference speed, the study evaluates compact and mid-scale LLMs as practical deployment configurations, where model choice must balance accuracy, throughput, power draw, and energy cost per generated token.

The novelty and main contributions of this work are summarized as follows:

• A systematic benchmark of twelve compact and mid-scale open-weight LLMs, ranging from sub-1B to 8B parameters, using a unified model set, prompt set, context-length limit, output-token limit, and deterministic decoding configuration.
• A joint accuracy–efficiency evaluation that links generation-based task performance with measured deployment cost on edge hardware.
• A two-part evaluation methodology in which task-level response quality is assessed on a more powerful desktop PC for faster benchmark execution, while deployment efficiency is measured directly on the target NVIDIA Jetson Orin Nano Super edge platform.
• A comparison of GPU-accelerated and CPU-only inference on the Jetson platform, quantifying the effect of execution mode on throughput, power consumption, and energy efficiency under the same workload conditions.
• A power-measurement setup that combines internal Jetson telemetry with external board-level input power measurement using a digital multimeter, enabling comparison between on-chip rail readings and total system-level input power.
• An empirical characterization of practically deployable Ollama/GGUF model configurations, showing how parameter scale and quantized deployment choices affect accuracy, throughput, power consumption, and energy efficiency on an 8 GB edge device.
• An empirical analysis of the accuracy–efficiency trade-off across model sizes, identifying practical regions where smaller models provide high throughput and energy efficiency, while larger models offer stronger response quality at a higher energy cost.

2. Materials and Methods

This section describes the experimental methodology used to evaluate compact and mid-scale large language models (LLMs) from two complementary perspectives: response quality, assessed using established automatic evaluation tasks, and deployment efficiency, assessed through inference speed and power-related measurements on edge hardware. The adopted methodology reflects the practical constraints of edge-oriented experimentation: model accuracy evaluation was performed on a more capable desktop platform in order to execute standardized benchmark suites efficiently, while speed and energy-related measurements were collected on the target embedded platform using a custom measurement harness. This split design allowed the use of a reproducible and tractable evaluation pipeline while preserving hardware-specific measurements on the actual deployment-oriented device.

2.1. Experimental Design

The study was organized into two experimentally distinct but model-aligned parts:

• An accuracy evaluation was performed on a desktop-class PC using the lm-evaluation-harness framework, together with Ollama’s OpenAI-compatible local API. This part was used to estimate the task performance of the evaluated models on generation-based benchmark tasks.
• Speed and power evaluation was performed on the Jetson platform using a custom Python (version 3.10.12) benchmark harness interfacing with the native Ollama REST API in non-streaming mode. This part was used to measure inference timing, throughput, device telemetry, and power consumption under controlled runtime conditions. In addition to Jetson internal rail measurements, external power measurements were also collected using a digital multimeter. This made it possible to estimate not only the behavior of internal power rails but also the total device-level power consumption, and it provides a measurement approach that can also be applied to platforms without built-in internal power monitoring capabilities.

The two parts were linked at the model level: the same model tags, deterministic decoding settings, and comparable inference configurations were used wherever possible. As a result, model quality and model efficiency could be analyzed jointly in the results section, even though they were measured on different hardware platforms. The overall structure of the experimental workflow is summarized in Figure 1.

Figure 1. Overall experimental workflow used in the study.

It should be noted that this split design means the accuracy–efficiency trade-off is established indirectly, at the model-configuration level, rather than through measurements conducted on the same hardware and software stack. Because the same GGUF model files and Ollama runtime are used with deterministic decoding, accuracy scores are expected to be identical across platforms; however, throughput, power, and energy measurements were collected exclusively on the Jetson device. The results should, therefore, not be interpreted as a single unified end-to-end Jetson benchmark: accuracy reflects a property of the model configuration, while deployment efficiency reflects the behavior of that configuration on the Jetson platform specifically.

2.2. Hardware Platforms

Two hardware environments were used in the study. The first was a desktop-class PC used for accuracy evaluation with lm-eval, chosen to reduce evaluation time for repeated benchmark runs. The second was the Jetson edge platform used for speed and power measurements, chosen as the target deployment-oriented device. The accuracy benchmarks were, therefore, intended to measure not hardware efficiency but model task performance. Conversely, the Jetson experiments were intended not to provide standardized benchmark accuracy scores but to characterize runtime behavior on limited-resource hardware. The main hardware characteristics of both platforms are summarized in Table 1.

Table 1. Hardware platforms used in the experiments.

Role	Platform	CPU	GPU	RAM	Storage
Accuracy evaluation	Desktop PC	Intel Core i5-14600K	NVIDIA RTX 4080 SUPER	128 GB	NVMe
Speed/power benchmark	Jetson Orin Nano Super	6-core Arm Cortex-A78AE	NVIDIA Ampere GPU (1024 CUDA cores, 32 Tensor Cores)	8 GB	External NVMe

For the Jetson-based measurements, the system was configured to reduce run-to-run variability. The device was operated in headless mode via SSH, unnecessary background services were minimized where possible, a fixed nvpmodel configuration was used, and maximum clocks were enabled with jetson_clocks. Stable ambient conditions and cooldown intervals between measured runs were used to reduce short-term thermal carryover effects.

2.3. Software Stack and Inference Environment

The experiments relied on a local inference setup based on the Ollama runtime. Two Ollama interfaces were used:

• the native Ollama /api/generate endpoint for the Jetson benchmark harness, and 
• the OpenAI-compatible local chat-completions interface for lm-eval during accuracy evaluation.

This distinction is important for reproducibility. The Jetson speed/power benchmark used Ollama in non-streaming text generation mode, while the accuracy benchmark used task-driven generated responses scored by lm-eval. The study, therefore, did not rely on token log-probability based evaluation or multiple-choice loglikelihood scoring. Instead, it focused on benchmarks that are compatible with generation-only evaluation. The main software components used in both environments are summarized in Table 2.

Table 2. Software stack used for accuracy evaluation and Jetson benchmarking.

Component	Accuracy Evaluation PC	Jetson Benchmark
Operating system	Windows 11 Education	Ubuntu 22.04.5 LTS (JetPack 6.2)
Ollama	0.20.6	0.11.10
Python	3.13.2	3.10.12
lm-eval	0.4.11	–
GPU runtime/CUDA	Docker GPU runtime; CUDA validation image “12.4.1-base-ubuntu22.04”	JetPack GPU runtime; CUDA 12.6

2.4. Evaluated Models

The experiments covered twelve Ollama-hosted compact and mid-scale models selected to represent realistic local-inference configurations for limited-resource edge hardware. The model set spans sub-1B, 1–4B, and 7–8B parameter ranges. Sub-1B models were included to represent lightweight configurations where high throughput and low energy costs are expected to be most important. The 7–8B models were included as a higher-quality but more resource-demanding upper range for quantized inference on an 8 GB Jetson-class device, while the 1–4B models capture the intermediate region where practical accuracy–efficiency compromises are often expected.

The selection also covers several widely used open-weight model families available in the Ollama/GGUF ecosystem, including Gemma, Qwen, Llama, Mistral, Phi, and DeepSeek-R1 distilled variants. These models provide diversity in the family, parameter scale, and deployment profile, including lightweight baseline models, general-purpose instruction-tuned models, and reasoning-oriented variants. The goal was not to isolate architecture as an independent variable, since the evaluated models differ simultaneously in parameter count, training procedure, tokenizer, instruction tuning, and quantization. Instead, the model set was chosen to reflect configurations that practitioners could realistically consider for edge deployment under the tested runtime stack.

The evaluated models were stored in GGUF format, which is commonly used for serving quantized large language models in the llama.cpp ecosystem. The evaluated model set, together with parameter scale and quantization level, is summarized in Table 3. The quantization labels indicate the approximate weight precision and encoding scheme used in these model files. For example, Q4_K_M denotes an approximately 4-bit k-quant scheme with a medium variant, Q4_0 denotes a basic 4-bit quantization scheme, and Q8_0 denotes an 8-bit quantization scheme. In general, lower-bit quantization reduces the model size and memory use, whereas higher-bit quantization typically preserves more accuracy at the cost of a larger memory footprint and potentially slower execution.

Table 3. Evaluated models.

Family	Model Name	Parameters	Quantization
phi3	phi3:3.8b	3.8 B	Q4_0
mistral	mistral:7b-instruct-v0.3-q4_K_M	7.2 B	Q4_K_M
llama	llama3.1:8b	8.0 B	Q4_K_M
qwen3	qwen3:0.6b-q4_K_M	751.63 M	Q4_K_M
	qwen3:1.7b-q4_K_M	2.0 B	Q4_K_M
	qwen3:4b-q4_K_M	4.0 B	Q4_K_M
	qwen3:8b-q4_K_M	8.2 B	Q4_K_M
DeepSeek-R1 distilled	deepseek-r1:1.5b-qwen-distill-q4_K_M	1.8 B	Q4_K_M
	deepseek-r1:8b-0528-qwen3-q4_K_M	8.2 B	Q4_K_M
gemma3	gemma3:270m	268.10 M	Q8_0
	gemma3:1b	999.89 M	Q4_K_M
	gemma3:4b-it-q4_K_M	4.3 B	Q4_K_M

2.5. Accuracy Evaluation

Model accuracy was evaluated using the lm-evaluation-harness framework, with Ollama exposed through its local OpenAI-compatible chat-completions API. Because the evaluation was based on generated responses rather than prompt log-likelihood scoring, the benchmark was restricted to tasks that support generation-based evaluation. All accuracy runs used deterministic decoding, no few-shot exemplars (num_fewshot = 0), non-tokenized API requests, and explicit chat-template application. Unless otherwise specified by the harness task definition, task-specific generation settings were left at their lm-eval defaults.

The final benchmark set was selected to cover complementary response-quality dimensions relevant to practical edge LLM use: instruction following, reasoning with extractable answers, and truthfulness-oriented free-form generation. The final benchmark set and primary reported metrics were:

• IFEval—instruction following: prompt_level_strict_acc, prompt_level_loose_ acc, inst_level_strict_acc, and inst_level_loose_acc.
• GSM8K—mathematical reasoning: exact_match with flexible-extract filtering.
• TruthfulQA_gen—truthfulness-oriented free-form generation: bleu_acc, rouge1_acc, rouge2_acc, and rougeL_acc.

Together, these tasks provide a compact and reproducible quality reference for comparing model configurations under the selected generation-based evaluation setup. However, they do not cover all possible edge deployment scenarios, such as long-context dialogue, summarization, domain-specific question answering, or interactive assistant behavior. Additional task-specific outputs produced by lm-eval, including strict GSM8K exact match and auxiliary TruthfulQA score variants, were retained for diagnostic interpretation but were not used as primary reported metrics.

The normalized composite accuracy score was used only as a compact indicator for the accuracy–efficiency analysis. It was computed from nine primary metrics: one GSM8K flexible-extract exact-match metric, four IFEval metrics, and four TruthfulQA_gen BLEU/ROUGE-based metrics. Each metric was min–max normalized across the evaluated models, and the final score was calculated as the unweighted mean of the normalized values. The strict-match GSM8K metric was excluded because flexible-extract exact match was used as the primary GSM8K result. Thus, the composite score should be interpreted as a relative, benchmark-specific indicator, not as a universal measure of LLM response quality.

2.6. Jetson Speed and Power Benchmark

Speed and power measurements were collected on the Jetson platform using a custom Python benchmark harness that interfaces with the native Ollama REST API. Each inference request was sent to the /api/generate endpoint with stream = False, so the full response was returned only after generation had completed. As a result, the methodology measures end-to-end request timing, prompt evaluation timing, and generation timing but does not directly measure time-to-first-token. To improve comparability across runs, deterministic decoding was used with throughout (temperature = 0.0), thereby reducing stochastic variability in model outputs. In the Jetson benchmark harness, the maximum context length was fixed to 1024 tokens (num_ctx = 1024), the maximum generated output length was fixed to 192 tokens (num_predict = 192), and the model keep-alive parameter was set to 5m.

For each model and prompt type, 3 warm-up runs were executed first, followed by 5 measured runs. Warm-up runs were excluded from the final analysis. A 5 s cooldown interval was inserted between measured runs. In addition, a 10 s idle baseline was recorded before each model configuration. To quantify run-to-run variability, standard deviations were computed after warm-up runs were excluded. For per-prompt results, the standard deviation was computed across repeated measured runs for each model-prompt combination. For model-level summary figures averaged across prompt categories, the error bars reflect variability across the measured run-level values included in each model-level average. These variability estimates are reported as error bars in the throughput, power, and energy-efficiency figures.

The benchmark workload consisted of seven fixed prompt categories defined directly in the benchmark script and kept identical across all evaluated models:

• short—concise factual response;
• medium—moderate-length explanatory response;
• long—extended explanatory response with more structured content;
• very_long—long-form generation representing heavier context usage;
• reasoning_final_only—reasoning-oriented task requiring a constrained final answer;
• code_small—short code generation task;
• struct_json—structured JSON output task.

For each measured request, the Ollama API returned run-level timing and tokenization statistics, including model load duration, prompt evaluation duration, generation duration, total request duration, prompt token count, and the generated token count. These values were used to derive the primary runtime indicators reported in the study, namely end-to-end latency and generation throughput in tokens per second, while also preserving phase-specific timing information for a later interpretation of performance differences across models and prompt types.

2.7. Jetson Inference Execution Modes

Jetson performance and power measurements were conducted in two inference execution modes. The first mode used GPU-accelerated inference, where Ollama was allowed to use the Jetson GPU through its default local inference backend. The second mode used CPU-only inference, where GPU offloading was disabled and inference was executed using the maximum available CPU thread count. For the CPU-only condition, the processor was configured for maximum sustained performance by using all available CPU cores, together with a performance-oriented CPU frequency configuration. This was done to evaluate a best-case CPU-only baseline rather than a dynamically frequency-scaled CPU configuration.

This distinction was introduced to separate the effect of accelerator use from the effect of model size and quantization. The GPU mode represents the intended accelerated deployment scenario on the Jetson platform, while the CPU-only maximum-thread, maximum-performance mode provides a controlled comparison point for cases where GPU offloading is unavailable, disabled, or reserved for other workloads. Both modes used the same benchmark harness, model set, prompt set, context length, output-token limit, and deterministic decoding settings. Therefore, differences in throughput, power consumption, and energy efficiency can be attributed primarily to the inference execution mode rather than to changes in the task workload.

2.8. Telemetry and Power Measurement

During each measured Jetson run, a parallel telemetry sampler recorded internal device statistics from tegrastats at 50 ms intervals. From these logs, the study derived average and peak power for the VDD_IN, VDD_CPU_GPU_CV, and VDD_SOC rails, average CPU utilization, maximum observed CPU frequency, average and peak GR3D activity, maximum CPU, GPU, and junction temperatures, as well as run-level changes in RAM and swap usage. These telemetry measurements were used to characterize both the internal energy-related behavior of each model and the broader system-level bottlenecks affecting runtime.

In addition to the internal Jetson telemetry, external board-level input power consumption was measured at the Jetson DC input using a TTi 1604 bench-top digital multimeter. The multimeter recorded voltage and current measurements, and instantaneous input power was computed as $P=V\times I$ for each sample. The measurements were logged on a Raspberry Pi 5 acting as a separate measurement host. Although a Raspberry Pi 5 was used in this work, the same external logging approach could, in principle, be implemented on any other host computer capable of reading and recording the multimeter output.

The external logger sampled at approximately 2 samples per second. The sampling interval was not strictly fixed because it was determined by the multimeter USB/serial polling rate. Synchronization between the Jetson benchmark and the external logger was performed using wall-clock timestamps.

No hardware trigger or separate laboratory recalibration of the multimeter was used. Therefore, the external measurements are interpreted as board-level average input-power estimates over each run interval, rather than as high-frequency transient measurements. This external path also captures total board-level input power and can be applied to platforms without built-in power telemetry. A per-model idle baseline was recorded for context, and the measurement arrangement is illustrated in Figure 2.

Figure 2. Photographs of the external power-measurement setup used in the Jetson experiments: (a) top view of the setup; (b) front view of the setup. The Jetson device was connected through a TTi 1604 digital multimeter, while measurement values were logged by a Raspberry Pi 5 acting as a separate measurement host.

2.9. Reproducibility

To improve reproducibility, all experiments used fixed model tags, deterministic decoding, stored prompt definitions, fixed benchmark parameters, and raw per-run logs. The Jetson-based measurements were executed under a controlled system configuration with headless operation via SSH, minimized background activity, fixed nvpmodel settings, enabled jetson_clocks, and explicit cooldown periods between measured runs. Accuracy evaluation outputs and Jetson benchmark outputs were retained separately in machine-readable form for later aggregation and verification.

All Jetson benchmark results were recorded at the individual-run level rather than only as aggregated summaries, preserving natural run-to-run variability and enabling later statistical analysis. Because model accuracy and hardware efficiency were evaluated on different platforms, all reported comparisons were made at the model configuration level. In this methodology, task performance is treated as a property of the evaluated model configuration, whereas throughput and power-related behavior are treated as properties of its execution on the target edge platform under the specified runtime conditions.

3. Results and Discussion

This section reports the results for the two Jetson execution modes: GPU-accelerated inference and CPU-only inference. Both modes followed the benchmark protocol described in Section 2.6, producing 420 measured runs per mode. Accuracy scores obtained on the desktop PC are used as a common model-quality reference when interpreting Jetson throughput, power, and energy efficiency. For readability, shortened model names are used in this section; full Ollama tags and quantization details are reported in Table 3.

3.1. Accuracy Results

The evaluated models span a wide range of accuracy scores across the selected generation-compatible benchmarks. Llama 3.1 8B achieved the highest normalized composite accuracy, followed by Gemma 3 4B. Phi-3 3.8B and Mistral 7B also returned competitive composite scores despite their relatively compact parameter counts. At the lower end, Gemma 3 270M and Qwen3 0.6B achieved substantially lower composite scores, although these same models later show the highest throughput and energy efficiency on the Jetson platform. The full task-level breakdown is shown in Figure 3.

Figure 3. Task-level accuracy heatmap for the evaluated models. Metrics include generation-compatible scores from IFEval, GSM8K, and TruthfulQA_gen, together with the normalized composite score.

3.2. Overall Jetson Performance and Power

Aggregate statistics across all measured Jetson runs are reported in Table 4. Throughput is expressed as the median over all runs, power as the mean, and energy efficiency as the ratio between throughput and power, as defined in Equation (1):

$${\displaystyle \frac{\mathrm{tok}/\mathrm{s}}{\mathrm{W}}}={\displaystyle \frac{\mathrm{tok}/\mathrm{s}}{\mathrm{J}/\mathrm{s}}}={\displaystyle \frac{\mathrm{tok}}{\mathrm{J}}}.$$

(1)

Table 4. Aggregate Jetson benchmark results for the two execution modes.

Mode	Median Throughput (tok/s)	Mean External Power (W)	Mean VDD_IN (W)	External Efficiency (tok/J)	VDD_IN Efficiency (tok/J)
GPU	18.13	22.03	17.63	0.823	1.006
CPU	7.12	15.11	12.45	0.453	0.565

Note: External power is board-level input power measured by the external multimeter. VDD_IN is the Jetson on-chip board input rail. CPU mode used all six Cortex-A78AE cores at maximum clock frequency.

This definition was applied separately to the external board-level power measurements and to the Jetson VDD_IN rail measurements. The resulting aggregate values for GPU-accelerated and CPU-only execution are shown in Table 4.

GPU execution produced approximately 2.55× higher median decoding throughput than CPU-only execution. Although the mean external input power increased by approximately 46%, the throughput gain was proportionally larger than the power increase, resulting in higher energy efficiency for GPU execution under both measurement paths.

The external meter reported higher power than the Jetson VDD_IN rail, consistent with board-level input-path and voltage-regulation losses not fully captured by on-chip telemetry. As both measurement paths yield the same relative conclusions, the external meter is used as the primary power reference throughout the following analyses.

3.3. Power Trace Characteristics

The full-session power traces provide diagnostic information about how the two execution modes behave over time. Because the full-session traces are visually dense at publication scale, the complete time-series plots are provided in Appendix A as full-page rotated figures. The GPU trace is split into three consecutive parts in Figure A1, Figure A2 and Figure A3 and the CPU-only trace into three consecutive parts in Figure A4, Figure A5 and Figure A6.

The traces show distinct power profiles for the two execution modes. GPU execution produces short, well-separated high-power bursts and returns closer to idle between model segments, whereas CPU-only execution produces broader and more sustained power plateaus. These differences help explain why GPU execution achieves higher throughput and energy efficiency despite its higher instantaneous power draw.

In the GPU-mode traces, the power envelope rises steeply during inference bursts and drops back between model segments. This burst-like profile is consistent with accelerator-based execution: the GPU completes weight-streaming operations more quickly and spends proportionally more time in a lower-power state. By contrast, the CPU-only traces show a flatter, more sustained power profile, reflecting the continuous memory and computational load of autoregressive decoding, where full weight matrices must be streamed through the memory hierarchy for each generated token.

Model-level summaries of throughput and power, averaged across all prompt categories, are reported in Section 3.4; readers seeking the corresponding temporal behavior are referred to Appendix A.

3.4. Throughput and Power by Model

Token generation throughput is primarily determined by model size rather than by prompt category. Across all models and both execution modes, prompt-to-prompt variation within a model was typically less than 1 tok/s. Therefore, Figure 4 reports the average throughput per model across all prompt categories. The error bars in Figure 4 and Figure 5 show ±1 standard deviation across the measured run-level values after warm-up runs are excluded; consequently, they reflect the combined variability introduced by repeated runs and prompt categories.

Figure 4. Average token generation throughput per model for GPU and CPU-only execution, averaged across all prompt categories. Error bars indicate ±1 standard deviation.

Figure 5. Average external-meter power draw per model for GPU and CPU-only execution, averaged across prompt categories. Error bars indicate ±1 standard deviation.

Figure 4 shows a clear throughput decrease as the model size increases. In GPU mode, the fastest models were Gemma 3 270M (55.2 tok/s) and Qwen3 0.6B (47.9 tok/s), while the 7–8B models clustered around 10–11 tok/s. CPU-only execution preserved the general model ranking but substantially reduced throughput. The relative GPU speedup was especially clear for larger models; for example, DeepSeek-R1 8B reached 10.2 tok/s on GPU compared with 4.0 tok/s on CPU, corresponding to approximately 2.6× speedup. This indicates that the benefit of GPU acceleration grows with the model size and the memory-bandwidth demand.

Figure 5 shows that power consumption also increases with model size, but less sharply than throughput decreases. In GPU mode, the 7–8B models draw the most power, around 25–26 W, while Gemma 3 270M draws the least power at 14.8 W. CPU-only execution draws less power than GPU mode for all models, but the reduction in power is smaller than the corresponding reduction in throughput. This explains why GPU execution remains more energy-efficient despite its higher absolute power draw.

3.5. Accuracy–Efficiency Trade-Off

The evaluated models show a clear trade-off between task performance and deployment efficiency. Higher-accuracy models are generally less energy-efficient because they tend to have larger parameter counts and lower token generation throughput. Figure 6 shows this relationship by comparing normalized composite accuracy with external-meter energy efficiency for both Jetson execution modes.

Figure 6. External-meter energy efficiency (tok/J) versus normalized composite accuracy for both Jetson execution modes. Dashed lines show the linear trend for each mode.

GPU execution improves the accuracy–efficiency trade-off for most models, especially in the larger 7–8B class, where the throughput gain outweighs the additional power draw. Llama 3.1 8B, the most accurate model, achieved approximately 0.41 tok/J in GPU mode and 0.26 tok/J in CPU-only mode. In contrast, Gemma 3 270M achieved approximately 3.68 tok/J in GPU mode and 3.76 tok/J in CPU-only mode, making it one of the few models where CPU and GPU efficiency were nearly equal.

Among the evaluated models, Gemma 3 4B represents one of the strongest compromise points. It combines a high normalized composite accuracy score of 0.836 with an external-meter efficiency of approximately 0.63 tok/J in GPU mode, placing it between the highly efficient small models and the higher-accuracy but more energy-demanding 7–8B models. These results confirm that model selection for edge deployment should not rely on accuracy alone; the appropriate operating point depends on the required balance among response quality, throughput, and energy cost per generated token.

3.6. Per-Prompt Efficiency Breakdown

The previous analyses used values averaged across the seven prompt categories. The per-prompt efficiency breakdown is included as a consistency check to verify that this averaging does not hide prompt-specific behavior. In GPU mode, Figure 7 shows that efficiency values form mostly model-wise bands, with only small variation across prompt categories. For both per-prompt efficiency plots, the error bars show ±1 standard deviation across repeated measured runs for each model–prompt combination.

Figure 7. External-meter energy efficiency (tok/J) per model and prompt category for GPU execution. Error bars indicate ±1 standard deviation.

The same pattern is visible in CPU-only mode, as shown in Figure 8. The relative ordering of models remains largely unchanged across prompt categories, indicating that the aggregate model-level comparisons are not driven by a specific prompt type.

Figure 8. External-meter energy efficiency (tok/J) per model and prompt category for CPU-only execution with all cores at maximum clock frequency. Error bars indicate ±1 standard deviation.

These plots support the use of averaged per-model efficiency values in the preceding analyses. Under the tested fixed context length and output-token limit, the prompt category had a smaller effect on energy efficiency than the model choice and the execution mode.

3.7. Limitations

Several limitations should be considered when interpreting the results. First, the accuracy–efficiency trade-off was established at the model-configuration level rather than through a single unified end-to-end benchmark on the Jetson device. Accuracy was evaluated on a desktop PC to make the generation-compatible lm-eval runs practical, whereas throughput, power consumption, and energy efficiency were measured on the Jetson Orin Nano Super. Because the same GGUF model files, Ollama version, and deterministic decoding settings were used, the accuracy scores are treated as properties of the evaluated model configurations; however, the deployment-efficiency values are specific to the Jetson platform and the tested runtime configuration.

Second, the reported performance and power results are specific to one edge device and one inference ecosystem: the Jetson Orin Nano Super using the Ollama/llama.cpp GGUF stack. Different edge accelerators, kernel backends, quantization implementations, or inference runtimes may produce different absolute throughput and energy-efficiency values. The selected accuracy tasks also provide only a compact quality reference based on generation-compatible benchmarks, and the normalized composite score should be interpreted as a relative, benchmark-specific indicator, rather than a universal measure of response quality.

Third, the Jetson benchmark used non-streaming generation, so it measured full-request completion behavior but did not directly capture time-to-first-token (TTFT), which is important for interactive assistant workloads. The experiments were also designed as controlled short-run benchmarks with cooldown intervals, not as sustained thermal endurance tests. Although temperatures, RAM usage, and swap behavior were logged, detailed memory-footprint profiling was outside the scope of the study, and the Jetson shared-memory architecture does not expose a separate VRAM metric in the same way as a discrete-GPU system. Finally, the external multimeter provided board-level average input-power estimates, but its sampling rate and lack of hardware triggering mean that high-frequency transient power behavior was not characterized.

3.8. Future Research

Future research should extend the benchmark beyond the present Jetson/Ollama setup by evaluating additional edge platforms and accelerators, including other Jetson-class modules, CPU-only single-board computers, mobile NPUs, and dedicated edge AI accelerator kits. Cross-runtime comparisons with alternative inference stacks, such as TensorRT-LLM, MLC-LLM, and optimized llama.cpp-based configurations, would help separate model-size effects from runtime- and kernel-specific behavior. Controlled quantization ablations should also be performed, where memory permits, by comparing multiple precision levels and GGUF variants of the same model family.

The workload model should also be broadened. Streaming inference should be included to measure TTFT and per-token latency directly, while benchmark settings should vary the context length, the generated output length, the prompt structure, and multi-turn interaction patterns. Longer continuous runs are also needed to quantify sustained power draw, thermal throttling, and stability under realistic operating conditions.

Finally, future evaluations should include deeper memory profiling, including model-loading behavior, KV-cache growth, RAM pressure, and swap activity under different context lengths and concurrent workloads. The quality evaluation should be expanded beyond the current compact task set to include summarization, domain-specific question answering, long-context dialogue, conversational consistency, and interactive assistant scenarios.

4. Conclusions

This study evaluated compact and mid-scale large language models from two complementary perspectives: task-level response quality and deployment efficiency on a resource-constrained edge platform. Accuracy was assessed on a desktop-class PC using generation-compatible lm-eval tasks, while throughput, power, and energy-efficiency measurements were collected on a Jetson Orin Nano Super using a custom non-streaming Ollama benchmark harness. Because a non-streaming request mode was used, the reported timing and throughput values reflect full-request completion behavior and do not directly capture time-to-first-token (TTFT); deployments where first-response latency is critical, such as interactive edge assistants, should therefore complement these results with streaming-mode TTFT measurements. The Jetson measurements combined internal tegrastats telemetry with external board-level power measurement using a digital multimeter.

The results show that GPU-accelerated inference provides a clear advantage over CPU-only execution on the Jetson platform. Across all evaluated models and prompt categories, GPU execution achieved a median throughput of 18.13 tok/s, compared with 7.12 tok/s for CPU-only execution. Although GPU execution increased the mean external input power from 15.11 W to 22.03 W, it also improved energy efficiency from 0.453 tok/J to 0.823 tok/J. Since the proportional throughput gain exceeds the proportional power increase at every tested model size, GPU offloading should be the default inference mode on Jetson-class platforms whenever the accelerator is available.

External power measurements were consistently higher than internal VDD_IN readings, reflecting board-level losses not captured by on-chip telemetry. However, both measurement methods led to the same relative conclusions, with the external meter providing the more conservative system-level estimate.

Model size largely determined the accuracy–efficiency trade-off. Smaller models delivered the highest throughput and energy efficiency but lower accuracy, whereas 7–8B models achieved stronger accuracy at substantially higher energy cost. Gemma 3 4B offered a practical middle ground.

The study confirms that edge LLM deployment should be evaluated as a multi-objective trade-off between accuracy, throughput, power consumption, and energy efficiency under explicitly stated hardware, runtime, and workload conditions. The reported throughput and latency figures characterize full-request completion rather than first-token responsiveness, and should not be used as a direct proxy for TTFT in interactive applications. Configurations that perform well on one metric often perform poorly on another, and deployment decisions that rely on a single headline value risk misaligning model choice with application requirements.