Abstract
Most of the energy consumed and carbon emitted by Large Language Models (LLMs) over their lifetimes is associated with inference as opposed to training. The device that receives each request decides the cost, carbon emitted, and how fast the response returns. We treat inference scheduling of LLMs as an optimization problem. Energy, carbon, and adherence to service-level agreements (SLA) are the objectives. We consider the optimization problem in single-objective and multi-objective form. Two key aspects that distinguish this work from previous attempts are: (1) treatment of each device as a distinct unit for measuring inference latency, and (2) SLA violations that depend on the schedule itself. For the optimization of LLM inference scheduling across heterogeneous, geo-distributed devices, we enforce capacity through a soft penalty that is normalized by total demand. This helps to keep the optimization meaningful when the demand exceeds the capacity of the device pool. The optimization problem is NP-hard. We present Improved Golden Jackal Optimization (IGJO) to solve this optimization problem. IGJO combines quasi-reflected opposition at initialization with a stagnation-driven controller that monitors the population’s coefficient of variation and raises the opposition rate as diversity collapses. We improved twelve metaheuristics by integrating the same mechanisms as IGJO. IGJO, therefore, competes against eleven other algorithms that have the same improvements. We conduct experiments on three synthetically generated workloads and two oversubscribed production workloads (Azure LLM inference and BurstGPT), and each workload has twenty common-seeded runs. We perform Friedman and Wilcoxon tests with Holm correction. The enhancement improves the algorithms broadly, but has some clear limits and those are reported. In single-objective mode, IGJO has the best average fitness and average Friedman rank. Of the 55 pairwise tests, IGJO won 47. Similarly, in the multi-objective case, IGJO also performs best. Here, IGJO won 45 out of the 55 pairwise comparisons. Against simple scheduling rules and the closely related FPC method, IGJO is better on every workload in both modes, with a fitness gap over FPC of 12.5–28.5%. IGJO shows the greatest advantage for the workloads that model the conditions faced in actual practice, namely, the constrained and oversubscribed settings.