Abstract
To generate each token, the Whisper decoder re-reads the encoder’s 1500-position cross-attention key–value (KV) cache on the CPU. In the Whisper-medium model, this cache occupies 281 MiB; for short recordings, 74–82% of it consists of padding positions. This paper proposes a calibratable framework that reduces the number of KV positions read at each step without retraining the model or changing the cache kept in memory. The saving therefore comes from reading fewer bytes from DRAM at each step. Its core method—PadSink-Track—preserves critical “padding-sink” positions, shifts the audio window at each step based on signals from Whisper’s alignment heads, and updates selected KV entries in a ring buffer. A fallback mechanism re-executes the step using the full cache when confidence is low. The budget, sink fraction, and fallback threshold (τ) are calibrated for each model on a validation set based on a specified non-inferiority criterion (δ = 0.2 × WER). When the calibrated budget was halved, none of the selection rules relying solely on the first step met the criterion. In contrast, the calibrated framework satisfied this criterion across all ten configurations, covering four languages (Uzbek, English, Russian, Turkish) and two model sizes. On the held-out sets, every no-fallback decision was reproduced, and the single fallback decision (medium/English) became “Inconclusive”. In three configurations, where 19–35% of the steps were re-executed, the speed condition selected the full cache. On the Intel Core i7-11850H processor, decoder-step time without fallback decreased by 32–34% (synthetic step; 23–25% with real decoding), the volume of data read from DRAM dropped by 36–40%, and the throughput of the eight parallel decoders increased by 53% (synthetic step; 33% with real decoding, median latency −26%). Time savings during whole-utterance decoding amounted to 13–14% for models without fallback, whereas for the original medium model that required fallback, this figure was approximately 4%.