Abstract
Large language model (LLM) accelerators are commonly optimized for fixed model structures and isolated inference workloads, whereas emerging artificial-intelligence operating systems (AI OSs) primarily virtualize agents, memory, and tools in software. Reusable hardware, phase-aware serving, operating-system-level composition, and automata-based formalisms each exist separately in prior work; this architecture paper treats them simultaneously to ask whether LLM inference can be exposed as a secure, reusable hardware service. We introduce HCDP-LLM, an AI-OS-managed hierarchical common datapath that composes reusable tensor, reduction, nonlinear, quantization, memory, and key–value (KV)-cache modules through model capsules and a Moore finite-state controller. The architecture is formalized as a finite-precision multi-tape computational transducer, enabling deterministic checkpointing, migration, isolation, and Turing-machine-based bit-time, space, and I/O analysis. An explicit derivation on this transducer yields quadratic prompt-attention work, context-linear per-token attention and KV-cache storage, and separates compute-dominated prefill from bandwidth-dominated decoding. The formal model separates finite-state physical realizations from a scalable family with extensible memory that can support universal computation; a worked example derives from it the admission limit, lane allocation, and rebinding granularity of a 7B-parameter model on one reconfigurable accelerator. No prototype measurements or simulations are reported; a pre-specified evaluation protocol with a control-plane-disabled baseline, which isolates the abstraction-layer cost, defines how a field-programmable gate array (FPGA) or system-on-chip prototype must test the resulting position: Hardware LLMs as runtime-composable, weight-programmable inference fabrics rather than permanently hardwired models.