Abstract
Low-rank compression shrinks large language models by factorizing their weights, guided by a small calibration set. When the factorization is learned by distillation, as in learned subspace projections (LSP), the calibration set becomes training data, yet is still chosen by convention. We compressed OPT-1.3B, Llama-3.2-1B, and Qwen3-1.7B by 30% and 50% with LSP, calibrated on six sources at a matched budget (WikiText-2, C4, SlimPajama, Alpaca, and two kinds of self-generated text) over three seeds, and repeated the comparison on Llama-3.1-8B and with a second compressor. Training removed most of the up to 600-fold perplexity differences between untrained models but turned them into specialization: each source best preserved its own text, and WikiText-2, the customary choice, gave the least general models. Zero-shot accuracy slightly favored instruction data, through content rather than template. Self-generated documents gave the most uniformly faithful Llama models without external data, whereas exact samples of the model, whose expected training objective is the sequence-level divergence, failed on Qwen3-1.7B. The initialization’s domain survived training, and compressed Llama models collapsed without their calibration beginning-of-sequence token. With every source, compression lost most instruction following and multi-step reasoning. These results indicate that calibration sources should be reported with every perplexity.